From e4cc2bf1c31eb3267b3541084929d769ed6e9648 Mon Sep 17 00:00:00 2001 From: Kartios Date: Fri, 21 Aug 2026 15:55:43 -0700 Subject: [PATCH] docs: the scope, the plan, the post-review execution plan, and the eval runbook MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MULTI_TURN.md scopes the multi-turn gap and records the finding that made it cheap: multi-turn needs no container and no real harness. vf.Env.run programs the control flow host-side and the simulator runs in-process as the user, with the harness left null. PLAN.md carries the settled decisions. EXECUTION.md is the post-adversarial-review plan — every position in it survived a reviewer, and where a design spec was overturned the corrected position is what is written down. Two corrections are marked in place rather than silently rewritten, because both were believed and acted on: - The provenance hole is src/arena/index.ts, not offices/sites.ts. Mutating offices/sites.ts produces a bit-identical episode; index.ts re-exports the scripted baselines that ARE the reward denominator, and a halved office-nav baseline moves return 4.744 -> 4.098 while every tera hash gate prints ok. - office-jobs-v1 is not phantom. It was absent at 13:00 and present at 15:50, because another session fast-forward-merged it into the shared tera tree mid-session. The counts are twelve, five and twenty. docs/FIRST_EVAL.md is the runbook for the first evaluation Arena ever completed: seven environments, 224 episodes, 223 scored, against brain-qwen38-dspark. It warns that the mean is sum(score x weight) — an unweighted mean over components is a different number, and it is how six of seven means were first misreported. Co-Authored-By: Claude Opus 5 (1M context) --- EXECUTION.md | 281 +++++++++++++++++++++++++++++++++++++++++++++ MULTI_TURN.md | 208 +++++++++++++++++++++++++++++++++ PLAN.md | 243 +++++++++++++++++++++++++++++++++++++++ docs/FIRST_EVAL.md | 185 +++++++++++++++++++++++++++++ 4 files changed, 917 insertions(+) create mode 100644 EXECUTION.md create mode 100644 MULTI_TURN.md create mode 100644 PLAN.md create mode 100644 docs/FIRST_EVAL.md diff --git a/EXECUTION.md b/EXECUTION.md new file mode 100644 index 0000000..dd58f51 --- /dev/null +++ b/EXECUTION.md @@ -0,0 +1,281 @@ +# Execution plan — Arena / Lumbridge + +Generated 2026-08-21 by a 13-agent design + adversarial-verification workflow. +Companion to `PLAN.md` (decisions) and `MULTI_TURN.md` (scope). + +⚠️ Every position here is the POST-REVIEW position. Where an adversarial reviewer +overturned a design spec, the spec's original claim was discarded. Do not resurrect +a claim from an earlier document that this one corrects. + +# Arena / Lumbridge — one execution plan + +Synthesis of six adversarially-reviewed workstream specs, 2026-08-21. Every position below is the **post-review** position. Where a reviewer overturned a spec, the spec's original claim is marked and discarded. + +Verified independently for this document (amd-server, 2026-08-21): `arena/probe.py` is 772 lines with a literal 7-name shim tuple at :89-93 and a literal `results` dict at :752-760; `tests/test_probe.py` has a 7-name `ENVIRONMENTS` tuple at :13-21 and `timeout=30` at :31; `.gitea/workflows/ci.yml` is 42 lines, its probe step is still named "Run four-environment floor and ceiling probe" (:26) for a seven-environment probe, its wheel loop (:36-40) is a hardcoded bash `for` over seven names, and its lock-policy check (:24) is failing on main because **three** `uv.lock` files are tracked (`bot_detection`, `drop_table_inference`, `grand_exchange`); `outputs/` contains exactly one file, a `config.toml` with `push = true` and `base_url = https://api.pinference.ai/api/v1` — **no `traces.jsonl` exists anywhere on this box**; `~/.prime/config.json` has an empty `api_key`; `tera/src/arena/index.ts:74-79` `ARENA_MANIFESTS` holds four. + +--- + +## 1. Verdicts applied — what each spec becomes + +### A — `grand-exchange-live` · feasible, but **not in the order written** + +Stands: the multi-turn mechanism (`Env.run` + `interaction.turn()`, null harness, no container), the delta-sheet action grammar, the malformed-reply policy, `roc` on **peak** capital employed (average-employed is a measured free-points bug), the vendored-generator equality assertion, no turn-count reward term. + +Corrected, blocking: + +1. **The measured ladder does not reconcile with the spec's own reward.** At ref 72,758 gp and `TARGET_SHARE = 0.85`, target = 61,844; impatient at 50,470 gp gives `profit_ratio` 0.816, `discipline` ≤ 0.816, `clean` false → max 0.612, not the reported 0.623. Churn and exhaustive *do* reconcile, so this is a specific defect. **Nothing in §1/§4 is locked until it is re-measured with the shipped engine.** +2. **`viable_market` retargeted at `live_reference` is circular as written.** Viability needs `freeze`; `freeze` is drawn in `Taskset.load()` after viability returns. Fix: derive `freeze` deterministically from `market.seed` (e.g. `random.Random(seed ^ FREEZE_SALT)`), inside `build_market`, so viability and scoring see the same value. This is also the fix for the unstated RNG source — without it, `scan()` recomputes a reference the model never played against. +3. **§6's held-out claim is inverted.** Live is *weaker* than one-shot: the model sees ~50 of 60 graded ticks. Rule 1 still passes; the README must say "held out: the tail after the last observation, ≥4 ticks", not "stronger by construction". Add a probe assertion on the *size* of the held-out slice, not only the frontier. +4. **The probe battery blows CI's own harness.** `probe.py` runs in 3.4 s today under `timeout=30`. §11 adds 9 policies × blocks × 240 baskets + two sweeps + a 240-seed generator check + a hostile battery replayed per turn. Raising the timeout is B1's job (below), but A must budget the cost, not append a name to a list. +5. **Taskset load cost is unbounded.** `num_tasks` defaults 48, `VIABILITY_TRIES = 64`; a live `viable_market` runs a full 8-turn stepped reference per candidate → ~3,072 simulations before the first token, on every eval and every `--dry-run`. Cache the reference per seed, or cap tries for live. +6. **`import` vs `copy` contradiction.** §3 says "import `grand_exchange.book.parse_orders`", §9 says vendor. It is a copy. Say copy. +7. **`[env.agent] max_turns = 8` is an off-by-one risk, not free insurance.** Verify how the framework counts, or set 9. +8. **`trace.info["episode"]` must be plain JSON types.** A dataclass or numpy scalar in `audit()` fails the trace write for the whole rollout. +9. **Effort is L/XL, not L** — it is A plus most of B. + +**Revised A build order (this is the change that matters):** `market.py`/`book.py` copies + equality assertion → **`live.py` + unit tests → re-measure the full ladder in the shipped engine** → only then lock `horizon`, `FREEZE_RANGE`, `TARGET_SHARE`, `TURNS` → `strategy.py` sweep → `ledger.py` → `protocol.py` → `taskset.py` → probe. + +### B — probe policies + de-hardcoding · **verdict: not feasible as written. Split it.** + +The reviewer is right and the split is the fix. + +**B1 — de-hardcode environment discovery. Sound, S/M, ships immediately, behaviour-preserving.** `tomllib` over `environments/*/pyproject.toml` replaces the shim tuple, the `main()` results dict (via a `@probes(name)` registry decorator), `ENVIRONMENTS` in `tests/test_probe.py`, and the bash wheel loop. **The CI environment list was never a matrix** — it is a `for` loop in one step (verified) — so no generator job, no `fromJSON` matrix, no act_runner risk. Take B1 now and the repo's known collision point disappears for one afternoon's work. + +Two mandatory amendments to B1: +- **Do not make "discovered but unregistered" a hard exit 1 while A is in flight.** The moment A commits a scaffold `pyproject.toml`, `probe.py` exits 1 and CI goes red for every session on the repo. Make it a warning until an env registers a probe, and flip it to exit 1 in the same commit as A's `exchange_live()`. +- **Make the two guards use the same denominator.** `ci.yml`'s "empty scan" counts directories; `discover()` counts `*/pyproject.toml`. A scaffold dir without a pyproject makes them disagree. Count `*/pyproject.toml` in both. +- Raise `tests/test_probe.py` `timeout=30 → 180`, keep the lock check *before* any `uv sync`, and fix the "four-environment" step name. + +**B2 — the `Interactive` policy machinery. Redesign required; do not build it before A's engine exists.** The structural claim ("oracle and exhaustive share `answer`, so rule 4 follows") is **false when the reward saturates**, and Arena's rewards all saturate: `profit_ratio` and `efficiency` clip at 1.0 and `clean` is a boolean band. If exhaustive also clears the band it scores 1.000 and the gate reads "budget is free". Further: +- `BIND_MARGIN = 0.02` is staked on realised-P&L differences over 24 seeds, against a block-to-block variance `probe.py:349-352` already documents at 0.48 vs 0.18. The five-block ladder is **required for all three rule-4 comparisons**, not an optional mitigation — multiply the cost estimate by ~5. +- `gate` is binary at weight 0.25, so the exhaustive/oracle gap is quantized in 0.25 steps. A 0.02 margin against a 0.25 quantum is a cliff, not a calibration. +- `play()` never terminates on an answer; "impatient answers turn 1" is an unstated precondition on A's `step()`. +- Seeds must come from A's `baskets()` skip-aware helper, not a contiguous `range`. +- B2's §1.8 dictates A's private module layout, class names, weight vector and turn mechanism. **Delete that.** A owns its contract; B2 adapts to it. This is the collision the workstream exists to remove. + +**Net: B1 is unblocked and independent of A. B2 depends on A's *measured* engine, not A's first draft.** PLAN.md's "B depends on A" edge survives for B2 and dies for B1. + +### D — Tera bridge · feasible, but **it is no longer parallel with A/B, and PLAN.md must be amended** + +Stands and is the strongest evidence in the six specs: 23-file / 283 KB / zero-bare-specifier TS closure; `node` type-strips raw `.ts`; **32/32 scenario × seed traces round-trip through Python `json` and replay to identical checksums**; one long-lived NDJSON worker (84 ms start, 22 µs RTT, 217 µs/step, 87 MB RSS) beats HTTP and beats per-rollout processes; the measured hold-length sweep. + +Corrected, blocking: + +1. **Role name vs config key — the spec will not load.** `agents.pilot` requires a config field named `pilot`, but all four TOMLs write `[env.agent.harness]`, and `EnvConfig` is `extra="forbid"`. **Name the role `agent`**, matching `SingleAgentEnvConfig`. Same trap A must resolve; settle it once for both (see Decision 1). +2. **`CONFIG = ...` classvar does not exist in verifiers v1.** Config is resolved by `concrete_type()` through `__orig_bases__`. Use the wordle form: `class CrowNavTaskset(TeraSpatialTaskset, vf.Taskset[TeraSpatialTask, CrowNavConfig])`. As written, every per-env `max_turns`/`max_hold` silently does nothing. +3. **The provenance claim is real but was mis-located — CORRECTED 2026-08-21.** `check-arena-source-hashes.mjs` pins 19 of 23 vendored files. The claim that `offices/sites.ts` is the material hole is **FALSE and was refuted twice independently**: mutating it (elevation 4→4000, heading 0→90, width 54→5400) produces a **bit-identical** episode, so it does not reach the simulation. ⚠️ **The real hole is `src/arena/index.ts` alone**, and it is worse than the original claim: rewriting it so `officeNavScriptedBaseline` re-exports a halved policy moves office-nav from 68 steps / return 4.744 to 136 steps / 4.098 — **while `npm run arena:source-hashes` prints "ok" for all four environments and exits 0**, `envHash` and `simulatorHash` unchanged. That file is the reward DENOMINATOR. Fix: pin `src/arena/index.ts` in `tera/scripts/check-arena-source-hashes.mjs` (a one-line `tera`-repo change) and regenerate the four pairs in `sourceHashes.ts`. +4. **`oracle.py` reimplementing the four TS baselines in Python contradicts the bridge's own rule** ("never recompute a number that came out of TypeScript") — and it does so for the reward *denominator*. Add an `oracle` op to `worker.mjs`; the four baselines are already exported from `src/arena/index.ts`. +5. **`drive-101` measures nothing, and the 0.9 ceiling band makes it worse than "no rule-4 claim".** Return is monotone in throttle, 12/12 goals at every value; `0.9 × 4.339 = 3.905` and throttle 0.2 already returns 3.723 → quality 0.953. Any positive constant throttle scores ~0.95–1.00. Same collapse on office-nav: any goal-reaching run clears the band, so `control` (0.50) and `gate` (0.20) are constants and only `efficiency` carries signal. +6. **office-nav's rule-4 argument is backwards under its own formula.** `efficiency = oracle_turns / max(turns, oracle_turns)` only ever *penalises* extra turns, and past `ceil(1.5 × 1.8) = 3` turns `clean` also drops. The formula punishes exactly the "precision buys turns" behaviour cited as evidence. +7. **`gate` is a separate additive term for goal-attainment that is already inside `ts_return` as the success bonus** — double-counted, additively. That is the failure `probe.py:15-17` records verbatim. Make it a multiplier on the sum. +8. **Missing:** no `@vf.stop`; no handling of a mid-loop `terminated` Segment (the loop as written hits `RuntimeError('the exchange is over')`); no `Taskset.load()` / prompt-vs-bare-`turn()` decision; an empty `{"plan":[]}` costs zero steps while a malformed reply costs 10, inverting the spec's own principle; no guard for `oracle_return <= 0`; `max_hold = 400` exceeds `MAX_STEPS = 360`. +9. **Scope.** PLAN.md:138 and the parallelization map say D lives in `tera` and shares no files with A/B/C. This spec puts everything in `arena` and edits `probe.py`, `tests/test_probe.py`, `ci.yml`, `README.md`, `configs/`, plus two `lumbridge-v4` files. That is a real decision (the packaging argument is correct — `tera` has no Python toolchain), but **PLAN.md must be amended to record it**, because it converts D from "parallel with everything" into "collides with A, B and W". + +**Revised D:** ship the bridge + replay gate + `tera_crow_nav` first. `drive-101` ships only as a labelled bridge-correctness environment or not at all (Decision 3). `office-nav` and `california-flight` follow. D's probe integration waits for B2. + +### P — `app.lumbridgecorp.com` · feasible; **Phase 1 as written cannot ship** + +Stands and is high-value: the endpoint inventory (29 routes, not 31), the 23 dead `api.ts` methods, one-build/two-hosts with app.* as the **only** auth origin, `bind 10.0.0.2` + the empty-200 verification, the `isAppHost` branch in `sendShell`, `WORLD_LAUNCHER_URL` convergence for `/go/:slug`, `World1` moved not deleted. + +Corrected, blocking: + +1. **"Ingest the runs already on amd-server" is impossible — there are none.** One `config.toml`, zero episodes, and that config points at `api.pinference.ai` with `PRIME_API_KEY`, which is empty. **Phase 1 must contain "execute one eval to completion against spark-1" as an explicit prerequisite**, or Arena's dashboard ships empty. +2. **The leaderboard is operator-gameable and mis-specified.** (a) The server must recompute every aggregate in SQL from the `arena_episodes` it just inserted, never trust the pusher's numbers. (b) Ranking on a mean over *scored* episodes is Arena's own "score what survived" trap. Store `reward_mean_scored` **and** `reward_mean_attempted` (errors as 0), rank on attempted, refuse to rank below a scored-fraction floor. +3. **`arena_episodes` flattens a list.** `Episode.traces` is a list; `reward`/`rewards`/`stop_condition`/`last_reply` are per-**Trace**. Primary key must be `(run_id, episode_id, trace_id)` with an `agent_name` column — otherwise the schema is wrong for `grand-exchange-live` (simulator as second role) and for `redaction-pressure` (genuine two-agent). +4. **The trace viewer publishes the held-out slice.** `GET /api/arena/runs/:id/episodes/:eid` at `user` scope returns full transcripts, which contain held-out task data and, for oracle runs, the answers. One leaked invite publishes the held-out corpus. **Transcripts are admin-only or redacted.** This settles the spec's open question on read scope. +5. **The compute-scene-vocabulary blocker is fabricated.** `computeScene` is a free-form slug regex, not `z.enum(SCENE_NAMES)`; `stock`, `nemotron` and `qwen38` all pass today. Delete that precondition. +6. **`portal_host_samples` and `portal_container_samples` have no node identity** — no `node_id`, no host column, no PK. Per-node charts need `ALTER TABLE … ADD COLUMN IF NOT EXISTS node_id text` plus an index first. +7. **`scored = ok AND rewards non-empty` is wrong.** `write_episode` dumps with `exclude_none=True`, so a `None` reward is dropped from the file entirely — "not measured" and "key absent" are indistinguishable on the wire. Require at least one non-null `Reward`, and carry the run's declared reward-name set from the config. +8. **`Reward.value` is a `@property`** and never appears in dumped JSON. Compute `score × weight` in the ingester. +9. **0.3.0 → 0.4.0 changes the directory layout, not just the config filename** — HEAD produces one flat `------` dir with no nested uuid, and `RunConfig.id` is a `PrivateAttr` absent from `configs/eval.json`. Derive the run id from the directory basename. +10. **Name the private console `/nodes`**, not `/compute` — `/compute` is already a public marketing section with its own nav. +11. Put `isAppHost` in `packages/shared` — two independent string checks that must agree, with a silent asymmetric failure mode (dashboard rendered under a 404 status). +12. `sendShell(url, reply)` takes no request; the signature and both call sites change. `spa-fallback.test.ts` needs a case. `@radix-ui/react-select` is **not** installed. + +**Revised P phasing.** The honest first ship is **P1: Caddy host + host-aware routing + `renderAppShell` + AppShell + Overview + Worlds + Org + Inbox + the `api.ts` trisection.** All of that runs on endpoints that exist today. **Arena (schema, 6 routes, ingester, 4 pages) is P2, a full workstream of its own**, and it is gated on a real eval having run. + +### W — website visual vocabulary · feasible; **the data pipeline must be redirected, not built** + +Stands: the diagnosis (the site draws what it hasn't measured and narrates what it has), `MeasureTable` / `Bar` / `FlowStrip` / `SpecSheet`, inline SVG with `currentColor` and **no charting dependency** (the `gold-500 → gold-700` theme flip alone settles this), the grouped `StateList`, and the four live factual errors. + +Corrected: + +1. **`arena-runs.json` has no machine source and must not be created.** `probe.py` does not run models; there is no `traces.jsonl`. **The site already has the right contract: `apps/web/src/data/eval-results.ts` + `eval-results.json` + `eval-results.parse.test.ts`, with a `{model, host, quant, serving, checkpoint}` target tuple and an explicit "null means not measured" rule.** The Nemotron run belongs there. Creating a second incompatible file guarantees the drift the pipeline exists to prevent — and putting it in `eval-results.json` also fixes **a fifth live error the audit missed**: `arena.tsx:519` says "No model scores are published yet, so nothing appears on Evals" while `README.md:88-105` publishes seven. +2. **The crude-maximiser labels have no machine source either.** `probe.py`'s functions return only `{inaction, crude, plausible, oracle}`; "redact everything" / "ban everyone" / "buy and sell at market" exist only as markdown italics. Emitting that column means **changing `probe.py`'s return shape — which is B's file.** Fold it into B1's registry work or accept it as a hand-typed literal, but do not schedule it as a third-party edit to `probe.py`. +3. **`check:arena` must not execute `probe.py`.** The `sync-scenes.mjs` precedent parses committed YAML from an anonymous public clone; running a 772-line probe under `uv run --with regex` is a slow, flaky gate that contradicts the hermetic-build rule. Emit JSON in `arena`'s CI, commit it, and have `check:arena` compare bytes. +4. **`Diagram` fails DESIGN.md §4.2's own gate** — no variants, no CVA variants object. It is a one-off `
` per page, exactly like `scaling-plot.tsx` / `spread-field.tsx` already are. Do not put it in `components/ui/`. +5. **Do not put a nine-verb FlowStrip above the fold on `/forge`.** Every StateList item on that page is `done: false`, including "No repository, no tag, no release." Drawing an unbuilt pipeline is the exact failure the audit indicts. +6. **The landing-page markup claim is wrong**, and it invalidates the landing plan's premise: `landing.tsx` contains **two** `` elements, not six. `ForgeChain` is an `
    `, `ArenaBand` is nested `
    `s and 32 ``s, `StreamSignal` and `SimulateViewport` are divs. The real coupling is ~33 class references into `machine-home.css` (35 KB) carrying reveal/scan animations. **Treat the landing as its own step with its own CSS audit.** +7. **`Bar` has one existing hand-rolled call site, not two** — `BudgetMeter` is an N-segment `flexGrow` track with a composed `aria-label`, and `BarProps` cannot express it. `Bar` earns its place through `MeasureTable`'s cells, not through BudgetMeter. +8. ~~**The scenario count is 16, not 20**~~ ⚠️ **WRONG as of 2026-08-21 15:50 — it is 20**, five registries × four, and `office-jobs-v1` is not phantom. Another session merged it into the shared `tera` tree mid-session. Counts are twelve / five / twenty; the site's original copy was right and W0's correction was reverted. +9. Four more table-shaped components exist that the "seven call sites" audit missed: `components/compute/model-table.tsx`, `components/publish/concurrency-table.tsx`, `ledger.tsx`, `rate-strip.tsx`. Design `MeasureTable`'s API against them. +10. No tests are proposed. New generated data + a check script are exactly what gets a parse test in this repo. + +**Revised W:** `/arena` first (it is also a truth fix), `/bench` second, `/stream` third — but `/arena`'s number pipeline routes through `eval-results.json`, and the `office-jobs-v1` deletion happens **once**, in one place, coordinated with D and P. + +### C-compute — rent to train, Spark to serve · feasible; **its headline correction is false** + +Stands and is the most immediately actionable: the policy itself; the measured spark-1 tenancy (121 GB total / 84 GB used / 36 GB free, SGLang + Chatterbox + vox + ASR co-resident, Chatterbox's ~8.4 GB/hr leak having killed brain before); `prime login` as step zero because `api_key` is empty (verified); **Arena → prime-rl needs no adapter** — `EnvConfig` is `SerializeAsAny[vf.EnvConfig]` and the taskset tree is the same shape, prefixed; the seat-name-is-a-TOML-key trap; Hosted Training being unavailable while E is parked (`rl.py:1522-1524` — bare v1 ids do not resolve through the Hub); the sm_121 lock contradiction in Forge; `TRAINERS = ("axolotl","trl")`; **no price is hardcoded anywhere and none should be invented.** + +Corrected, blocking: + +1. **`--model brain` does NOT 404.** SGLang does not validate the model field on the chat route; it returns 200 with a real completion. Two of the four "stale facts" and two proposed file edits rest on a false claim. **The real hazard is the opposite and worse: it silently succeeds.** Re-running the `README.md:88-89` baseline today produces **Qwen3.8-27B** numbers under a command that reads `--model brain`, and they will be tabulated beside Nemotron numbers as comparable. The correction the arena docs need is: **pin the served id `brain-qwen38-dspark` and the scene name in every reproduction command**, not "fix a 404". +2. **`uv run eval grand-exchange-live --harness.id null` is not valid.** `EvalConfig` has no top-level `harness`; `--harness.` is admitted only to render a did-you-mean. Canonical: `--env.agent.harness.id null`, or the TOML block `configs/grand_exchange.toml` already uses. +3. **`uv sync --package prime-rl --package grand-exchange-live` cannot work.** prime-rl's `[tool.uv.workspace] members` is a closed list of its own vendored envs. Use `uv pip install` of a built wheel or git URL into the env-server venv, or add a path source. +4. **`forge.tsx:114` is not stale.** The remote is private, `git tag` is empty, `status="In development"`, no `source=` prop. Nobody can clone it. The page's own README warns "This section has been wrong three times" — this edit would make it four. +5. **Workstream-letter collision:** PLAN.md already defines a "workstream C" (more interactive environments). Rename this one — **"Compute Policy"**, no letter. +6. The wordle wall-clock anchors are fabricated (the README says 2 GPUs, the toml declares 8 and 200 steps; neither states a duration). Delete them and produce our own. +7. `torchao` and `nixl-cu12` are pinned to **x86_64-only** wheel URLs while `torchao` is a hard runtime dep — the "aarch64 wheels for every heavy dependency" claim does not hold, which sharpens the aarch64-variance risk. +8. `prime disks` attach with `--disks`, not `--attach-disk`. +9. **Two things nobody has planned:** how the DCP checkpoint gets *off* the pod before `prime pods terminate` destroys the disk (needs a `--disks` volume at create time), and the DCP → HF shards → NVFP4 → Compute model-id conversion, which nothing on disk automates end-to-end (prime-rl has `convert_to_hf` but no CLI entrypoint). +10. **The link nobody drew:** rental hours are driven by whether the turn budget binds. `group_size × batch × steps` of *multi-turn* rollouts against a rented inference fleet is real money. If B2's exhaustive policy scores at the oracle, the run burns GPU hours measuring nothing. **Rule 4 is a compute-policy constraint, not only a design one.** + +--- + +## 2. Parallelization map and file collisions + +### What can start today, concurrently + +| Lane | Work | Repo | Blocked by | +|---|---|---|---| +| **1** | **B1** — discovery de-hardcode + timeout + step name + `uv.lock` cleanup | `arena` | nothing | +| **2** | **A step 2** — `live.py` engine + unit tests + **re-measure the ladder** | `arena` | nothing (own worktree) | +| **3** | **D step 1** — worker, vendored TS, hashes, `test_replay.py` | `arena` (new dir only) | nothing | +| **4** | **Compute Policy step 0** — `prime login`, catalogue + prices, doc the corrected spark-1 facts | none / `arena/docs` | nothing | +| **5** | **P1** — Caddy host, host-aware routing, AppShell, Overview/Worlds/Org/Inbox | `lumbridge-v4` | nothing | +| **6** | **W step 0** — the five factual corrections + `eval-results.json` decision | `lumbridge-v4` | nothing | + +Six lanes, and only lanes 1/2/3 share a repo. **Use a worktree per lane** — this repo took divergent concurrent pushes on 2026-08-19. Merge, never force. + +### True dependency edges + +``` +B1 ──────────────────────────────────────────┐ + ├──► A step 7 (probe row) ──► A ships +A steps 1-2 (engine) ──► A re-measure ──► A steps 3-6 ─┘ + │ + └──► B2 (policy machinery, calibrated on real numbers) + │ + ├──► C1 drop-table + ├──► C2 bot-detection + ├──► C3 fault-localisation + ├──► C4 redaction (2-agent) + └──► D step 5 (tera probe rows) + +D step 1 (replay gate) ──► D crow-nav ──► D flight/office + +A ships ──► one real eval run ──► traces.jsonl exists ──► P2 (Arena dashboard) + │ + └──► W /arena numbers (via eval-results.json) + +prime login ──► rented-pod runbook ──► first RL run ──► Forge prime-rl renderer +``` + +Two edges that were wrong in the source specs and are now correct: +- **B1 does not depend on A** (PLAN.md's "B depends on A" applies only to B2). +- **D is no longer file-independent of A/B/W** — it lives in `arena` and edits `probe.py`, `ci.yml`, `test_probe.py`, `README.md`, plus `arena.tsx` and `seo.ts`. + +### Merge-conflict map — the specific paths + +**Tier 1 — will conflict, must be sequenced or de-hardcoded first:** + +| Path | Claimed by | Resolution | +|---|---|---| +| `arena/probe.py` | A (§11), B1 (registry), B2 (machinery), D (§7), C1-C4, W (needs a return-shape change for crude labels) | **B1 lands first.** After B1, adding an env is one appended function at EOF — a mergeable case. Nothing else touches this file until B1 is merged. | +| `arena/tests/test_probe.py` | A, B1, D | B1 removes it from every future workstream's diff. Never edited again. | +| `arena/.gitea/workflows/ci.yml` | A, B1, D (needs Node + replay test + a one-dir-four-wheels change) | B1 lands the glob; D then adds *steps*, not list entries. **D's one-directory-four-tasksets layout actively breaks B1's one-dir-one-wheel discovery** — D must add a `[tool.arena] tasksets = [...]` key or equivalent, and B1 should be told to read it. | +| `arena/configs/grand_exchange_live.toml` | **A, B(§1.7.2), Compute Policy** — three specs create the same file | **A owns it.** Delete it from the other two file lists. | +| `arena/environments/grand_exchange_live/*/taskset.py` and `session.py` | **A and B both specify the module layout and class signatures** | **A owns the contract.** B2 adapts. | +| `arena/README.md` | A (rows), D (rows + house rule 4), Compute Policy (baseline qualifier), W (source of truth for the site tables) | Sequence: A → D → Compute Policy. W reads it, never writes it. | +| `arena/PLAN.md` + `MULTI_TURN.md` | B, D (scope amendment), Compute Policy | One session, one pass, after the D-scope decision lands. | + +**Tier 2 — cross-repo, same file:** + +| Path | Claimed by | Resolution | +|---|---|---| +| `lumbridge-v4/apps/web/src/pages/v4/arena.tsx` | **D** (delete `office-jobs-v1` at :203 and :511) and **W** (full rebuild of the page) | **W owns the file.** D contributes the *fact* (four manifests, 16 scenarios), not the edit. | +| `lumbridge-v4/services/api/src/seo.ts` | **D** (:271 twelve→eleven, five→four) and **P** (`renderAppShell`, drop `/app` from `APP_ROUTE_PATTERNS` at :698) | Different regions; sequence P after D's one-line copy fix, or vice versa. Low risk but both must run `seo.test.ts` (`ROUTER_PATHS` at :36) and `spa-fallback.test.ts`. | +| `lumbridge-v4/apps/web/src/pages/v4/forge.tsx` | **W** (FlowStrip, StateList) and **Compute Policy** (:59-62 delegation copy) | W owns it. Compute Policy contributes one sentence, **only after a rented run has actually happened**. The `:114` edit is cancelled (see verdict). | +| `lumbridge-v4/apps/web/src/data/eval-results.json` | **W** (Nemotron run) and **P2** (the dashboard's run store) | The public file is a curated summary; the dashboard is the archive. Decide the direction (Decision 6) before either writes. | + +**Tier 3 — three tracked `uv.lock` files are failing CI on main right now.** `git rm --cached` them in B1's commit, before anyone blames a new failure on a new environment. + +--- + +## 3. Sequencing + +### Phase 1 — genuinely shippable, ~2 weeks, six parallel lanes + +1. **B1** (arena): tomllib discovery, `@probes` registry, timeout 180, `uv.lock` untracking, step rename. *Gate: `probe.py` still prints seven `ok` rows; CI green with zero list edits.* +2. **A steps 1-2 + re-measurement** (arena, worktree): frozen generator copy + equality assertion; `LiveBook`/`Leg`/`Ledger` with unit tests; **then re-run the full policy ladder and publish the corrected numbers.** *Gate: the ladder reconciles with the reward formula; `horizon`, `FREEZE_RANGE`, `TARGET_SHARE`, `TURNS` are chosen from that measurement, not from the spec.* +3. **D step 1** (arena, new dir): worker + vendored TS + `hashes.py` + `sync_tera.py` + `test_replay.py`. *Gate: 32/32 replay exact, tampering rejected, hash check green — and a decision on the 4 unpinned files.* +4. **Compute Policy step 0**: `prime login`; record real `availability` output and prices; write `arena/docs/COMPUTE_POLICY.md` with the corrected spark-1 facts (`brain-qwen38-dspark`, silent-wrong-model hazard, 36 GB headroom). +5. **P1** (lumbridge-v4): Caddy `app.` block with `bind 10.0.0.2` and the non-empty-body check; host-aware `App.tsx`; `renderAppShell` + `sendShell` signature change + `seo.test.ts` + `spa-fallback.test.ts`; AppShell; Overview / Worlds / Org / Inbox; delete the 23 dead `api.ts` methods. +6. **W step 0** (lumbridge-v4): the five factual corrections — `arena.tsx:330` "Nine tasks", `landing.tsx` card 02, the 4-row probe table → 7 rows, `office-jobs-v1` deleted, `arena.tsx:519` "no model scores published". Ship the Nemotron run into `eval-results.json`. + +**Phase 1 ends with**: a green de-hardcoded CI, a measured live engine, a proven TS replay bridge, a live private console on real endpoints, a truthful public site, and a Prime Intellect account that can rent. + +### Phase 2 + +7. **A steps 3-9** → `grand-exchange-live` ships with its probe row (needs B1 merged). +8. **One real eval run** against spark-1, pinned to `brain-qwen38-dspark`, with the empty-rewards count checked. *This is a hard prerequisite for P2 and for any site number.* +9. **D crow-nav** end to end (role named `agent`, `vf.Taskset[...]` generic form, `oracle` op in the worker, `gate` as a multiplier). +10. **W /arena rebuild** + `/bench` on the new primitives. +11. **First rented run**: pod create with an attached disk, `install.sh`, wheel install, baseline eval, `uv run rl`, re-eval, before/after table. + +### Phase 3 + +12. **B2** — policy machinery, calibrated against A's measured exhaustive/oracle gap, with block-wise ladders replacing the single 0.02 margin. +13. **P2** — Arena schema (per-trace PK), server-side aggregates, admin-only transcripts, ingester for both output layouts, the four pages. +14. **C1-C4** — the four remaining interactive environments, fanning out from B2. +15. **D flight + office-nav**; drive-101 per Decision 3. +16. **Forge**: `prime-rl` renderer (LoRA RL first), x86_64 train lock, retitled aarch64 lock, DECISIONS entry, checkpoint→HF→NVFP4→Compute-model-id stage. +17. **W /stream**, landing (with its own `machine-home.css` audit), compute/simulate/about. +18. **E** — Hub push, when un-parked. + +--- + +## 4. Decisions Karti still owes, ranked by what they block + +| # | Decision | Blocks | Recommendation | +|---|---|---|---| +| **1** | **The agent seat name.** `agent` (matching `SingleAgentEnvConfig`) or a custom name? It becomes the TOML key in `configs/*.toml` **and** in every prime-rl `rl.toml`, on both A and D. Renaming later parses fine and silently has no seat. | A, D, every future env, every training config | **`agent`, everywhere.** One word, decided once, written into each env's README. | +| **2** | **Does D live in `arena` or `tera`?** The spec's packaging argument is strong (`tera` has no Python toolchain) but it overturns PLAN.md:138 and makes D collide with A/B/W. | D's entire file layout, the CI wheel loop, PLAN.md | **`arena`**, and amend PLAN.md's parallelization map in the same commit. | +| **3** | **Does `tera-drive-101` ship?** Measured: monotone in throttle, optimal policy is one constant string, and the 0.9 band makes any positive throttle score ~0.95+. Also decides whether the site says four spatial environments or three. | D's scope, `arena.tsx`/`seo.ts` copy, the Hub listing | **Ship it labelled a bridge-correctness environment with no rule-4 and no rule-3 claim, or defer.** Do not list it as a fifth RL environment. | +| **4** | **A's freeze mechanic: flat per-look, or proportional to items re-quoted?** Flat is measured and works; per-item is the better mechanism, makes the look count genuinely vary, and forces a much larger surgical reference. | A's engine, and therefore B2's calibration | **Ship flat.** Record per-item as the identified strengthening. Do not build an unmeasured mechanism on the critical path. | +| **5** | **Arena transcript visibility.** Full transcripts contain the held-out slice and, for oracle runs, the answers. | P2's route scoping, and house rule 1's integrity | **Admin-only.** This is not an org-politics question; it is a contamination question, and that settles it. | +| **6** | **Where does model-run data live?** Public `eval-results.json` (already typed, tested, rendered on `/evals`) vs P2's `arena_runs`/`arena_episodes` tables. | W's whole data pipeline, P2's schema | **Both, with a direction:** the dashboard is the archive, `eval-results.json` is a curated public summary emitted from it. Never two independent hand-maintained files. | +| **7** | **Is `nemotron` coming back as the production brain, or is `qwen38` the default?** `README.md`'s baseline, D23/D24's Forge preflight, and the site's Arena numbers all assume Nemotron; the box serves Qwen3.8-27B. | Every reproduction of the baseline; what "inference in production on the Spark" names | Needs an answer before anyone re-runs a benchmark. | +| **8** | **First Forge/prime-rl integration: LoRA RL or full-FT?** LoRA reuses merge + attest unchanged and matches `alphabet-sort`; but LoRA rejects `--gpu-type` and falls back to filesystem weight broadcast. Full-FT needs a new node kind and the tensor-delta-map gate never fires. | Forge's roadmap, and what its manifest is for | **LoRA first** — it keeps Forge's headline claim intact. | +| **9** | **Is app.\* the only authenticated origin?** Yes means the apex loses `/login`, `/signup`, `/app`, `/go`, and all three existing sessions are signed out on cutover day. | P1's scope, apex route deletions | **Yes**, and state the sign-out as an accepted cost for three accounts. | +| **10** | **W's audience.** Arena/Bench/Stream (brokenness × argument) vs Arena/Landing/Bench (the Prime Intellect Hub reader). | W's priority order only | Answer when P1 lands; it does not block anything upstream. | + +--- + +## 5. Risk register, highest first + +| # | Risk | Retired by | +|---|---|---| +| **1** | **The whole reward design of `grand-exchange-live` rests on numbers from a throwaway engine that no longer exists, and the ladder does not reconcile with the reward formula.** `horizon=60`, `FREEZE_RANGE=(1,4)`, `TARGET_SHARE=0.85` and `TURNS=8` are all locked against arithmetic that is off by 0.011 in the one place it can be checked. | Building `live.py` FIRST and re-measuring every row before any constant is frozen. This is Phase 1, lane 2, and it is the single most important task in the programme. | +| **2** | **Every reward in Arena saturates, so "exhaustive < oracle" may be structurally unreachable.** `profit_ratio` and `efficiency` clip at 1.0; `clean` is a boolean band. A 0.02 margin against a 0.25 quantum is a cliff. If the gate can't open, rule 4 is unenforceable and every interactive environment is unfalsifiable. | A's re-measurement producing a real exhaustive/oracle gap; B2 then calibrated against it with block-wise ladders. If no gap exists, the reward shape must change before C1-C4 are built on it. | +| **3** | **No `traces.jsonl` exists anywhere.** Everything downstream — the dashboard, the site's numbers, the leaderboard, the before/after training table — is waiting on one eval that has never completed. The one run that was attempted pointed at Prime's inference API with an empty key. | One completed eval against `brain-qwen38-dspark` with the empty-rewards count checked. Cheap, and it unblocks P2, W and Compute Policy simultaneously. | +| **4** | **`eval` exits 0 when every rollout errors, and multi-turn has far more ways to error than one-shot.** `turn()` on a closed interaction raises; `turn(msg)` on turn 1 of a prompted task raises. Combined with risk 3, a "successful" run can be entirely empty. | A CI/runbook step that asserts non-empty `rewards` on ≥N episodes, in `arena`, in D's replay gate, and in P2's ingester. Never trust the exit code. | +| **5** | **Silent wrong-model comparison.** `--model brain` returns 200 from a box serving Qwen3.8-27B. Re-running the Nemotron baseline today produces different-model numbers under an identical command, and they will be tabulated as comparable. | Pinning `brain-qwen38-dspark` (and the scene) in every command in `README.md`, `PLAN.md`, `MULTI_TURN.md` and `COMPUTE_POLICY.md`. Trivial; do it in Phase 1. | +| **6** | **`probe.py`, `tests/test_probe.py` and `ci.yml` are edited by five of six workstreams, in a repo that already took divergent concurrent pushes.** | B1, first, before anything else touches `arena`. | +| **7** | **The reward denominator is editable with every `tera` gate green.** ⚠️ CORRECTED 2026-08-21: not `offices/sites.ts` (mutating it produces a bit-identical episode — refuted twice). The hole is **`src/arena/index.ts`**, which re-exports the scripted baselines that ARE the denominator. Demonstrated: a halved office-nav baseline changes return 4.744 → 4.098 while `arena:source-hashes` prints ok and exits 0. | Pinning `src/arena/index.ts` in `tera/scripts/check-arena-source-hashes.mjs` and regenerating `sourceHashes.ts`. One line, in a repo this session did not own. | +| **8** | **spark-1 has 36 GB of 121 GB free with four resident model services, one of which leaks ~8.4 GB/hour and has killed the brain before.** "Production inference on the Spark" is a tenancy claim held together by the governor on :8011. | A mandatory headroom check before any new resident model, and Compute's watchdog verified live. | +| **9** | **`PRIME_API_KEY` is empty**, blocking not just the parked Hub push but every rental command, every price, and the entire training half of the compute policy. | `prime login`. One command. | +| **10** | **Two of D's four environments cannot discriminate under the 0.9 ceiling band** — drive-101 gives ~0.95+ to any constant throttle, office-nav collapses to `efficiency` alone. | Re-deriving the band per environment against the measured return distribution, or shipping fewer environments. | +| **11** | **The held-out corpus leaks through P2's trace viewer at `user` scope.** One invite code. | Decision 5 — admin-only. | +| **12** | **Effort is understated across the board.** A is L/XL; B2 is not B's to size (its §1.8 is L/XL and belongs to A); D is L/XL; P's "Phase 1" is XL; W's item 1 alone is L/XL across two repos. Read as written, this programme looks like six L's and is closer to four XLs. | The phasing in §3, which cuts P1 out of P, B1 out of B, and D-step-1 out of D. | +| **13** | **Rented GPU hours are burned by a non-binding turn budget.** `group_size × batch × steps` of multi-turn rollouts against a rented inference fleet is real money, and nothing in the compute doc connects to rule 4. | Making "exhaustive < oracle by a measured margin" a precondition of dispatching any RL run — write it into `COMPUTE_POLICY.md`. | +| **14** | **The checkpoint has no exit path.** `prime pods terminate` destroys ephemeral disk; the DCP → HF → NVFP4 → Compute-model-id conversion is unautomated. | `--disks` at pod-create time, plus a `forge quantize/publish` stage. Discover this before the first successful training run, not after. | +| **15** | **GH200 may not appear in `prime availability` at all** — the CLI names only H100_80GB, H200_141GB, B200_180GB, RTX_PRO_6000. "GH200-class" may mean H200_141GB in practice. And `perf.py` has no GH200 branch, so MFU may silently fall back to A100's 312 TFLOPS. | `prime login`, then one `availability gpu-types`, then one `nvidia-smi --query-gpu=name` on a real pod. | + +--- + +## 6. Build this first + +**`environments/grand_exchange_live/grand_exchange_live/live.py` — the stepped engine and its unit tests — and, concurrently, workstream B1's discovery de-hardcoding.** + +Everything else in the programme is downstream of two facts we do not yet have: whether the live market's reward ladder actually reconciles (risk 1, which invalidates A's horizon, freeze, TARGET_SHARE and turn budget if it does not, and takes B2, C1-C4 and half the compute argument with it), and whether three files can stop being a five-way merge conflict before five workstreams reach for them. B1 is an afternoon and removes the repo's only structural collision permanently; `live.py` is the smallest artifact that can turn six specs' worth of `/tmp` measurements into something the shipped code agrees with. \ No newline at end of file diff --git a/MULTI_TURN.md b/MULTI_TURN.md new file mode 100644 index 0000000..73c2394 --- /dev/null +++ b/MULTI_TURN.md @@ -0,0 +1,208 @@ +# Scope: the multi-turn gap + +Status: scoping document, 2026-08-21. Nothing here is built yet. + +Every environment in this repository sets `[env.agent.harness] id = "null"` — one +completion, no tools, no runtime. That is the right choice for a pure-trace data +environment and it is also the whole of what Arena currently measures. Nothing here +scores an agent *acting*: choosing what to look at next, paying for a turn, or +revising after seeing a consequence. + +This document scopes closing that gap. The headline finding is that it is three +separable tiers, not one, and the cheapest tier needs no new infrastructure at all. + +## What upstream actually requires + +Read from `~/vendor/prime-intellect/verifiers` (v1, installed version 0.3.0), not +from blog posts. + +**Multi-turn does not imply a container.** `vf.Env.run(task, agents)` programs the +control flow host-side. `agents..interaction(task)` opens a rollout and holds +it open; `await interaction.turn(observation)` sends one user turn, runs one harness +segment, and returns a `Segment` with `.last_reply` and `.terminated`. The simulator +runs in-process, as the user. The harness can stay `null` throughout. + +The reference implementation is `verifiers/v1/tasksets/textarena/taskset.py` — 90 +lines, drives a real game engine, scores off the engine directly. Read it before +writing anything. + +**An Env ships inside the taskset's own wheel.** `EnvConfig.id` empty means "the +taskset's own env, else `SingleAgentEnv`". Export the `Env` subclass from the +package `__init__.py`, exactly as `environments/wordle` exports `TextArenaEnv`, and +`[env.taskset] id = "..."` picks it up with no extra config. + +**Available in the installed version** — confirmed, not assumed: + +- harnesses: `null`, `bash`, `browser_use`, `claude_code`, `codex`, `hermes_agent`, + `kimi_code`, `mini_swe_agent`, `openclaw`, `pi`, `pool`, `rlm`, `terminus_2` +- runtimes: `subprocess` (debug only — rollouts share side effects), `docker`, + `modal`, `prime` +- prebuilt envs: `single_agent`, `best_of_n`, `agentic_judge`, + `shared_agentic_judge`, `user_sim` + +**The interception server** sits between harness and provider. It speaks each +harness's native dialect (Anthropic Messages for Claude Code, OpenAI Responses for +Codex), builds the trace live, forces sampling params the harness will not expose, +and can rewrite tool results and web-search results to block reward hacks. It is +mandatory for tiers 2 and 3 and free for tier 1. + +## The three tiers + +| | Mechanism | Harness | Runtime | New infra | +|---|---|---|---|---| +| **1. Interactive** | `vf.Env.run` + `interaction.turn`, simulator in-process | `null` | none | **none** | +| **2. Tool-using** | Taskset exports a `Toolset`, served as MCP | `bash` | `subprocess` / `docker` | image per env | +| **3. Agentic** | Full coding agent in a sandbox | `claude_code`, `codex`, `openclaw`, `pi` | `docker` / `prime` | image + interception + budget | + +Tier 1 is the one to build first, and it is where our existing simulators already +sit. Tiers 2 and 3 are real work with real cost and should follow evidence from +tier 1, not precede it. + +## What we already have that plugs in + +**Six of the seven environments here own a simulator that is already steppable.** +They are one-shot only because the taskset hands the model the whole observation up +front. Each has an obvious interactive form: + +| environment | one-shot today | interactive form | tier | +|---|---|---|---| +| `grand-exchange` | read 30 ticks of history, place one basket of orders | quote, see fills and the next tick, re-quote, over a turn budget — inventory and unfilled orders carry | 1 | +| `drop-table-inference` | read a fixed kill log, estimate the table | **buy** observations: each turn spends budget on more kills, then answers — measures when to stop looking | 1 | +| `fault-localisation` | read the whole incident log, name the cause | the log is *not* given: query it by service, time window and level under a query budget | 1 → 2 | +| `redaction-pressure` | write rules, scored by a scanner | two agents: a redactor and an extractor scored on what it recovers — the counterweight becomes an opponent | 1 (multi-agent) | +| `schema-migration` | emit a migration, executed against held-out rows | a real SQLite in a container the agent runs SQL against, with rollback | 3 | +| `bot-detection` | classify from six labelled examples | request labels one at a time under a budget (active learning) | 1 | +| `canary-trap` | write probes | no natural interactive form — leave one-shot | — | + +**Tera's four spatial environments are already multi-turn — in the wrong language.** +`~/repos/gitea/tera/src/arena/` implements `tera.arena/v1`: Gym-style +`reset(seed, scenario)` / `step(action)` / `snapshot` / `restore` / `trace` / +`replay`, FNV-1a-64 checksums, SHA-256 source pins, train/dev splits, counterweighted +rewards, and a CI gate that proves inaction finishes below zero and a scripted +baseline finishes positive. That is a stricter contract than most of what is on the +Hub — and none of it is reachable from `uv run eval`, because it is TypeScript with +its own contract rather than a verifiers taskset. + +The work there is a **bridge**, not a simulator: a Python `vf.Env` whose `run()` +drives the TS package over a subprocess or a small local HTTP loop, turning +`step()` observations into `interaction.turn()` messages and the existing component +rewards into `trace.record_reward` calls. The replay checksum makes the bridge +verifiable — a bridged episode must replay identically in the TS package or the +bridge is wrong. That is a rare property and worth building the bridge around. + +⚠️ **RESOLVED 2026-08-21, 15:50 — `office-jobs-v1` EXISTS.** It was absent at 13:00 and +present at 15:50, because another session fast-forward-merged `feat/tera-ultracode-integration` +into `~/repos/gitea/tera` mid-session (reflog `8738367`). It is committed (`e378a03`, +`120eac8`, dated 2026-08-19), is in `ARENA_MANIFESTS`, is documented in `tera/ARENA.md`, and +passes `npm run arena:source-hashes`. **The true counts are twelve environments, five spatial, +twenty scenarios** — measured, not inferred. The site's original copy was right; the +correction to eleven/four/sixteen was right when made and wrong an hour later, and has been +reverted. Lesson: `tera` is a shared tree with live concurrent sessions — re-check its HEAD +before acting on a claim about what it contains. + +~~⚠️ **`office-jobs-v1` does not exist.** `lumbridgecorp.com/arena` and +`services/api/src/seo.ts` both advertise five spatial environments; `ARENA_MANIFESTS` +in `src/arena/index.ts` holds four (`drive-101-v1`, `office-nav-v1`, `crow-nav-v1`, +`california-flight-v1`) and there is no `officeJobs.ts`. The real total is eleven, +not twelve. Decide whether to build it or correct the page before publishing +anything that cites the count.~~ (superseded — see above) + +## The house rules under multi-turn + +The three rules were written for one-shot scoring. Two survive unchanged and one +does not, and there is a fourth the interactive tier forces. + +1. **Graded on what the model did not see** — unchanged and easier: a stepped + simulator generates its held-out continuation by construction. +2. **No single-sided reward** — *harder*. Multi-turn adds degenerate strategies that + have no one-shot analogue: stalling to burn the clock when the score is + time-averaged, spamming cheap turns when observations are free, refusing to act + when acting risks a penalty. Every new reward needs a counterweight in the *turn* + dimension, not just the answer dimension. +3. **A zero floor and a reachable ceiling, demonstrated** — unchanged, but the floor + now has to hold across a whole trajectory, not one reply. +4. **New: the budget must bind.** An interactive environment where the optimal + policy is "use every turn" is measuring nothing the one-shot version did not. + Observations must cost something, and the oracle must be shown to stop early. + +**`probe.py` must grow degenerate *policies*, not just degenerate answers.** Today +it scores four fixed strings per environment. The interactive tier needs at minimum: +the staller (never acts), the spammer (acts every turn at random), the impatient +(answers on turn 1 with no observation), and the exhaustive (spends the entire +budget then answers). The gate stays the same shape — inaction 0.000, oracle 1.000, +exit 1 otherwise — plus a new assertion that the exhaustive policy scores *below* +the oracle. If it does not, rule 4 is violated and the environment does not ship. + +This is the differentiator. Nothing in `prime-envs` (88 environments) or +`community-environments` (109) ships a degenerate-policy gate. + +## Dual publishing + +Both targets take the same wheel; the repo layout already mirrors verifiers' own. + +- **Prime Intellect Hub**, our own namespace: `prime env push -v PUBLIC`. + We keep authorship and versioning, and the listing can link back to + lumbridgecorp.com/arena. +- **`community-environments`**: a PR to that repo, published under the + `primeintellect` org by their CI on a version bump. Extra distribution, but it + hands the namespace to them. Secondary, not primary. +- **lumbridgecorp.com/arena**: unchanged — the page documents the same wheel. + +⚠️ Its README still documents the **legacy** flow (`load_environment`, `vf-eval`, +`uv pip install -e .`). Ignore it; follow `verifiers/docs/v1/`. + +⚠️ **Blocker: `PRIME_API_KEY` is not set on amd-server.** `prime config view` shows +`API Key: Not set`. The `pit_` key lives in karti's memory note on cloud-1 +(`.claude/projects/-home-ubuntu/memory/prime-intellect.md`) and is the same +credential for `api.pinference.ai` and `api.primeintellect.ai`. Nothing pushes until +it is configured here. + +## Preconditions, verified on this box + +- `uv run eval @ configs/canary_trap.toml --dry-run` → exit 0, config written ✓ +- verifiers 0.3.0 with all 13 harnesses and 4 runtimes present ✓ +- Docker 29.1.3 running (tier 2/3 runtime) ✓ +- spark-1 `brain` on `http://100.127.247.67:8001/v1`, model id `brain` ✓ +- `prime` CLI installed (0.6.25; 0.6.26 available) ✓ +- `PRIME_API_KEY` ✗ — see above + +## Proposed sequence + +**Phase 1 — one interactive environment, end to end.** `grand-exchange` is the +candidate: the market simulation already steps, the counterweight (liquidity, the +random-walk trap) already exists, and turning one basket into a quote/fill loop is +the smallest change that produces a genuinely new measurement. Deliverable: a +`vf.Env` in the existing wheel, a second config, and probe coverage for the four +degenerate policies. This is where rule 4 gets written for real. + +**Phase 2 — extend `probe.py` to policies and re-gate everything.** Whatever +phase 1 teaches about degenerate trajectories applies to every later environment. +Do this before building a second one, not after. Note that `tests/test_probe.py` +and the wheel-build loop in `.gitea/workflows/ci.yml` hardcode the environment +list — both need editing or CI goes red. + +**Phase 3 — the Tera bridge.** One Python `vf.Env` that drives the existing TS +package, validated by replaying every bridged episode through `replay()` and +requiring an identical checksum. Lands four environments at once and makes the +strongest existing asset publishable. Larger and riskier than phase 1; the +process boundary and the action encoding are the unknowns. + +**Phase 4 — first Hub push**, once phases 1–3 give three to four interactive +environments alongside the seven one-shot ones. Needs the API key. + +**Phase 5 — tier 2, then tier 3.** `fault-localisation` as a query-budgeted +tool-using environment, then `schema-migration` against a real database in a +container. Only after tier 1 has produced measured scores worth extending, and +with the concurrency cost of a container per rollout understood first. + +## Open questions + +- Does a bridged Tera episode replay to an identical checksum through a process + boundary? If not, the bridge cannot be verified and phase 3 gets much more + expensive. +- What is the per-rollout wall-clock cost of the `docker` runtime here, at the + concurrency an eval sweep needs? Unmeasured. It gates tier 3 entirely. +- Should the interactive form of an environment be a **separate** taskset id + (`grand-exchange-live`) or a config flag on the existing one? Separate ids keep + the one-shot scores comparable across time; a flag keeps the wheel count down. + Leaning separate. diff --git a/PLAN.md b/PLAN.md new file mode 100644 index 0000000..54059ef --- /dev/null +++ b/PLAN.md @@ -0,0 +1,243 @@ +# Arena build plan + +Companion to `MULTI_TURN.md` (the scope). This is the sequencing: what gets built, +in what order, what can run in parallel, and what collides. + +Status: plan, 2026-08-21. ✅ = settled. ❓ = blocks the work under it. ⛔ = parked by decision. + +## Where Arena sits + +Lumbridge is an open-source company shipping six self-hostable things: +**Compute** (governed compute for unified-memory hardware), **Bench** (model quality +and serving measured against one pinned host × quant × config tuple), **Simulate** +(the Tera/Spaces 3D engine), **Stream** (a headless broadcast rig), **Forge** +(post-training as a content-addressed build), and **Arena**. Four of the six are +public. `apps/web/src/data/projects.ts` in `lumbridge-v4` is the canonical list. + +**Arena is the RL-environment project — yes, `/arena` is the section.** It is +appended last in the project list for numbering reasons, but semantically it sits +beside Bench and feeds Forge: Bench measures a model, Arena produces the gradient, +Forge runs the training. That triangle is the reason Arena is worth doing well +rather than as a side quest — we own all three corners, which nobody on the Hub does. + +Arena spans **two repositories on one page**, and that is deliberate: + +- `git.karti.ai/lumbridge-public/arena` — the Python data environments (this repo) +- `git.karti.ai/lumbridge-public/tera` — the TypeScript spatial environments + +## What we take from Prime Intellect, and what stays ours + +All of PI's stack is open source and we use it rather than rebuild it. Checkouts are +at `~/vendor/prime-intellect/` (46 repos; see its README). + +| We use, unmodified | We build | +|---|---| +| `verifiers` — the v1 spec, the taskset/harness/trace model, `vf.Env` | The environments | +| `prime-rl` — the trainer traces feed | The house rules | +| `prime-agent` (`pi`) — a harness, already embedded in Piggy | `probe.py` — the degenerate-policy gate | +| `prime` CLI — `env push`, `env action` | The Tera bridge | +| The Environments Hub — distribution | lumbridgecorp.com/arena — documentation | + +We do **not** build a harness, a trainer, or a hub. Every one of those exists, +is permissively licensed, and is better than what we would write. + +## Decisions + +**✅ 1 — Hub org handle: `lumbridge`.** Settled 2026-08-21. Environments will +publish as `lumbridge/`. + +**✅ 2 — Attribution: Lumbridge as author.** Settled 2026-08-21. `karti.ai` stays +the code host (`git.karti.ai`), Lumbridge is the author on every surface. One +identity across all six projects; no per-project byline. + +**⛔ Hub push is ON HOLD.** Decided 2026-08-21: keep building, publish nothing to +the Prime Intellect Hub yet. Workstream E is parked — not cancelled, not blocked on +anything technical. `PRIME_API_KEY` is therefore not urgent. + +**✅ 7 — Production brain: Qwen3.8-27B.** Settled 2026-08-21. spark-1 serves +`brain-qwen38-dspark`; that is the default. The Nemotron baseline is ARCHIVED, not +deleted, and Arena's seven scores get re-run and republished against Qwen. +⚠️ **`--model brain` does NOT 404 — it silently succeeds and records `brain` in the +trace while the box serves Qwen.** Verified by direct call 2026-08-21. Pin +`brain-qwen38-dspark` in every reproduction command, everywhere. + +**✅ 8 — `tera-drive-101` ships labelled.** Settled 2026-08-21. Published as a +**bridge-correctness** environment with NO rule-3 and NO rule-4 claim (its return is +monotone in throttle; any positive constant scores ~0.95+). Site says **four** +spatial environments. It is not listed as a fifth RL environment. + +**✅ 9 — `app.lumbridgecorp.com` is the ONLY authenticated origin.** Settled +2026-08-21. The apex loses `/login`, `/signup`, `/app`, `/go`. All existing sessions +sign out once on cutover; accepted cost for three accounts. + +**✅ 10 — Adopted defaults** (technical, recommended by the synthesis, settled +2026-08-21): the agent seat is named **`agent`** everywhere (matching +`SingleAgentEnvConfig`) — it becomes the TOML key in every `configs/*.toml` AND every +prime-rl `rl.toml`, and renaming later parses fine while silently having no seat; +**D lives in the `arena` repo**, not `tera` (amend the parallelization map); +**A ships the FLAT freeze**, with per-item recorded as the identified strengthening; +**Arena transcripts are ADMIN-ONLY** (they carry the held-out slice and oracle +answers — a contamination question, not an org-politics one); **the dashboard is the +archive and `eval-results.json` a curated summary emitted from it**, never two +hand-maintained files; **Forge integrates prime-rl with LoRA RL first**. + +**✅ 3 — `office-jobs-v1` EXISTS; the counts are twelve / five / twenty.** Resolved +2026-08-21 15:50 when another session fast-forward-merged it into the shared `tera` tree +mid-session. It is committed, in `ARENA_MANIFESTS`, and passes `arena:source-hashes`. +The site's original copy was correct. ⚠️ `tera` has live concurrent sessions — re-check +its HEAD before acting on any claim about its contents. + +~~**❓ 3 — `office-jobs-v1`.** Advertised on the live page and in `seo.ts`; absent +from `ARENA_MANIFESTS`. Build it, or correct the page to eleven. Blocks any copy +that cites the count, including a Hub listing. + +**✅ 4 — Interactive envs get their own taskset id.** Settled 2026-08-21. The first +is **`grand-exchange-live`**, a separate wheel beside `grand-exchange`. One-shot +scores stay comparable across time; the two forms are measured, not conflated. + +**✅ 5 — App is a subdomain: `app.lumbridgecorp.com`.** Settled 2026-08-21. Matches +the two-surface model (public site + private platform). Public site stays a site. + +**✅ 6 — Compute policy.** Settled 2026-08-21. **Training runs on rented GPUs** +(GH200-class, from Prime Intellect) — rented because it is faster, and because +renting keeps the hardware choice open per run (the GB300-vs-alternatives call is +one we want to keep making). **Inference in production runs on the DGX Spark.** +We do **not** train on the Spark. + +## Workstreams + +Five, with the dependency edges called out. **A and D can start immediately and in +parallel. B must land before C or E.** + +### A — `grand-exchange` interactive *(start now)* + +The first tier-1 environment: quote → see fills → re-quote over a turn budget, +inventory and unfilled orders carrying between turns. Chosen because the market +simulation already steps, the counterweight (liquidity in gp/tick, the random-walk +trap) already exists, and it is the smallest change that yields a genuinely new +measurement. + +Deliverable: a `vf.Env` subclass exported from the wheel's `__init__.py`, a config, +and a measured run against spark-1's `brain`. Pattern to copy: +`verifiers/v1/tasksets/textarena/taskset.py`. + +This workstream is where **rule 4 — the budget must bind** gets written for real. +Whatever it learns about degenerate trajectories is the input to B. + +Depends on: nothing — ✅4 settled. + +### B — `probe.py` grows degenerate *policies* *(after A's first draft)* + +Today the probe scores four fixed strings per environment. Tier 1 needs policies: +the **staller** (never acts), the **spammer** (acts every turn at random), the +**impatient** (answers turn 1 with no observation), the **exhaustive** (spends the +whole budget, then answers). Same gate shape — inaction 0.000, oracle 1.000, exit 1 +otherwise — plus a new assertion that **exhaustive scores below oracle**. If it does +not, rule 4 is violated and the environment does not ship. + +Do this before building a second interactive environment, not after. It is the +differentiator: nothing across `prime-envs` (88) or `community-environments` (109) +ships a degenerate-policy gate. + +Depends on: A (needs one real interactive env to gate against). +Blocks: C, E. + +### C — More interactive environments *(after B)* + +In rough order of value: + +1. `drop-table-inference` — buy observations under budget; measures when to stop looking +2. `bot-detection` — request labels one at a time (active learning) +3. `fault-localisation` — the log is not given; query it by service, window and level +4. `redaction-pressure` — two agents, redactor vs extractor; the counterweight becomes an opponent + +Each is independently parallelizable **once B exists**. `redaction-pressure` is the +most interesting and the least certain — a genuine multi-agent env, not a stepped one. + +Depends on: B. + +### D — The Tera bridge *(start now, independent repo)* + +One Python `vf.Env` driving the existing TypeScript `tera.arena/v1` package over a +process boundary, turning `step()` observations into `interaction.turn()` messages +and existing component rewards into `trace.record_reward` calls. + +Validated by replay: every bridged episode must `replay()` through the TS package to +an identical FNV-1a-64 checksum, or the bridge is wrong. That property is rare and +the bridge should be built around it. + +Lands four environments at once and makes our strongest existing asset publishable. +Larger and riskier than A — the process boundary and the action encoding are the +unknowns. **Touches the `tera` repo, not this one, so it does not collide with A/B/C.** + +Depends on: nothing. + +### E — First Hub push *(PARKED)* + +Push once there are three to four interactive environments alongside the seven +one-shot ones. `prime env push -v PUBLIC` to our own namespace is primary — +we keep authorship and versioning. A PR into `community-environments` is optional +extra distribution that hands the namespace to `primeintellect`; secondary. + +Then update `lumbridgecorp.com/arena` to document the same wheels. + +⛔ On hold by decision, 2026-08-21. Resume when Karti says so; ❓3 still gates any +copy that cites the environment count. + +## ⚠️ The collision point + +**`tests/test_probe.py` and the wheel-build loop in `.gitea/workflows/ci.yml` +hardcode the environment list.** Every workstream that adds an environment edits +both files. Two sessions adding environments concurrently *will* conflict there, and +CI goes red if either is missed. + +Mitigation, in order of preference: (1) B refactors both to discover environments +from `environments/*/pyproject.toml` instead of a literal list — do this as part of +B and the collision disappears; (2) failing that, one session at a time touches +those two files. + +Separately: this repo already took divergent remote pushes from concurrent sessions +on 2026-08-19. **Merge, never force.** Use a worktree per workstream. + +## Parallelization map + +``` +now ──┬── A (grand-exchange interactive) ──┬── B (probe → policies, de-hardcode CI) ──┬── C1 drop-table + │ │ ├── C2 bot-detection + │ │ ├── C3 fault-localisation + │ │ └── C4 redaction (2-agent) + │ │ + └── D (Tera bridge, tera repo) ──────┴──────────────────────────────── E (Hub push) ⛔ PARKED +``` + +Two sessions can start today: **A** in this repo, **D** in `tera`. They share no +files. Everything downstream of B fans out four ways. + +## Preconditions, verified on amd-server + +- `uv run eval @ configs/.toml --dry-run` → exit 0 ✓ +- verifiers 0.3.0, all 13 harnesses and 4 runtimes present ✓ +- Docker 29.1.3 (tier 2/3 only; tier 1 needs none) ✓ +- spark-1 `brain` at `http://100.127.247.67:8001/v1`, model id `brain` ✓ +- `prime` CLI 0.6.25 ✓ +- `PRIME_API_KEY` ✗ — blocks E, which is parked anyway + +--- + +## ⚠️ Superseded by `EXECUTION.md` + +The workstream descriptions and the parallelization map above were written before the +2026-08-21 design workflow. **`EXECUTION.md` is now authoritative** for sequencing, +file ownership and risk. Three structural corrections it makes: + +1. **B splits.** **B1** (de-hardcode environment discovery) does NOT depend on A and + ships immediately; the CI env list was never a matrix, just a bash `for` loop. + **B2** (degenerate-policy machinery) was found NOT FEASIBLE as written and must be + calibrated against A's *measured* engine, not A's first draft. +2. **D is no longer file-independent.** It lives in `arena` and edits `probe.py`, + `ci.yml`, `test_probe.py`, `README.md` plus two `lumbridge-v4` files. +3. **The top risk is that every Arena reward SATURATES**, so house rule 4 + ("exhaustive < oracle") may be structurally unreachable. If it is, every + interactive environment is unfalsifiable and the reward shape must change before + C1–C4 are built on it. This is also a compute-policy constraint: a non-binding + turn budget burns rented GPU hours measuring nothing. diff --git a/docs/FIRST_EVAL.md b/docs/FIRST_EVAL.md new file mode 100644 index 0000000..488bfb7 --- /dev/null +++ b/docs/FIRST_EVAL.md @@ -0,0 +1,185 @@ +# Running an eval — the runbook + +Written 2026-08-21, the day Arena first measured anything end to end. Before this, +`outputs/` held one `config.toml` and zero episodes: every number downstream of an +eval — the dashboard, `eval-results.json`, the leaderboard, the before/after +training table — was waiting on a run that had never completed. + +This is that run, written down so the next one is a copy-paste. + +--- + +## 1. The command that works + +```bash +cd ~/repos/gitea/arena +export SPARK_API_KEY=dummy + +uv run --project environments/canary_trap eval @ configs/canary_trap.toml \ + --model brain-qwen38-dspark \ + --client.base-url http://100.127.247.67:8001/v1 \ + --client.api-key-var SPARK_API_KEY \ + --no-push --no-rich \ + -c 8 -o outputs/run-/canary-trap +``` + +Swap `environments/` and `configs/.toml` for any of the seven. The +sweep script that ran all of them is `outputs/run_all.sh` (gitignored); the +counter is `outputs/count_rewards.py`. + +### Why each flag is there + +| flag | why | +|---|---| +| `--model brain-qwen38-dspark` | **The served id, pinned.** `--model brain` does *not* 404. SGLang does not validate the model field on the chat route: it returns 200, Qwen answers, and the trace records `brain`. The old `outputs/canary-trap--brain--null/` dir is exactly that mistake. Pin the id `GET /v1/models` reports. | +| `--client.base-url` | The default is `https://api.pinference.ai/api/v1`, which needs a `PRIME_API_KEY` this box does not have. | +| `--client.api-key-var SPARK_API_KEY` | spark-1 requires **no** key — a bare `POST /v1/chat/completions` returns 200. But the client still resolves *some* env var, so point it at a dummy one rather than leaving it on `PRIME_API_KEY` (empty → the failure mode of the one earlier attempt). | +| `--no-push` | Hub push is parked (PLAN.md). Push is **on by default** and needs `$PRIME_API_KEY`. | +| `--no-rich` | The live dashboard is in-process only and unreadable in a log. Without it you get one `rollout done: … reward=…` line per rollout. | +| `-c 8` | spark-1 is shared with Chatterbox, vox and ASR. The config's own `max_concurrent = 128` will hammer it. | +| `-o ` | Otherwise the run lands in `outputs/----//` and you have to go hunting for the uuid. | + +### Flags our own docs get wrong + +- `README.md`'s "Running one" line omits `--client.base-url` / `--client.api-key-var` + entirely, so copy-pasting it points at Prime's inference API with an empty key. + That is the one attempt that was ever made, and why `outputs/` was empty. +- `MULTI_TURN.md`'s preconditions say the spark-1 model id is `brain`. The served + id is `brain-qwen38-dspark`; `brain` silently mislabels the trace. +- `EXECUTION.md` is right that `--harness.id null` is not valid — `EvalConfig` has + no top-level `harness`. The configs already carry `[env.agent.harness] id = "null"`, + so nothing needs passing on the command line. The CLI form, if you ever need it, + is `--env.agent.harness.id null`. +- The `@ config.toml` form is a *positional* argument, not a flag: `eval @ file.toml`. + Inside a shell script quote it as `"@"` so it is not glob-expanded. + +--- + +## 2. The output layout verifiers 0.3.0 actually produces + +`verifiers/v1/cli/output.py:38` — `output_path()`: + +``` +outputs/----// +├── config.toml # the RESOLVED config, re-runnable as `eval @ config.toml` +├── traces.jsonl # one JSON *Episode* per line, appended as rollouts land +└── eval.log # only when you redirect it there yourself +``` + +`--output-dir` replaces the whole `/` pair, so an explicit `-o` gives +you a flat dir with no uuid leaf. Slashes in the model id become `--`. + +**One Episode per line, not one Trace.** Shape: + +```jsonc +{"id": "...", "env": {...}, "ok": true, "errors": [], + "traces": [ { "id": "...", "agent": "...", "rewards": {...}, + "metrics": {...}, "stop_condition": "agent_completed", + "errors": [], "timing": {...} } ] } +``` + +Two traps in that file, both of which bite an ingester: + +1. **`Reward.value` is a `@property` and never appears in the JSON.** Each entry + in `rewards` is `{"score": float, "weight": float}`. The episode total is + `sum(score * weight)`. The `reward=0.525` in the log line is computed, not stored. +2. **`write_episode` dumps with `exclude_none=True`.** A `None` reward is dropped + from the file entirely, so "not measured" and "key absent" are indistinguishable + on the wire. Carry the run's declared reward-name set from `config.toml` if you + need to tell them apart. + +### ⚠️ 0.4.0 moves this + +Upstream HEAD (`~/vendor/prime-intellect/verifiers`) produces one **flat** +`------` directory with no nested uuid, and the +resolved config moves to `configs/.json` (JSON, because JSON keeps nulls). +`RunConfig.id` is a `PrivateAttr` and is absent from that file — derive the run id +from the directory basename, not from the config. + +--- + +## 3. Counting what actually scored + +**`eval` exits 0 even when every rollout errors.** An errored trace is written with +`rewards: {}`. Never report a run on its exit code. + +```bash +uv run python outputs/count_rewards.py outputs/run-/ +``` + +It prints `traces_scored` (non-empty `rewards`), `traces_errored` (`rewards: {}`), +`provider_errors`, the stop-condition histogram, and two means: + +- `reward_mean_scored` — the mean over what survived. This is Arena's own + "score what survived" trap in reporting form: a run where 30 of 32 rollouts + errored and the 2 survivors scored 0.9 reads as 0.900. +- `reward_mean_attempted` — errors counted as 0. **Rank on this one**, and refuse + to rank at all below a scored-fraction floor. + +A quick eyeball without the script: + +```bash +jq -r '.traces[] | if (.rewards|length)>0 then "scored" else "errored" end' \ + outputs/run-//traces.jsonl | sort | uniq -c +``` + +--- + +## 4. What it cost, and what it scored + + +**The run: `outputs/run-20260821-1401/`.** Seven data environments, 32 rollouts each, +224 episodes, **223 scored**. Model `brain-qwen38-dspark` (Qwen3.8-27B) on spark-1. +Wall-clock 14:03 → 14:58, 55 minutes, `-c 8`. + +⚠️ **Compute the mean as `sum(score × weight)` per trace.** An unweighted mean over the +reward components is a different number and it is wrong — it disagrees with what the +harness itself prints. Cross-check against the `reward=` field on each env's 32 +`rollout done` lines in `eval.log`; the two must agree exactly. The first transcription +of this run got six of seven means wrong by taking the unweighted mean. + +| environment | traces | scored | errored | reward_mean_attempted | reward_mean_scored | +|---|---|---|---|---|---| +| `fault-localisation` | 32 | 32 | 0 | **0.9531** | 0.9531 | +| `canary-trap` | 32 | 32 | 0 | **0.6289** | 0.6289 | +| `schema-migration` | 32 | 32 | 0 | **0.4432** | 0.4432 | +| `drop-table-inference` | 32 | 31 | 1 | **0.4174** | 0.4309 | +| `redaction-pressure` | 32 | 32 | 0 | **0.4040** | 0.4040 | +| `bot-detection` | 32 | 32 | 0 | **0.3488** | 0.3488 | +| `grand-exchange` | 32 | 32 | 0 | **0.0055** | 0.0055 | + +The single errored episode is a genuine spark-1 gateway timeout (`ProviderError`, 504) in +`drop-table-inference` — not a scoring bug. **Rank on `_attempted`** (errors as 0); the +`_scored` column is Arena's own "score what survived" trap in reporting form. + +### Two numbers not to publish as capability scores + +⚠️ **`grand-exchange` 0.0055 measures a sampling configuration, not a capability.** +In **31 of 32 rollouts** the model returned a valid, parseable, degenerate answer — +`{"expected_profit": 0, "orders": []}` in 18 completion tokens, `finish_reason=stop`, +never entering a thinking block. The one rollout that did reason (4,954 completion tokens) +placed 4 real orders and scored 0.176 weighted, with `filled=1233` against `planned=2100`. +**The parser, the execution engine and the reward all work end to end** — the environment +is not broken. But `probe.py` rates the *crude* baseline here at 0.082, so 0.0055 sits +**below crude** and barely above inaction. Re-run with thinking forced on before this +number goes anywhere near `/evals`. + +⚠️ **`fault-localisation` 0.9531 has two of four components pegged at maximum.** +`evidence` mean 1.000 (max 1.000) and `fault` mean 1.000 (max 1.000) across all 32 +rollouts; only `gate` (0.9062) and `service` (0.9062) still move. Half the reward mass +no longer discriminates at this capability level. Nemotron scored 0.352 here on 19 August — +a 2.7× swing that needs separating into a stronger model, a prompting difference, or an +environment that has become too easy, before it appears on a scoreboard. + +### The `gate` component carried no gradient in four of seven environments + +Across all 32 rollouts each, `gate` scored **exactly 0.000, max 0.000**, in +`bot-detection` (weight 0.25), `grand-exchange` (0.25), `redaction-pressure` (0.30) and +`schema-migration` (0.25). It fired elsewhere — `canary-trap` 0.2188, `drop-table` 0.2258, +`fault-localisation` 0.9062. + +This is **not** a house-rule-3 violation: `probe.py`'s oracle row is 1.000 for every one of +those environments, which requires `gate = 1`, so it is reachable. But a quarter to a third +of the reward mass produced no gradient in this run, and any ranking or before/after +training table over those four environments is really a ranking over their two soft +components. Understand that before training against them.