lumbridge-public/arena
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: the scope, the plan, the post-review execution plan, and the eval runbook

Browse Source 
MULTI_TURN.md scopes the multi-turn gap and records the finding that made it
cheap: multi-turn needs no container and no real harness. vf.Env.run programs the
control flow host-side and the simulator runs in-process as the user, with the
harness left null.

PLAN.md carries the settled decisions. EXECUTION.md is the post-adversarial-review
plan — every position in it survived a reviewer, and where a design spec was
overturned the corrected position is what is written down.

Two corrections are marked in place rather than silently rewritten, because both
were believed and acted on:

- The provenance hole is src/arena/index.ts, not offices/sites.ts. Mutating
  offices/sites.ts produces a bit-identical episode; index.ts re-exports the
  scripted baselines that ARE the reward denominator, and a halved office-nav
  baseline moves return 4.744 -> 4.098 while every tera hash gate prints ok.
- office-jobs-v1 is not phantom. It was absent at 13:00 and present at 15:50,
  because another session fast-forward-merged it into the shared tera tree
  mid-session. The counts are twelve, five and twenty.

docs/FIRST_EVAL.md is the runbook for the first evaluation Arena ever completed:
seven environments, 224 episodes, 223 scored, against brain-qwen38-dspark. It
warns that the mean is sum(score x weight) — an unweighted mean over components
is a different number, and it is how six of seven means were first misreported.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
karti
2026-08-21 15:55:43 -07:00
co-authored by Claude Opus 5
parent 162d67a83c
commit e4cc2bf1c3
4 changed files with 917 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
EXECUTION.md
+281
View File
@@ -0,0 +1,281 @@
# Execution plan — Arena / Lumbridge
Generated 2026-08-21 by a 13-agent design + adversarial-verification workflow.
Companion to `PLAN.md` (decisions) and `MULTI_TURN.md` (scope).
⚠️ Every position here is the POST-REVIEW position. Where an adversarial reviewer
overturned a design spec, the spec's original claim was discarded. Do not resurrect
a claim from an earlier document that this one corrects.
# Arena / Lumbridge — one execution plan
Synthesis of six adversarially-reviewed workstream specs, 2026-08-21. Every position below is the **post-review** position. Where a reviewer overturned a spec, the spec's original claim is marked and discarded.
Verified independently for this document (amd-server, 2026-08-21): `arena/probe.py` is 772 lines with a literal 7-name shim tuple at :89-93 and a literal `results` dict at :752-760; `tests/test_probe.py` has a 7-name `ENVIRONMENTS` tuple at :13-21 and `timeout=30` at :31; `.gitea/workflows/ci.yml` is 42 lines, its probe step is still named "Run four-environment floor and ceiling probe" (:26) for a seven-environment probe, its wheel loop (:36-40) is a hardcoded bash `for` over seven names, and its lock-policy check (:24) is failing on main because **three** `uv.lock` files are tracked (`bot_detection`, `drop_table_inference`, `grand_exchange`); `outputs/` contains exactly one file, a `config.toml` with `push = true` and `base_url = https://api.pinference.ai/api/v1` — **no `traces.jsonl` exists anywhere on this box**; `~/.prime/config.json` has an empty `api_key`; `tera/src/arena/index.ts:74-79` `ARENA_MANIFESTS` holds four.
---
## 1. Verdicts applied — what each spec becomes
### A — `grand-exchange-live` · feasible, but **not in the order written**
Stands: the multi-turn mechanism (`Env.run` + `interaction.turn()`, null harness, no container), the delta-sheet action grammar, the malformed-reply policy, `roc` on **peak** capital employed (average-employed is a measured free-points bug), the vendored-generator equality assertion, no turn-count reward term.
Corrected, blocking:
1. **The measured ladder does not reconcile with the spec's own reward.** At ref 72,758 gp and `TARGET_SHARE = 0.85`, target = 61,844; impatient at 50,470 gp gives `profit_ratio` 0.816, `discipline` ≤ 0.816, `clean` false → max 0.612, not the reported 0.623. Churn and exhaustive *do* reconcile, so this is a specific defect. **Nothing in §1/§4 is locked until it is re-measured with the shipped engine.**
2. **`viable_market` retargeted at `live_reference` is circular as written.** Viability needs `freeze`; `freeze` is drawn in `Taskset.load()` after viability returns. Fix: derive `freeze` deterministically from `market.seed` (e.g. `random.Random(seed ^ FREEZE_SALT)`), inside `build_market`, so viability and scoring see the same value. This is also the fix for the unstated RNG source — without it, `scan()` recomputes a reference the model never played against.
3. **§6's held-out claim is inverted.** Live is *weaker* than one-shot: the model sees ~50 of 60 graded ticks. Rule 1 still passes; the README must say "held out: the tail after the last observation, ≥4 ticks", not "stronger by construction". Add a probe assertion on the *size* of the held-out slice, not only the frontier.
4. **The probe battery blows CI's own harness.** `probe.py` runs in 3.4 s today under `timeout=30`. §11 adds 9 policies × blocks × 240 baskets + two sweeps + a 240-seed generator check + a hostile battery replayed per turn. Raising the timeout is B1's job (below), but A must budget the cost, not append a name to a list.
5. **Taskset load cost is unbounded.** `num_tasks` defaults 48, `VIABILITY_TRIES = 64`; a live `viable_market` runs a full 8-turn stepped reference per candidate → ~3,072 simulations before the first token, on every eval and every `--dry-run`. Cache the reference per seed, or cap tries for live.
6. **`import` vs `copy` contradiction.** §3 says "import `grand_exchange.book.parse_orders`", §9 says vendor. It is a copy. Say copy.
7. **`[env.agent] max_turns = 8` is an off-by-one risk, not free insurance.** Verify how the framework counts, or set 9.
8. **`trace.info["episode"]` must be plain JSON types.** A dataclass or numpy scalar in `audit()` fails the trace write for the whole rollout.
9. **Effort is L/XL, not L** — it is A plus most of B.
**Revised A build order (this is the change that matters):** `market.py`/`book.py` copies + equality assertion → **`live.py` + unit tests → re-measure the full ladder in the shipped engine** → only then lock `horizon`, `FREEZE_RANGE`, `TARGET_SHARE`, `TURNS` → `strategy.py` sweep → `ledger.py` → `protocol.py` → `taskset.py` → probe.
### B — probe policies + de-hardcoding · **verdict: not feasible as written. Split it.**
The reviewer is right and the split is the fix.
**B1 — de-hardcode environment discovery. Sound, S/M, ships immediately, behaviour-preserving.** `tomllib` over `environments/*/pyproject.toml` replaces the shim tuple, the `main()` results dict (via a `@probes(name)` registry decorator), `ENVIRONMENTS` in `tests/test_probe.py`, and the bash wheel loop. **The CI environment list was never a matrix** — it is a `for` loop in one step (verified) — so no generator job, no `fromJSON` matrix, no act_runner risk. Take B1 now and the repo's known collision point disappears for one afternoon's work.
Two mandatory amendments to B1:
- **Do not make "discovered but unregistered" a hard exit 1 while A is in flight.** The moment A commits a scaffold `pyproject.toml`, `probe.py` exits 1 and CI goes red for every session on the repo. Make it a warning until an env registers a probe, and flip it to exit 1 in the same commit as A's `exchange_live()`.
- **Make the two guards use the same denominator.** `ci.yml`'s "empty scan" counts directories; `discover()` counts `*/pyproject.toml`. A scaffold dir without a pyproject makes them disagree. Count `*/pyproject.toml` in both.
- Raise `tests/test_probe.py` `timeout=30 → 180`, keep the lock check *before* any `uv sync`, and fix the "four-environment" step name.
**B2 — the `Interactive` policy machinery. Redesign required; do not build it before A's engine exists.** The structural claim ("oracle and exhaustive share `answer`, so rule 4 follows") is **false when the reward saturates**, and Arena's rewards all saturate: `profit_ratio` and `efficiency` clip at 1.0 and `clean` is a boolean band. If exhaustive also clears the band it scores 1.000 and the gate reads "budget is free". Further:
- `BIND_MARGIN = 0.02` is staked on realised-P&L differences over 24 seeds, against a block-to-block variance `probe.py:349-352` already documents at 0.48 vs 0.18. The five-block ladder is **required for all three rule-4 comparisons**, not an optional mitigation — multiply the cost estimate by ~5.
- `gate` is binary at weight 0.25, so the exhaustive/oracle gap is quantized in 0.25 steps. A 0.02 margin against a 0.25 quantum is a cliff, not a calibration.
- `play()` never terminates on an answer; "impatient answers turn 1" is an unstated precondition on A's `step()`.
- Seeds must come from A's `baskets()` skip-aware helper, not a contiguous `range`.
- B2's §1.8 dictates A's private module layout, class names, weight vector and turn mechanism. **Delete that.** A owns its contract; B2 adapts to it. This is the collision the workstream exists to remove.
**Net: B1 is unblocked and independent of A. B2 depends on A's *measured* engine, not A's first draft.** PLAN.md's "B depends on A" edge survives for B2 and dies for B1.
### D — Tera bridge · feasible, but **it is no longer parallel with A/B, and PLAN.md must be amended**
Stands and is the strongest evidence in the six specs: 23-file / 283 KB / zero-bare-specifier TS closure; `node` type-strips raw `.ts`; **32/32 scenario × seed traces round-trip through Python `json` and replay to identical checksums**; one long-lived NDJSON worker (84 ms start, 22 µs RTT, 217 µs/step, 87 MB RSS) beats HTTP and beats per-rollout processes; the measured hold-length sweep.
Corrected, blocking:
1. **Role name vs config key — the spec will not load.** `agents.pilot` requires a config field named `pilot`, but all four TOMLs write `[env.agent.harness]`, and `EnvConfig` is `extra="forbid"`. **Name the role `agent`**, matching `SingleAgentEnvConfig`. Same trap A must resolve; settle it once for both (see Decision 1).
2. **`CONFIG = ...` classvar does not exist in verifiers v1.** Config is resolved by `concrete_type()` through `__orig_bases__`. Use the wordle form: `class CrowNavTaskset(TeraSpatialTaskset, vf.Taskset[TeraSpatialTask, CrowNavConfig])`. As written, every per-env `max_turns`/`max_hold` silently does nothing.
3. **The provenance claim is real but was mis-located — CORRECTED 2026-08-21.** `check-arena-source-hashes.mjs` pins 19 of 23 vendored files. The claim that `offices/sites.ts` is the material hole is **FALSE and was refuted twice independently**: mutating it (elevation 4→4000, heading 0→90, width 54→5400) produces a **bit-identical** episode, so it does not reach the simulation. ⚠️ **The real hole is `src/arena/index.ts` alone**, and it is worse than the original claim: rewriting it so `officeNavScriptedBaseline` re-exports a halved policy moves office-nav from 68 steps / return 4.744 to 136 steps / 4.098 — **while `npm run arena:source-hashes` prints "ok" for all four environments and exits 0**, `envHash` and `simulatorHash` unchanged. That file is the reward DENOMINATOR. Fix: pin `src/arena/index.ts` in `tera/scripts/check-arena-source-hashes.mjs` (a one-line `tera`-repo change) and regenerate the four pairs in `sourceHashes.ts`.
4. **`oracle.py` reimplementing the four TS baselines in Python contradicts the bridge's own rule** ("never recompute a number that came out of TypeScript") — and it does so for the reward *denominator*. Add an `oracle` op to `worker.mjs`; the four baselines are already exported from `src/arena/index.ts`.
5. **`drive-101` measures nothing, and the 0.9 ceiling band makes it worse than "no rule-4 claim".** Return is monotone in throttle, 12/12 goals at every value; `0.9 × 4.339 = 3.905` and throttle 0.2 already returns 3.723 → quality 0.953. Any positive constant throttle scores ~0.95–1.00. Same collapse on office-nav: any goal-reaching run clears the band, so `control` (0.50) and `gate` (0.20) are constants and only `efficiency` carries signal.
6. **office-nav's rule-4 argument is backwards under its own formula.** `efficiency = oracle_turns / max(turns, oracle_turns)` only ever *penalises* extra turns, and past `ceil(1.5 × 1.8) = 3` turns `clean` also drops. The formula punishes exactly the "precision buys turns" behaviour cited as evidence.
7. **`gate` is a separate additive term for goal-attainment that is already inside `ts_return` as the success bonus** — double-counted, additively. That is the failure `probe.py:15-17` records verbatim. Make it a multiplier on the sum.
8. **Missing:** no `@vf.stop`; no handling of a mid-loop `terminated` Segment (the loop as written hits `RuntimeError('the exchange is over')`); no `Taskset.load()` / prompt-vs-bare-`turn()` decision; an empty `{"plan":[]}` costs zero steps while a malformed reply costs 10, inverting the spec's own principle; no guard for `oracle_return <= 0`; `max_hold = 400` exceeds `MAX_STEPS = 360`.
9. **Scope.** PLAN.md:138 and the parallelization map say D lives in `tera` and shares no files with A/B/C. This spec puts everything in `arena` and edits `probe.py`, `tests/test_probe.py`, `ci.yml`, `README.md`, `configs/`, plus two `lumbridge-v4` files. That is a real decision (the packaging argument is correct — `tera` has no Python toolchain), but **PLAN.md must be amended to record it**, because it converts D from "parallel with everything" into "collides with A, B and W".
**Revised D:** ship the bridge + replay gate + `tera_crow_nav` first. `drive-101` ships only as a labelled bridge-correctness environment or not at all (Decision 3). `office-nav` and `california-flight` follow. D's probe integration waits for B2.
### P — `app.lumbridgecorp.com` · feasible; **Phase 1 as written cannot ship**
Stands and is high-value: the endpoint inventory (29 routes, not 31), the 23 dead `api.ts` methods, one-build/two-hosts with app.* as the **only** auth origin, `bind 10.0.0.2` + the empty-200 verification, the `isAppHost` branch in `sendShell`, `WORLD_LAUNCHER_URL` convergence for `/go/:slug`, `World1` moved not deleted.
Corrected, blocking:
1. **"Ingest the runs already on amd-server" is impossible — there are none.** One `config.toml`, zero episodes, and that config points at `api.pinference.ai` with `PRIME_API_KEY`, which is empty. **Phase 1 must contain "execute one eval to completion against spark-1" as an explicit prerequisite**, or Arena's dashboard ships empty.
2. **The leaderboard is operator-gameable and mis-specified.** (a) The server must recompute every aggregate in SQL from the `arena_episodes` it just inserted, never trust the pusher's numbers. (b) Ranking on a mean over *scored* episodes is Arena's own "score what survived" trap. Store `reward_mean_scored` **and** `reward_mean_attempted` (errors as 0), rank on attempted, refuse to rank below a scored-fraction floor.
3. **`arena_episodes` flattens a list.** `Episode.traces` is a list; `reward`/`rewards`/`stop_condition`/`last_reply` are per-**Trace**. Primary key must be `(run_id, episode_id, trace_id)` with an `agent_name` column — otherwise the schema is wrong for `grand-exchange-live` (simulator as second role) and for `redaction-pressure` (genuine two-agent).
4. **The trace viewer publishes the held-out slice.** `GET /api/arena/runs/:id/episodes/:eid` at `user` scope returns full transcripts, which contain held-out task data and, for oracle runs, the answers. One leaked invite publishes the held-out corpus. **Transcripts are admin-only or redacted.** This settles the spec's open question on read scope.
5. **The compute-scene-vocabulary blocker is fabricated.** `computeScene` is a free-form slug regex, not `z.enum(SCENE_NAMES)`; `stock`, `nemotron` and `qwen38` all pass today. Delete that precondition.
6. **`portal_host_samples` and `portal_container_samples` have no node identity** — no `node_id`, no host column, no PK. Per-node charts need `ALTER TABLE … ADD COLUMN IF NOT EXISTS node_id text` plus an index first.
7. **`scored = ok AND rewards non-empty` is wrong.** `write_episode` dumps with `exclude_none=True`, so a `None` reward is dropped from the file entirely — "not measured" and "key absent" are indistinguishable on the wire. Require at least one non-null `Reward`, and carry the run's declared reward-name set from the config.
8. **`Reward.value` is a `@property`** and never appears in dumped JSON. Compute `score × weight` in the ingester.
9. **0.3.0 → 0.4.0 changes the directory layout, not just the config filename** — HEAD produces one flat `<env>--<model>--<harness>--<hex8>` dir with no nested uuid, and `RunConfig.id` is a `PrivateAttr` absent from `configs/eval.json`. Derive the run id from the directory basename.
10. **Name the private console `/nodes`**, not `/compute` — `/compute` is already a public marketing section with its own nav.
11. Put `isAppHost` in `packages/shared` — two independent string checks that must agree, with a silent asymmetric failure mode (dashboard rendered under a 404 status).
12. `sendShell(url, reply)` takes no request; the signature and both call sites change. `spa-fallback.test.ts` needs a case. `@radix-ui/react-select` is **not** installed.
**Revised P phasing.** The honest first ship is **P1: Caddy host + host-aware routing + `renderAppShell` + AppShell + Overview + Worlds + Org + Inbox + the `api.ts` trisection.** All of that runs on endpoints that exist today. **Arena (schema, 6 routes, ingester, 4 pages) is P2, a full workstream of its own**, and it is gated on a real eval having run.
### W — website visual vocabulary · feasible; **the data pipeline must be redirected, not built**
Stands: the diagnosis (the site draws what it hasn't measured and narrates what it has), `MeasureTable` / `Bar` / `FlowStrip` / `SpecSheet`, inline SVG with `currentColor` and **no charting dependency** (the `gold-500 → gold-700` theme flip alone settles this), the grouped `StateList`, and the four live factual errors.
Corrected:
1. **`arena-runs.json` has no machine source and must not be created.** `probe.py` does not run models; there is no `traces.jsonl`. **The site already has the right contract: `apps/web/src/data/eval-results.ts` + `eval-results.json` + `eval-results.parse.test.ts`, with a `{model, host, quant, serving, checkpoint}` target tuple and an explicit "null means not measured" rule.** The Nemotron run belongs there. Creating a second incompatible file guarantees the drift the pipeline exists to prevent — and putting it in `eval-results.json` also fixes **a fifth live error the audit missed**: `arena.tsx:519` says "No model scores are published yet, so nothing appears on Evals" while `README.md:88-105` publishes seven.
2. **The crude-maximiser labels have no machine source either.** `probe.py`'s functions return only `{inaction, crude, plausible, oracle}`; "redact everything" / "ban everyone" / "buy and sell at market" exist only as markdown italics. Emitting that column means **changing `probe.py`'s return shape — which is B's file.** Fold it into B1's registry work or accept it as a hand-typed literal, but do not schedule it as a third-party edit to `probe.py`.
3. **`check:arena` must not execute `probe.py`.** The `sync-scenes.mjs` precedent parses committed YAML from an anonymous public clone; running a 772-line probe under `uv run --with regex` is a slow, flaky gate that contradicts the hermetic-build rule. Emit JSON in `arena`'s CI, commit it, and have `check:arena` compare bytes.
4. **`Diagram` fails DESIGN.md §4.2's own gate** — no variants, no CVA variants object. It is a one-off `<figure>` per page, exactly like `scaling-plot.tsx` / `spread-field.tsx` already are. Do not put it in `components/ui/`.
5. **Do not put a nine-verb FlowStrip above the fold on `/forge`.** Every StateList item on that page is `done: false`, including "No repository, no tag, no release." Drawing an unbuilt pipeline is the exact failure the audit indicts.
6. **The landing-page markup claim is wrong**, and it invalidates the landing plan's premise: `landing.tsx` contains **two** `<svg>` elements, not six. `ForgeChain` is an `<ol>`, `ArenaBand` is nested `<div>`s and 32 `<span>`s, `StreamSignal` and `SimulateViewport` are divs. The real coupling is ~33 class references into `machine-home.css` (35 KB) carrying reveal/scan animations. **Treat the landing as its own step with its own CSS audit.**
7. **`Bar` has one existing hand-rolled call site, not two** — `BudgetMeter` is an N-segment `flexGrow` track with a composed `aria-label`, and `BarProps` cannot express it. `Bar` earns its place through `MeasureTable`'s cells, not through BudgetMeter.
8. ~~**The scenario count is 16, not 20**~~ ⚠️ **WRONG as of 2026-08-21 15:50 — it is 20**, five registries × four, and `office-jobs-v1` is not phantom. Another session merged it into the shared `tera` tree mid-session. Counts are twelve / five / twenty; the site's original copy was right and W0's correction was reverted.
9. Four more table-shaped components exist that the "seven call sites" audit missed: `components/compute/model-table.tsx`, `components/publish/concurrency-table.tsx`, `ledger.tsx`, `rate-strip.tsx`. Design `MeasureTable`'s API against them.
10. No tests are proposed. New generated data + a check script are exactly what gets a parse test in this repo.
**Revised W:** `/arena` first (it is also a truth fix), `/bench` second, `/stream` third — but `/arena`'s number pipeline routes through `eval-results.json`, and the `office-jobs-v1` deletion happens **once**, in one place, coordinated with D and P.
### C-compute — rent to train, Spark to serve · feasible; **its headline correction is false**
Stands and is the most immediately actionable: the policy itself; the measured spark-1 tenancy (121 GB total / 84 GB used / 36 GB free, SGLang + Chatterbox + vox + ASR co-resident, Chatterbox's ~8.4 GB/hr leak having killed brain before); `prime login` as step zero because `api_key` is empty (verified); **Arena → prime-rl needs no adapter** — `EnvConfig` is `SerializeAsAny[vf.EnvConfig]` and the taskset tree is the same shape, prefixed; the seat-name-is-a-TOML-key trap; Hosted Training being unavailable while E is parked (`rl.py:1522-1524` — bare v1 ids do not resolve through the Hub); the sm_121 lock contradiction in Forge; `TRAINERS = ("axolotl","trl")`; **no price is hardcoded anywhere and none should be invented.**
Corrected, blocking:
1. **`--model brain` does NOT 404.** SGLang does not validate the model field on the chat route; it returns 200 with a real completion. Two of the four "stale facts" and two proposed file edits rest on a false claim. **The real hazard is the opposite and worse: it silently succeeds.** Re-running the `README.md:88-89` baseline today produces **Qwen3.8-27B** numbers under a command that reads `--model brain`, and they will be tabulated beside Nemotron numbers as comparable. The correction the arena docs need is: **pin the served id `brain-qwen38-dspark` and the scene name in every reproduction command**, not "fix a 404".
2. **`uv run eval grand-exchange-live --harness.id null` is not valid.** `EvalConfig` has no top-level `harness`; `--harness.` is admitted only to render a did-you-mean. Canonical: `--env.agent.harness.id null`, or the TOML block `configs/grand_exchange.toml` already uses.
3. **`uv sync --package prime-rl --package grand-exchange-live` cannot work.** prime-rl's `[tool.uv.workspace] members` is a closed list of its own vendored envs. Use `uv pip install` of a built wheel or git URL into the env-server venv, or add a path source.
4. **`forge.tsx:114` is not stale.** The remote is private, `git tag` is empty, `status="In development"`, no `source=` prop. Nobody can clone it. The page's own README warns "This section has been wrong three times" — this edit would make it four.
5. **Workstream-letter collision:** PLAN.md already defines a "workstream C" (more interactive environments). Rename this one — **"Compute Policy"**, no letter.
6. The wordle wall-clock anchors are fabricated (the README says 2 GPUs, the toml declares 8 and 200 steps; neither states a duration). Delete them and produce our own.
7. `torchao` and `nixl-cu12` are pinned to **x86_64-only** wheel URLs while `torchao` is a hard runtime dep — the "aarch64 wheels for every heavy dependency" claim does not hold, which sharpens the aarch64-variance risk.
8. `prime disks` attach with `--disks`, not `--attach-disk`.
9. **Two things nobody has planned:** how the DCP checkpoint gets *off* the pod before `prime pods terminate` destroys the disk (needs a `--disks` volume at create time), and the DCP → HF shards → NVFP4 → Compute model-id conversion, which nothing on disk automates end-to-end (prime-rl has `convert_to_hf` but no CLI entrypoint).
10. **The link nobody drew:** rental hours are driven by whether the turn budget binds. `group_size × batch × steps` of *multi-turn* rollouts against a rented inference fleet is real money. If B2's exhaustive policy scores at the oracle, the run burns GPU hours measuring nothing. **Rule 4 is a compute-policy constraint, not only a design one.**
---
## 2. Parallelization map and file collisions
### What can start today, concurrently
| Lane | Work | Repo | Blocked by |
|---|---|---|---|
| **1** | **B1** — discovery de-hardcode + timeout + step name + `uv.lock` cleanup | `arena` | nothing |
| **2** | **A step 2** — `live.py` engine + unit tests + **re-measure the ladder** | `arena` | nothing (own worktree) |
| **3** | **D step 1** — worker, vendored TS, hashes, `test_replay.py` | `arena` (new dir only) | nothing |
| **4** | **Compute Policy step 0** — `prime login`, catalogue + prices, doc the corrected spark-1 facts | none / `arena/docs` | nothing |
| **5** | **P1** — Caddy host, host-aware routing, AppShell, Overview/Worlds/Org/Inbox | `lumbridge-v4` | nothing |
| **6** | **W step 0** — the five factual corrections + `eval-results.json` decision | `lumbridge-v4` | nothing |
Six lanes, and only lanes 1/2/3 share a repo. **Use a worktree per lane** — this repo took divergent concurrent pushes on 2026-08-19. Merge, never force.
### True dependency edges
```
B1 ──────────────────────────────────────────┐
├──► A step 7 (probe row) ──► A ships
A steps 1-2 (engine) ──► A re-measure ──► A steps 3-6 ─┘
│
└──► B2 (policy machinery, calibrated on real numbers)
│
├──► C1 drop-table
├──► C2 bot-detection
├──► C3 fault-localisation
├──► C4 redaction (2-agent)
└──► D step 5 (tera probe rows)
D step 1 (replay gate) ──► D crow-nav ──► D flight/office
A ships ──► one real eval run ──► traces.jsonl exists ──► P2 (Arena dashboard)
│
└──► W /arena numbers (via eval-results.json)
prime login ──► rented-pod runbook ──► first RL run ──► Forge prime-rl renderer
```
Two edges that were wrong in the source specs and are now correct:
- **B1 does not depend on A** (PLAN.md's "B depends on A" applies only to B2).
- **D is no longer file-independent of A/B/W** — it lives in `arena` and edits `probe.py`, `ci.yml`, `test_probe.py`, `README.md`, plus `arena.tsx` and `seo.ts`.
### Merge-conflict map — the specific paths
**Tier 1 — will conflict, must be sequenced or de-hardcoded first:**
| Path | Claimed by | Resolution |
|---|---|---|
| `arena/probe.py` | A (§11), B1 (registry), B2 (machinery), D (§7), C1-C4, W (needs a return-shape change for crude labels) | **B1 lands first.** After B1, adding an env is one appended function at EOF — a mergeable case. Nothing else touches this file until B1 is merged. |
| `arena/tests/test_probe.py` | A, B1, D | B1 removes it from every future workstream's diff. Never edited again. |
| `arena/.gitea/workflows/ci.yml` | A, B1, D (needs Node + replay test + a one-dir-four-wheels change) | B1 lands the glob; D then adds *steps*, not list entries. **D's one-directory-four-tasksets layout actively breaks B1's one-dir-one-wheel discovery** — D must add a `[tool.arena] tasksets = [...]` key or equivalent, and B1 should be told to read it. |
| `arena/configs/grand_exchange_live.toml` | **A, B(§1.7.2), Compute Policy** — three specs create the same file | **A owns it.** Delete it from the other two file lists. |
| `arena/environments/grand_exchange_live/*/taskset.py` and `session.py` | **A and B both specify the module layout and class signatures** | **A owns the contract.** B2 adapts. |
| `arena/README.md` | A (rows), D (rows + house rule 4), Compute Policy (baseline qualifier), W (source of truth for the site tables) | Sequence: A → D → Compute Policy. W reads it, never writes it. |
| `arena/PLAN.md` + `MULTI_TURN.md` | B, D (scope amendment), Compute Policy | One session, one pass, after the D-scope decision lands. |
**Tier 2 — cross-repo, same file:**
| Path | Claimed by | Resolution |
|---|---|---|
| `lumbridge-v4/apps/web/src/pages/v4/arena.tsx` | **D** (delete `office-jobs-v1` at :203 and :511) and **W** (full rebuild of the page) | **W owns the file.** D contributes the *fact* (four manifests, 16 scenarios), not the edit. |
| `lumbridge-v4/services/api/src/seo.ts` | **D** (:271 twelve→eleven, five→four) and **P** (`renderAppShell`, drop `/app` from `APP_ROUTE_PATTERNS` at :698) | Different regions; sequence P after D's one-line copy fix, or vice versa. Low risk but both must run `seo.test.ts` (`ROUTER_PATHS` at :36) and `spa-fallback.test.ts`. |
| `lumbridge-v4/apps/web/src/pages/v4/forge.tsx` | **W** (FlowStrip, StateList) and **Compute Policy** (:59-62 delegation copy) | W owns it. Compute Policy contributes one sentence, **only after a rented run has actually happened**. The `:114` edit is cancelled (see verdict). |
| `lumbridge-v4/apps/web/src/data/eval-results.json` | **W** (Nemotron run) and **P2** (the dashboard's run store) | The public file is a curated summary; the dashboard is the archive. Decide the direction (Decision 6) before either writes. |
**Tier 3 — three tracked `uv.lock` files are failing CI on main right now.** `git rm --cached` them in B1's commit, before anyone blames a new failure on a new environment.
---
## 3. Sequencing
### Phase 1 — genuinely shippable, ~2 weeks, six parallel lanes
1. **B1** (arena): tomllib discovery, `@probes` registry, timeout 180, `uv.lock` untracking, step rename. *Gate: `probe.py` still prints seven `ok` rows; CI green with zero list edits.*
2. **A steps 1-2 + re-measurement** (arena, worktree): frozen generator copy + equality assertion; `LiveBook`/`Leg`/`Ledger` with unit tests; **then re-run the full policy ladder and publish the corrected numbers.** *Gate: the ladder reconciles with the reward formula; `horizon`, `FREEZE_RANGE`, `TARGET_SHARE`, `TURNS` are chosen from that measurement, not from the spec.*
3. **D step 1** (arena, new dir): worker + vendored TS + `hashes.py` + `sync_tera.py` + `test_replay.py`. *Gate: 32/32 replay exact, tampering rejected, hash check green — and a decision on the 4 unpinned files.*
4. **Compute Policy step 0**: `prime login`; record real `availability` output and prices; write `arena/docs/COMPUTE_POLICY.md` with the corrected spark-1 facts (`brain-qwen38-dspark`, silent-wrong-model hazard, 36 GB headroom).
5. **P1** (lumbridge-v4): Caddy `app.` block with `bind 10.0.0.2` and the non-empty-body check; host-aware `App.tsx`; `renderAppShell` + `sendShell` signature change + `seo.test.ts` + `spa-fallback.test.ts`; AppShell; Overview / Worlds / Org / Inbox; delete the 23 dead `api.ts` methods.
6. **W step 0** (lumbridge-v4): the five factual corrections — `arena.tsx:330` "Nine tasks", `landing.tsx` card 02, the 4-row probe table → 7 rows, `office-jobs-v1` deleted, `arena.tsx:519` "no model scores published". Ship the Nemotron run into `eval-results.json`.
**Phase 1 ends with**: a green de-hardcoded CI, a measured live engine, a proven TS replay bridge, a live private console on real endpoints, a truthful public site, and a Prime Intellect account that can rent.
### Phase 2
7. **A steps 3-9** → `grand-exchange-live` ships with its probe row (needs B1 merged).
8. **One real eval run** against spark-1, pinned to `brain-qwen38-dspark`, with the empty-rewards count checked. *This is a hard prerequisite for P2 and for any site number.*
9. **D crow-nav** end to end (role named `agent`, `vf.Taskset[...]` generic form, `oracle` op in the worker, `gate` as a multiplier).
10. **W /arena rebuild** + `/bench` on the new primitives.
11. **First rented run**: pod create with an attached disk, `install.sh`, wheel install, baseline eval, `uv run rl`, re-eval, before/after table.
### Phase 3
12. **B2** — policy machinery, calibrated against A's measured exhaustive/oracle gap, with block-wise ladders replacing the single 0.02 margin.
13. **P2** — Arena schema (per-trace PK), server-side aggregates, admin-only transcripts, ingester for both output layouts, the four pages.
14. **C1-C4** — the four remaining interactive environments, fanning out from B2.
15. **D flight + office-nav**; drive-101 per Decision 3.
16. **Forge**: `prime-rl` renderer (LoRA RL first), x86_64 train lock, retitled aarch64 lock, DECISIONS entry, checkpoint→HF→NVFP4→Compute-model-id stage.
17. **W /stream**, landing (with its own `machine-home.css` audit), compute/simulate/about.
18. **E** — Hub push, when un-parked.
---
## 4. Decisions Karti still owes, ranked by what they block
| # | Decision | Blocks | Recommendation |
|---|---|---|---|
| **1** | **The agent seat name.** `agent` (matching `SingleAgentEnvConfig`) or a custom name? It becomes the TOML key in `configs/*.toml` **and** in every prime-rl `rl.toml`, on both A and D. Renaming later parses fine and silently has no seat. | A, D, every future env, every training config | **`agent`, everywhere.** One word, decided once, written into each env's README. |
| **2** | **Does D live in `arena` or `tera`?** The spec's packaging argument is strong (`tera` has no Python toolchain) but it overturns PLAN.md:138 and makes D collide with A/B/W. | D's entire file layout, the CI wheel loop, PLAN.md | **`arena`**, and amend PLAN.md's parallelization map in the same commit. |
| **3** | **Does `tera-drive-101` ship?** Measured: monotone in throttle, optimal policy is one constant string, and the 0.9 band makes any positive throttle score ~0.95+. Also decides whether the site says four spatial environments or three. | D's scope, `arena.tsx`/`seo.ts` copy, the Hub listing | **Ship it labelled a bridge-correctness environment with no rule-4 and no rule-3 claim, or defer.** Do not list it as a fifth RL environment. |
| **4** | **A's freeze mechanic: flat per-look, or proportional to items re-quoted?** Flat is measured and works; per-item is the better mechanism, makes the look count genuinely vary, and forces a much larger surgical reference. | A's engine, and therefore B2's calibration | **Ship flat.** Record per-item as the identified strengthening. Do not build an unmeasured mechanism on the critical path. |
| **5** | **Arena transcript visibility.** Full transcripts contain the held-out slice and, for oracle runs, the answers. | P2's route scoping, and house rule 1's integrity | **Admin-only.** This is not an org-politics question; it is a contamination question, and that settles it. |
| **6** | **Where does model-run data live?** Public `eval-results.json` (already typed, tested, rendered on `/evals`) vs P2's `arena_runs`/`arena_episodes` tables. | W's whole data pipeline, P2's schema | **Both, with a direction:** the dashboard is the archive, `eval-results.json` is a curated public summary emitted from it. Never two independent hand-maintained files. |
| **7** | **Is `nemotron` coming back as the production brain, or is `qwen38` the default?** `README.md`'s baseline, D23/D24's Forge preflight, and the site's Arena numbers all assume Nemotron; the box serves Qwen3.8-27B. | Every reproduction of the baseline; what "inference in production on the Spark" names | Needs an answer before anyone re-runs a benchmark. |
| **8** | **First Forge/prime-rl integration: LoRA RL or full-FT?** LoRA reuses merge + attest unchanged and matches `alphabet-sort`; but LoRA rejects `--gpu-type` and falls back to filesystem weight broadcast. Full-FT needs a new node kind and the tensor-delta-map gate never fires. | Forge's roadmap, and what its manifest is for | **LoRA first** — it keeps Forge's headline claim intact. |
| **9** | **Is app.\* the only authenticated origin?** Yes means the apex loses `/login`, `/signup`, `/app`, `/go`, and all three existing sessions are signed out on cutover day. | P1's scope, apex route deletions | **Yes**, and state the sign-out as an accepted cost for three accounts. |
| **10** | **W's audience.** Arena/Bench/Stream (brokenness × argument) vs Arena/Landing/Bench (the Prime Intellect Hub reader). | W's priority order only | Answer when P1 lands; it does not block anything upstream. |
---
## 5. Risk register, highest first
| # | Risk | Retired by |
|---|---|---|
| **1** | **The whole reward design of `grand-exchange-live` rests on numbers from a throwaway engine that no longer exists, and the ladder does not reconcile with the reward formula.** `horizon=60`, `FREEZE_RANGE=(1,4)`, `TARGET_SHARE=0.85` and `TURNS=8` are all locked against arithmetic that is off by 0.011 in the one place it can be checked. | Building `live.py` FIRST and re-measuring every row before any constant is frozen. This is Phase 1, lane 2, and it is the single most important task in the programme. |
| **2** | **Every reward in Arena saturates, so "exhaustive < oracle" may be structurally unreachable.** `profit_ratio` and `efficiency` clip at 1.0; `clean` is a boolean band. A 0.02 margin against a 0.25 quantum is a cliff. If the gate can't open, rule 4 is unenforceable and every interactive environment is unfalsifiable. | A's re-measurement producing a real exhaustive/oracle gap; B2 then calibrated against it with block-wise ladders. If no gap exists, the reward shape must change before C1-C4 are built on it. |
| **3** | **No `traces.jsonl` exists anywhere.** Everything downstream — the dashboard, the site's numbers, the leaderboard, the before/after training table — is waiting on one eval that has never completed. The one run that was attempted pointed at Prime's inference API with an empty key. | One completed eval against `brain-qwen38-dspark` with the empty-rewards count checked. Cheap, and it unblocks P2, W and Compute Policy simultaneously. |
| **4** | **`eval` exits 0 when every rollout errors, and multi-turn has far more ways to error than one-shot.** `turn()` on a closed interaction raises; `turn(msg)` on turn 1 of a prompted task raises. Combined with risk 3, a "successful" run can be entirely empty. | A CI/runbook step that asserts non-empty `rewards` on ≥N episodes, in `arena`, in D's replay gate, and in P2's ingester. Never trust the exit code. |
| **5** | **Silent wrong-model comparison.** `--model brain` returns 200 from a box serving Qwen3.8-27B. Re-running the Nemotron baseline today produces different-model numbers under an identical command, and they will be tabulated as comparable. | Pinning `brain-qwen38-dspark` (and the scene) in every command in `README.md`, `PLAN.md`, `MULTI_TURN.md` and `COMPUTE_POLICY.md`. Trivial; do it in Phase 1. |
| **6** | **`probe.py`, `tests/test_probe.py` and `ci.yml` are edited by five of six workstreams, in a repo that already took divergent concurrent pushes.** | B1, first, before anything else touches `arena`. |
| **7** | **The reward denominator is editable with every `tera` gate green.** ⚠️ CORRECTED 2026-08-21: not `offices/sites.ts` (mutating it produces a bit-identical episode — refuted twice). The hole is **`src/arena/index.ts`**, which re-exports the scripted baselines that ARE the denominator. Demonstrated: a halved office-nav baseline changes return 4.744 → 4.098 while `arena:source-hashes` prints ok and exits 0. | Pinning `src/arena/index.ts` in `tera/scripts/check-arena-source-hashes.mjs` and regenerating `sourceHashes.ts`. One line, in a repo this session did not own. |
| **8** | **spark-1 has 36 GB of 121 GB free with four resident model services, one of which leaks ~8.4 GB/hour and has killed the brain before.** "Production inference on the Spark" is a tenancy claim held together by the governor on :8011. | A mandatory headroom check before any new resident model, and Compute's watchdog verified live. |
| **9** | **`PRIME_API_KEY` is empty**, blocking not just the parked Hub push but every rental command, every price, and the entire training half of the compute policy. | `prime login`. One command. |
| **10** | **Two of D's four environments cannot discriminate under the 0.9 ceiling band** — drive-101 gives ~0.95+ to any constant throttle, office-nav collapses to `efficiency` alone. | Re-deriving the band per environment against the measured return distribution, or shipping fewer environments. |
| **11** | **The held-out corpus leaks through P2's trace viewer at `user` scope.** One invite code. | Decision 5 — admin-only. |
| **12** | **Effort is understated across the board.** A is L/XL; B2 is not B's to size (its §1.8 is L/XL and belongs to A); D is L/XL; P's "Phase 1" is XL; W's item 1 alone is L/XL across two repos. Read as written, this programme looks like six L's and is closer to four XLs. | The phasing in §3, which cuts P1 out of P, B1 out of B, and D-step-1 out of D. |
| **13** | **Rented GPU hours are burned by a non-binding turn budget.** `group_size × batch × steps` of multi-turn rollouts against a rented inference fleet is real money, and nothing in the compute doc connects to rule 4. | Making "exhaustive < oracle by a measured margin" a precondition of dispatching any RL run — write it into `COMPUTE_POLICY.md`. |
| **14** | **The checkpoint has no exit path.** `prime pods terminate` destroys ephemeral disk; the DCP → HF → NVFP4 → Compute-model-id conversion is unautomated. | `--disks` at pod-create time, plus a `forge quantize/publish` stage. Discover this before the first successful training run, not after. |
| **15** | **GH200 may not appear in `prime availability` at all** — the CLI names only H100_80GB, H200_141GB, B200_180GB, RTX_PRO_6000. "GH200-class" may mean H200_141GB in practice. And `perf.py` has no GH200 branch, so MFU may silently fall back to A100's 312 TFLOPS. | `prime login`, then one `availability gpu-types`, then one `nvidia-smi --query-gpu=name` on a real pod. |
---
## 6. Build this first
**`environments/grand_exchange_live/grand_exchange_live/live.py` — the stepped engine and its unit tests — and, concurrently, workstream B1's discovery de-hardcoding.**
Everything else in the programme is downstream of two facts we do not yet have: whether the live market's reward ladder actually reconciles (risk 1, which invalidates A's horizon, freeze, TARGET_SHARE and turn budget if it does not, and takes B2, C1-C4 and half the compute argument with it), and whether three files can stop being a five-way merge conflict before five workstreams reach for them. B1 is an afternoon and removes the repo's only structural collision permanently; `live.py` is the smallest artifact that can turn six specs' worth of `/tmp` measurements into something the shipped code agrees with.
MULTI_TURN.md
+208
View File
@@ -0,0 +1,208 @@
# Scope: the multi-turn gap
Status: scoping document, 2026-08-21. Nothing here is built yet.
Every environment in this repository sets `[env.agent.harness] id = "null"` — one
completion, no tools, no runtime. That is the right choice for a pure-trace data
environment and it is also the whole of what Arena currently measures. Nothing here
scores an agent *acting*: choosing what to look at next, paying for a turn, or
revising after seeing a consequence.
This document scopes closing that gap. The headline finding is that it is three
separable tiers, not one, and the cheapest tier needs no new infrastructure at all.
## What upstream actually requires
Read from `~/vendor/prime-intellect/verifiers` (v1, installed version 0.3.0), not
from blog posts.
**Multi-turn does not imply a container.** `vf.Env.run(task, agents)` programs the
control flow host-side. `agents.<role>.interaction(task)` opens a rollout and holds
it open; `await interaction.turn(observation)` sends one user turn, runs one harness
segment, and returns a `Segment` with `.last_reply` and `.terminated`. The simulator
runs in-process, as the user. The harness can stay `null` throughout.
The reference implementation is `verifiers/v1/tasksets/textarena/taskset.py` — 90
lines, drives a real game engine, scores off the engine directly. Read it before
writing anything.
**An Env ships inside the taskset's own wheel.** `EnvConfig.id` empty means "the
taskset's own env, else `SingleAgentEnv`". Export the `Env` subclass from the
package `__init__.py`, exactly as `environments/wordle` exports `TextArenaEnv`, and
`[env.taskset] id = "..."` picks it up with no extra config.
**Available in the installed version** — confirmed, not assumed:
- harnesses: `null`, `bash`, `browser_use`, `claude_code`, `codex`, `hermes_agent`,
`kimi_code`, `mini_swe_agent`, `openclaw`, `pi`, `pool`, `rlm`, `terminus_2`
- runtimes: `subprocess` (debug only — rollouts share side effects), `docker`,
`modal`, `prime`
- prebuilt envs: `single_agent`, `best_of_n`, `agentic_judge`,
`shared_agentic_judge`, `user_sim`
**The interception server** sits between harness and provider. It speaks each
harness's native dialect (Anthropic Messages for Claude Code, OpenAI Responses for
Codex), builds the trace live, forces sampling params the harness will not expose,
and can rewrite tool results and web-search results to block reward hacks. It is
mandatory for tiers 2 and 3 and free for tier 1.
## The three tiers
| | Mechanism | Harness | Runtime | New infra |
|---|---|---|---|---|
| **1. Interactive** | `vf.Env.run` + `interaction.turn`, simulator in-process | `null` | none | **none** |
| **2. Tool-using** | Taskset exports a `Toolset`, served as MCP | `bash` | `subprocess` / `docker` | image per env |
| **3. Agentic** | Full coding agent in a sandbox | `claude_code`, `codex`, `openclaw`, `pi` | `docker` / `prime` | image + interception + budget |
Tier 1 is the one to build first, and it is where our existing simulators already
sit. Tiers 2 and 3 are real work with real cost and should follow evidence from
tier 1, not precede it.
## What we already have that plugs in
**Six of the seven environments here own a simulator that is already steppable.**
They are one-shot only because the taskset hands the model the whole observation up
front. Each has an obvious interactive form:
| environment | one-shot today | interactive form | tier |
|---|---|---|---|
| `grand-exchange` | read 30 ticks of history, place one basket of orders | quote, see fills and the next tick, re-quote, over a turn budget — inventory and unfilled orders carry | 1 |
| `drop-table-inference` | read a fixed kill log, estimate the table | **buy** observations: each turn spends budget on more kills, then answers — measures when to stop looking | 1 |
| `fault-localisation` | read the whole incident log, name the cause | the log is *not* given: query it by service, time window and level under a query budget | 1 → 2 |
| `redaction-pressure` | write rules, scored by a scanner | two agents: a redactor and an extractor scored on what it recovers — the counterweight becomes an opponent | 1 (multi-agent) |
| `schema-migration` | emit a migration, executed against held-out rows | a real SQLite in a container the agent runs SQL against, with rollback | 3 |
| `bot-detection` | classify from six labelled examples | request labels one at a time under a budget (active learning) | 1 |
| `canary-trap` | write probes | no natural interactive form — leave one-shot | — |
**Tera's four spatial environments are already multi-turn — in the wrong language.**
`~/repos/gitea/tera/src/arena/` implements `tera.arena/v1`: Gym-style
`reset(seed, scenario)` / `step(action)` / `snapshot` / `restore` / `trace` /
`replay`, FNV-1a-64 checksums, SHA-256 source pins, train/dev splits, counterweighted
rewards, and a CI gate that proves inaction finishes below zero and a scripted
baseline finishes positive. That is a stricter contract than most of what is on the
Hub — and none of it is reachable from `uv run eval`, because it is TypeScript with
its own contract rather than a verifiers taskset.
The work there is a **bridge**, not a simulator: a Python `vf.Env` whose `run()`
drives the TS package over a subprocess or a small local HTTP loop, turning
`step()` observations into `interaction.turn()` messages and the existing component
rewards into `trace.record_reward` calls. The replay checksum makes the bridge
verifiable — a bridged episode must replay identically in the TS package or the
bridge is wrong. That is a rare property and worth building the bridge around.
⚠️ **RESOLVED 2026-08-21, 15:50 — `office-jobs-v1` EXISTS.** It was absent at 13:00 and
present at 15:50, because another session fast-forward-merged `feat/tera-ultracode-integration`
into `~/repos/gitea/tera` mid-session (reflog `8738367`). It is committed (`e378a03`,
`120eac8`, dated 2026-08-19), is in `ARENA_MANIFESTS`, is documented in `tera/ARENA.md`, and
passes `npm run arena:source-hashes`. **The true counts are twelve environments, five spatial,
twenty scenarios** — measured, not inferred. The site's original copy was right; the
correction to eleven/four/sixteen was right when made and wrong an hour later, and has been
reverted. Lesson: `tera` is a shared tree with live concurrent sessions — re-check its HEAD
before acting on a claim about what it contains.
~~⚠️ **`office-jobs-v1` does not exist.** `lumbridgecorp.com/arena` and
`services/api/src/seo.ts` both advertise five spatial environments; `ARENA_MANIFESTS`
in `src/arena/index.ts` holds four (`drive-101-v1`, `office-nav-v1`, `crow-nav-v1`,
`california-flight-v1`) and there is no `officeJobs.ts`. The real total is eleven,
not twelve. Decide whether to build it or correct the page before publishing
anything that cites the count.~~ (superseded — see above)
## The house rules under multi-turn
The three rules were written for one-shot scoring. Two survive unchanged and one
does not, and there is a fourth the interactive tier forces.
1. **Graded on what the model did not see** — unchanged and easier: a stepped
simulator generates its held-out continuation by construction.
2. **No single-sided reward** — *harder*. Multi-turn adds degenerate strategies that
have no one-shot analogue: stalling to burn the clock when the score is
time-averaged, spamming cheap turns when observations are free, refusing to act
when acting risks a penalty. Every new reward needs a counterweight in the *turn*
dimension, not just the answer dimension.
3. **A zero floor and a reachable ceiling, demonstrated** — unchanged, but the floor
now has to hold across a whole trajectory, not one reply.
4. **New: the budget must bind.** An interactive environment where the optimal
policy is "use every turn" is measuring nothing the one-shot version did not.
Observations must cost something, and the oracle must be shown to stop early.
**`probe.py` must grow degenerate *policies*, not just degenerate answers.** Today
it scores four fixed strings per environment. The interactive tier needs at minimum:
the staller (never acts), the spammer (acts every turn at random), the impatient
(answers on turn 1 with no observation), and the exhaustive (spends the entire
budget then answers). The gate stays the same shape — inaction 0.000, oracle 1.000,
exit 1 otherwise — plus a new assertion that the exhaustive policy scores *below*
the oracle. If it does not, rule 4 is violated and the environment does not ship.
This is the differentiator. Nothing in `prime-envs` (88 environments) or
`community-environments` (109) ships a degenerate-policy gate.
## Dual publishing
Both targets take the same wheel; the repo layout already mirrors verifiers' own.
- **Prime Intellect Hub**, our own namespace: `prime env push <name> -v PUBLIC`.
We keep authorship and versioning, and the listing can link back to
lumbridgecorp.com/arena.
- **`community-environments`**: a PR to that repo, published under the
`primeintellect` org by their CI on a version bump. Extra distribution, but it
hands the namespace to them. Secondary, not primary.
- **lumbridgecorp.com/arena**: unchanged — the page documents the same wheel.
⚠️ Its README still documents the **legacy** flow (`load_environment`, `vf-eval`,
`uv pip install -e .`). Ignore it; follow `verifiers/docs/v1/`.
⚠️ **Blocker: `PRIME_API_KEY` is not set on amd-server.** `prime config view` shows
`API Key: Not set`. The `pit_` key lives in karti's memory note on cloud-1
(`.claude/projects/-home-ubuntu/memory/prime-intellect.md`) and is the same
credential for `api.pinference.ai` and `api.primeintellect.ai`. Nothing pushes until
it is configured here.
## Preconditions, verified on this box
- `uv run eval @ configs/canary_trap.toml --dry-run` → exit 0, config written ✓
- verifiers 0.3.0 with all 13 harnesses and 4 runtimes present ✓
- Docker 29.1.3 running (tier 2/3 runtime) ✓
- spark-1 `brain` on `http://100.127.247.67:8001/v1`, model id `brain` ✓
- `prime` CLI installed (0.6.25; 0.6.26 available) ✓
- `PRIME_API_KEY` ✗ — see above
## Proposed sequence
**Phase 1 — one interactive environment, end to end.** `grand-exchange` is the
candidate: the market simulation already steps, the counterweight (liquidity, the
random-walk trap) already exists, and turning one basket into a quote/fill loop is
the smallest change that produces a genuinely new measurement. Deliverable: a
`vf.Env` in the existing wheel, a second config, and probe coverage for the four
degenerate policies. This is where rule 4 gets written for real.
**Phase 2 — extend `probe.py` to policies and re-gate everything.** Whatever
phase 1 teaches about degenerate trajectories applies to every later environment.
Do this before building a second one, not after. Note that `tests/test_probe.py`
and the wheel-build loop in `.gitea/workflows/ci.yml` hardcode the environment
list — both need editing or CI goes red.
**Phase 3 — the Tera bridge.** One Python `vf.Env` that drives the existing TS
package, validated by replaying every bridged episode through `replay()` and
requiring an identical checksum. Lands four environments at once and makes the
strongest existing asset publishable. Larger and riskier than phase 1; the
process boundary and the action encoding are the unknowns.
**Phase 4 — first Hub push**, once phases 1–3 give three to four interactive
environments alongside the seven one-shot ones. Needs the API key.
**Phase 5 — tier 2, then tier 3.** `fault-localisation` as a query-budgeted
tool-using environment, then `schema-migration` against a real database in a
container. Only after tier 1 has produced measured scores worth extending, and
with the concurrency cost of a container per rollout understood first.
## Open questions
- Does a bridged Tera episode replay to an identical checksum through a process
boundary? If not, the bridge cannot be verified and phase 3 gets much more
expensive.
- What is the per-rollout wall-clock cost of the `docker` runtime here, at the
concurrency an eval sweep needs? Unmeasured. It gates tier 3 entirely.
- Should the interactive form of an environment be a **separate** taskset id
(`grand-exchange-live`) or a config flag on the existing one? Separate ids keep
the one-shot scores comparable across time; a flag keeps the wheel count down.
Leaning separate.
PLAN.md
+243
View File
@@ -0,0 +1,243 @@
# Arena build plan
Companion to `MULTI_TURN.md` (the scope). This is the sequencing: what gets built,
in what order, what can run in parallel, and what collides.
Status: plan, 2026-08-21. ✅ = settled. ❓ = blocks the work under it. ⛔ = parked by decision.
## Where Arena sits
Lumbridge is an open-source company shipping six self-hostable things:
**Compute** (governed compute for unified-memory hardware), **Bench** (model quality
and serving measured against one pinned host × quant × config tuple), **Simulate**
(the Tera/Spaces 3D engine), **Stream** (a headless broadcast rig), **Forge**
(post-training as a content-addressed build), and **Arena**. Four of the six are
public. `apps/web/src/data/projects.ts` in `lumbridge-v4` is the canonical list.
**Arena is the RL-environment project — yes, `/arena` is the section.** It is
appended last in the project list for numbering reasons, but semantically it sits
beside Bench and feeds Forge: Bench measures a model, Arena produces the gradient,
Forge runs the training. That triangle is the reason Arena is worth doing well
rather than as a side quest — we own all three corners, which nobody on the Hub does.
Arena spans **two repositories on one page**, and that is deliberate:
- `git.karti.ai/lumbridge-public/arena` — the Python data environments (this repo)
- `git.karti.ai/lumbridge-public/tera` — the TypeScript spatial environments
## What we take from Prime Intellect, and what stays ours
All of PI's stack is open source and we use it rather than rebuild it. Checkouts are
at `~/vendor/prime-intellect/` (46 repos; see its README).
| We use, unmodified | We build |
|---|---|
| `verifiers` — the v1 spec, the taskset/harness/trace model, `vf.Env` | The environments |
| `prime-rl` — the trainer traces feed | The house rules |
| `prime-agent` (`pi`) — a harness, already embedded in Piggy | `probe.py` — the degenerate-policy gate |
| `prime` CLI — `env push`, `env action` | The Tera bridge |
| The Environments Hub — distribution | lumbridgecorp.com/arena — documentation |
We do **not** build a harness, a trainer, or a hub. Every one of those exists,
is permissively licensed, and is better than what we would write.
## Decisions
**✅ 1 — Hub org handle: `lumbridge`.** Settled 2026-08-21. Environments will
publish as `lumbridge/<env-name>`.
**✅ 2 — Attribution: Lumbridge as author.** Settled 2026-08-21. `karti.ai` stays
the code host (`git.karti.ai`), Lumbridge is the author on every surface. One
identity across all six projects; no per-project byline.
**⛔ Hub push is ON HOLD.** Decided 2026-08-21: keep building, publish nothing to
the Prime Intellect Hub yet. Workstream E is parked — not cancelled, not blocked on
anything technical. `PRIME_API_KEY` is therefore not urgent.
**✅ 7 — Production brain: Qwen3.8-27B.** Settled 2026-08-21. spark-1 serves
`brain-qwen38-dspark`; that is the default. The Nemotron baseline is ARCHIVED, not
deleted, and Arena's seven scores get re-run and republished against Qwen.
⚠️ **`--model brain` does NOT 404 — it silently succeeds and records `brain` in the
trace while the box serves Qwen.** Verified by direct call 2026-08-21. Pin
`brain-qwen38-dspark` in every reproduction command, everywhere.
**✅ 8 — `tera-drive-101` ships labelled.** Settled 2026-08-21. Published as a
**bridge-correctness** environment with NO rule-3 and NO rule-4 claim (its return is
monotone in throttle; any positive constant scores ~0.95+). Site says **four**
spatial environments. It is not listed as a fifth RL environment.
**✅ 9 — `app.lumbridgecorp.com` is the ONLY authenticated origin.** Settled
2026-08-21. The apex loses `/login`, `/signup`, `/app`, `/go`. All existing sessions
sign out once on cutover; accepted cost for three accounts.
**✅ 10 — Adopted defaults** (technical, recommended by the synthesis, settled
2026-08-21): the agent seat is named **`agent`** everywhere (matching
`SingleAgentEnvConfig`) — it becomes the TOML key in every `configs/*.toml` AND every
prime-rl `rl.toml`, and renaming later parses fine while silently having no seat;
**D lives in the `arena` repo**, not `tera` (amend the parallelization map);
**A ships the FLAT freeze**, with per-item recorded as the identified strengthening;
**Arena transcripts are ADMIN-ONLY** (they carry the held-out slice and oracle
answers — a contamination question, not an org-politics one); **the dashboard is the
archive and `eval-results.json` a curated summary emitted from it**, never two
hand-maintained files; **Forge integrates prime-rl with LoRA RL first**.
**✅ 3 — `office-jobs-v1` EXISTS; the counts are twelve / five / twenty.** Resolved
2026-08-21 15:50 when another session fast-forward-merged it into the shared `tera` tree
mid-session. It is committed, in `ARENA_MANIFESTS`, and passes `arena:source-hashes`.
The site's original copy was correct. ⚠️ `tera` has live concurrent sessions — re-check
its HEAD before acting on any claim about its contents.
~~**❓ 3 — `office-jobs-v1`.** Advertised on the live page and in `seo.ts`; absent
from `ARENA_MANIFESTS`. Build it, or correct the page to eleven. Blocks any copy
that cites the count, including a Hub listing.
**✅ 4 — Interactive envs get their own taskset id.** Settled 2026-08-21. The first
is **`grand-exchange-live`**, a separate wheel beside `grand-exchange`. One-shot
scores stay comparable across time; the two forms are measured, not conflated.
**✅ 5 — App is a subdomain: `app.lumbridgecorp.com`.** Settled 2026-08-21. Matches
the two-surface model (public site + private platform). Public site stays a site.
**✅ 6 — Compute policy.** Settled 2026-08-21. **Training runs on rented GPUs**
(GH200-class, from Prime Intellect) — rented because it is faster, and because
renting keeps the hardware choice open per run (the GB300-vs-alternatives call is
one we want to keep making). **Inference in production runs on the DGX Spark.**
We do **not** train on the Spark.
## Workstreams
Five, with the dependency edges called out. **A and D can start immediately and in
parallel. B must land before C or E.**
### A — `grand-exchange` interactive *(start now)*
The first tier-1 environment: quote → see fills → re-quote over a turn budget,
inventory and unfilled orders carrying between turns. Chosen because the market
simulation already steps, the counterweight (liquidity in gp/tick, the random-walk
trap) already exists, and it is the smallest change that yields a genuinely new
measurement.
Deliverable: a `vf.Env` subclass exported from the wheel's `__init__.py`, a config,
and a measured run against spark-1's `brain`. Pattern to copy:
`verifiers/v1/tasksets/textarena/taskset.py`.
This workstream is where **rule 4 — the budget must bind** gets written for real.
Whatever it learns about degenerate trajectories is the input to B.
Depends on: nothing — ✅4 settled.
### B — `probe.py` grows degenerate *policies* *(after A's first draft)*
Today the probe scores four fixed strings per environment. Tier 1 needs policies:
the **staller** (never acts), the **spammer** (acts every turn at random), the
**impatient** (answers turn 1 with no observation), the **exhaustive** (spends the
whole budget, then answers). Same gate shape — inaction 0.000, oracle 1.000, exit 1
otherwise — plus a new assertion that **exhaustive scores below oracle**. If it does
not, rule 4 is violated and the environment does not ship.
Do this before building a second interactive environment, not after. It is the
differentiator: nothing across `prime-envs` (88) or `community-environments` (109)
ships a degenerate-policy gate.
Depends on: A (needs one real interactive env to gate against).
Blocks: C, E.
### C — More interactive environments *(after B)*
In rough order of value:
1. `drop-table-inference` — buy observations under budget; measures when to stop looking
2. `bot-detection` — request labels one at a time (active learning)
3. `fault-localisation` — the log is not given; query it by service, window and level
4. `redaction-pressure` — two agents, redactor vs extractor; the counterweight becomes an opponent
Each is independently parallelizable **once B exists**. `redaction-pressure` is the
most interesting and the least certain — a genuine multi-agent env, not a stepped one.
Depends on: B.
### D — The Tera bridge *(start now, independent repo)*
One Python `vf.Env` driving the existing TypeScript `tera.arena/v1` package over a
process boundary, turning `step()` observations into `interaction.turn()` messages
and existing component rewards into `trace.record_reward` calls.
Validated by replay: every bridged episode must `replay()` through the TS package to
an identical FNV-1a-64 checksum, or the bridge is wrong. That property is rare and
the bridge should be built around it.
Lands four environments at once and makes our strongest existing asset publishable.
Larger and riskier than A — the process boundary and the action encoding are the
unknowns. **Touches the `tera` repo, not this one, so it does not collide with A/B/C.**
Depends on: nothing.
### E — First Hub push *(PARKED)*
Push once there are three to four interactive environments alongside the seven
one-shot ones. `prime env push <name> -v PUBLIC` to our own namespace is primary —
we keep authorship and versioning. A PR into `community-environments` is optional
extra distribution that hands the namespace to `primeintellect`; secondary.
Then update `lumbridgecorp.com/arena` to document the same wheels.
⛔ On hold by decision, 2026-08-21. Resume when Karti says so; ❓3 still gates any
copy that cites the environment count.
## ⚠️ The collision point
**`tests/test_probe.py` and the wheel-build loop in `.gitea/workflows/ci.yml`
hardcode the environment list.** Every workstream that adds an environment edits
both files. Two sessions adding environments concurrently *will* conflict there, and
CI goes red if either is missed.
Mitigation, in order of preference: (1) B refactors both to discover environments
from `environments/*/pyproject.toml` instead of a literal list — do this as part of
B and the collision disappears; (2) failing that, one session at a time touches
those two files.
Separately: this repo already took divergent remote pushes from concurrent sessions
on 2026-08-19. **Merge, never force.** Use a worktree per workstream.
## Parallelization map
```
now ──┬── A (grand-exchange interactive) ──┬── B (probe → policies, de-hardcode CI) ──┬── C1 drop-table
│ │ ├── C2 bot-detection
│ │ ├── C3 fault-localisation
│ │ └── C4 redaction (2-agent)
│ │
└── D (Tera bridge, tera repo) ──────┴──────────────────────────────── E (Hub push) ⛔ PARKED
```
Two sessions can start today: **A** in this repo, **D** in `tera`. They share no
files. Everything downstream of B fans out four ways.
## Preconditions, verified on amd-server
- `uv run eval @ configs/<env>.toml --dry-run` → exit 0 ✓
- verifiers 0.3.0, all 13 harnesses and 4 runtimes present ✓
- Docker 29.1.3 (tier 2/3 only; tier 1 needs none) ✓
- spark-1 `brain` at `http://100.127.247.67:8001/v1`, model id `brain` ✓
- `prime` CLI 0.6.25 ✓
- `PRIME_API_KEY` ✗ — blocks E, which is parked anyway
---
## ⚠️ Superseded by `EXECUTION.md`
The workstream descriptions and the parallelization map above were written before the
2026-08-21 design workflow. **`EXECUTION.md` is now authoritative** for sequencing,
file ownership and risk. Three structural corrections it makes:
1. **B splits.** **B1** (de-hardcode environment discovery) does NOT depend on A and
ships immediately; the CI env list was never a matrix, just a bash `for` loop.
**B2** (degenerate-policy machinery) was found NOT FEASIBLE as written and must be
calibrated against A's *measured* engine, not A's first draft.
2. **D is no longer file-independent.** It lives in `arena` and edits `probe.py`,
`ci.yml`, `test_probe.py`, `README.md` plus two `lumbridge-v4` files.
3. **The top risk is that every Arena reward SATURATES**, so house rule 4
("exhaustive < oracle") may be structurally unreachable. If it is, every
interactive environment is unfalsifiable and the reward shape must change before
C1–C4 are built on it. This is also a compute-policy constraint: a non-binding
turn budget burns rented GPU hours measuring nothing.
docs/FIRST_EVAL.md
+185
View File
@@ -0,0 +1,185 @@
# Running an eval — the runbook
Written 2026-08-21, the day Arena first measured anything end to end. Before this,
`outputs/` held one `config.toml` and zero episodes: every number downstream of an
eval — the dashboard, `eval-results.json`, the leaderboard, the before/after
training table — was waiting on a run that had never completed.
This is that run, written down so the next one is a copy-paste.
---
## 1. The command that works
```bash
cd ~/repos/gitea/arena
export SPARK_API_KEY=dummy
uv run --project environments/canary_trap eval @ configs/canary_trap.toml \
--model brain-qwen38-dspark \
--client.base-url http://100.127.247.67:8001/v1 \
--client.api-key-var SPARK_API_KEY \
--no-push --no-rich \
-c 8 -o outputs/run-<stamp>/canary-trap
```
Swap `environments/<pkg>` and `configs/<pkg>.toml` for any of the seven. The
sweep script that ran all of them is `outputs/run_all.sh` (gitignored); the
counter is `outputs/count_rewards.py`.
### Why each flag is there
| flag | why |
|---|---|
| `--model brain-qwen38-dspark` | **The served id, pinned.** `--model brain` does *not* 404. SGLang does not validate the model field on the chat route: it returns 200, Qwen answers, and the trace records `brain`. The old `outputs/canary-trap--brain--null/` dir is exactly that mistake. Pin the id `GET /v1/models` reports. |
| `--client.base-url` | The default is `https://api.pinference.ai/api/v1`, which needs a `PRIME_API_KEY` this box does not have. |
| `--client.api-key-var SPARK_API_KEY` | spark-1 requires **no** key — a bare `POST /v1/chat/completions` returns 200. But the client still resolves *some* env var, so point it at a dummy one rather than leaving it on `PRIME_API_KEY` (empty → the failure mode of the one earlier attempt). |
| `--no-push` | Hub push is parked (PLAN.md). Push is **on by default** and needs `$PRIME_API_KEY`. |
| `--no-rich` | The live dashboard is in-process only and unreadable in a log. Without it you get one `rollout done: … reward=…` line per rollout. |
| `-c 8` | spark-1 is shared with Chatterbox, vox and ASR. The config's own `max_concurrent = 128` will hammer it. |
| `-o <dir>` | Otherwise the run lands in `outputs/<env>--<model>--<harness>/<uuid>/` and you have to go hunting for the uuid. |
### Flags our own docs get wrong
- `README.md`'s "Running one" line omits `--client.base-url` / `--client.api-key-var`
entirely, so copy-pasting it points at Prime's inference API with an empty key.
That is the one attempt that was ever made, and why `outputs/` was empty.
- `MULTI_TURN.md`'s preconditions say the spark-1 model id is `brain`. The served
id is `brain-qwen38-dspark`; `brain` silently mislabels the trace.
- `EXECUTION.md` is right that `--harness.id null` is not valid — `EvalConfig` has
no top-level `harness`. The configs already carry `[env.agent.harness] id = "null"`,
so nothing needs passing on the command line. The CLI form, if you ever need it,
is `--env.agent.harness.id null`.
- The `@ config.toml` form is a *positional* argument, not a flag: `eval @ file.toml`.
Inside a shell script quote it as `"@"` so it is not glob-expanded.
---
## 2. The output layout verifiers 0.3.0 actually produces
`verifiers/v1/cli/output.py:38` — `output_path()`:
```
outputs/<env>--<model>--<harness>/<uuid>/
├── config.toml # the RESOLVED config, re-runnable as `eval @ config.toml`
├── traces.jsonl # one JSON *Episode* per line, appended as rollouts land
└── eval.log # only when you redirect it there yourself
```
`--output-dir` replaces the whole `<name>/<uuid>` pair, so an explicit `-o` gives
you a flat dir with no uuid leaf. Slashes in the model id become `--`.
**One Episode per line, not one Trace.** Shape:
```jsonc
{"id": "...", "env": {...}, "ok": true, "errors": [],
"traces": [ { "id": "...", "agent": "...", "rewards": {...},
"metrics": {...}, "stop_condition": "agent_completed",
"errors": [], "timing": {...} } ] }
```
Two traps in that file, both of which bite an ingester:
1. **`Reward.value` is a `@property` and never appears in the JSON.** Each entry
in `rewards` is `{"score": float, "weight": float}`. The episode total is
`sum(score * weight)`. The `reward=0.525` in the log line is computed, not stored.
2. **`write_episode` dumps with `exclude_none=True`.** A `None` reward is dropped
from the file entirely, so "not measured" and "key absent" are indistinguishable
on the wire. Carry the run's declared reward-name set from `config.toml` if you
need to tell them apart.
### ⚠️ 0.4.0 moves this
Upstream HEAD (`~/vendor/prime-intellect/verifiers`) produces one **flat**
`<env>--<model>--<harness>--<short-id>` directory with no nested uuid, and the
resolved config moves to `configs/<cli>.json` (JSON, because JSON keeps nulls).
`RunConfig.id` is a `PrivateAttr` and is absent from that file — derive the run id
from the directory basename, not from the config.
---
## 3. Counting what actually scored
**`eval` exits 0 even when every rollout errors.** An errored trace is written with
`rewards: {}`. Never report a run on its exit code.
```bash
uv run python outputs/count_rewards.py outputs/run-<stamp>/<env>
```
It prints `traces_scored` (non-empty `rewards`), `traces_errored` (`rewards: {}`),
`provider_errors`, the stop-condition histogram, and two means:
- `reward_mean_scored` — the mean over what survived. This is Arena's own
"score what survived" trap in reporting form: a run where 30 of 32 rollouts
errored and the 2 survivors scored 0.9 reads as 0.900.
- `reward_mean_attempted` — errors counted as 0. **Rank on this one**, and refuse
to rank at all below a scored-fraction floor.
A quick eyeball without the script:
```bash
jq -r '.traces[] | if (.rewards|length)>0 then "scored" else "errored" end' \
outputs/run-<stamp>/<env>/traces.jsonl | sort | uniq -c
```
---
## 4. What it cost, and what it scored
**The run: `outputs/run-20260821-1401/`.** Seven data environments, 32 rollouts each,
224 episodes, **223 scored**. Model `brain-qwen38-dspark` (Qwen3.8-27B) on spark-1.
Wall-clock 14:03 → 14:58, 55 minutes, `-c 8`.
⚠️ **Compute the mean as `sum(score × weight)` per trace.** An unweighted mean over the
reward components is a different number and it is wrong — it disagrees with what the
harness itself prints. Cross-check against the `reward=` field on each env's 32
`rollout done` lines in `eval.log`; the two must agree exactly. The first transcription
of this run got six of seven means wrong by taking the unweighted mean.
| environment | traces | scored | errored | reward_mean_attempted | reward_mean_scored |
|---|---|---|---|---|---|
| `fault-localisation` | 32 | 32 | 0 | **0.9531** | 0.9531 |
| `canary-trap` | 32 | 32 | 0 | **0.6289** | 0.6289 |
| `schema-migration` | 32 | 32 | 0 | **0.4432** | 0.4432 |
| `drop-table-inference` | 32 | 31 | 1 | **0.4174** | 0.4309 |
| `redaction-pressure` | 32 | 32 | 0 | **0.4040** | 0.4040 |
| `bot-detection` | 32 | 32 | 0 | **0.3488** | 0.3488 |
| `grand-exchange` | 32 | 32 | 0 | **0.0055** | 0.0055 |
The single errored episode is a genuine spark-1 gateway timeout (`ProviderError`, 504) in
`drop-table-inference` — not a scoring bug. **Rank on `_attempted`** (errors as 0); the
`_scored` column is Arena's own "score what survived" trap in reporting form.
### Two numbers not to publish as capability scores
⚠️ **`grand-exchange` 0.0055 measures a sampling configuration, not a capability.**
In **31 of 32 rollouts** the model returned a valid, parseable, degenerate answer —
`{"expected_profit": 0, "orders": []}` in 18 completion tokens, `finish_reason=stop`,
never entering a thinking block. The one rollout that did reason (4,954 completion tokens)
placed 4 real orders and scored 0.176 weighted, with `filled=1233` against `planned=2100`.
**The parser, the execution engine and the reward all work end to end** — the environment
is not broken. But `probe.py` rates the *crude* baseline here at 0.082, so 0.0055 sits
**below crude** and barely above inaction. Re-run with thinking forced on before this
number goes anywhere near `/evals`.
⚠️ **`fault-localisation` 0.9531 has two of four components pegged at maximum.**
`evidence` mean 1.000 (max 1.000) and `fault` mean 1.000 (max 1.000) across all 32
rollouts; only `gate` (0.9062) and `service` (0.9062) still move. Half the reward mass
no longer discriminates at this capability level. Nemotron scored 0.352 here on 19 August —
a 2.7× swing that needs separating into a stronger model, a prompting difference, or an
environment that has become too easy, before it appears on a scoreboard.
### The `gate` component carried no gradient in four of seven environments
Across all 32 rollouts each, `gate` scored **exactly 0.000, max 0.000**, in
`bot-detection` (weight 0.25), `grand-exchange` (0.25), `redaction-pressure` (0.30) and
`schema-migration` (0.25). It fired elsewhere — `canary-trap` 0.2188, `drop-table` 0.2258,
`fault-localisation` 0.9062.
This is **not** a house-rule-3 violation: `probe.py`'s oracle row is 1.000 for every one of
those environments, which requires `gate = 1`, so it is reachable. But a quarter to a third
of the reward mass produced no gradient in this run, and any ranking or before/after
training table over those four environments is really a ranking over their two soft
components. Understand that before training against them.