lumbridge-public/arena
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: the scope, the plan, the post-review execution plan, and the eval runbook

Browse Source 
MULTI_TURN.md scopes the multi-turn gap and records the finding that made it
cheap: multi-turn needs no container and no real harness. vf.Env.run programs the
control flow host-side and the simulator runs in-process as the user, with the
harness left null.

PLAN.md carries the settled decisions. EXECUTION.md is the post-adversarial-review
plan — every position in it survived a reviewer, and where a design spec was
overturned the corrected position is what is written down.

Two corrections are marked in place rather than silently rewritten, because both
were believed and acted on:

- The provenance hole is src/arena/index.ts, not offices/sites.ts. Mutating
  offices/sites.ts produces a bit-identical episode; index.ts re-exports the
  scripted baselines that ARE the reward denominator, and a halved office-nav
  baseline moves return 4.744 -> 4.098 while every tera hash gate prints ok.
- office-jobs-v1 is not phantom. It was absent at 13:00 and present at 15:50,
  because another session fast-forward-merged it into the shared tera tree
  mid-session. The counts are twelve, five and twenty.

docs/FIRST_EVAL.md is the runbook for the first evaluation Arena ever completed:
seven environments, 224 episodes, 223 scored, against brain-qwen38-dspark. It
warns that the mean is sum(score x weight) — an unweighted mean over components
is a different number, and it is how six of seven means were first misreported.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
karti
2026-08-21 15:55:43 -07:00
co-authored by Claude Opus 5
parent 162d67a83c
commit e4cc2bf1c3
4 changed files with 917 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
EXECUTION.md
+281
View File
@@ -0,0 +1,281 @@
# Execution plan — Arena / Lumbridge
Generated 2026-08-21 by a 13-agent design + adversarial-verification workflow.
Companion to `PLAN.md` (decisions) and `MULTI_TURN.md` (scope).
⚠️ Every position here is the POST-REVIEW position. Where an adversarial reviewer
overturned a design spec, the spec's original claim was discarded. Do not resurrect
a claim from an earlier document that this one corrects.
# Arena / Lumbridge — one execution plan
Synthesis of six adversarially-reviewed workstream specs, 2026-08-21. Every position below is the **post-review** position. Where a reviewer overturned a spec, the spec's original claim is marked and discarded.
Verified independently for this document (amd-server, 2026-08-21): `arena/probe.py` is 772 lines with a literal 7-name shim tuple at :89-93 and a literal `results` dict at :752-760; `tests/test_probe.py` has a 7-name `ENVIRONMENTS` tuple at :13-21 and `timeout=30` at :31; `.gitea/workflows/ci.yml` is 42 lines, its probe step is still named "Run four-environment floor and ceiling probe" (:26) for a seven-environment probe, its wheel loop (:36-40) is a hardcoded bash `for` over seven names, and its lock-policy check (:24) is failing on main because **three** `uv.lock` files are tracked (`bot_detection`, `drop_table_inference`, `grand_exchange`); `outputs/` contains exactly one file, a `config.toml` with `push = true` and `base_url = https://api.pinference.ai/api/v1` — **no `traces.jsonl` exists anywhere on this box**; `~/.prime/config.json` has an empty `api_key`; `tera/src/arena/index.ts:74-79` `ARENA_MANIFESTS` holds four.
---
## 1. Verdicts applied — what each spec becomes
### A — `grand-exchange-live` · feasible, but **not in the order written**
Stands: the multi-turn mechanism (`Env.run` + `interaction.turn()`, null harness, no container), the delta-sheet action grammar, the malformed-reply policy, `roc` on **peak** capital employed (average-employed is a measured free-points bug), the vendored-generator equality assertion, no turn-count reward term.
Corrected, blocking:
1. **The measured ladder does not reconcile with the spec's own reward.** At ref 72,758 gp and `TARGET_SHARE = 0.85`, target = 61,844; impatient at 50,470 gp gives `profit_ratio` 0.816, `discipline` ≤ 0.816, `clean` false → max 0.612, not the reported 0.623. Churn and exhaustive *do* reconcile, so this is a specific defect. **Nothing in §1/§4 is locked until it is re-measured with the shipped engine.**
2. **`viable_market` retargeted at `live_reference` is circular as written.** Viability needs `freeze`; `freeze` is drawn in `Taskset.load()` after viability returns. Fix: derive `freeze` deterministically from `market.seed` (e.g. `random.Random(seed ^ FREEZE_SALT)`), inside `build_market`, so viability and scoring see the same value. This is also the fix for the unstated RNG source — without it, `scan()` recomputes a reference the model never played against.
3. **§6's held-out claim is inverted.** Live is *weaker* than one-shot: the model sees ~50 of 60 graded ticks. Rule 1 still passes; the README must say "held out: the tail after the last observation, ≥4 ticks", not "stronger by construction". Add a probe assertion on the *size* of the held-out slice, not only the frontier.
4. **The probe battery blows CI's own harness.** `probe.py` runs in 3.4 s today under `timeout=30`. §11 adds 9 policies × blocks × 240 baskets + two sweeps + a 240-seed generator check + a hostile battery replayed per turn. Raising the timeout is B1's job (below), but A must budget the cost, not append a name to a list.
5. **Taskset load cost is unbounded.** `num_tasks` defaults 48, `VIABILITY_TRIES = 64`; a live `viable_market` runs a full 8-turn stepped reference per candidate → ~3,072 simulations before the first token, on every eval and every `--dry-run`. Cache the reference per seed, or cap tries for live.
6. **`import` vs `copy` contradiction.** §3 says "import `grand_exchange.book.parse_orders`", §9 says vendor. It is a copy. Say copy.
7. **`[env.agent] max_turns = 8` is an off-by-one risk, not free insurance.** Verify how the framework counts, or set 9.
8. **`trace.info["episode"]` must be plain JSON types.** A dataclass or numpy scalar in `audit()` fails the trace write for the whole rollout.
9. **Effort is L/XL, not L** — it is A plus most of B.
**Revised A build order (this is the change that matters):** `market.py`/`book.py` copies + equality assertion → **`live.py` + unit tests → re-measure the full ladder in the shipped engine** → only then lock `horizon`, `FREEZE_RANGE`, `TARGET_SHARE`, `TURNS` → `strategy.py` sweep → `ledger.py` → `protocol.py` → `taskset.py` → probe.
### B — probe policies + de-hardcoding · **verdict: not feasible as written. Split it.**
The reviewer is right and the split is the fix.
**B1 — de-hardcode environment discovery. Sound, S/M, ships immediately, behaviour-preserving.** `tomllib` over `environments/*/pyproject.toml` replaces the shim tuple, the `main()` results dict (via a `@probes(name)` registry decorator), `ENVIRONMENTS` in `tests/test_probe.py`, and the bash wheel loop. **The CI environment list was never a matrix** — it is a `for` loop in one step (verified) — so no generator job, no `fromJSON` matrix, no act_runner risk. Take B1 now and the repo's known collision point disappears for one afternoon's work.
Two mandatory amendments to B1:
- **Do not make "discovered but unregistered" a hard exit 1 while A is in flight.** The moment A commits a scaffold `pyproject.toml`, `probe.py` exits 1 and CI goes red for every session on the repo. Make it a warning until an env registers a probe, and flip it to exit 1 in the same commit as A's `exchange_live()`.
- **Make the two guards use the same denominator.** `ci.yml`'s "empty scan" counts directories; `discover()` counts `*/pyproject.toml`. A scaffold dir without a pyproject makes them disagree. Count `*/pyproject.toml` in both.
- Raise `tests/test_probe.py` `timeout=30 → 180`, keep the lock check *before* any `uv sync`, and fix the "four-environment" step name.
**B2 — the `Interactive` policy machinery. Redesign required; do not build it before A's engine exists.** The structural claim ("oracle and exhaustive share `answer`, so rule 4 follows") is **false when the reward saturates**, and Arena's rewards all saturate: `profit_ratio` and `efficiency` clip at 1.0 and `clean` is a boolean band. If exhaustive also clears the band it scores 1.000 and the gate reads "budget is free". Further:
- `BIND_MARGIN = 0.02` is staked on realised-P&L differences over 24 seeds, against a block-to-block variance `probe.py:349-352` already documents at 0.48 vs 0.18. The five-block ladder is **required for all three rule-4 comparisons**, not an optional mitigation — multiply the cost estimate by ~5.
- `gate` is binary at weight 0.25, so the exhaustive/oracle gap is quantized in 0.25 steps. A 0.02 margin against a 0.25 quantum is a cliff, not a calibration.
- `play()` never terminates on an answer; "impatient answers turn 1" is an unstated precondition on A's `step()`.
- Seeds must come from A's `baskets()` skip-aware helper, not a contiguous `range`.
- B2's §1.8 dictates A's private module layout, class names, weight vector and turn mechanism. **Delete that.** A owns its contract; B2 adapts to it. This is the collision the workstream exists to remove.
**Net: B1 is unblocked and independent of A. B2 depends on A's *measured* engine, not A's first draft.** PLAN.md's "B depends on A" edge survives for B2 and dies for B1.
### D — Tera bridge · feasible, but **it is no longer parallel with A/B, and PLAN.md must be amended**
Stands and is the strongest evidence in the six specs: 23-file / 283 KB / zero-bare-specifier TS closure; `node` type-strips raw `.ts`; **32/32 scenario × seed traces round-trip through Python `json` and replay to identical checksums**; one long-lived NDJSON worker (84 ms start, 22 µs RTT, 217 µs/step, 87 MB RSS) beats HTTP and beats per-rollout processes; the measured hold-length sweep.
Corrected, blocking:
1. **Role name vs config key — the spec will not load.** `agents.pilot` requires a config field named `pilot`, but all four TOMLs write `[env.agent.harness]`, and `EnvConfig` is `extra="forbid"`. **Name the role `agent`**, matching `SingleAgentEnvConfig`. Same trap A must resolve; settle it once for both (see Decision 1).
2. **`CONFIG = ...` classvar does not exist in verifiers v1.** Config is resolved by `concrete_type()` through `__orig_bases__`. Use the wordle form: `class CrowNavTaskset(TeraSpatialTaskset, vf.Taskset[TeraSpatialTask, CrowNavConfig])`. As written, every per-env `max_turns`/`max_hold` silently does nothing.
3. **The provenance claim is real but was mis-located — CORRECTED 2026-08-21.** `check-arena-source-hashes.mjs` pins 19 of 23 vendored files. The claim that `offices/sites.ts` is the material hole is **FALSE and was refuted twice independently**: mutating it (elevation 4→4000, heading 0→90, width 54→5400) produces a **bit-identical** episode, so it does not reach the simulation. ⚠️ **The real hole is `src/arena/index.ts` alone**, and it is worse than the original claim: rewriting it so `officeNavScriptedBaseline` re-exports a halved policy moves office-nav from 68 steps / return 4.744 to 136 steps / 4.098 — **while `npm run arena:source-hashes` prints "ok" for all four environments and exits 0**, `envHash` and `simulatorHash` unchanged. That file is the reward DENOMINATOR. Fix: pin `src/arena/index.ts` in `tera/scripts/check-arena-source-hashes.mjs` (a one-line `tera`-repo change) and regenerate the four pairs in `sourceHashes.ts`.
4. **`oracle.py` reimplementing the four TS baselines in Python contradicts the bridge's own rule** ("never recompute a number that came out of TypeScript") — and it does so for the reward *denominator*. Add an `oracle` op to `worker.mjs`; the four baselines are already exported from `src/arena/index.ts`.
5. **`drive-101` measures nothing, and the 0.9 ceiling band makes it worse than "no rule-4 claim".** Return is monotone in throttle, 12/12 goals at every value; `0.9 × 4.339 = 3.905` and throttle 0.2 already returns 3.723 → quality 0.953. Any positive constant throttle scores ~0.95–1.00. Same collapse on office-nav: any goal-reaching run clears the band, so `control` (0.50) and `gate` (0.20) are constants and only `efficiency` carries signal.
6. **office-nav's rule-4 argument is backwards under its own formula.** `efficiency = oracle_turns / max(turns, oracle_turns)` only ever *penalises* extra turns, and past `ceil(1.5 × 1.8) = 3` turns `clean` also drops. The formula punishes exactly the "precision buys turns" behaviour cited as evidence.
7. **`gate` is a separate additive term for goal-attainment that is already inside `ts_return` as the success bonus** — double-counted, additively. That is the failure `probe.py:15-17` records verbatim. Make it a multiplier on the sum.
8. **Missing:** no `@vf.stop`; no handling of a mid-loop `terminated` Segment (the loop as written hits `RuntimeError('the exchange is over')`); no `Taskset.load()` / prompt-vs-bare-`turn()` decision; an empty `{"plan":[]}` costs zero steps while a malformed reply costs 10, inverting the spec's own principle; no guard for `oracle_return <= 0`; `max_hold = 400` exceeds `MAX_STEPS = 360`.
9. **Scope.** PLAN.md:138 and the parallelization map say D lives in `tera` and shares no files with A/B/C. This spec puts everything in `arena` and edits `probe.py`, `tests/test_probe.py`, `ci.yml`, `README.md`, `configs/`, plus two `lumbridge-v4` files. That is a real decision (the packaging argument is correct — `tera` has no Python toolchain), but **PLAN.md must be amended to record it**, because it converts D from "parallel with everything" into "collides with A, B and W".
**Revised D:** ship the bridge + replay gate + `tera_crow_nav` first. `drive-101` ships only as a labelled bridge-correctness environment or not at all (Decision 3). `office-nav` and `california-flight` follow. D's probe integration waits for B2.
### P — `app.lumbridgecorp.com` · feasible; **Phase 1 as written cannot ship**
Stands and is high-value: the endpoint inventory (29 routes, not 31), the 23 dead `api.ts` methods, one-build/two-hosts with app.* as the **only** auth origin, `bind 10.0.0.2` + the empty-200 verification, the `isAppHost` branch in `sendShell`, `WORLD_LAUNCHER_URL` convergence for `/go/:slug`, `World1` moved not deleted.
Corrected, blocking:
1. **"Ingest the runs already on amd-server" is impossible — there are none.** One `config.toml`, zero episodes, and that config points at `api.pinference.ai` with `PRIME_API_KEY`, which is empty. **Phase 1 must contain "execute one eval to completion against spark-1" as an explicit prerequisite**, or Arena's dashboard ships empty.
2. **The leaderboard is operator-gameable and mis-specified.** (a) The server must recompute every aggregate in SQL from the `arena_episodes` it just inserted, never trust the pusher's numbers. (b) Ranking on a mean over *scored* episodes is Arena's own "score what survived" trap. Store `reward_mean_scored` **and** `reward_mean_attempted` (errors as 0), rank on attempted, refuse to rank below a scored-fraction floor.
3. **`arena_episodes` flattens a list.** `Episode.traces` is a list; `reward`/`rewards`/`stop_condition`/`last_reply` are per-**Trace**. Primary key must be `(run_id, episode_id, trace_id)` with an `agent_name` column — otherwise the schema is wrong for `grand-exchange-live` (simulator as second role) and for `redaction-pressure` (genuine two-agent).
4. **The trace viewer publishes the held-out slice.** `GET /api/arena/runs/:id/episodes/:eid` at `user` scope returns full transcripts, which contain held-out task data and, for oracle runs, the answers. One leaked invite publishes the held-out corpus. **Transcripts are admin-only or redacted.** This settles the spec's open question on read scope.
5. **The compute-scene-vocabulary blocker is fabricated.** `computeScene` is a free-form slug regex, not `z.enum(SCENE_NAMES)`; `stock`, `nemotron` and `qwen38` all pass today. Delete that precondition.
6. **`portal_host_samples` and `portal_container_samples` have no node identity** — no `node_id`, no host column, no PK. Per-node charts need `ALTER TABLE … ADD COLUMN IF NOT EXISTS node_id text` plus an index first.
7. **`scored = ok AND rewards non-empty` is wrong.** `write_episode` dumps with `exclude_none=True`, so a `None` reward is dropped from the file entirely — "not measured" and "key absent" are indistinguishable on the wire. Require at least one non-null `Reward`, and carry the run's declared reward-name set from the config.
8. **`Reward.value` is a `@property`** and never appears in dumped JSON. Compute `score × weight` in the ingester.
9. **0.3.0 → 0.4.0 changes the directory layout, not just the config filename** — HEAD produces one flat `<env>--<model>--<harness>--<hex8>` dir with no nested uuid, and `RunConfig.id` is a `PrivateAttr` absent from `configs/eval.json`. Derive the run id from the directory basename.
10. **Name the private console `/nodes`**, not `/compute` — `/compute` is already a public marketing section with its own nav.
11. Put `isAppHost` in `packages/shared` — two independent string checks that must agree, with a silent asymmetric failure mode (dashboard rendered under a 404 status).
12. `sendShell(url, reply)` takes no request; the signature and both call sites change. `spa-fallback.test.ts` needs a case. `@radix-ui/react-select` is **not** installed.
**Revised P phasing.** The honest first ship is **P1: Caddy host + host-aware routing + `renderAppShell` + AppShell + Overview + Worlds + Org + Inbox + the `api.ts` trisection.** All of that runs on endpoints that exist today. **Arena (schema, 6 routes, ingester, 4 pages) is P2, a full workstream of its own**, and it is gated on a real eval having run.
### W — website visual vocabulary · feasible; **the data pipeline must be redirected, not built**
Stands: the diagnosis (the site draws what it hasn't measured and narrates what it has), `MeasureTable` / `Bar` / `FlowStrip` / `SpecSheet`, inline SVG with `currentColor` and **no charting dependency** (the `gold-500 → gold-700` theme flip alone settles this), the grouped `StateList`, and the four live factual errors.
Corrected:
1. **`arena-runs.json` has no machine source and must not be created.** `probe.py` does not run models; there is no `traces.jsonl`. **The site already has the right contract: `apps/web/src/data/eval-results.ts` + `eval-results.json` + `eval-results.parse.test.ts`, with a `{model, host, quant, serving, checkpoint}` target tuple and an explicit "null means not measured" rule.** The Nemotron run belongs there. Creating a second incompatible file guarantees the drift the pipeline exists to prevent — and putting it in `eval-results.json` also fixes **a fifth live error the audit missed**: `arena.tsx:519` says "No model scores are published yet, so nothing appears on Evals" while `README.md:88-105` publishes seven.
2. **The crude-maximiser labels have no machine source either.** `probe.py`'s functions return only `{inaction, crude, plausible, oracle}`; "redact everything" / "ban everyone" / "buy and sell at market" exist only as markdown italics. Emitting that column means **changing `probe.py`'s return shape — which is B's file.** Fold it into B1's registry work or accept it as a hand-typed literal, but do not schedule it as a third-party edit to `probe.py`.
3. **`check:arena` must not execute `probe.py`.** The `sync-scenes.mjs` precedent parses committed YAML from an anonymous public clone; running a 772-line probe under `uv run --with regex` is a slow, flaky gate that contradicts the hermetic-build rule. Emit JSON in `arena`'s CI, commit it, and have `check:arena` compare bytes.
4. **`Diagram` fails DESIGN.md §4.2's own gate** — no variants, no CVA variants object. It is a one-off `<figure>` per page, exactly like `scaling-plot.tsx` / `spread-field.tsx` already are. Do not put it in `components/ui/`.
5. **Do not put a nine-verb FlowStrip above the fold on `/forge`.** Every StateList item on that page is `done: false`, including "No repository, no tag, no release." Drawing an unbuilt pipeline is the exact failure the audit indicts.
6. **The landing-page markup claim is wrong**, and it invalidates the landing plan's premise: `landing.tsx` contains **two** `<svg>` elements, not six. `ForgeChain` is an `<ol>`, `ArenaBand` is nested `<div>`s and 32 `<span>`s, `StreamSignal` and `SimulateViewport` are divs. The real coupling is ~33 class references into `machine-home.css` (35 KB) carrying reveal/scan animations. **Treat the landing as its own step with its own CSS audit.**
7. **`Bar` has one existing hand-rolled call site, not two** — `BudgetMeter` is an N-segment `flexGrow` track with a composed `aria-label`, and `BarProps` cannot express it. `Bar` earns its place through `MeasureTable`'s cells, not through BudgetMeter.
8. ~~**The scenario count is 16, not 20**~~ ⚠️ **WRONG as of 2026-08-21 15:50 — it is 20**, five registries × four, and `office-jobs-v1` is not phantom. Another session merged it into the shared `tera` tree mid-session. Counts are twelve / five / twenty; the site's original copy was right and W0's correction was reverted.
9. Four more table-shaped components exist that the "seven call sites" audit missed: `components/compute/model-table.tsx`, `components/publish/concurrency-table.tsx`, `ledger.tsx`, `rate-strip.tsx`. Design `MeasureTable`'s API against them.
10. No tests are proposed. New generated data + a check script are exactly what gets a parse test in this repo.
**Revised W:** `/arena` first (it is also a truth fix), `/bench` second, `/stream` third — but `/arena`'s number pipeline routes through `eval-results.json`, and the `office-jobs-v1` deletion happens **once**, in one place, coordinated with D and P.
### C-compute — rent to train, Spark to serve · feasible; **its headline correction is false**
Stands and is the most immediately actionable: the policy itself; the measured spark-1 tenancy (121 GB total / 84 GB used / 36 GB free, SGLang + Chatterbox + vox + ASR co-resident, Chatterbox's ~8.4 GB/hr leak having killed brain before); `prime login` as step zero because `api_key` is empty (verified); **Arena → prime-rl needs no adapter** — `EnvConfig` is `SerializeAsAny[vf.EnvConfig]` and the taskset tree is the same shape, prefixed; the seat-name-is-a-TOML-key trap; Hosted Training being unavailable while E is parked (`rl.py:1522-1524` — bare v1 ids do not resolve through the Hub); the sm_121 lock contradiction in Forge; `TRAINERS = ("axolotl","trl")`; **no price is hardcoded anywhere and none should be invented.**
Corrected, blocking:
1. **`--model brain` does NOT 404.** SGLang does not validate the model field on the chat route; it returns 200 with a real completion. Two of the four "stale facts" and two proposed file edits rest on a false claim. **The real hazard is the opposite and worse: it silently succeeds.** Re-running the `README.md:88-89` baseline today produces **Qwen3.8-27B** numbers under a command that reads `--model brain`, and they will be tabulated beside Nemotron numbers as comparable. The correction the arena docs need is: **pin the served id `brain-qwen38-dspark` and the scene name in every reproduction command**, not "fix a 404".
2. **`uv run eval grand-exchange-live --harness.id null` is not valid.** `EvalConfig` has no top-level `harness`; `--harness.` is admitted only to render a did-you-mean. Canonical: `--env.agent.harness.id null`, or the TOML block `configs/grand_exchange.toml` already uses.
3. **`uv sync --package prime-rl --package grand-exchange-live` cannot work.** prime-rl's `[tool.uv.workspace] members` is a closed list of its own vendored envs. Use `uv pip install` of a built wheel or git URL into the env-server venv, or add a path source.
4. **`forge.tsx:114` is not stale.** The remote is private, `git tag` is empty, `status="In development"`, no `source=` prop. Nobody can clone it. The page's own README warns "This section has been wrong three times" — this edit would make it four.
5. **Workstream-letter collision:** PLAN.md already defines a "workstream C" (more interactive environments). Rename this one — **"Compute Policy"**, no letter.
6. The wordle wall-clock anchors are fabricated (the README says 2 GPUs, the toml declares 8 and 200 steps; neither states a duration). Delete them and produce our own.
7. `torchao` and `nixl-cu12` are pinned to **x86_64-only** wheel URLs while `torchao` is a hard runtime dep — the "aarch64 wheels for every heavy dependency" claim does not hold, which sharpens the aarch64-variance risk.
8. `prime disks` attach with `--disks`, not `--attach-disk`.
9. **Two things nobody has planned:** how the DCP checkpoint gets *off* the pod before `prime pods terminate` destroys the disk (needs a `--disks` volume at create time), and the DCP → HF shards → NVFP4 → Compute model-id conversion, which nothing on disk automates end-to-end (prime-rl has `convert_to_hf` but no CLI entrypoint).
10. **The link nobody drew:** rental hours are driven by whether the turn budget binds. `group_size × batch × steps` of *multi-turn* rollouts against a rented inference fleet is real money. If B2's exhaustive policy scores at the oracle, the run burns GPU hours measuring nothing. **Rule 4 is a compute-policy constraint, not only a design one.**
---
## 2. Parallelization map and file collisions
### What can start today, concurrently
| Lane | Work | Repo | Blocked by |
|---|---|---|---|
| **1** | **B1** — discovery de-hardcode + timeout + step name + `uv.lock` cleanup | `arena` | nothing |
| **2** | **A step 2** — `live.py` engine + unit tests + **re-measure the ladder** | `arena` | nothing (own worktree) |
| **3** | **D step 1** — worker, vendored TS, hashes, `test_replay.py` | `arena` (new dir only) | nothing |
| **4** | **Compute Policy step 0** — `prime login`, catalogue + prices, doc the corrected spark-1 facts | none / `arena/docs` | nothing |
| **5** | **P1** — Caddy host, host-aware routing, AppShell, Overview/Worlds/Org/Inbox | `lumbridge-v4` | nothing |
| **6** | **W step 0** — the five factual corrections + `eval-results.json` decision | `lumbridge-v4` | nothing |
Six lanes, and only lanes 1/2/3 share a repo. **Use a worktree per lane** — this repo took divergent concurrent pushes on 2026-08-19. Merge, never force.
### True dependency edges
```
B1 ──────────────────────────────────────────┐
├──► A step 7 (probe row) ──► A ships
A steps 1-2 (engine) ──► A re-measure ──► A steps 3-6 ─┘
│
└──► B2 (policy machinery, calibrated on real numbers)
│
├──► C1 drop-table
├──► C2 bot-detection
├──► C3 fault-localisation
├──► C4 redaction (2-agent)
└──► D step 5 (tera probe rows)
D step 1 (replay gate) ──► D crow-nav ──► D flight/office
A ships ──► one real eval run ──► traces.jsonl exists ──► P2 (Arena dashboard)
│
└──► W /arena numbers (via eval-results.json)
prime login ──► rented-pod runbook ──► first RL run ──► Forge prime-rl renderer
```
Two edges that were wrong in the source specs and are now correct:
- **B1 does not depend on A** (PLAN.md's "B depends on A" applies only to B2).
- **D is no longer file-independent of A/B/W** — it lives in `arena` and edits `probe.py`, `ci.yml`, `test_probe.py`, `README.md`, plus `arena.tsx` and `seo.ts`.
### Merge-conflict map — the specific paths
**Tier 1 — will conflict, must be sequenced or de-hardcoded first:**
| Path | Claimed by | Resolution |
|---|---|---|
| `arena/probe.py` | A (§11), B1 (registry), B2 (machinery), D (§7), C1-C4, W (needs a return-shape change for crude labels) | **B1 lands first.** After B1, adding an env is one appended function at EOF — a mergeable case. Nothing else touches this file until B1 is merged. |
| `arena/tests/test_probe.py` | A, B1, D | B1 removes it from every future workstream's diff. Never edited again. |
| `arena/.gitea/workflows/ci.yml` | A, B1, D (needs Node + replay test + a one-dir-four-wheels change) | B1 lands the glob; D then adds *steps*, not list entries. **D's one-directory-four-tasksets layout actively breaks B1's one-dir-one-wheel discovery** — D must add a `[tool.arena] tasksets = [...]` key or equivalent, and B1 should be told to read it. |
| `arena/configs/grand_exchange_live.toml` | **A, B(§1.7.2), Compute Policy** — three specs create the same file | **A owns it.** Delete it from the other two file lists. |
| `arena/environments/grand_exchange_live/*/taskset.py` and `session.py` | **A and B both specify the module layout and class signatures** | **A owns the contract.** B2 adapts. |
| `arena/README.md` | A (rows), D (rows + house rule 4), Compute Policy (baseline qualifier), W (source of truth for the site tables) | Sequence: A → D → Compute Policy. W reads it, never writes it. |
| `arena/PLAN.md` + `MULTI_TURN.md` | B, D (scope amendment), Compute Policy | One session, one pass, after the D-scope decision lands. |
**Tier 2 — cross-repo, same file:**
| Path | Claimed by | Resolution |
|---|---|---|
| `lumbridge-v4/apps/web/src/pages/v4/arena.tsx` | **D** (delete `office-jobs-v1` at :203 and :511) and **W** (full rebuild of the page) | **W owns the file.** D contributes the *fact* (four manifests, 16 scenarios), not the edit. |
| `lumbridge-v4/services/api/src/seo.ts` | **D** (:271 twelve→eleven, five→four) and **P** (`renderAppShell`, drop `/app` from `APP_ROUTE_PATTERNS` at :698) | Different regions; sequence P after D's one-line copy fix, or vice versa. Low risk but both must run `seo.test.ts` (`ROUTER_PATHS` at :36) and `spa-fallback.test.ts`. |
| `lumbridge-v4/apps/web/src/pages/v4/forge.tsx` | **W** (FlowStrip, StateList) and **Compute Policy** (:59-62 delegation copy) | W owns it. Compute Policy contributes one sentence, **only after a rented run has actually happened**. The `:114` edit is cancelled (see verdict). |
| `lumbridge-v4/apps/web/src/data/eval-results.json` | **W** (Nemotron run) and **P2** (the dashboard's run store) | The public file is a curated summary; the dashboard is the archive. Decide the direction (Decision 6) before either writes. |
**Tier 3 — three tracked `uv.lock` files are failing CI on main right now.** `git rm --cached` them in B1's commit, before anyone blames a new failure on a new environment.
---
## 3. Sequencing
### Phase 1 — genuinely shippable, ~2 weeks, six parallel lanes
1. **B1** (arena): tomllib discovery, `@probes` registry, timeout 180, `uv.lock` untracking, step rename. *Gate: `probe.py` still prints seven `ok` rows; CI green with zero list edits.*
2. **A steps 1-2 + re-measurement** (arena, worktree): frozen generator copy + equality assertion; `LiveBook`/`Leg`/`Ledger` with unit tests; **then re-run the full policy ladder and publish the corrected numbers.** *Gate: the ladder reconciles with the reward formula; `horizon`, `FREEZE_RANGE`, `TARGET_SHARE`, `TURNS` are chosen from that measurement, not from the spec.*
3. **D step 1** (arena, new dir): worker + vendored TS + `hashes.py` + `sync_tera.py` + `test_replay.py`. *Gate: 32/32 replay exact, tampering rejected, hash check green — and a decision on the 4 unpinned files.*
4. **Compute Policy step 0**: `prime login`; record real `availability` output and prices; write `arena/docs/COMPUTE_POLICY.md` with the corrected spark-1 facts (`brain-qwen38-dspark`, silent-wrong-model hazard, 36 GB headroom).
5. **P1** (lumbridge-v4): Caddy `app.` block with `bind 10.0.0.2` and the non-empty-body check; host-aware `App.tsx`; `renderAppShell` + `sendShell` signature change + `seo.test.ts` + `spa-fallback.test.ts`; AppShell; Overview / Worlds / Org / Inbox; delete the 23 dead `api.ts` methods.
6. **W step 0** (lumbridge-v4): the five factual corrections — `arena.tsx:330` "Nine tasks", `landing.tsx` card 02, the 4-row probe table → 7 rows, `office-jobs-v1` deleted, `arena.tsx:519` "no model scores published". Ship the Nemotron run into `eval-results.json`.
**Phase 1 ends with**: a green de-hardcoded CI, a measured live engine, a proven TS replay bridge, a live private console on real endpoints, a truthful public site, and a Prime Intellect account that can rent.
### Phase 2
7. **A steps 3-9** → `grand-exchange-live` ships with its probe row (needs B1 merged).
8. **One real eval run** against spark-1, pinned to `brain-qwen38-dspark`, with the empty-rewards count checked. *This is a hard prerequisite for P2 and for any site number.*
9. **D crow-nav** end to end (role named `agent`, `vf.Taskset[...]` generic form, `oracle` op in the worker, `gate` as a multiplier).
10. **W /arena rebuild** + `/bench` on the new primitives.
11. **First rented run**: pod create with an attached disk, `install.sh`, wheel install, baseline eval, `uv run rl`, re-eval, before/after table.
### Phase 3
12. **B2** — policy machinery, calibrated against A's measured exhaustive/oracle gap, with block-wise ladders replacing the single 0.02 margin.
13. **P2** — Arena schema (per-trace PK), server-side aggregates, admin-only transcripts, ingester for both output layouts, the four pages.
14. **C1-C4** — the four remaining interactive environments, fanning out from B2.
15. **D flight + office-nav**; drive-101 per Decision 3.
16. **Forge**: `prime-rl` renderer (LoRA RL first), x86_64 train lock, retitled aarch64 lock, DECISIONS entry, checkpoint→HF→NVFP4→Compute-model-id stage.
17. **W /stream**, landing (with its own `machine-home.css` audit), compute/simulate/about.
18. **E** — Hub push, when un-parked.
---
## 4. Decisions Karti still owes, ranked by what they block
| # | Decision | Blocks | Recommendation |
|---|---|---|---|
| **1** | **The agent seat name.** `agent` (matching `SingleAgentEnvConfig`) or a custom name? It becomes the TOML key in `configs/*.toml` **and** in every prime-rl `rl.toml`, on both A and D. Renaming later parses fine and silently has no seat. | A, D, every future env, every training config | **`agent`, everywhere.** One word, decided once, written into each env's README. |
| **2** | **Does D live in `arena` or `tera`?** The spec's packaging argument is strong (`tera` has no Python toolchain) but it overturns PLAN.md:138 and makes D collide with A/B/W. | D's entire file layout, the CI wheel loop, PLAN.md | **`arena`**, and amend PLAN.md's parallelization map in the same commit. |
| **3** | **Does `tera-drive-101` ship?** Measured: monotone in throttle, optimal policy is one constant string, and the 0.9 band makes any positive throttle score ~0.95+. Also decides whether the site says four spatial environments or three. | D's scope, `arena.tsx`/`seo.ts` copy, the Hub listing | **Ship it labelled a bridge-correctness environment with no rule-4 and no rule-3 claim, or defer.** Do not list it as a fifth RL environment. |
| **4** | **A's freeze mechanic: flat per-look, or proportional to items re-quoted?** Flat is measured and works; per-item is the better mechanism, makes the look count genuinely vary, and forces a much larger surgical reference. | A's engine, and therefore B2's calibration | **Ship flat.** Record per-item as the identified strengthening. Do not build an unmeasured mechanism on the critical path. |
| **5** | **Arena transcript visibility.** Full transcripts contain the held-out slice and, for oracle runs, the answers. | P2's route scoping, and house rule 1's integrity | **Admin-only.** This is not an org-politics question; it is a contamination question, and that settles it. |
| **6** | **Where does model-run data live?** Public `eval-results.json` (already typed, tested, rendered on `/evals`) vs P2's `arena_runs`/`arena_episodes` tables. | W's whole data pipeline, P2's schema | **Both, with a direction:** the dashboard is the archive, `eval-results.json` is a curated public summary emitted from it. Never two independent hand-maintained files. |
| **7** | **Is `nemotron` coming back as the production brain, or is `qwen38` the default?** `README.md`'s baseline, D23/D24's Forge preflight, and the site's Arena numbers all assume Nemotron; the box serves Qwen3.8-27B. | Every reproduction of the baseline; what "inference in production on the Spark" names | Needs an answer before anyone re-runs a benchmark. |
| **8** | **First Forge/prime-rl integration: LoRA RL or full-FT?** LoRA reuses merge + attest unchanged and matches `alphabet-sort`; but LoRA rejects `--gpu-type` and falls back to filesystem weight broadcast. Full-FT needs a new node kind and the tensor-delta-map gate never fires. | Forge's roadmap, and what its manifest is for | **LoRA first** — it keeps Forge's headline claim intact. |
| **9** | **Is app.\* the only authenticated origin?** Yes means the apex loses `/login`, `/signup`, `/app`, `/go`, and all three existing sessions are signed out on cutover day. | P1's scope, apex route deletions | **Yes**, and state the sign-out as an accepted cost for three accounts. |
| **10** | **W's audience.** Arena/Bench/Stream (brokenness × argument) vs Arena/Landing/Bench (the Prime Intellect Hub reader). | W's priority order only | Answer when P1 lands; it does not block anything upstream. |
---
## 5. Risk register, highest first
| # | Risk | Retired by |
|---|---|---|
| **1** | **The whole reward design of `grand-exchange-live` rests on numbers from a throwaway engine that no longer exists, and the ladder does not reconcile with the reward formula.** `horizon=60`, `FREEZE_RANGE=(1,4)`, `TARGET_SHARE=0.85` and `TURNS=8` are all locked against arithmetic that is off by 0.011 in the one place it can be checked. | Building `live.py` FIRST and re-measuring every row before any constant is frozen. This is Phase 1, lane 2, and it is the single most important task in the programme. |
| **2** | **Every reward in Arena saturates, so "exhaustive < oracle" may be structurally unreachable.** `profit_ratio` and `efficiency` clip at 1.0; `clean` is a boolean band. A 0.02 margin against a 0.25 quantum is a cliff. If the gate can't open, rule 4 is unenforceable and every interactive environment is unfalsifiable. | A's re-measurement producing a real exhaustive/oracle gap; B2 then calibrated against it with block-wise ladders. If no gap exists, the reward shape must change before C1-C4 are built on it. |
| **3** | **No `traces.jsonl` exists anywhere.** Everything downstream — the dashboard, the site's numbers, the leaderboard, the before/after training table — is waiting on one eval that has never completed. The one run that was attempted pointed at Prime's inference API with an empty key. | One completed eval against `brain-qwen38-dspark` with the empty-rewards count checked. Cheap, and it unblocks P2, W and Compute Policy simultaneously. |
| **4** | **`eval` exits 0 when every rollout errors, and multi-turn has far more ways to error than one-shot.** `turn()` on a closed interaction raises; `turn(msg)` on turn 1 of a prompted task raises. Combined with risk 3, a "successful" run can be entirely empty. | A CI/runbook step that asserts non-empty `rewards` on ≥N episodes, in `arena`, in D's replay gate, and in P2's ingester. Never trust the exit code. |
| **5** | **Silent wrong-model comparison.** `--model brain` returns 200 from a box serving Qwen3.8-27B. Re-running the Nemotron baseline today produces different-model numbers under an identical command, and they will be tabulated as comparable. | Pinning `brain-qwen38-dspark` (and the scene) in every command in `README.md`, `PLAN.md`, `MULTI_TURN.md` and `COMPUTE_POLICY.md`. Trivial; do it in Phase 1. |
| **6** | **`probe.py`, `tests/test_probe.py` and `ci.yml` are edited by five of six workstreams, in a repo that already took divergent concurrent pushes.** | B1, first, before anything else touches `arena`. |
| **7** | **The reward denominator is editable with every `tera` gate green.** ⚠️ CORRECTED 2026-08-21: not `offices/sites.ts` (mutating it produces a bit-identical episode — refuted twice). The hole is **`src/arena/index.ts`**, which re-exports the scripted baselines that ARE the denominator. Demonstrated: a halved office-nav baseline changes return 4.744 → 4.098 while `arena:source-hashes` prints ok and exits 0. | Pinning `src/arena/index.ts` in `tera/scripts/check-arena-source-hashes.mjs` and regenerating `sourceHashes.ts`. One line, in a repo this session did not own. |
| **8** | **spark-1 has 36 GB of 121 GB free with four resident model services, one of which leaks ~8.4 GB/hour and has killed the brain before.** "Production inference on the Spark" is a tenancy claim held together by the governor on :8011. | A mandatory headroom check before any new resident model, and Compute's watchdog verified live. |
| **9** | **`PRIME_API_KEY` is empty**, blocking not just the parked Hub push but every rental command, every price, and the entire training half of the compute policy. | `prime login`. One command. |
| **10** | **Two of D's four environments cannot discriminate under the 0.9 ceiling band** — drive-101 gives ~0.95+ to any constant throttle, office-nav collapses to `efficiency` alone. | Re-deriving the band per environment against the measured return distribution, or shipping fewer environments. |
| **11** | **The held-out corpus leaks through P2's trace viewer at `user` scope.** One invite code. | Decision 5 — admin-only. |
| **12** | **Effort is understated across the board.** A is L/XL; B2 is not B's to size (its §1.8 is L/XL and belongs to A); D is L/XL; P's "Phase 1" is XL; W's item 1 alone is L/XL across two repos. Read as written, this programme looks like six L's and is closer to four XLs. | The phasing in §3, which cuts P1 out of P, B1 out of B, and D-step-1 out of D. |
| **13** | **Rented GPU hours are burned by a non-binding turn budget.** `group_size × batch × steps` of multi-turn rollouts against a rented inference fleet is real money, and nothing in the compute doc connects to rule 4. | Making "exhaustive < oracle by a measured margin" a precondition of dispatching any RL run — write it into `COMPUTE_POLICY.md`. |
| **14** | **The checkpoint has no exit path.** `prime pods terminate` destroys ephemeral disk; the DCP → HF → NVFP4 → Compute-model-id conversion is unautomated. | `--disks` at pod-create time, plus a `forge quantize/publish` stage. Discover this before the first successful training run, not after. |
| **15** | **GH200 may not appear in `prime availability` at all** — the CLI names only H100_80GB, H200_141GB, B200_180GB, RTX_PRO_6000. "GH200-class" may mean H200_141GB in practice. And `perf.py` has no GH200 branch, so MFU may silently fall back to A100's 312 TFLOPS. | `prime login`, then one `availability gpu-types`, then one `nvidia-smi --query-gpu=name` on a real pod. |
---
## 6. Build this first
**`environments/grand_exchange_live/grand_exchange_live/live.py` — the stepped engine and its unit tests — and, concurrently, workstream B1's discovery de-hardcoding.**
Everything else in the programme is downstream of two facts we do not yet have: whether the live market's reward ladder actually reconciles (risk 1, which invalidates A's horizon, freeze, TARGET_SHARE and turn budget if it does not, and takes B2, C1-C4 and half the compute argument with it), and whether three files can stop being a five-way merge conflict before five workstreams reach for them. B1 is an afternoon and removes the repo's only structural collision permanently; `live.py` is the smallest artifact that can turn six specs' worth of `/tmp` measurements into something the shipped code agrees with.