docs: the scope, the plan, the post-review execution plan, and the eval runbook
MULTI_TURN.md scopes the multi-turn gap and records the finding that made it cheap: multi-turn needs no container and no real harness. vf.Env.run programs the control flow host-side and the simulator runs in-process as the user, with the harness left null. PLAN.md carries the settled decisions. EXECUTION.md is the post-adversarial-review plan — every position in it survived a reviewer, and where a design spec was overturned the corrected position is what is written down. Two corrections are marked in place rather than silently rewritten, because both were believed and acted on: - The provenance hole is src/arena/index.ts, not offices/sites.ts. Mutating offices/sites.ts produces a bit-identical episode; index.ts re-exports the scripted baselines that ARE the reward denominator, and a halved office-nav baseline moves return 4.744 -> 4.098 while every tera hash gate prints ok. - office-jobs-v1 is not phantom. It was absent at 13:00 and present at 15:50, because another session fast-forward-merged it into the shared tera tree mid-session. The counts are twelve, five and twenty. docs/FIRST_EVAL.md is the runbook for the first evaluation Arena ever completed: seven environments, 224 episodes, 223 scored, against brain-qwen38-dspark. It warns that the mean is sum(score x weight) — an unweighted mean over components is a different number, and it is how six of seven means were first misreported. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
+281
@@ -0,0 +1,281 @@
|
||||
# Execution plan — Arena / Lumbridge
|
||||
|
||||
Generated 2026-08-21 by a 13-agent design + adversarial-verification workflow.
|
||||
Companion to `PLAN.md` (decisions) and `MULTI_TURN.md` (scope).
|
||||
|
||||
⚠️ Every position here is the POST-REVIEW position. Where an adversarial reviewer
|
||||
overturned a design spec, the spec's original claim was discarded. Do not resurrect
|
||||
a claim from an earlier document that this one corrects.
|
||||
|
||||
# Arena / Lumbridge — one execution plan
|
||||
|
||||
Synthesis of six adversarially-reviewed workstream specs, 2026-08-21. Every position below is the **post-review** position. Where a reviewer overturned a spec, the spec's original claim is marked and discarded.
|
||||
|
||||
Verified independently for this document (amd-server, 2026-08-21): `arena/probe.py` is 772 lines with a literal 7-name shim tuple at :89-93 and a literal `results` dict at :752-760; `tests/test_probe.py` has a 7-name `ENVIRONMENTS` tuple at :13-21 and `timeout=30` at :31; `.gitea/workflows/ci.yml` is 42 lines, its probe step is still named "Run four-environment floor and ceiling probe" (:26) for a seven-environment probe, its wheel loop (:36-40) is a hardcoded bash `for` over seven names, and its lock-policy check (:24) is failing on main because **three** `uv.lock` files are tracked (`bot_detection`, `drop_table_inference`, `grand_exchange`); `outputs/` contains exactly one file, a `config.toml` with `push = true` and `base_url = https://api.pinference.ai/api/v1` — **no `traces.jsonl` exists anywhere on this box**; `~/.prime/config.json` has an empty `api_key`; `tera/src/arena/index.ts:74-79` `ARENA_MANIFESTS` holds four.
|
||||
|
||||
---
|
||||
|
||||
## 1. Verdicts applied — what each spec becomes
|
||||
|
||||
### A — `grand-exchange-live` · feasible, but **not in the order written**
|
||||
|
||||
Stands: the multi-turn mechanism (`Env.run` + `interaction.turn()`, null harness, no container), the delta-sheet action grammar, the malformed-reply policy, `roc` on **peak** capital employed (average-employed is a measured free-points bug), the vendored-generator equality assertion, no turn-count reward term.
|
||||
|
||||
Corrected, blocking:
|
||||
|
||||
1. **The measured ladder does not reconcile with the spec's own reward.** At ref 72,758 gp and `TARGET_SHARE = 0.85`, target = 61,844; impatient at 50,470 gp gives `profit_ratio` 0.816, `discipline` ≤ 0.816, `clean` false → max 0.612, not the reported 0.623. Churn and exhaustive *do* reconcile, so this is a specific defect. **Nothing in §1/§4 is locked until it is re-measured with the shipped engine.**
|
||||
2. **`viable_market` retargeted at `live_reference` is circular as written.** Viability needs `freeze`; `freeze` is drawn in `Taskset.load()` after viability returns. Fix: derive `freeze` deterministically from `market.seed` (e.g. `random.Random(seed ^ FREEZE_SALT)`), inside `build_market`, so viability and scoring see the same value. This is also the fix for the unstated RNG source — without it, `scan()` recomputes a reference the model never played against.
|
||||
3. **§6's held-out claim is inverted.** Live is *weaker* than one-shot: the model sees ~50 of 60 graded ticks. Rule 1 still passes; the README must say "held out: the tail after the last observation, ≥4 ticks", not "stronger by construction". Add a probe assertion on the *size* of the held-out slice, not only the frontier.
|
||||
4. **The probe battery blows CI's own harness.** `probe.py` runs in 3.4 s today under `timeout=30`. §11 adds 9 policies × blocks × 240 baskets + two sweeps + a 240-seed generator check + a hostile battery replayed per turn. Raising the timeout is B1's job (below), but A must budget the cost, not append a name to a list.
|
||||
5. **Taskset load cost is unbounded.** `num_tasks` defaults 48, `VIABILITY_TRIES = 64`; a live `viable_market` runs a full 8-turn stepped reference per candidate → ~3,072 simulations before the first token, on every eval and every `--dry-run`. Cache the reference per seed, or cap tries for live.
|
||||
6. **`import` vs `copy` contradiction.** §3 says "import `grand_exchange.book.parse_orders`", §9 says vendor. It is a copy. Say copy.
|
||||
7. **`[env.agent] max_turns = 8` is an off-by-one risk, not free insurance.** Verify how the framework counts, or set 9.
|
||||
8. **`trace.info["episode"]` must be plain JSON types.** A dataclass or numpy scalar in `audit()` fails the trace write for the whole rollout.
|
||||
9. **Effort is L/XL, not L** — it is A plus most of B.
|
||||
|
||||
**Revised A build order (this is the change that matters):** `market.py`/`book.py` copies + equality assertion → **`live.py` + unit tests → re-measure the full ladder in the shipped engine** → only then lock `horizon`, `FREEZE_RANGE`, `TARGET_SHARE`, `TURNS` → `strategy.py` sweep → `ledger.py` → `protocol.py` → `taskset.py` → probe.
|
||||
|
||||
### B — probe policies + de-hardcoding · **verdict: not feasible as written. Split it.**
|
||||
|
||||
The reviewer is right and the split is the fix.
|
||||
|
||||
**B1 — de-hardcode environment discovery. Sound, S/M, ships immediately, behaviour-preserving.** `tomllib` over `environments/*/pyproject.toml` replaces the shim tuple, the `main()` results dict (via a `@probes(name)` registry decorator), `ENVIRONMENTS` in `tests/test_probe.py`, and the bash wheel loop. **The CI environment list was never a matrix** — it is a `for` loop in one step (verified) — so no generator job, no `fromJSON` matrix, no act_runner risk. Take B1 now and the repo's known collision point disappears for one afternoon's work.
|
||||
|
||||
Two mandatory amendments to B1:
|
||||
- **Do not make "discovered but unregistered" a hard exit 1 while A is in flight.** The moment A commits a scaffold `pyproject.toml`, `probe.py` exits 1 and CI goes red for every session on the repo. Make it a warning until an env registers a probe, and flip it to exit 1 in the same commit as A's `exchange_live()`.
|
||||
- **Make the two guards use the same denominator.** `ci.yml`'s "empty scan" counts directories; `discover()` counts `*/pyproject.toml`. A scaffold dir without a pyproject makes them disagree. Count `*/pyproject.toml` in both.
|
||||
- Raise `tests/test_probe.py` `timeout=30 → 180`, keep the lock check *before* any `uv sync`, and fix the "four-environment" step name.
|
||||
|
||||
**B2 — the `Interactive` policy machinery. Redesign required; do not build it before A's engine exists.** The structural claim ("oracle and exhaustive share `answer`, so rule 4 follows") is **false when the reward saturates**, and Arena's rewards all saturate: `profit_ratio` and `efficiency` clip at 1.0 and `clean` is a boolean band. If exhaustive also clears the band it scores 1.000 and the gate reads "budget is free". Further:
|
||||
- `BIND_MARGIN = 0.02` is staked on realised-P&L differences over 24 seeds, against a block-to-block variance `probe.py:349-352` already documents at 0.48 vs 0.18. The five-block ladder is **required for all three rule-4 comparisons**, not an optional mitigation — multiply the cost estimate by ~5.
|
||||
- `gate` is binary at weight 0.25, so the exhaustive/oracle gap is quantized in 0.25 steps. A 0.02 margin against a 0.25 quantum is a cliff, not a calibration.
|
||||
- `play()` never terminates on an answer; "impatient answers turn 1" is an unstated precondition on A's `step()`.
|
||||
- Seeds must come from A's `baskets()` skip-aware helper, not a contiguous `range`.
|
||||
- B2's §1.8 dictates A's private module layout, class names, weight vector and turn mechanism. **Delete that.** A owns its contract; B2 adapts to it. This is the collision the workstream exists to remove.
|
||||
|
||||
**Net: B1 is unblocked and independent of A. B2 depends on A's *measured* engine, not A's first draft.** PLAN.md's "B depends on A" edge survives for B2 and dies for B1.
|
||||
|
||||
### D — Tera bridge · feasible, but **it is no longer parallel with A/B, and PLAN.md must be amended**
|
||||
|
||||
Stands and is the strongest evidence in the six specs: 23-file / 283 KB / zero-bare-specifier TS closure; `node` type-strips raw `.ts`; **32/32 scenario × seed traces round-trip through Python `json` and replay to identical checksums**; one long-lived NDJSON worker (84 ms start, 22 µs RTT, 217 µs/step, 87 MB RSS) beats HTTP and beats per-rollout processes; the measured hold-length sweep.
|
||||
|
||||
Corrected, blocking:
|
||||
|
||||
1. **Role name vs config key — the spec will not load.** `agents.pilot` requires a config field named `pilot`, but all four TOMLs write `[env.agent.harness]`, and `EnvConfig` is `extra="forbid"`. **Name the role `agent`**, matching `SingleAgentEnvConfig`. Same trap A must resolve; settle it once for both (see Decision 1).
|
||||
2. **`CONFIG = ...` classvar does not exist in verifiers v1.** Config is resolved by `concrete_type()` through `__orig_bases__`. Use the wordle form: `class CrowNavTaskset(TeraSpatialTaskset, vf.Taskset[TeraSpatialTask, CrowNavConfig])`. As written, every per-env `max_turns`/`max_hold` silently does nothing.
|
||||
3. **The provenance claim is real but was mis-located — CORRECTED 2026-08-21.** `check-arena-source-hashes.mjs` pins 19 of 23 vendored files. The claim that `offices/sites.ts` is the material hole is **FALSE and was refuted twice independently**: mutating it (elevation 4→4000, heading 0→90, width 54→5400) produces a **bit-identical** episode, so it does not reach the simulation. ⚠️ **The real hole is `src/arena/index.ts` alone**, and it is worse than the original claim: rewriting it so `officeNavScriptedBaseline` re-exports a halved policy moves office-nav from 68 steps / return 4.744 to 136 steps / 4.098 — **while `npm run arena:source-hashes` prints "ok" for all four environments and exits 0**, `envHash` and `simulatorHash` unchanged. That file is the reward DENOMINATOR. Fix: pin `src/arena/index.ts` in `tera/scripts/check-arena-source-hashes.mjs` (a one-line `tera`-repo change) and regenerate the four pairs in `sourceHashes.ts`.
|
||||
4. **`oracle.py` reimplementing the four TS baselines in Python contradicts the bridge's own rule** ("never recompute a number that came out of TypeScript") — and it does so for the reward *denominator*. Add an `oracle` op to `worker.mjs`; the four baselines are already exported from `src/arena/index.ts`.
|
||||
5. **`drive-101` measures nothing, and the 0.9 ceiling band makes it worse than "no rule-4 claim".** Return is monotone in throttle, 12/12 goals at every value; `0.9 × 4.339 = 3.905` and throttle 0.2 already returns 3.723 → quality 0.953. Any positive constant throttle scores ~0.95–1.00. Same collapse on office-nav: any goal-reaching run clears the band, so `control` (0.50) and `gate` (0.20) are constants and only `efficiency` carries signal.
|
||||
6. **office-nav's rule-4 argument is backwards under its own formula.** `efficiency = oracle_turns / max(turns, oracle_turns)` only ever *penalises* extra turns, and past `ceil(1.5 × 1.8) = 3` turns `clean` also drops. The formula punishes exactly the "precision buys turns" behaviour cited as evidence.
|
||||
7. **`gate` is a separate additive term for goal-attainment that is already inside `ts_return` as the success bonus** — double-counted, additively. That is the failure `probe.py:15-17` records verbatim. Make it a multiplier on the sum.
|
||||
8. **Missing:** no `@vf.stop`; no handling of a mid-loop `terminated` Segment (the loop as written hits `RuntimeError('the exchange is over')`); no `Taskset.load()` / prompt-vs-bare-`turn()` decision; an empty `{"plan":[]}` costs zero steps while a malformed reply costs 10, inverting the spec's own principle; no guard for `oracle_return <= 0`; `max_hold = 400` exceeds `MAX_STEPS = 360`.
|
||||
9. **Scope.** PLAN.md:138 and the parallelization map say D lives in `tera` and shares no files with A/B/C. This spec puts everything in `arena` and edits `probe.py`, `tests/test_probe.py`, `ci.yml`, `README.md`, `configs/`, plus two `lumbridge-v4` files. That is a real decision (the packaging argument is correct — `tera` has no Python toolchain), but **PLAN.md must be amended to record it**, because it converts D from "parallel with everything" into "collides with A, B and W".
|
||||
|
||||
**Revised D:** ship the bridge + replay gate + `tera_crow_nav` first. `drive-101` ships only as a labelled bridge-correctness environment or not at all (Decision 3). `office-nav` and `california-flight` follow. D's probe integration waits for B2.
|
||||
|
||||
### P — `app.lumbridgecorp.com` · feasible; **Phase 1 as written cannot ship**
|
||||
|
||||
Stands and is high-value: the endpoint inventory (29 routes, not 31), the 23 dead `api.ts` methods, one-build/two-hosts with app.* as the **only** auth origin, `bind 10.0.0.2` + the empty-200 verification, the `isAppHost` branch in `sendShell`, `WORLD_LAUNCHER_URL` convergence for `/go/:slug`, `World1` moved not deleted.
|
||||
|
||||
Corrected, blocking:
|
||||
|
||||
1. **"Ingest the runs already on amd-server" is impossible — there are none.** One `config.toml`, zero episodes, and that config points at `api.pinference.ai` with `PRIME_API_KEY`, which is empty. **Phase 1 must contain "execute one eval to completion against spark-1" as an explicit prerequisite**, or Arena's dashboard ships empty.
|
||||
2. **The leaderboard is operator-gameable and mis-specified.** (a) The server must recompute every aggregate in SQL from the `arena_episodes` it just inserted, never trust the pusher's numbers. (b) Ranking on a mean over *scored* episodes is Arena's own "score what survived" trap. Store `reward_mean_scored` **and** `reward_mean_attempted` (errors as 0), rank on attempted, refuse to rank below a scored-fraction floor.
|
||||
3. **`arena_episodes` flattens a list.** `Episode.traces` is a list; `reward`/`rewards`/`stop_condition`/`last_reply` are per-**Trace**. Primary key must be `(run_id, episode_id, trace_id)` with an `agent_name` column — otherwise the schema is wrong for `grand-exchange-live` (simulator as second role) and for `redaction-pressure` (genuine two-agent).
|
||||
4. **The trace viewer publishes the held-out slice.** `GET /api/arena/runs/:id/episodes/:eid` at `user` scope returns full transcripts, which contain held-out task data and, for oracle runs, the answers. One leaked invite publishes the held-out corpus. **Transcripts are admin-only or redacted.** This settles the spec's open question on read scope.
|
||||
5. **The compute-scene-vocabulary blocker is fabricated.** `computeScene` is a free-form slug regex, not `z.enum(SCENE_NAMES)`; `stock`, `nemotron` and `qwen38` all pass today. Delete that precondition.
|
||||
6. **`portal_host_samples` and `portal_container_samples` have no node identity** — no `node_id`, no host column, no PK. Per-node charts need `ALTER TABLE … ADD COLUMN IF NOT EXISTS node_id text` plus an index first.
|
||||
7. **`scored = ok AND rewards non-empty` is wrong.** `write_episode` dumps with `exclude_none=True`, so a `None` reward is dropped from the file entirely — "not measured" and "key absent" are indistinguishable on the wire. Require at least one non-null `Reward`, and carry the run's declared reward-name set from the config.
|
||||
8. **`Reward.value` is a `@property`** and never appears in dumped JSON. Compute `score × weight` in the ingester.
|
||||
9. **0.3.0 → 0.4.0 changes the directory layout, not just the config filename** — HEAD produces one flat `<env>--<model>--<harness>--<hex8>` dir with no nested uuid, and `RunConfig.id` is a `PrivateAttr` absent from `configs/eval.json`. Derive the run id from the directory basename.
|
||||
10. **Name the private console `/nodes`**, not `/compute` — `/compute` is already a public marketing section with its own nav.
|
||||
11. Put `isAppHost` in `packages/shared` — two independent string checks that must agree, with a silent asymmetric failure mode (dashboard rendered under a 404 status).
|
||||
12. `sendShell(url, reply)` takes no request; the signature and both call sites change. `spa-fallback.test.ts` needs a case. `@radix-ui/react-select` is **not** installed.
|
||||
|
||||
**Revised P phasing.** The honest first ship is **P1: Caddy host + host-aware routing + `renderAppShell` + AppShell + Overview + Worlds + Org + Inbox + the `api.ts` trisection.** All of that runs on endpoints that exist today. **Arena (schema, 6 routes, ingester, 4 pages) is P2, a full workstream of its own**, and it is gated on a real eval having run.
|
||||
|
||||
### W — website visual vocabulary · feasible; **the data pipeline must be redirected, not built**
|
||||
|
||||
Stands: the diagnosis (the site draws what it hasn't measured and narrates what it has), `MeasureTable` / `Bar` / `FlowStrip` / `SpecSheet`, inline SVG with `currentColor` and **no charting dependency** (the `gold-500 → gold-700` theme flip alone settles this), the grouped `StateList`, and the four live factual errors.
|
||||
|
||||
Corrected:
|
||||
|
||||
1. **`arena-runs.json` has no machine source and must not be created.** `probe.py` does not run models; there is no `traces.jsonl`. **The site already has the right contract: `apps/web/src/data/eval-results.ts` + `eval-results.json` + `eval-results.parse.test.ts`, with a `{model, host, quant, serving, checkpoint}` target tuple and an explicit "null means not measured" rule.** The Nemotron run belongs there. Creating a second incompatible file guarantees the drift the pipeline exists to prevent — and putting it in `eval-results.json` also fixes **a fifth live error the audit missed**: `arena.tsx:519` says "No model scores are published yet, so nothing appears on Evals" while `README.md:88-105` publishes seven.
|
||||
2. **The crude-maximiser labels have no machine source either.** `probe.py`'s functions return only `{inaction, crude, plausible, oracle}`; "redact everything" / "ban everyone" / "buy and sell at market" exist only as markdown italics. Emitting that column means **changing `probe.py`'s return shape — which is B's file.** Fold it into B1's registry work or accept it as a hand-typed literal, but do not schedule it as a third-party edit to `probe.py`.
|
||||
3. **`check:arena` must not execute `probe.py`.** The `sync-scenes.mjs` precedent parses committed YAML from an anonymous public clone; running a 772-line probe under `uv run --with regex` is a slow, flaky gate that contradicts the hermetic-build rule. Emit JSON in `arena`'s CI, commit it, and have `check:arena` compare bytes.
|
||||
4. **`Diagram` fails DESIGN.md §4.2's own gate** — no variants, no CVA variants object. It is a one-off `<figure>` per page, exactly like `scaling-plot.tsx` / `spread-field.tsx` already are. Do not put it in `components/ui/`.
|
||||
5. **Do not put a nine-verb FlowStrip above the fold on `/forge`.** Every StateList item on that page is `done: false`, including "No repository, no tag, no release." Drawing an unbuilt pipeline is the exact failure the audit indicts.
|
||||
6. **The landing-page markup claim is wrong**, and it invalidates the landing plan's premise: `landing.tsx` contains **two** `<svg>` elements, not six. `ForgeChain` is an `<ol>`, `ArenaBand` is nested `<div>`s and 32 `<span>`s, `StreamSignal` and `SimulateViewport` are divs. The real coupling is ~33 class references into `machine-home.css` (35 KB) carrying reveal/scan animations. **Treat the landing as its own step with its own CSS audit.**
|
||||
7. **`Bar` has one existing hand-rolled call site, not two** — `BudgetMeter` is an N-segment `flexGrow` track with a composed `aria-label`, and `BarProps` cannot express it. `Bar` earns its place through `MeasureTable`'s cells, not through BudgetMeter.
|
||||
8. ~~**The scenario count is 16, not 20**~~ ⚠️ **WRONG as of 2026-08-21 15:50 — it is 20**, five registries × four, and `office-jobs-v1` is not phantom. Another session merged it into the shared `tera` tree mid-session. Counts are twelve / five / twenty; the site's original copy was right and W0's correction was reverted.
|
||||
9. Four more table-shaped components exist that the "seven call sites" audit missed: `components/compute/model-table.tsx`, `components/publish/concurrency-table.tsx`, `ledger.tsx`, `rate-strip.tsx`. Design `MeasureTable`'s API against them.
|
||||
10. No tests are proposed. New generated data + a check script are exactly what gets a parse test in this repo.
|
||||
|
||||
**Revised W:** `/arena` first (it is also a truth fix), `/bench` second, `/stream` third — but `/arena`'s number pipeline routes through `eval-results.json`, and the `office-jobs-v1` deletion happens **once**, in one place, coordinated with D and P.
|
||||
|
||||
### C-compute — rent to train, Spark to serve · feasible; **its headline correction is false**
|
||||
|
||||
Stands and is the most immediately actionable: the policy itself; the measured spark-1 tenancy (121 GB total / 84 GB used / 36 GB free, SGLang + Chatterbox + vox + ASR co-resident, Chatterbox's ~8.4 GB/hr leak having killed brain before); `prime login` as step zero because `api_key` is empty (verified); **Arena → prime-rl needs no adapter** — `EnvConfig` is `SerializeAsAny[vf.EnvConfig]` and the taskset tree is the same shape, prefixed; the seat-name-is-a-TOML-key trap; Hosted Training being unavailable while E is parked (`rl.py:1522-1524` — bare v1 ids do not resolve through the Hub); the sm_121 lock contradiction in Forge; `TRAINERS = ("axolotl","trl")`; **no price is hardcoded anywhere and none should be invented.**
|
||||
|
||||
Corrected, blocking:
|
||||
|
||||
1. **`--model brain` does NOT 404.** SGLang does not validate the model field on the chat route; it returns 200 with a real completion. Two of the four "stale facts" and two proposed file edits rest on a false claim. **The real hazard is the opposite and worse: it silently succeeds.** Re-running the `README.md:88-89` baseline today produces **Qwen3.8-27B** numbers under a command that reads `--model brain`, and they will be tabulated beside Nemotron numbers as comparable. The correction the arena docs need is: **pin the served id `brain-qwen38-dspark` and the scene name in every reproduction command**, not "fix a 404".
|
||||
2. **`uv run eval grand-exchange-live --harness.id null` is not valid.** `EvalConfig` has no top-level `harness`; `--harness.` is admitted only to render a did-you-mean. Canonical: `--env.agent.harness.id null`, or the TOML block `configs/grand_exchange.toml` already uses.
|
||||
3. **`uv sync --package prime-rl --package grand-exchange-live` cannot work.** prime-rl's `[tool.uv.workspace] members` is a closed list of its own vendored envs. Use `uv pip install` of a built wheel or git URL into the env-server venv, or add a path source.
|
||||
4. **`forge.tsx:114` is not stale.** The remote is private, `git tag` is empty, `status="In development"`, no `source=` prop. Nobody can clone it. The page's own README warns "This section has been wrong three times" — this edit would make it four.
|
||||
5. **Workstream-letter collision:** PLAN.md already defines a "workstream C" (more interactive environments). Rename this one — **"Compute Policy"**, no letter.
|
||||
6. The wordle wall-clock anchors are fabricated (the README says 2 GPUs, the toml declares 8 and 200 steps; neither states a duration). Delete them and produce our own.
|
||||
7. `torchao` and `nixl-cu12` are pinned to **x86_64-only** wheel URLs while `torchao` is a hard runtime dep — the "aarch64 wheels for every heavy dependency" claim does not hold, which sharpens the aarch64-variance risk.
|
||||
8. `prime disks` attach with `--disks`, not `--attach-disk`.
|
||||
9. **Two things nobody has planned:** how the DCP checkpoint gets *off* the pod before `prime pods terminate` destroys the disk (needs a `--disks` volume at create time), and the DCP → HF shards → NVFP4 → Compute model-id conversion, which nothing on disk automates end-to-end (prime-rl has `convert_to_hf` but no CLI entrypoint).
|
||||
10. **The link nobody drew:** rental hours are driven by whether the turn budget binds. `group_size × batch × steps` of *multi-turn* rollouts against a rented inference fleet is real money. If B2's exhaustive policy scores at the oracle, the run burns GPU hours measuring nothing. **Rule 4 is a compute-policy constraint, not only a design one.**
|
||||
|
||||
---
|
||||
|
||||
## 2. Parallelization map and file collisions
|
||||
|
||||
### What can start today, concurrently
|
||||
|
||||
| Lane | Work | Repo | Blocked by |
|
||||
|---|---|---|---|
|
||||
| **1** | **B1** — discovery de-hardcode + timeout + step name + `uv.lock` cleanup | `arena` | nothing |
|
||||
| **2** | **A step 2** — `live.py` engine + unit tests + **re-measure the ladder** | `arena` | nothing (own worktree) |
|
||||
| **3** | **D step 1** — worker, vendored TS, hashes, `test_replay.py` | `arena` (new dir only) | nothing |
|
||||
| **4** | **Compute Policy step 0** — `prime login`, catalogue + prices, doc the corrected spark-1 facts | none / `arena/docs` | nothing |
|
||||
| **5** | **P1** — Caddy host, host-aware routing, AppShell, Overview/Worlds/Org/Inbox | `lumbridge-v4` | nothing |
|
||||
| **6** | **W step 0** — the five factual corrections + `eval-results.json` decision | `lumbridge-v4` | nothing |
|
||||
|
||||
Six lanes, and only lanes 1/2/3 share a repo. **Use a worktree per lane** — this repo took divergent concurrent pushes on 2026-08-19. Merge, never force.
|
||||
|
||||
### True dependency edges
|
||||
|
||||
```
|
||||
B1 ──────────────────────────────────────────┐
|
||||
├──► A step 7 (probe row) ──► A ships
|
||||
A steps 1-2 (engine) ──► A re-measure ──► A steps 3-6 ─┘
|
||||
│
|
||||
└──► B2 (policy machinery, calibrated on real numbers)
|
||||
│
|
||||
├──► C1 drop-table
|
||||
├──► C2 bot-detection
|
||||
├──► C3 fault-localisation
|
||||
├──► C4 redaction (2-agent)
|
||||
└──► D step 5 (tera probe rows)
|
||||
|
||||
D step 1 (replay gate) ──► D crow-nav ──► D flight/office
|
||||
|
||||
A ships ──► one real eval run ──► traces.jsonl exists ──► P2 (Arena dashboard)
|
||||
│
|
||||
└──► W /arena numbers (via eval-results.json)
|
||||
|
||||
prime login ──► rented-pod runbook ──► first RL run ──► Forge prime-rl renderer
|
||||
```
|
||||
|
||||
Two edges that were wrong in the source specs and are now correct:
|
||||
- **B1 does not depend on A** (PLAN.md's "B depends on A" applies only to B2).
|
||||
- **D is no longer file-independent of A/B/W** — it lives in `arena` and edits `probe.py`, `ci.yml`, `test_probe.py`, `README.md`, plus `arena.tsx` and `seo.ts`.
|
||||
|
||||
### Merge-conflict map — the specific paths
|
||||
|
||||
**Tier 1 — will conflict, must be sequenced or de-hardcoded first:**
|
||||
|
||||
| Path | Claimed by | Resolution |
|
||||
|---|---|---|
|
||||
| `arena/probe.py` | A (§11), B1 (registry), B2 (machinery), D (§7), C1-C4, W (needs a return-shape change for crude labels) | **B1 lands first.** After B1, adding an env is one appended function at EOF — a mergeable case. Nothing else touches this file until B1 is merged. |
|
||||
| `arena/tests/test_probe.py` | A, B1, D | B1 removes it from every future workstream's diff. Never edited again. |
|
||||
| `arena/.gitea/workflows/ci.yml` | A, B1, D (needs Node + replay test + a one-dir-four-wheels change) | B1 lands the glob; D then adds *steps*, not list entries. **D's one-directory-four-tasksets layout actively breaks B1's one-dir-one-wheel discovery** — D must add a `[tool.arena] tasksets = [...]` key or equivalent, and B1 should be told to read it. |
|
||||
| `arena/configs/grand_exchange_live.toml` | **A, B(§1.7.2), Compute Policy** — three specs create the same file | **A owns it.** Delete it from the other two file lists. |
|
||||
| `arena/environments/grand_exchange_live/*/taskset.py` and `session.py` | **A and B both specify the module layout and class signatures** | **A owns the contract.** B2 adapts. |
|
||||
| `arena/README.md` | A (rows), D (rows + house rule 4), Compute Policy (baseline qualifier), W (source of truth for the site tables) | Sequence: A → D → Compute Policy. W reads it, never writes it. |
|
||||
| `arena/PLAN.md` + `MULTI_TURN.md` | B, D (scope amendment), Compute Policy | One session, one pass, after the D-scope decision lands. |
|
||||
|
||||
**Tier 2 — cross-repo, same file:**
|
||||
|
||||
| Path | Claimed by | Resolution |
|
||||
|---|---|---|
|
||||
| `lumbridge-v4/apps/web/src/pages/v4/arena.tsx` | **D** (delete `office-jobs-v1` at :203 and :511) and **W** (full rebuild of the page) | **W owns the file.** D contributes the *fact* (four manifests, 16 scenarios), not the edit. |
|
||||
| `lumbridge-v4/services/api/src/seo.ts` | **D** (:271 twelve→eleven, five→four) and **P** (`renderAppShell`, drop `/app` from `APP_ROUTE_PATTERNS` at :698) | Different regions; sequence P after D's one-line copy fix, or vice versa. Low risk but both must run `seo.test.ts` (`ROUTER_PATHS` at :36) and `spa-fallback.test.ts`. |
|
||||
| `lumbridge-v4/apps/web/src/pages/v4/forge.tsx` | **W** (FlowStrip, StateList) and **Compute Policy** (:59-62 delegation copy) | W owns it. Compute Policy contributes one sentence, **only after a rented run has actually happened**. The `:114` edit is cancelled (see verdict). |
|
||||
| `lumbridge-v4/apps/web/src/data/eval-results.json` | **W** (Nemotron run) and **P2** (the dashboard's run store) | The public file is a curated summary; the dashboard is the archive. Decide the direction (Decision 6) before either writes. |
|
||||
|
||||
**Tier 3 — three tracked `uv.lock` files are failing CI on main right now.** `git rm --cached` them in B1's commit, before anyone blames a new failure on a new environment.
|
||||
|
||||
---
|
||||
|
||||
## 3. Sequencing
|
||||
|
||||
### Phase 1 — genuinely shippable, ~2 weeks, six parallel lanes
|
||||
|
||||
1. **B1** (arena): tomllib discovery, `@probes` registry, timeout 180, `uv.lock` untracking, step rename. *Gate: `probe.py` still prints seven `ok` rows; CI green with zero list edits.*
|
||||
2. **A steps 1-2 + re-measurement** (arena, worktree): frozen generator copy + equality assertion; `LiveBook`/`Leg`/`Ledger` with unit tests; **then re-run the full policy ladder and publish the corrected numbers.** *Gate: the ladder reconciles with the reward formula; `horizon`, `FREEZE_RANGE`, `TARGET_SHARE`, `TURNS` are chosen from that measurement, not from the spec.*
|
||||
3. **D step 1** (arena, new dir): worker + vendored TS + `hashes.py` + `sync_tera.py` + `test_replay.py`. *Gate: 32/32 replay exact, tampering rejected, hash check green — and a decision on the 4 unpinned files.*
|
||||
4. **Compute Policy step 0**: `prime login`; record real `availability` output and prices; write `arena/docs/COMPUTE_POLICY.md` with the corrected spark-1 facts (`brain-qwen38-dspark`, silent-wrong-model hazard, 36 GB headroom).
|
||||
5. **P1** (lumbridge-v4): Caddy `app.` block with `bind 10.0.0.2` and the non-empty-body check; host-aware `App.tsx`; `renderAppShell` + `sendShell` signature change + `seo.test.ts` + `spa-fallback.test.ts`; AppShell; Overview / Worlds / Org / Inbox; delete the 23 dead `api.ts` methods.
|
||||
6. **W step 0** (lumbridge-v4): the five factual corrections — `arena.tsx:330` "Nine tasks", `landing.tsx` card 02, the 4-row probe table → 7 rows, `office-jobs-v1` deleted, `arena.tsx:519` "no model scores published". Ship the Nemotron run into `eval-results.json`.
|
||||
|
||||
**Phase 1 ends with**: a green de-hardcoded CI, a measured live engine, a proven TS replay bridge, a live private console on real endpoints, a truthful public site, and a Prime Intellect account that can rent.
|
||||
|
||||
### Phase 2
|
||||
|
||||
7. **A steps 3-9** → `grand-exchange-live` ships with its probe row (needs B1 merged).
|
||||
8. **One real eval run** against spark-1, pinned to `brain-qwen38-dspark`, with the empty-rewards count checked. *This is a hard prerequisite for P2 and for any site number.*
|
||||
9. **D crow-nav** end to end (role named `agent`, `vf.Taskset[...]` generic form, `oracle` op in the worker, `gate` as a multiplier).
|
||||
10. **W /arena rebuild** + `/bench` on the new primitives.
|
||||
11. **First rented run**: pod create with an attached disk, `install.sh`, wheel install, baseline eval, `uv run rl`, re-eval, before/after table.
|
||||
|
||||
### Phase 3
|
||||
|
||||
12. **B2** — policy machinery, calibrated against A's measured exhaustive/oracle gap, with block-wise ladders replacing the single 0.02 margin.
|
||||
13. **P2** — Arena schema (per-trace PK), server-side aggregates, admin-only transcripts, ingester for both output layouts, the four pages.
|
||||
14. **C1-C4** — the four remaining interactive environments, fanning out from B2.
|
||||
15. **D flight + office-nav**; drive-101 per Decision 3.
|
||||
16. **Forge**: `prime-rl` renderer (LoRA RL first), x86_64 train lock, retitled aarch64 lock, DECISIONS entry, checkpoint→HF→NVFP4→Compute-model-id stage.
|
||||
17. **W /stream**, landing (with its own `machine-home.css` audit), compute/simulate/about.
|
||||
18. **E** — Hub push, when un-parked.
|
||||
|
||||
---
|
||||
|
||||
## 4. Decisions Karti still owes, ranked by what they block
|
||||
|
||||
| # | Decision | Blocks | Recommendation |
|
||||
|---|---|---|---|
|
||||
| **1** | **The agent seat name.** `agent` (matching `SingleAgentEnvConfig`) or a custom name? It becomes the TOML key in `configs/*.toml` **and** in every prime-rl `rl.toml`, on both A and D. Renaming later parses fine and silently has no seat. | A, D, every future env, every training config | **`agent`, everywhere.** One word, decided once, written into each env's README. |
|
||||
| **2** | **Does D live in `arena` or `tera`?** The spec's packaging argument is strong (`tera` has no Python toolchain) but it overturns PLAN.md:138 and makes D collide with A/B/W. | D's entire file layout, the CI wheel loop, PLAN.md | **`arena`**, and amend PLAN.md's parallelization map in the same commit. |
|
||||
| **3** | **Does `tera-drive-101` ship?** Measured: monotone in throttle, optimal policy is one constant string, and the 0.9 band makes any positive throttle score ~0.95+. Also decides whether the site says four spatial environments or three. | D's scope, `arena.tsx`/`seo.ts` copy, the Hub listing | **Ship it labelled a bridge-correctness environment with no rule-4 and no rule-3 claim, or defer.** Do not list it as a fifth RL environment. |
|
||||
| **4** | **A's freeze mechanic: flat per-look, or proportional to items re-quoted?** Flat is measured and works; per-item is the better mechanism, makes the look count genuinely vary, and forces a much larger surgical reference. | A's engine, and therefore B2's calibration | **Ship flat.** Record per-item as the identified strengthening. Do not build an unmeasured mechanism on the critical path. |
|
||||
| **5** | **Arena transcript visibility.** Full transcripts contain the held-out slice and, for oracle runs, the answers. | P2's route scoping, and house rule 1's integrity | **Admin-only.** This is not an org-politics question; it is a contamination question, and that settles it. |
|
||||
| **6** | **Where does model-run data live?** Public `eval-results.json` (already typed, tested, rendered on `/evals`) vs P2's `arena_runs`/`arena_episodes` tables. | W's whole data pipeline, P2's schema | **Both, with a direction:** the dashboard is the archive, `eval-results.json` is a curated public summary emitted from it. Never two independent hand-maintained files. |
|
||||
| **7** | **Is `nemotron` coming back as the production brain, or is `qwen38` the default?** `README.md`'s baseline, D23/D24's Forge preflight, and the site's Arena numbers all assume Nemotron; the box serves Qwen3.8-27B. | Every reproduction of the baseline; what "inference in production on the Spark" names | Needs an answer before anyone re-runs a benchmark. |
|
||||
| **8** | **First Forge/prime-rl integration: LoRA RL or full-FT?** LoRA reuses merge + attest unchanged and matches `alphabet-sort`; but LoRA rejects `--gpu-type` and falls back to filesystem weight broadcast. Full-FT needs a new node kind and the tensor-delta-map gate never fires. | Forge's roadmap, and what its manifest is for | **LoRA first** — it keeps Forge's headline claim intact. |
|
||||
| **9** | **Is app.\* the only authenticated origin?** Yes means the apex loses `/login`, `/signup`, `/app`, `/go`, and all three existing sessions are signed out on cutover day. | P1's scope, apex route deletions | **Yes**, and state the sign-out as an accepted cost for three accounts. |
|
||||
| **10** | **W's audience.** Arena/Bench/Stream (brokenness × argument) vs Arena/Landing/Bench (the Prime Intellect Hub reader). | W's priority order only | Answer when P1 lands; it does not block anything upstream. |
|
||||
|
||||
---
|
||||
|
||||
## 5. Risk register, highest first
|
||||
|
||||
| # | Risk | Retired by |
|
||||
|---|---|---|
|
||||
| **1** | **The whole reward design of `grand-exchange-live` rests on numbers from a throwaway engine that no longer exists, and the ladder does not reconcile with the reward formula.** `horizon=60`, `FREEZE_RANGE=(1,4)`, `TARGET_SHARE=0.85` and `TURNS=8` are all locked against arithmetic that is off by 0.011 in the one place it can be checked. | Building `live.py` FIRST and re-measuring every row before any constant is frozen. This is Phase 1, lane 2, and it is the single most important task in the programme. |
|
||||
| **2** | **Every reward in Arena saturates, so "exhaustive < oracle" may be structurally unreachable.** `profit_ratio` and `efficiency` clip at 1.0; `clean` is a boolean band. A 0.02 margin against a 0.25 quantum is a cliff. If the gate can't open, rule 4 is unenforceable and every interactive environment is unfalsifiable. | A's re-measurement producing a real exhaustive/oracle gap; B2 then calibrated against it with block-wise ladders. If no gap exists, the reward shape must change before C1-C4 are built on it. |
|
||||
| **3** | **No `traces.jsonl` exists anywhere.** Everything downstream — the dashboard, the site's numbers, the leaderboard, the before/after training table — is waiting on one eval that has never completed. The one run that was attempted pointed at Prime's inference API with an empty key. | One completed eval against `brain-qwen38-dspark` with the empty-rewards count checked. Cheap, and it unblocks P2, W and Compute Policy simultaneously. |
|
||||
| **4** | **`eval` exits 0 when every rollout errors, and multi-turn has far more ways to error than one-shot.** `turn()` on a closed interaction raises; `turn(msg)` on turn 1 of a prompted task raises. Combined with risk 3, a "successful" run can be entirely empty. | A CI/runbook step that asserts non-empty `rewards` on ≥N episodes, in `arena`, in D's replay gate, and in P2's ingester. Never trust the exit code. |
|
||||
| **5** | **Silent wrong-model comparison.** `--model brain` returns 200 from a box serving Qwen3.8-27B. Re-running the Nemotron baseline today produces different-model numbers under an identical command, and they will be tabulated as comparable. | Pinning `brain-qwen38-dspark` (and the scene) in every command in `README.md`, `PLAN.md`, `MULTI_TURN.md` and `COMPUTE_POLICY.md`. Trivial; do it in Phase 1. |
|
||||
| **6** | **`probe.py`, `tests/test_probe.py` and `ci.yml` are edited by five of six workstreams, in a repo that already took divergent concurrent pushes.** | B1, first, before anything else touches `arena`. |
|
||||
| **7** | **The reward denominator is editable with every `tera` gate green.** ⚠️ CORRECTED 2026-08-21: not `offices/sites.ts` (mutating it produces a bit-identical episode — refuted twice). The hole is **`src/arena/index.ts`**, which re-exports the scripted baselines that ARE the denominator. Demonstrated: a halved office-nav baseline changes return 4.744 → 4.098 while `arena:source-hashes` prints ok and exits 0. | Pinning `src/arena/index.ts` in `tera/scripts/check-arena-source-hashes.mjs` and regenerating `sourceHashes.ts`. One line, in a repo this session did not own. |
|
||||
| **8** | **spark-1 has 36 GB of 121 GB free with four resident model services, one of which leaks ~8.4 GB/hour and has killed the brain before.** "Production inference on the Spark" is a tenancy claim held together by the governor on :8011. | A mandatory headroom check before any new resident model, and Compute's watchdog verified live. |
|
||||
| **9** | **`PRIME_API_KEY` is empty**, blocking not just the parked Hub push but every rental command, every price, and the entire training half of the compute policy. | `prime login`. One command. |
|
||||
| **10** | **Two of D's four environments cannot discriminate under the 0.9 ceiling band** — drive-101 gives ~0.95+ to any constant throttle, office-nav collapses to `efficiency` alone. | Re-deriving the band per environment against the measured return distribution, or shipping fewer environments. |
|
||||
| **11** | **The held-out corpus leaks through P2's trace viewer at `user` scope.** One invite code. | Decision 5 — admin-only. |
|
||||
| **12** | **Effort is understated across the board.** A is L/XL; B2 is not B's to size (its §1.8 is L/XL and belongs to A); D is L/XL; P's "Phase 1" is XL; W's item 1 alone is L/XL across two repos. Read as written, this programme looks like six L's and is closer to four XLs. | The phasing in §3, which cuts P1 out of P, B1 out of B, and D-step-1 out of D. |
|
||||
| **13** | **Rented GPU hours are burned by a non-binding turn budget.** `group_size × batch × steps` of multi-turn rollouts against a rented inference fleet is real money, and nothing in the compute doc connects to rule 4. | Making "exhaustive < oracle by a measured margin" a precondition of dispatching any RL run — write it into `COMPUTE_POLICY.md`. |
|
||||
| **14** | **The checkpoint has no exit path.** `prime pods terminate` destroys ephemeral disk; the DCP → HF → NVFP4 → Compute-model-id conversion is unautomated. | `--disks` at pod-create time, plus a `forge quantize/publish` stage. Discover this before the first successful training run, not after. |
|
||||
| **15** | **GH200 may not appear in `prime availability` at all** — the CLI names only H100_80GB, H200_141GB, B200_180GB, RTX_PRO_6000. "GH200-class" may mean H200_141GB in practice. And `perf.py` has no GH200 branch, so MFU may silently fall back to A100's 312 TFLOPS. | `prime login`, then one `availability gpu-types`, then one `nvidia-smi --query-gpu=name` on a real pod. |
|
||||
|
||||
---
|
||||
|
||||
## 6. Build this first
|
||||
|
||||
**`environments/grand_exchange_live/grand_exchange_live/live.py` — the stepped engine and its unit tests — and, concurrently, workstream B1's discovery de-hardcoding.**
|
||||
|
||||
Everything else in the programme is downstream of two facts we do not yet have: whether the live market's reward ladder actually reconciles (risk 1, which invalidates A's horizon, freeze, TARGET_SHARE and turn budget if it does not, and takes B2, C1-C4 and half the compute argument with it), and whether three files can stop being a five-way merge conflict before five workstreams reach for them. B1 is an afternoon and removes the repo's only structural collision permanently; `live.py` is the smallest artifact that can turn six specs' worth of `/tmp` measurements into something the shipped code agrees with.
|
||||
+208
@@ -0,0 +1,208 @@
|
||||
# Scope: the multi-turn gap
|
||||
|
||||
Status: scoping document, 2026-08-21. Nothing here is built yet.
|
||||
|
||||
Every environment in this repository sets `[env.agent.harness] id = "null"` — one
|
||||
completion, no tools, no runtime. That is the right choice for a pure-trace data
|
||||
environment and it is also the whole of what Arena currently measures. Nothing here
|
||||
scores an agent *acting*: choosing what to look at next, paying for a turn, or
|
||||
revising after seeing a consequence.
|
||||
|
||||
This document scopes closing that gap. The headline finding is that it is three
|
||||
separable tiers, not one, and the cheapest tier needs no new infrastructure at all.
|
||||
|
||||
## What upstream actually requires
|
||||
|
||||
Read from `~/vendor/prime-intellect/verifiers` (v1, installed version 0.3.0), not
|
||||
from blog posts.
|
||||
|
||||
**Multi-turn does not imply a container.** `vf.Env.run(task, agents)` programs the
|
||||
control flow host-side. `agents.<role>.interaction(task)` opens a rollout and holds
|
||||
it open; `await interaction.turn(observation)` sends one user turn, runs one harness
|
||||
segment, and returns a `Segment` with `.last_reply` and `.terminated`. The simulator
|
||||
runs in-process, as the user. The harness can stay `null` throughout.
|
||||
|
||||
The reference implementation is `verifiers/v1/tasksets/textarena/taskset.py` — 90
|
||||
lines, drives a real game engine, scores off the engine directly. Read it before
|
||||
writing anything.
|
||||
|
||||
**An Env ships inside the taskset's own wheel.** `EnvConfig.id` empty means "the
|
||||
taskset's own env, else `SingleAgentEnv`". Export the `Env` subclass from the
|
||||
package `__init__.py`, exactly as `environments/wordle` exports `TextArenaEnv`, and
|
||||
`[env.taskset] id = "..."` picks it up with no extra config.
|
||||
|
||||
**Available in the installed version** — confirmed, not assumed:
|
||||
|
||||
- harnesses: `null`, `bash`, `browser_use`, `claude_code`, `codex`, `hermes_agent`,
|
||||
`kimi_code`, `mini_swe_agent`, `openclaw`, `pi`, `pool`, `rlm`, `terminus_2`
|
||||
- runtimes: `subprocess` (debug only — rollouts share side effects), `docker`,
|
||||
`modal`, `prime`
|
||||
- prebuilt envs: `single_agent`, `best_of_n`, `agentic_judge`,
|
||||
`shared_agentic_judge`, `user_sim`
|
||||
|
||||
**The interception server** sits between harness and provider. It speaks each
|
||||
harness's native dialect (Anthropic Messages for Claude Code, OpenAI Responses for
|
||||
Codex), builds the trace live, forces sampling params the harness will not expose,
|
||||
and can rewrite tool results and web-search results to block reward hacks. It is
|
||||
mandatory for tiers 2 and 3 and free for tier 1.
|
||||
|
||||
## The three tiers
|
||||
|
||||
| | Mechanism | Harness | Runtime | New infra |
|
||||
|---|---|---|---|---|
|
||||
| **1. Interactive** | `vf.Env.run` + `interaction.turn`, simulator in-process | `null` | none | **none** |
|
||||
| **2. Tool-using** | Taskset exports a `Toolset`, served as MCP | `bash` | `subprocess` / `docker` | image per env |
|
||||
| **3. Agentic** | Full coding agent in a sandbox | `claude_code`, `codex`, `openclaw`, `pi` | `docker` / `prime` | image + interception + budget |
|
||||
|
||||
Tier 1 is the one to build first, and it is where our existing simulators already
|
||||
sit. Tiers 2 and 3 are real work with real cost and should follow evidence from
|
||||
tier 1, not precede it.
|
||||
|
||||
## What we already have that plugs in
|
||||
|
||||
**Six of the seven environments here own a simulator that is already steppable.**
|
||||
They are one-shot only because the taskset hands the model the whole observation up
|
||||
front. Each has an obvious interactive form:
|
||||
|
||||
| environment | one-shot today | interactive form | tier |
|
||||
|---|---|---|---|
|
||||
| `grand-exchange` | read 30 ticks of history, place one basket of orders | quote, see fills and the next tick, re-quote, over a turn budget — inventory and unfilled orders carry | 1 |
|
||||
| `drop-table-inference` | read a fixed kill log, estimate the table | **buy** observations: each turn spends budget on more kills, then answers — measures when to stop looking | 1 |
|
||||
| `fault-localisation` | read the whole incident log, name the cause | the log is *not* given: query it by service, time window and level under a query budget | 1 → 2 |
|
||||
| `redaction-pressure` | write rules, scored by a scanner | two agents: a redactor and an extractor scored on what it recovers — the counterweight becomes an opponent | 1 (multi-agent) |
|
||||
| `schema-migration` | emit a migration, executed against held-out rows | a real SQLite in a container the agent runs SQL against, with rollback | 3 |
|
||||
| `bot-detection` | classify from six labelled examples | request labels one at a time under a budget (active learning) | 1 |
|
||||
| `canary-trap` | write probes | no natural interactive form — leave one-shot | — |
|
||||
|
||||
**Tera's four spatial environments are already multi-turn — in the wrong language.**
|
||||
`~/repos/gitea/tera/src/arena/` implements `tera.arena/v1`: Gym-style
|
||||
`reset(seed, scenario)` / `step(action)` / `snapshot` / `restore` / `trace` /
|
||||
`replay`, FNV-1a-64 checksums, SHA-256 source pins, train/dev splits, counterweighted
|
||||
rewards, and a CI gate that proves inaction finishes below zero and a scripted
|
||||
baseline finishes positive. That is a stricter contract than most of what is on the
|
||||
Hub — and none of it is reachable from `uv run eval`, because it is TypeScript with
|
||||
its own contract rather than a verifiers taskset.
|
||||
|
||||
The work there is a **bridge**, not a simulator: a Python `vf.Env` whose `run()`
|
||||
drives the TS package over a subprocess or a small local HTTP loop, turning
|
||||
`step()` observations into `interaction.turn()` messages and the existing component
|
||||
rewards into `trace.record_reward` calls. The replay checksum makes the bridge
|
||||
verifiable — a bridged episode must replay identically in the TS package or the
|
||||
bridge is wrong. That is a rare property and worth building the bridge around.
|
||||
|
||||
⚠️ **RESOLVED 2026-08-21, 15:50 — `office-jobs-v1` EXISTS.** It was absent at 13:00 and
|
||||
present at 15:50, because another session fast-forward-merged `feat/tera-ultracode-integration`
|
||||
into `~/repos/gitea/tera` mid-session (reflog `8738367`). It is committed (`e378a03`,
|
||||
`120eac8`, dated 2026-08-19), is in `ARENA_MANIFESTS`, is documented in `tera/ARENA.md`, and
|
||||
passes `npm run arena:source-hashes`. **The true counts are twelve environments, five spatial,
|
||||
twenty scenarios** — measured, not inferred. The site's original copy was right; the
|
||||
correction to eleven/four/sixteen was right when made and wrong an hour later, and has been
|
||||
reverted. Lesson: `tera` is a shared tree with live concurrent sessions — re-check its HEAD
|
||||
before acting on a claim about what it contains.
|
||||
|
||||
~~⚠️ **`office-jobs-v1` does not exist.** `lumbridgecorp.com/arena` and
|
||||
`services/api/src/seo.ts` both advertise five spatial environments; `ARENA_MANIFESTS`
|
||||
in `src/arena/index.ts` holds four (`drive-101-v1`, `office-nav-v1`, `crow-nav-v1`,
|
||||
`california-flight-v1`) and there is no `officeJobs.ts`. The real total is eleven,
|
||||
not twelve. Decide whether to build it or correct the page before publishing
|
||||
anything that cites the count.~~ (superseded — see above)
|
||||
|
||||
## The house rules under multi-turn
|
||||
|
||||
The three rules were written for one-shot scoring. Two survive unchanged and one
|
||||
does not, and there is a fourth the interactive tier forces.
|
||||
|
||||
1. **Graded on what the model did not see** — unchanged and easier: a stepped
|
||||
simulator generates its held-out continuation by construction.
|
||||
2. **No single-sided reward** — *harder*. Multi-turn adds degenerate strategies that
|
||||
have no one-shot analogue: stalling to burn the clock when the score is
|
||||
time-averaged, spamming cheap turns when observations are free, refusing to act
|
||||
when acting risks a penalty. Every new reward needs a counterweight in the *turn*
|
||||
dimension, not just the answer dimension.
|
||||
3. **A zero floor and a reachable ceiling, demonstrated** — unchanged, but the floor
|
||||
now has to hold across a whole trajectory, not one reply.
|
||||
4. **New: the budget must bind.** An interactive environment where the optimal
|
||||
policy is "use every turn" is measuring nothing the one-shot version did not.
|
||||
Observations must cost something, and the oracle must be shown to stop early.
|
||||
|
||||
**`probe.py` must grow degenerate *policies*, not just degenerate answers.** Today
|
||||
it scores four fixed strings per environment. The interactive tier needs at minimum:
|
||||
the staller (never acts), the spammer (acts every turn at random), the impatient
|
||||
(answers on turn 1 with no observation), and the exhaustive (spends the entire
|
||||
budget then answers). The gate stays the same shape — inaction 0.000, oracle 1.000,
|
||||
exit 1 otherwise — plus a new assertion that the exhaustive policy scores *below*
|
||||
the oracle. If it does not, rule 4 is violated and the environment does not ship.
|
||||
|
||||
This is the differentiator. Nothing in `prime-envs` (88 environments) or
|
||||
`community-environments` (109) ships a degenerate-policy gate.
|
||||
|
||||
## Dual publishing
|
||||
|
||||
Both targets take the same wheel; the repo layout already mirrors verifiers' own.
|
||||
|
||||
- **Prime Intellect Hub**, our own namespace: `prime env push <name> -v PUBLIC`.
|
||||
We keep authorship and versioning, and the listing can link back to
|
||||
lumbridgecorp.com/arena.
|
||||
- **`community-environments`**: a PR to that repo, published under the
|
||||
`primeintellect` org by their CI on a version bump. Extra distribution, but it
|
||||
hands the namespace to them. Secondary, not primary.
|
||||
- **lumbridgecorp.com/arena**: unchanged — the page documents the same wheel.
|
||||
|
||||
⚠️ Its README still documents the **legacy** flow (`load_environment`, `vf-eval`,
|
||||
`uv pip install -e .`). Ignore it; follow `verifiers/docs/v1/`.
|
||||
|
||||
⚠️ **Blocker: `PRIME_API_KEY` is not set on amd-server.** `prime config view` shows
|
||||
`API Key: Not set`. The `pit_` key lives in karti's memory note on cloud-1
|
||||
(`.claude/projects/-home-ubuntu/memory/prime-intellect.md`) and is the same
|
||||
credential for `api.pinference.ai` and `api.primeintellect.ai`. Nothing pushes until
|
||||
it is configured here.
|
||||
|
||||
## Preconditions, verified on this box
|
||||
|
||||
- `uv run eval @ configs/canary_trap.toml --dry-run` → exit 0, config written ✓
|
||||
- verifiers 0.3.0 with all 13 harnesses and 4 runtimes present ✓
|
||||
- Docker 29.1.3 running (tier 2/3 runtime) ✓
|
||||
- spark-1 `brain` on `http://100.127.247.67:8001/v1`, model id `brain` ✓
|
||||
- `prime` CLI installed (0.6.25; 0.6.26 available) ✓
|
||||
- `PRIME_API_KEY` ✗ — see above
|
||||
|
||||
## Proposed sequence
|
||||
|
||||
**Phase 1 — one interactive environment, end to end.** `grand-exchange` is the
|
||||
candidate: the market simulation already steps, the counterweight (liquidity, the
|
||||
random-walk trap) already exists, and turning one basket into a quote/fill loop is
|
||||
the smallest change that produces a genuinely new measurement. Deliverable: a
|
||||
`vf.Env` in the existing wheel, a second config, and probe coverage for the four
|
||||
degenerate policies. This is where rule 4 gets written for real.
|
||||
|
||||
**Phase 2 — extend `probe.py` to policies and re-gate everything.** Whatever
|
||||
phase 1 teaches about degenerate trajectories applies to every later environment.
|
||||
Do this before building a second one, not after. Note that `tests/test_probe.py`
|
||||
and the wheel-build loop in `.gitea/workflows/ci.yml` hardcode the environment
|
||||
list — both need editing or CI goes red.
|
||||
|
||||
**Phase 3 — the Tera bridge.** One Python `vf.Env` that drives the existing TS
|
||||
package, validated by replaying every bridged episode through `replay()` and
|
||||
requiring an identical checksum. Lands four environments at once and makes the
|
||||
strongest existing asset publishable. Larger and riskier than phase 1; the
|
||||
process boundary and the action encoding are the unknowns.
|
||||
|
||||
**Phase 4 — first Hub push**, once phases 1–3 give three to four interactive
|
||||
environments alongside the seven one-shot ones. Needs the API key.
|
||||
|
||||
**Phase 5 — tier 2, then tier 3.** `fault-localisation` as a query-budgeted
|
||||
tool-using environment, then `schema-migration` against a real database in a
|
||||
container. Only after tier 1 has produced measured scores worth extending, and
|
||||
with the concurrency cost of a container per rollout understood first.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Does a bridged Tera episode replay to an identical checksum through a process
|
||||
boundary? If not, the bridge cannot be verified and phase 3 gets much more
|
||||
expensive.
|
||||
- What is the per-rollout wall-clock cost of the `docker` runtime here, at the
|
||||
concurrency an eval sweep needs? Unmeasured. It gates tier 3 entirely.
|
||||
- Should the interactive form of an environment be a **separate** taskset id
|
||||
(`grand-exchange-live`) or a config flag on the existing one? Separate ids keep
|
||||
the one-shot scores comparable across time; a flag keeps the wheel count down.
|
||||
Leaning separate.
|
||||
@@ -0,0 +1,243 @@
|
||||
# Arena build plan
|
||||
|
||||
Companion to `MULTI_TURN.md` (the scope). This is the sequencing: what gets built,
|
||||
in what order, what can run in parallel, and what collides.
|
||||
|
||||
Status: plan, 2026-08-21. ✅ = settled. ❓ = blocks the work under it. ⛔ = parked by decision.
|
||||
|
||||
## Where Arena sits
|
||||
|
||||
Lumbridge is an open-source company shipping six self-hostable things:
|
||||
**Compute** (governed compute for unified-memory hardware), **Bench** (model quality
|
||||
and serving measured against one pinned host × quant × config tuple), **Simulate**
|
||||
(the Tera/Spaces 3D engine), **Stream** (a headless broadcast rig), **Forge**
|
||||
(post-training as a content-addressed build), and **Arena**. Four of the six are
|
||||
public. `apps/web/src/data/projects.ts` in `lumbridge-v4` is the canonical list.
|
||||
|
||||
**Arena is the RL-environment project — yes, `/arena` is the section.** It is
|
||||
appended last in the project list for numbering reasons, but semantically it sits
|
||||
beside Bench and feeds Forge: Bench measures a model, Arena produces the gradient,
|
||||
Forge runs the training. That triangle is the reason Arena is worth doing well
|
||||
rather than as a side quest — we own all three corners, which nobody on the Hub does.
|
||||
|
||||
Arena spans **two repositories on one page**, and that is deliberate:
|
||||
|
||||
- `git.karti.ai/lumbridge-public/arena` — the Python data environments (this repo)
|
||||
- `git.karti.ai/lumbridge-public/tera` — the TypeScript spatial environments
|
||||
|
||||
## What we take from Prime Intellect, and what stays ours
|
||||
|
||||
All of PI's stack is open source and we use it rather than rebuild it. Checkouts are
|
||||
at `~/vendor/prime-intellect/` (46 repos; see its README).
|
||||
|
||||
| We use, unmodified | We build |
|
||||
|---|---|
|
||||
| `verifiers` — the v1 spec, the taskset/harness/trace model, `vf.Env` | The environments |
|
||||
| `prime-rl` — the trainer traces feed | The house rules |
|
||||
| `prime-agent` (`pi`) — a harness, already embedded in Piggy | `probe.py` — the degenerate-policy gate |
|
||||
| `prime` CLI — `env push`, `env action` | The Tera bridge |
|
||||
| The Environments Hub — distribution | lumbridgecorp.com/arena — documentation |
|
||||
|
||||
We do **not** build a harness, a trainer, or a hub. Every one of those exists,
|
||||
is permissively licensed, and is better than what we would write.
|
||||
|
||||
## Decisions
|
||||
|
||||
**✅ 1 — Hub org handle: `lumbridge`.** Settled 2026-08-21. Environments will
|
||||
publish as `lumbridge/<env-name>`.
|
||||
|
||||
**✅ 2 — Attribution: Lumbridge as author.** Settled 2026-08-21. `karti.ai` stays
|
||||
the code host (`git.karti.ai`), Lumbridge is the author on every surface. One
|
||||
identity across all six projects; no per-project byline.
|
||||
|
||||
**⛔ Hub push is ON HOLD.** Decided 2026-08-21: keep building, publish nothing to
|
||||
the Prime Intellect Hub yet. Workstream E is parked — not cancelled, not blocked on
|
||||
anything technical. `PRIME_API_KEY` is therefore not urgent.
|
||||
|
||||
**✅ 7 — Production brain: Qwen3.8-27B.** Settled 2026-08-21. spark-1 serves
|
||||
`brain-qwen38-dspark`; that is the default. The Nemotron baseline is ARCHIVED, not
|
||||
deleted, and Arena's seven scores get re-run and republished against Qwen.
|
||||
⚠️ **`--model brain` does NOT 404 — it silently succeeds and records `brain` in the
|
||||
trace while the box serves Qwen.** Verified by direct call 2026-08-21. Pin
|
||||
`brain-qwen38-dspark` in every reproduction command, everywhere.
|
||||
|
||||
**✅ 8 — `tera-drive-101` ships labelled.** Settled 2026-08-21. Published as a
|
||||
**bridge-correctness** environment with NO rule-3 and NO rule-4 claim (its return is
|
||||
monotone in throttle; any positive constant scores ~0.95+). Site says **four**
|
||||
spatial environments. It is not listed as a fifth RL environment.
|
||||
|
||||
**✅ 9 — `app.lumbridgecorp.com` is the ONLY authenticated origin.** Settled
|
||||
2026-08-21. The apex loses `/login`, `/signup`, `/app`, `/go`. All existing sessions
|
||||
sign out once on cutover; accepted cost for three accounts.
|
||||
|
||||
**✅ 10 — Adopted defaults** (technical, recommended by the synthesis, settled
|
||||
2026-08-21): the agent seat is named **`agent`** everywhere (matching
|
||||
`SingleAgentEnvConfig`) — it becomes the TOML key in every `configs/*.toml` AND every
|
||||
prime-rl `rl.toml`, and renaming later parses fine while silently having no seat;
|
||||
**D lives in the `arena` repo**, not `tera` (amend the parallelization map);
|
||||
**A ships the FLAT freeze**, with per-item recorded as the identified strengthening;
|
||||
**Arena transcripts are ADMIN-ONLY** (they carry the held-out slice and oracle
|
||||
answers — a contamination question, not an org-politics one); **the dashboard is the
|
||||
archive and `eval-results.json` a curated summary emitted from it**, never two
|
||||
hand-maintained files; **Forge integrates prime-rl with LoRA RL first**.
|
||||
|
||||
**✅ 3 — `office-jobs-v1` EXISTS; the counts are twelve / five / twenty.** Resolved
|
||||
2026-08-21 15:50 when another session fast-forward-merged it into the shared `tera` tree
|
||||
mid-session. It is committed, in `ARENA_MANIFESTS`, and passes `arena:source-hashes`.
|
||||
The site's original copy was correct. ⚠️ `tera` has live concurrent sessions — re-check
|
||||
its HEAD before acting on any claim about its contents.
|
||||
|
||||
~~**❓ 3 — `office-jobs-v1`.** Advertised on the live page and in `seo.ts`; absent
|
||||
from `ARENA_MANIFESTS`. Build it, or correct the page to eleven. Blocks any copy
|
||||
that cites the count, including a Hub listing.
|
||||
|
||||
**✅ 4 — Interactive envs get their own taskset id.** Settled 2026-08-21. The first
|
||||
is **`grand-exchange-live`**, a separate wheel beside `grand-exchange`. One-shot
|
||||
scores stay comparable across time; the two forms are measured, not conflated.
|
||||
|
||||
**✅ 5 — App is a subdomain: `app.lumbridgecorp.com`.** Settled 2026-08-21. Matches
|
||||
the two-surface model (public site + private platform). Public site stays a site.
|
||||
|
||||
**✅ 6 — Compute policy.** Settled 2026-08-21. **Training runs on rented GPUs**
|
||||
(GH200-class, from Prime Intellect) — rented because it is faster, and because
|
||||
renting keeps the hardware choice open per run (the GB300-vs-alternatives call is
|
||||
one we want to keep making). **Inference in production runs on the DGX Spark.**
|
||||
We do **not** train on the Spark.
|
||||
|
||||
## Workstreams
|
||||
|
||||
Five, with the dependency edges called out. **A and D can start immediately and in
|
||||
parallel. B must land before C or E.**
|
||||
|
||||
### A — `grand-exchange` interactive *(start now)*
|
||||
|
||||
The first tier-1 environment: quote → see fills → re-quote over a turn budget,
|
||||
inventory and unfilled orders carrying between turns. Chosen because the market
|
||||
simulation already steps, the counterweight (liquidity in gp/tick, the random-walk
|
||||
trap) already exists, and it is the smallest change that yields a genuinely new
|
||||
measurement.
|
||||
|
||||
Deliverable: a `vf.Env` subclass exported from the wheel's `__init__.py`, a config,
|
||||
and a measured run against spark-1's `brain`. Pattern to copy:
|
||||
`verifiers/v1/tasksets/textarena/taskset.py`.
|
||||
|
||||
This workstream is where **rule 4 — the budget must bind** gets written for real.
|
||||
Whatever it learns about degenerate trajectories is the input to B.
|
||||
|
||||
Depends on: nothing — ✅4 settled.
|
||||
|
||||
### B — `probe.py` grows degenerate *policies* *(after A's first draft)*
|
||||
|
||||
Today the probe scores four fixed strings per environment. Tier 1 needs policies:
|
||||
the **staller** (never acts), the **spammer** (acts every turn at random), the
|
||||
**impatient** (answers turn 1 with no observation), the **exhaustive** (spends the
|
||||
whole budget, then answers). Same gate shape — inaction 0.000, oracle 1.000, exit 1
|
||||
otherwise — plus a new assertion that **exhaustive scores below oracle**. If it does
|
||||
not, rule 4 is violated and the environment does not ship.
|
||||
|
||||
Do this before building a second interactive environment, not after. It is the
|
||||
differentiator: nothing across `prime-envs` (88) or `community-environments` (109)
|
||||
ships a degenerate-policy gate.
|
||||
|
||||
Depends on: A (needs one real interactive env to gate against).
|
||||
Blocks: C, E.
|
||||
|
||||
### C — More interactive environments *(after B)*
|
||||
|
||||
In rough order of value:
|
||||
|
||||
1. `drop-table-inference` — buy observations under budget; measures when to stop looking
|
||||
2. `bot-detection` — request labels one at a time (active learning)
|
||||
3. `fault-localisation` — the log is not given; query it by service, window and level
|
||||
4. `redaction-pressure` — two agents, redactor vs extractor; the counterweight becomes an opponent
|
||||
|
||||
Each is independently parallelizable **once B exists**. `redaction-pressure` is the
|
||||
most interesting and the least certain — a genuine multi-agent env, not a stepped one.
|
||||
|
||||
Depends on: B.
|
||||
|
||||
### D — The Tera bridge *(start now, independent repo)*
|
||||
|
||||
One Python `vf.Env` driving the existing TypeScript `tera.arena/v1` package over a
|
||||
process boundary, turning `step()` observations into `interaction.turn()` messages
|
||||
and existing component rewards into `trace.record_reward` calls.
|
||||
|
||||
Validated by replay: every bridged episode must `replay()` through the TS package to
|
||||
an identical FNV-1a-64 checksum, or the bridge is wrong. That property is rare and
|
||||
the bridge should be built around it.
|
||||
|
||||
Lands four environments at once and makes our strongest existing asset publishable.
|
||||
Larger and riskier than A — the process boundary and the action encoding are the
|
||||
unknowns. **Touches the `tera` repo, not this one, so it does not collide with A/B/C.**
|
||||
|
||||
Depends on: nothing.
|
||||
|
||||
### E — First Hub push *(PARKED)*
|
||||
|
||||
Push once there are three to four interactive environments alongside the seven
|
||||
one-shot ones. `prime env push <name> -v PUBLIC` to our own namespace is primary —
|
||||
we keep authorship and versioning. A PR into `community-environments` is optional
|
||||
extra distribution that hands the namespace to `primeintellect`; secondary.
|
||||
|
||||
Then update `lumbridgecorp.com/arena` to document the same wheels.
|
||||
|
||||
⛔ On hold by decision, 2026-08-21. Resume when Karti says so; ❓3 still gates any
|
||||
copy that cites the environment count.
|
||||
|
||||
## ⚠️ The collision point
|
||||
|
||||
**`tests/test_probe.py` and the wheel-build loop in `.gitea/workflows/ci.yml`
|
||||
hardcode the environment list.** Every workstream that adds an environment edits
|
||||
both files. Two sessions adding environments concurrently *will* conflict there, and
|
||||
CI goes red if either is missed.
|
||||
|
||||
Mitigation, in order of preference: (1) B refactors both to discover environments
|
||||
from `environments/*/pyproject.toml` instead of a literal list — do this as part of
|
||||
B and the collision disappears; (2) failing that, one session at a time touches
|
||||
those two files.
|
||||
|
||||
Separately: this repo already took divergent remote pushes from concurrent sessions
|
||||
on 2026-08-19. **Merge, never force.** Use a worktree per workstream.
|
||||
|
||||
## Parallelization map
|
||||
|
||||
```
|
||||
now ──┬── A (grand-exchange interactive) ──┬── B (probe → policies, de-hardcode CI) ──┬── C1 drop-table
|
||||
│ │ ├── C2 bot-detection
|
||||
│ │ ├── C3 fault-localisation
|
||||
│ │ └── C4 redaction (2-agent)
|
||||
│ │
|
||||
└── D (Tera bridge, tera repo) ──────┴──────────────────────────────── E (Hub push) ⛔ PARKED
|
||||
```
|
||||
|
||||
Two sessions can start today: **A** in this repo, **D** in `tera`. They share no
|
||||
files. Everything downstream of B fans out four ways.
|
||||
|
||||
## Preconditions, verified on amd-server
|
||||
|
||||
- `uv run eval @ configs/<env>.toml --dry-run` → exit 0 ✓
|
||||
- verifiers 0.3.0, all 13 harnesses and 4 runtimes present ✓
|
||||
- Docker 29.1.3 (tier 2/3 only; tier 1 needs none) ✓
|
||||
- spark-1 `brain` at `http://100.127.247.67:8001/v1`, model id `brain` ✓
|
||||
- `prime` CLI 0.6.25 ✓
|
||||
- `PRIME_API_KEY` ✗ — blocks E, which is parked anyway
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ Superseded by `EXECUTION.md`
|
||||
|
||||
The workstream descriptions and the parallelization map above were written before the
|
||||
2026-08-21 design workflow. **`EXECUTION.md` is now authoritative** for sequencing,
|
||||
file ownership and risk. Three structural corrections it makes:
|
||||
|
||||
1. **B splits.** **B1** (de-hardcode environment discovery) does NOT depend on A and
|
||||
ships immediately; the CI env list was never a matrix, just a bash `for` loop.
|
||||
**B2** (degenerate-policy machinery) was found NOT FEASIBLE as written and must be
|
||||
calibrated against A's *measured* engine, not A's first draft.
|
||||
2. **D is no longer file-independent.** It lives in `arena` and edits `probe.py`,
|
||||
`ci.yml`, `test_probe.py`, `README.md` plus two `lumbridge-v4` files.
|
||||
3. **The top risk is that every Arena reward SATURATES**, so house rule 4
|
||||
("exhaustive < oracle") may be structurally unreachable. If it is, every
|
||||
interactive environment is unfalsifiable and the reward shape must change before
|
||||
C1–C4 are built on it. This is also a compute-policy constraint: a non-binding
|
||||
turn budget burns rented GPU hours measuring nothing.
|
||||
@@ -0,0 +1,185 @@
|
||||
# Running an eval — the runbook
|
||||
|
||||
Written 2026-08-21, the day Arena first measured anything end to end. Before this,
|
||||
`outputs/` held one `config.toml` and zero episodes: every number downstream of an
|
||||
eval — the dashboard, `eval-results.json`, the leaderboard, the before/after
|
||||
training table — was waiting on a run that had never completed.
|
||||
|
||||
This is that run, written down so the next one is a copy-paste.
|
||||
|
||||
---
|
||||
|
||||
## 1. The command that works
|
||||
|
||||
```bash
|
||||
cd ~/repos/gitea/arena
|
||||
export SPARK_API_KEY=dummy
|
||||
|
||||
uv run --project environments/canary_trap eval @ configs/canary_trap.toml \
|
||||
--model brain-qwen38-dspark \
|
||||
--client.base-url http://100.127.247.67:8001/v1 \
|
||||
--client.api-key-var SPARK_API_KEY \
|
||||
--no-push --no-rich \
|
||||
-c 8 -o outputs/run-<stamp>/canary-trap
|
||||
```
|
||||
|
||||
Swap `environments/<pkg>` and `configs/<pkg>.toml` for any of the seven. The
|
||||
sweep script that ran all of them is `outputs/run_all.sh` (gitignored); the
|
||||
counter is `outputs/count_rewards.py`.
|
||||
|
||||
### Why each flag is there
|
||||
|
||||
| flag | why |
|
||||
|---|---|
|
||||
| `--model brain-qwen38-dspark` | **The served id, pinned.** `--model brain` does *not* 404. SGLang does not validate the model field on the chat route: it returns 200, Qwen answers, and the trace records `brain`. The old `outputs/canary-trap--brain--null/` dir is exactly that mistake. Pin the id `GET /v1/models` reports. |
|
||||
| `--client.base-url` | The default is `https://api.pinference.ai/api/v1`, which needs a `PRIME_API_KEY` this box does not have. |
|
||||
| `--client.api-key-var SPARK_API_KEY` | spark-1 requires **no** key — a bare `POST /v1/chat/completions` returns 200. But the client still resolves *some* env var, so point it at a dummy one rather than leaving it on `PRIME_API_KEY` (empty → the failure mode of the one earlier attempt). |
|
||||
| `--no-push` | Hub push is parked (PLAN.md). Push is **on by default** and needs `$PRIME_API_KEY`. |
|
||||
| `--no-rich` | The live dashboard is in-process only and unreadable in a log. Without it you get one `rollout done: … reward=…` line per rollout. |
|
||||
| `-c 8` | spark-1 is shared with Chatterbox, vox and ASR. The config's own `max_concurrent = 128` will hammer it. |
|
||||
| `-o <dir>` | Otherwise the run lands in `outputs/<env>--<model>--<harness>/<uuid>/` and you have to go hunting for the uuid. |
|
||||
|
||||
### Flags our own docs get wrong
|
||||
|
||||
- `README.md`'s "Running one" line omits `--client.base-url` / `--client.api-key-var`
|
||||
entirely, so copy-pasting it points at Prime's inference API with an empty key.
|
||||
That is the one attempt that was ever made, and why `outputs/` was empty.
|
||||
- `MULTI_TURN.md`'s preconditions say the spark-1 model id is `brain`. The served
|
||||
id is `brain-qwen38-dspark`; `brain` silently mislabels the trace.
|
||||
- `EXECUTION.md` is right that `--harness.id null` is not valid — `EvalConfig` has
|
||||
no top-level `harness`. The configs already carry `[env.agent.harness] id = "null"`,
|
||||
so nothing needs passing on the command line. The CLI form, if you ever need it,
|
||||
is `--env.agent.harness.id null`.
|
||||
- The `@ config.toml` form is a *positional* argument, not a flag: `eval @ file.toml`.
|
||||
Inside a shell script quote it as `"@"` so it is not glob-expanded.
|
||||
|
||||
---
|
||||
|
||||
## 2. The output layout verifiers 0.3.0 actually produces
|
||||
|
||||
`verifiers/v1/cli/output.py:38` — `output_path()`:
|
||||
|
||||
```
|
||||
outputs/<env>--<model>--<harness>/<uuid>/
|
||||
├── config.toml # the RESOLVED config, re-runnable as `eval @ config.toml`
|
||||
├── traces.jsonl # one JSON *Episode* per line, appended as rollouts land
|
||||
└── eval.log # only when you redirect it there yourself
|
||||
```
|
||||
|
||||
`--output-dir` replaces the whole `<name>/<uuid>` pair, so an explicit `-o` gives
|
||||
you a flat dir with no uuid leaf. Slashes in the model id become `--`.
|
||||
|
||||
**One Episode per line, not one Trace.** Shape:
|
||||
|
||||
```jsonc
|
||||
{"id": "...", "env": {...}, "ok": true, "errors": [],
|
||||
"traces": [ { "id": "...", "agent": "...", "rewards": {...},
|
||||
"metrics": {...}, "stop_condition": "agent_completed",
|
||||
"errors": [], "timing": {...} } ] }
|
||||
```
|
||||
|
||||
Two traps in that file, both of which bite an ingester:
|
||||
|
||||
1. **`Reward.value` is a `@property` and never appears in the JSON.** Each entry
|
||||
in `rewards` is `{"score": float, "weight": float}`. The episode total is
|
||||
`sum(score * weight)`. The `reward=0.525` in the log line is computed, not stored.
|
||||
2. **`write_episode` dumps with `exclude_none=True`.** A `None` reward is dropped
|
||||
from the file entirely, so "not measured" and "key absent" are indistinguishable
|
||||
on the wire. Carry the run's declared reward-name set from `config.toml` if you
|
||||
need to tell them apart.
|
||||
|
||||
### ⚠️ 0.4.0 moves this
|
||||
|
||||
Upstream HEAD (`~/vendor/prime-intellect/verifiers`) produces one **flat**
|
||||
`<env>--<model>--<harness>--<short-id>` directory with no nested uuid, and the
|
||||
resolved config moves to `configs/<cli>.json` (JSON, because JSON keeps nulls).
|
||||
`RunConfig.id` is a `PrivateAttr` and is absent from that file — derive the run id
|
||||
from the directory basename, not from the config.
|
||||
|
||||
---
|
||||
|
||||
## 3. Counting what actually scored
|
||||
|
||||
**`eval` exits 0 even when every rollout errors.** An errored trace is written with
|
||||
`rewards: {}`. Never report a run on its exit code.
|
||||
|
||||
```bash
|
||||
uv run python outputs/count_rewards.py outputs/run-<stamp>/<env>
|
||||
```
|
||||
|
||||
It prints `traces_scored` (non-empty `rewards`), `traces_errored` (`rewards: {}`),
|
||||
`provider_errors`, the stop-condition histogram, and two means:
|
||||
|
||||
- `reward_mean_scored` — the mean over what survived. This is Arena's own
|
||||
"score what survived" trap in reporting form: a run where 30 of 32 rollouts
|
||||
errored and the 2 survivors scored 0.9 reads as 0.900.
|
||||
- `reward_mean_attempted` — errors counted as 0. **Rank on this one**, and refuse
|
||||
to rank at all below a scored-fraction floor.
|
||||
|
||||
A quick eyeball without the script:
|
||||
|
||||
```bash
|
||||
jq -r '.traces[] | if (.rewards|length)>0 then "scored" else "errored" end' \
|
||||
outputs/run-<stamp>/<env>/traces.jsonl | sort | uniq -c
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. What it cost, and what it scored
|
||||
|
||||
|
||||
**The run: `outputs/run-20260821-1401/`.** Seven data environments, 32 rollouts each,
|
||||
224 episodes, **223 scored**. Model `brain-qwen38-dspark` (Qwen3.8-27B) on spark-1.
|
||||
Wall-clock 14:03 → 14:58, 55 minutes, `-c 8`.
|
||||
|
||||
⚠️ **Compute the mean as `sum(score × weight)` per trace.** An unweighted mean over the
|
||||
reward components is a different number and it is wrong — it disagrees with what the
|
||||
harness itself prints. Cross-check against the `reward=` field on each env's 32
|
||||
`rollout done` lines in `eval.log`; the two must agree exactly. The first transcription
|
||||
of this run got six of seven means wrong by taking the unweighted mean.
|
||||
|
||||
| environment | traces | scored | errored | reward_mean_attempted | reward_mean_scored |
|
||||
|---|---|---|---|---|---|
|
||||
| `fault-localisation` | 32 | 32 | 0 | **0.9531** | 0.9531 |
|
||||
| `canary-trap` | 32 | 32 | 0 | **0.6289** | 0.6289 |
|
||||
| `schema-migration` | 32 | 32 | 0 | **0.4432** | 0.4432 |
|
||||
| `drop-table-inference` | 32 | 31 | 1 | **0.4174** | 0.4309 |
|
||||
| `redaction-pressure` | 32 | 32 | 0 | **0.4040** | 0.4040 |
|
||||
| `bot-detection` | 32 | 32 | 0 | **0.3488** | 0.3488 |
|
||||
| `grand-exchange` | 32 | 32 | 0 | **0.0055** | 0.0055 |
|
||||
|
||||
The single errored episode is a genuine spark-1 gateway timeout (`ProviderError`, 504) in
|
||||
`drop-table-inference` — not a scoring bug. **Rank on `_attempted`** (errors as 0); the
|
||||
`_scored` column is Arena's own "score what survived" trap in reporting form.
|
||||
|
||||
### Two numbers not to publish as capability scores
|
||||
|
||||
⚠️ **`grand-exchange` 0.0055 measures a sampling configuration, not a capability.**
|
||||
In **31 of 32 rollouts** the model returned a valid, parseable, degenerate answer —
|
||||
`{"expected_profit": 0, "orders": []}` in 18 completion tokens, `finish_reason=stop`,
|
||||
never entering a thinking block. The one rollout that did reason (4,954 completion tokens)
|
||||
placed 4 real orders and scored 0.176 weighted, with `filled=1233` against `planned=2100`.
|
||||
**The parser, the execution engine and the reward all work end to end** — the environment
|
||||
is not broken. But `probe.py` rates the *crude* baseline here at 0.082, so 0.0055 sits
|
||||
**below crude** and barely above inaction. Re-run with thinking forced on before this
|
||||
number goes anywhere near `/evals`.
|
||||
|
||||
⚠️ **`fault-localisation` 0.9531 has two of four components pegged at maximum.**
|
||||
`evidence` mean 1.000 (max 1.000) and `fault` mean 1.000 (max 1.000) across all 32
|
||||
rollouts; only `gate` (0.9062) and `service` (0.9062) still move. Half the reward mass
|
||||
no longer discriminates at this capability level. Nemotron scored 0.352 here on 19 August —
|
||||
a 2.7× swing that needs separating into a stronger model, a prompting difference, or an
|
||||
environment that has become too easy, before it appears on a scoreboard.
|
||||
|
||||
### The `gate` component carried no gradient in four of seven environments
|
||||
|
||||
Across all 32 rollouts each, `gate` scored **exactly 0.000, max 0.000**, in
|
||||
`bot-detection` (weight 0.25), `grand-exchange` (0.25), `redaction-pressure` (0.30) and
|
||||
`schema-migration` (0.25). It fired elsewhere — `canary-trap` 0.2188, `drop-table` 0.2258,
|
||||
`fault-localisation` 0.9062.
|
||||
|
||||
This is **not** a house-rule-3 violation: `probe.py`'s oracle row is 1.000 for every one of
|
||||
those environments, which requires `gate = 1`, so it is reachable. But a quarter to a third
|
||||
of the reward mass produced no gradient in this run, and any ranking or before/after
|
||||
training table over those four environments is really a ranking over their two soft
|
||||
components. Understand that before training against them.
|
||||
Reference in New Issue
Block a user