Two measurement findings, both larger than the lanes that found them.
FIRST_EVAL.md: every number in the first run was sampled with thinking OFF —
0 of 224 traces carry reasoning_content or a <think> block, and spark-1 serves
with enable_thinking False. No artefact of the run recorded that. Five of the
seven numbers still stand; two do not. grand-exchange 0.0055 measured nothing
about the environment — with thinking on it scores 0.1667 on six rollouts and
one of them scored a clean 1.000, so it is fully solvable. drop-table-inference
0.4174 is a mixture of two regimes (0.0894 short, 0.6465 long) and should be
reported split or not at all. The defect is the missing sampling footnote, not
the values.
The 600-second per-call ceiling that blocked an n=32 thinking-on run is the
INSTALLED wheel, not upstream: verifiers fixed it in a298bcfe on 2026-08-08 and
0.3.0 predates it. The remedy is a dependency bump, not a wait.
GATE_DIAGNOSIS.md: the `gate` component scored exactly 0.000, max 0.000, across
all 32 rollouts in four environments. One shape explains three of them — exact
equality. Nothing below the oracle earns a fraction: not the plausible strategy,
not six of redaction's seven rules, not the oracle bot list minus one account.
bot-detection and schema-migration are a threshold set at the ceiling;
redaction-pressure is that and genuinely hard; grand-exchange's zero was the
sampling artefact above. The repo already contains the fix and already uses it
twice — drop-table's GATE_MARGIN and grand-exchange's TARGET_SHARE are margins,
not equalities. Recommended, not yet measured.
Consequence worth stating plainly: 25-30% of the reward mass on three
environments carries identically zero gradient, so they train against a
0.70-0.75 objective while being scored out of 1.00. The mirror failure exists
too — fault-localisation's evidence and fault components are pinned at exactly
1.0 on all 32 rollouts, so half its headline 0.9531 is constant.
tools/gate_probe.py is the reproduction, moved out of the gitignored outputs/
so the finding survives.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MULTI_TURN.md scopes the multi-turn gap and records the finding that made it
cheap: multi-turn needs no container and no real harness. vf.Env.run programs the
control flow host-side and the simulator runs in-process as the user, with the
harness left null.
PLAN.md carries the settled decisions. EXECUTION.md is the post-adversarial-review
plan — every position in it survived a reviewer, and where a design spec was
overturned the corrected position is what is written down.
Two corrections are marked in place rather than silently rewritten, because both
were believed and acted on:
- The provenance hole is src/arena/index.ts, not offices/sites.ts. Mutating
offices/sites.ts produces a bit-identical episode; index.ts re-exports the
scripted baselines that ARE the reward denominator, and a halved office-nav
baseline moves return 4.744 -> 4.098 while every tera hash gate prints ok.
- office-jobs-v1 is not phantom. It was absent at 13:00 and present at 15:50,
because another session fast-forward-merged it into the shared tera tree
mid-session. The counts are twelve, five and twenty.
docs/FIRST_EVAL.md is the runbook for the first evaluation Arena ever completed:
seven environments, 224 episodes, 223 scored, against brain-qwen38-dspark. It
warns that the mean is sum(score x weight) — an unweighted mean over components
is a different number, and it is how six of seven means were first misreported.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kartiandClaude Opus 5