The 600-second per-model-call ceiling that capped the thinking-on run at n=6 is
gone: DEFAULT_TIMEOUT is read=None in all nine venvs.
There is NO verifiers 0.4.0. The `version = "0.4.0"` in the vendored tree belongs
to a [[tool.uv.dependency-metadata]] block for nemo-gym, and reading it as
verifiers' own sends you hunting a release that does not exist and concluding the
bump is blocked. The vendored checkout is v0.3.0-59-g4bcb48e5, published to PyPI
as 0.3.1.dev59, and the installed wheel diffs clean against the whole vendored
v1 tree. Pinned as an ordinary index dependency: a path source is reproducible
only on this box and a git URL pins a commit no wheel matches.
⚠️ dev59 changes the default agent runtime from subprocess to `prime`, a REMOTE
sandbox. Without an explicit runtime every eval here exits "not authenticated
with prime" before a single rollout — having already created its output directory
and a zero-line traces.jsonl, so it reads as a failed eval rather than an
unauthenticated one. All nine configs now say subprocess. Nothing here needs a
sandbox: the reward is a pure function of the trace and the harness is null.
Output layout is flat now — <env>--<model>--<harness>--<hex8>/ with configs/,
traces.jsonl and a logs/eval.log that 0.3.0 never wrote. `-o` is the output_dir,
not the run dir; `--run.dir` pins the leaf.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two measurement findings, both larger than the lanes that found them.
FIRST_EVAL.md: every number in the first run was sampled with thinking OFF —
0 of 224 traces carry reasoning_content or a <think> block, and spark-1 serves
with enable_thinking False. No artefact of the run recorded that. Five of the
seven numbers still stand; two do not. grand-exchange 0.0055 measured nothing
about the environment — with thinking on it scores 0.1667 on six rollouts and
one of them scored a clean 1.000, so it is fully solvable. drop-table-inference
0.4174 is a mixture of two regimes (0.0894 short, 0.6465 long) and should be
reported split or not at all. The defect is the missing sampling footnote, not
the values.
The 600-second per-call ceiling that blocked an n=32 thinking-on run is the
INSTALLED wheel, not upstream: verifiers fixed it in a298bcfe on 2026-08-08 and
0.3.0 predates it. The remedy is a dependency bump, not a wait.
GATE_DIAGNOSIS.md: the `gate` component scored exactly 0.000, max 0.000, across
all 32 rollouts in four environments. One shape explains three of them — exact
equality. Nothing below the oracle earns a fraction: not the plausible strategy,
not six of redaction's seven rules, not the oracle bot list minus one account.
bot-detection and schema-migration are a threshold set at the ceiling;
redaction-pressure is that and genuinely hard; grand-exchange's zero was the
sampling artefact above. The repo already contains the fix and already uses it
twice — drop-table's GATE_MARGIN and grand-exchange's TARGET_SHARE are margins,
not equalities. Recommended, not yet measured.
Consequence worth stating plainly: 25-30% of the reward mass on three
environments carries identically zero gradient, so they train against a
0.70-0.75 objective while being scored out of 1.00. The mirror failure exists
too — fault-localisation's evidence and fault components are pinned at exactly
1.0 on all 32 rollouts, so half its headline 0.9531 is constant.
tools/gate_probe.py is the reproduction, moved out of the gitignored outputs/
so the finding survives.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MULTI_TURN.md scopes the multi-turn gap and records the finding that made it
cheap: multi-turn needs no container and no real harness. vf.Env.run programs the
control flow host-side and the simulator runs in-process as the user, with the
harness left null.
PLAN.md carries the settled decisions. EXECUTION.md is the post-adversarial-review
plan — every position in it survived a reviewer, and where a design spec was
overturned the corrected position is what is written down.
Two corrections are marked in place rather than silently rewritten, because both
were believed and acted on:
- The provenance hole is src/arena/index.ts, not offices/sites.ts. Mutating
offices/sites.ts produces a bit-identical episode; index.ts re-exports the
scripted baselines that ARE the reward denominator, and a halved office-nav
baseline moves return 4.744 -> 4.098 while every tera hash gate prints ok.
- office-jobs-v1 is not phantom. It was absent at 13:00 and present at 15:50,
because another session fast-forward-merged it into the shared tera tree
mid-session. The counts are twelve, five and twenty.
docs/FIRST_EVAL.md is the runbook for the first evaluation Arena ever completed:
seven environments, 224 episodes, 223 scored, against brain-qwen38-dspark. It
warns that the mean is sum(score x weight) — an unweighted mean over components
is a different number, and it is how six of seven means were first misreported.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kartiandClaude Opus 5