arena/docs/FIRST_EVAL.md

Running an eval — the runbook

Written 2026-08-21, the day Arena first measured anything end to end. Before this, outputs/ held one config.toml and zero episodes: every number downstream of an eval — the dashboard, eval-results.json, the leaderboard, the before/after training table — was waiting on a run that had never completed.

This is that run, written down so the next one is a copy-paste.

⚠️ Updated 2026-08-25. The pin is now the stable verifiers==0.3.1 release published 2026-08-24. It removes the 600-second model-call ceiling §5 blames for every truncation here, and moves the output layout. §1 and §2 are the post-bump instructions; §4 and §5 are kept as written because they are the record of what the thinking-off run measured, and §6 is the thinking-on re-measurement the bump made possible. Every reward number in this file now carries its sampling configuration.

---

1. The command that works

cd ~/repos/gitea/arena
export SPARK_API_KEY=dummy
# The OpenAI-compatible endpoint of the reference node, e.g. http://<host>:8001/v1.
# Kept out of this repo on purpose: it is an internal address, not a public one.
export SPARK_BASE_URL=...

uv run --project environments/canary_trap eval @ configs/canary_trap.toml \
  --model brain-qwen38-dspark \
  --client.base-url "$SPARK_BASE_URL" \
  --client.api-key-var SPARK_API_KEY \
  --env.agent.runtime.type subprocess \
  --no-push --no-rich \
  -c 8 -o outputs/run-<stamp> --run.dir canary-trap

Swap environments/<pkg> and configs/<pkg>.toml for any of the eight in configs/. The thinking-on sweep is outputs/thinking-n32/run_all.sh (gitignored, and it takes a flock — see below); the counters are outputs/count_rewards.py and outputs/summarise_run.py.

Thinking is off unless you ask for it, and asking goes in a config file, not a flag — see §5. The seven data environments have thinking-on copies in outputs/thinking-n32/*.toml; §6 is what they measured.

Why each flag is there

flag	why
--model brain-qwen38-dspark	The served id, pinned. --model brain does not 404. SGLang does not validate the model field on the chat route: it returns 200, Qwen answers, and the trace records brain. The old outputs/canary-trap--brain--null/ dir is exactly that mistake. Pin the id GET /v1/models reports.
--client.base-url	The default is https://api.pinference.ai/api/v1, which needs a PRIME_API_KEY this box does not have.
--client.api-key-var SPARK_API_KEY	The reference node does not authenticate with a Prime key. The client still resolves some env var, so point it at a dummy one rather than leaving it on PRIME_API_KEY (empty → the failure mode of the one earlier attempt).
--no-push	Hub push is parked (PLAN.md). Push is on by default and needs $PRIME_API_KEY.
--no-rich	The live dashboard is in-process only and unreadable in a log. Without it you get one rollout done: … reward=… line per rollout.
-c 8	The reference node is shared with Chatterbox, vox and ASR. The config's own max_concurrent = 128 will hammer it.
--env.agent.runtime.type subprocess	⚠️ New, and not optional. 0.3.0 defaulted the agent runtime to subprocess; 0.3.1 defaults it to prime — a remote sandbox. A remote role makes Env._runs_local() false, which makes the interception pool mint a Prime tunnel, which calls ensure_prime_auth() and raises SystemExit: not authenticated with prime. The run dies before a single rollout, having already created its output directory and an empty traces.jsonl. The null harness needs no runtime at all; naming a local one is the whole fix.
-o <dir> + --run.dir <name>	-o is output_dir and the run writes to output_dir / run.dir. On 0.3.0 a bare -o gave you a flat directory; on 0.3.1 it gives you outputs/<your-dir>/<env>--<model>--<harness>--<hex8>/. Pass --run.dir to name the leaf yourself.

Flags our own docs get wrong

• README.md's "Running one" line omits --client.base-url / --client.api-key-var entirely, so copy-pasting it points at Prime's inference API with an empty key. That is the one attempt that was ever made, and why outputs/ was empty.
• MULTI_TURN.md's preconditions say the the reference node model id is brain. The served id is brain-qwen38-dspark; brain silently mislabels the trace.
• EXECUTION.md is right that --harness.id null is not valid — EvalConfig has no top-level harness. The configs already carry [env.agent.harness] id = "null", so nothing needs passing on the command line. The CLI form, if you ever need it, is --env.agent.harness.id null.
• The @ config.toml form is a positional argument, not a flag: eval @ file.toml. Inside a shell script quote it as "@" so it is not glob-expanded.

---

2. The pin, and the output layout it produces

There is no verifiers 0.4.0

~/vendor/prime-intellect/verifiers is not a 0.4.0 release, and nothing on any index is. git describe --tags there says v0.3.0-59-g4bcb48e5: fifty-nine commits past the 0.3.0 tag, version derived from git tags by hatch-vcs. The version = "0.4.0" that earlier notes read off its pyproject.toml:136 belongs to a [[tool.uv.dependency-metadata]] block for nemo-gym, not to verifiers. Anyone who goes looking for verifiers==0.4.0 will not find it.

The stable 0.3.1 wheel was published to PyPI on 2026-08-24 after the dev stream. It contains the unbounded model-call timeout and output-layout changes this runbook requires; verifiers/v1/clients/base.py reads:

DEFAULT_TIMEOUT = httpx.Timeout(connect=5.0, read=None, write=None, pool=None)

So the pin is an ordinary index dependency, not a path or git source. Every environments/*/pyproject.toml now reads verifiers==0.3.1. It is an ordinary stable index dependency — no --prerelease=allow, no [tool.uv.sources], no tool.uv block, and nothing that resolves differently in CI than it does here. A path dependency on ~/vendor would have been reproducible only on this box; a git URL would have pinned a commit no wheel matches. Neither was necessary.

Resolved 114 packages in 620ms
 - verifiers==0.3.0
 + verifiers==0.3.1

uv sync each environment after pulling. Verify the thing you bumped for, not the version string:

for d in environments/*/; do printf "%-34s " "$d"
  grep -h "DEFAULT_TIMEOUT = " $d/.venv/lib/python*/site-packages/verifiers/v1/clients/base.py
done
# environments/bot_detection/       DEFAULT_TIMEOUT = httpx.Timeout(connect=5.0, read=None, write=None, pool=None)
# ... eight of eight

uv sync rewrites each environments/*/uv.lock; none of them are tracked any more (.gitignore:5 is environments/*/uv.lock, and git ls-files environments | grep uv.lock is empty), so the bump does not put a lock file into a commit. CI's lock-policy step is test -z "$(find environments -name uv.lock -print -quit)" — it looks on disk, not in git, so it passes on a fresh checkout and would fail on a working tree that has been synced. Do not run it locally and conclude the repo is broken.

The layout 0.3.1 writes

verifiers/v1/cli/output.py — output_path() is config.output_dir / config.run.dir, and run.dir defaults to run.name, which is <env>--<model>--<harness>--<uuid4-hex8>, lowercased, with slashes in the model id becoming --. Measured, not read off the source:

outputs/<output-dir>/<env>--<model>--<harness>--<hex8>/
├── configs/eval.json   # the RESOLVED config, re-runnable as `eval @ configs/eval.json`
├── traces.jsonl        # one JSON *Episode* per line, appended as rollouts land
└── logs/eval.log       # written for you now — no redirect needed

Three differences from 0.3.0 that an ingester or a script has to handle:

1. The uuid leaf is gone. 0.3.0 wrote <env>--<model>--<harness>/<uuid>/; 0.3.1 writes one flat directory with an 8-hex suffix on the name. -o no longer flattens anything — it sets output_dir, and the run still makes its own leaf underneath. --run.dir <name> is what pins the leaf, and it is what outputs/thinking-n32/run_all.sh uses.
2. config.toml → configs/eval.json. JSON because JSON keeps nulls, so an explicit None round-trips on re-parse. RunConfig._id is a PrivateAttr and is not in it — derive the run id from the directory basename.
3. logs/eval.log is written unconditionally. On 0.3.0 you only got it if you redirected stdout yourself, which is why outputs/run-20260821-1401/*.log sit beside their run dirs rather than inside them. Anything that globs for a log should look in both places.
4. --resume exists now, and it is the answer to §5's "a timed-out run leaves no evidence": uv run eval @ <run-dir>/configs/eval.json --resume re-runs only the missing and errored rollouts, in place. (Its own --help still says @ <run-dir>/config.toml — the 0.3.0 filename. The file is configs/eval.json.)

The episode schema did not move, despite 4bcb48e5 feat(v1): make episodes training-native landing in this range. One Episode per line, not one Trace:

{"id": "...", "env": {...}, "ok": true, "errors": [],
 "traces": [ { "id": "...", "agent": "...", "rewards": {...},
               "metrics": {...}, "stop_condition": "agent_completed",
               "errors": [], "timing": {...} } ] }

outputs/count_rewards.py, written against 0.3.0, reproduces every published number from run-20260821-1401 unchanged against a 0.3.1 run. The two traps in that file are also unchanged, and both still bite an ingester:

1. Reward.value is a @property and never appears in the JSON. Each entry in rewards is {"score": float, "weight": float}. The episode total is sum(score * weight). The reward=0.525 in the log line is computed, not stored.
2. write_episode dumps with exclude_none=True. A None reward is dropped from the file entirely, so "not measured" and "key absent" are indistinguishable on the wire. Carry the run's declared reward-name set from the resolved config if you need to tell them apart.

What the bump does not fix

sampling in configs/eval.json now carries chat_template_kwargs through resolution, so a run's own config records whether thinking was requested. The trace still does not. Every calls[].sampling is {} — ChatDialect.parse_sampling keeps an allow-list of known keys — and usage.reasoning_tokens is still null, because SGLang reports it at the top level of usage while Usage.from_openai reads it out of completion_tokens_details. The only in-trace evidence that a model thought is a reasoning_content field on the assistant message. outputs/summarise_run.py counts exactly that, and prints the run's resolved sampling beside every mean it reports.

3. Counting what actually scored

eval exits 0 even when every rollout errors. An errored trace is written with rewards: {}. Never report a run on its exit code.

uv run python outputs/count_rewards.py outputs/run-<stamp>/<env>

It prints traces_scored (non-empty rewards), traces_errored (rewards: {}), provider_errors, the stop-condition histogram, and two means:

• reward_mean_scored — the mean over what survived. This is Arena's own "score what survived" trap in reporting form: a run where 30 of 32 rollouts errored and the 2 survivors scored 0.9 reads as 0.900.
• reward_mean_attempted — errors counted as 0. Rank on this one, and refuse to rank at all below a scored-fraction floor.

A quick eyeball without the script:

jq -r '.traces[] | if (.rewards|length)>0 then "scored" else "errored" end' \
  outputs/run-<stamp>/<env>/traces.jsonl | sort | uniq -c

And under what sampling configuration

count_rewards.py answers did anything score. It does not answer the question that made run-20260821-1401 unpublishable: how was it sampled. outputs/summarise_run.py adds that, and nothing else you cannot get from the first script:

uv run --project environments/<pkg> python outputs/summarise_run.py outputs/<run-dir>

It reads the run's own resolved config (configs/eval.json on dev59, config.toml on 0.3.0) and prints, beside the means:

• sampling — the run's own record of max_tokens / chat_template_kwargs. A run whose sampling is {} was sampled at the reference node's server-side default, which is enable_thinking: false.
• traces_with_reasoning — assistant messages carrying reasoning_content (or a literal <think> block). This is the only in-trace evidence that the model was allowed to think. Verify the flag from here, never from the fact that you passed it.
• finish_reasons and the completion-token quartiles — length in that histogram means truncation, and a truncated answer scores whatever an empty answer scores.
• reward_mean_from_eval_log — the mean of the reward= field on the harness's own rollout done lines. It must equal reward_mean_scored exactly. An unweighted mean over reward components does not, which is how six of seven means were misreported once.

Its arithmetic is checked against the published run: it reproduces fault-localisation 0.9531, canary-trap 0.6289, grand-exchange 0.0055 and drop-table-inference 0.4174 from run-20260821-1401, with reward_mean_from_eval_log agreeing in every case, and traces_with_reasoning 0/32 in every case.

---

4. What it cost, and what it scored

The run: outputs/run-20260821-1401/. Seven data environments, 32 rollouts each, 224 episodes, 223 scored. Model brain-qwen38-dspark (Qwen3.8-27B) on the reference node. Wall-clock 14:03 → 14:58, 55 minutes, -c 8.

⚠️ Sampling configuration for every number in this section — read it before quoting one. verifiers==0.3.0; sampling = {}, i.e. no max_tokens and no chat_template_kwargs, so the reference node's server-side default applied and enable_thinking was false; temperature, top_p and reasoning_effort all unset; -c 8; harness null; agent runtime subprocess. traces_with_reasoning is 0 of 224. These are thinking-off numbers. §6 re-measures the same eight environments with thinking on, and the two columns are not interchangeable — grand-exchange moves by two orders of magnitude between them.

⚠️ Compute the mean as sum(score × weight) per trace. An unweighted mean over the reward components is a different number and it is wrong — it disagrees with what the harness itself prints. Cross-check against the reward= field on each env's 32 rollout done lines in eval.log; the two must agree exactly. The first transcription of this run got six of seven means wrong by taking the unweighted mean.

environment	traces	scored	errored	reward_mean_attempted	reward_mean_scored
fault-localisation	32	32	0	0.9531	0.9531
canary-trap	32	32	0	0.6289	0.6289
schema-migration	32	32	0	0.4432	0.4432
drop-table-inference	32	31	1	0.4174	0.4309
redaction-pressure	32	32	0	0.4040	0.4040
bot-detection	32	32	0	0.3488	0.3488
grand-exchange	32	32	0	0.0055	0.0055

The single errored episode is a genuine the reference node gateway timeout (ProviderError, 504) in drop-table-inference — not a scoring bug. Rank on _attempted (errors as 0); the _scored column is Arena's own "score what survived" trap in reporting form.

Two numbers not to publish as capability scores

⚠️ grand-exchange 0.0055 measures a sampling configuration, not a capability. In 31 of 32 rollouts the model returned a valid, parseable, degenerate answer (27 of them in exactly 18 completion tokens; the full distribution is {18: 27, 19: 1, 24: 3, 4954: 1}) — {"expected_profit": 0, "orders": []} in 18 completion tokens, finish_reason=stop, never entering a thinking block. The one rollout that did reason (4,954 completion tokens) placed 4 real orders and scored 0.176 weighted, with filled=1233 against planned=2100. The parser, the execution engine and the reward all work end to end — the environment is not broken. But probe.py rates the crude baseline here at 0.082, so 0.0055 sits below crude and barely above inaction. Re-run with thinking forced on before this number goes anywhere near /evals. That re-run happened — see §5. With thinking on the same model scored 1.000 on every component of the one rollout that finished inside its token budget.

⚠️ drop-table-inference 0.4174 is two populations, not one. Twelve of its 31 scored rollouts answered in 116–174 tokens and averaged 0.0894; the other nineteen wrote 4,585–11,034 tokens and averaged 0.6465. §5 has the distribution.

⚠️ fault-localisation 0.9531 has two of four components pegged at maximum. evidence mean 1.000 (max 1.000) and fault mean 1.000 (max 1.000) across all 32 rollouts; only gate (0.9062) and service (0.9062) still move. Half the reward mass no longer discriminates at this capability level. Nemotron scored 0.352 here on 19 August — a 2.7× swing that needs separating into a stronger model, a prompting difference, or an environment that has become too easy, before it appears on a scoreboard.

The gate component carried no gradient in four of seven environments

Across all 32 rollouts each, gate scored exactly 0.000, max 0.000, in bot-detection (weight 0.25), grand-exchange (0.25), redaction-pressure (0.30) and schema-migration (0.25). It fired elsewhere — canary-trap 0.2188, drop-table 0.2258, fault-localisation 0.9062.

This is not a house-rule-3 violation: probe.py's oracle row is 1.000 for every one of those environments, which requires gate = 1, so it is reachable. But a quarter to a third of the reward mass produced no gradient in this run, and any ranking or before/after training table over those four environments is really a ranking over their two soft components. Understand that before training against them.

---

5. The run was measured with thinking OFF, and nothing in the run says so

Appended 2026-08-21, after re-measuring grand-exchange.

The cause

The reference node's SGLang is started with a default that overrides the chat template:

curl -s "${SPARK_BASE_URL%/v1}/get_server_info" | jq '.default_chat_template_kwargs, .reasoning_parser'
# {"enable_thinking": false}
# "qwen3"

Every one of the 224 episodes in run-20260821-1401 was sampled with Qwen3's thinking mode disabled, and not one of them requested otherwise. Confirmed from the traces: no assistant message in any of the seven environments carries a reasoning_content field, and none contains a <think> block — 0 of 223 scored traces.

Nothing in the run's own artefacts records this. The resolved config.toml says sampling = {}, and ChatDialect.parse_sampling keeps only an allow-list of known keys, so even a run that does pass chat_template_kwargs writes a calls[].sampling that does not mention it. usage.reasoning_tokens is null too: SGLang reports reasoning_tokens at the top level of usage, while Usage.from_openai reads it out of completion_tokens_details. A trace in this repository cannot tell you whether the model was allowed to think.

Turning it on

SamplingConfig is extra="allow" and ChatDialect.apply_overrides merges the whole dump into the request body, so an untyped key rides through to the provider untouched. There is no CLI flag — --sampling.* is typed — so it goes in a config file:

[sampling]
chat_template_kwargs = { enable_thinking = true }
max_tokens = 16384

outputs/thinking/grand_exchange_thinking.toml is that file. Verify it worked from the trace, not from the flag: the assistant message gains a reasoning_content field.

What thinking on actually did to grand-exchange

outputs/thinking/ge-think-n6/ — 6 tasks × 1 rollout, -c 2, max_tokens = 16384, thinking on. All 6 traces carry reasoning_content, so the flag reached the model.

tokens	finish_reason	content	weighted reward
16384	length	0 chars	0.000
16384	length	0 chars	0.000
16384	length	0 chars	0.000
16384	length	0 chars	0.000
16384	length	0 chars	0.000
8335	stop	1089 chars	1.000

Mean 0.1667 against 0.0055 with thinking off — a 30× move, and above probe.py's crude baseline of 0.082 for the first time. But read the column, not the mean: it is one perfect run and five truncations.

The run that finished scored 1.000 on every component — profit 1.000, discipline 1.000, gate 1.000 — with realised 40,081 gp against a reference_realised of 36,157, committed 0.994 of the purse and conversion 0.835. Four orders, a written rationale, orders_dropped 0:

{"expected_profit": 15000, "orders": [
  {"item": "Chipped bone charm",        "quantity": 400, "buy":  120, "sell":   134},
  {"item": "Coarse fletching feather",  "quantity": 700, "buy":   97, "sell":   114},
  {"item": "Emberglass shard",          "quantity":  80, "buy": 1420, "sell":  1640},
  {"item": "Heart of the sunken cairn", "quantity":   2, "buy": 9500, "sell": 13500}]}

⚠️ So the environment is not merely unbroken — it is fully solvable by this model, gate included, and the thinking-off number measured none of that. arith_ok was still 0.0 on that rollout (expected_profit 15,000 against an implied 40,184, arith_error 0.627), which is exactly the line the taskset docstring says it records for this reason.

The five truncations are not junk either. The reasoning is the analysis the environment asks for, item by item — counting crossings, applying the 25%-of-volume cap, sizing against the buy limit — it simply does not converge inside 16k:

**Heart of the sunken cairn**:
  Buy ticks (<=10000): 9267, 8846, 9656, 9541, 9703, 8853, 9945 = 7 ticks
  Volumes: 2, 2, 3, 6, 2, 3, 3   25%: 0.5, 0.5, 0.75, 1.5, 0.5, 0.75, 0.75
  Total: 5.25 in 56 ticks. In 30 ticks: ~2.8.   So I can only buy about 2-3 units!

A single rollout at -c 1 (outputs/thinking/smoke/) does the same: 486 seconds, completion_tokens 16384, reasoning_content 30,121 characters, content empty, reward 0.000. So does the same prompt sent straight at the reference node with curl, no verifiers in the path — reasoning_tokens 16390, finish_reason length, content empty. This is not a harness bug, and it is not fixed by raising max_tokens — see the next section for why.

⚠️ grand-exchange 0.0055 was a sampling artefact and 0.1667 is a truncation artefact. Neither is the environment's number. The number is somewhere at or above 0.1667, taken with a budget large enough that the model finishes, and this box cannot currently run that.

⚠️ The harness has a hard 600-second ceiling on one model call

verifiers/v1/clients/base.py:11:

DEFAULT_TIMEOUT = httpx.Timeout(connect=5.0, read=600.0, write=600.0, pool=600.0)

BaseClientConfig exposes base_url, api_key_var and headers and nothing else. There is no config key and no CLI flag for this timeout. At the reference node's ~34 tok/s single-stream it caps one call at roughly 20,000 completion tokens, and far fewer under concurrency.

That is not theoretical. outputs/thinking/ge-think-n8.log is 8 tasks at -c 8 with max_tokens = 32768:

16:09:27 INFO rollout start: ... (×8)
16:19:30 WARNING model call failed: id=... ProviderError:      (×8)

Eight failures, all at 16:19:30, exactly 603 seconds after the starts, with an empty ProviderError message and traces.jsonl left at zero lines. eval still exited 0. The -c 1 smoke run survived at 486 seconds; the same work at -c 8 did not.

-c 2 is not safe either. ge-think-n6.log at max_tokens = 16384 still logged 4 ProviderErrors in 6 rollouts, each exactly 601–603 seconds after its rollout started — 16:36:12 → 16:46:15, 16:46:15 → 16:56:16. Those rollouts survived only because the null harness's program retries the call itself (tenacity), which is why their traces carry two calls entries, the first with a null finish_reason. A retry that also lands at 16k tokens costs another ten minutes and scores zero anyway.

Raising max_tokens therefore does not fix the truncations. The five rollouts that ran out at 16,384 tokens want more budget; the wall says they cannot have it, because 16,384 tokens is already 486–603 seconds of wall clock on this box. The two limits point in opposite directions and there is no setting that satisfies both.

And the thinking block cannot be bounded from the request either. SGLang's thinking_budget is accepted (HTTP 200) and ignored by this build: the same prompt with {"max_tokens": 8192, "thinking_budget": 3000} came back with reasoning_tokens 8195, finish_reason length, content empty. The server also reports enable_strict_thinking: false. The only lever is max_tokens, and max_tokens cuts the answer, not the reasoning.

Three consequences for anyone re-running this suite against a reasoning model:

1. Keep -c low and max_tokens under the wall, or budget the run knowing calls will be cut at ten minutes. The reference node is shared — another lane's eval was running against it during this measurement, and contention alone moves a rollout across the line.
2. A timed-out run leaves no evidence. Zero traces, exit code 0. count_rewards.py reports on a file that does not exist. This is the same trap as §3 with a bigger blast radius: check the log for ProviderError and check wc -l traces.jsonl before believing any run.
3. ⚠️ CORRECTED: the 600-second ceiling is the INSTALLED WHEEL, not upstream. The remedy stated here first — "the 600-second default gets a config key upstream" — is wrong, and it is wrong in the expensive direction, because it reads as blocked on someone else. Upstream already removed it: verifiers/v1/clients/base.py:12 in ~/vendor/prime-intellect/verifiers reads httpx.Timeout(connect=5.0, read=None, ...), commit a298bcfe fix(v1): restore the unbounded model-call timeout (#2304), 2026-08-08. Every environments/*/.venv here pins read=600.0 because it resolves verifiers 0.3.0. The fix is a dependency bump, and an n=32 thinking-on run is achievable today. That makes the "grand-exchange is solvable" claim — currently resting on six rollouts — cheap to settle properly.

✅ RETIRED 2026-08-21 evening. The bump happened — verifiers==0.3.1, §2 — and every venv on this box now reads read=None. One correction to the paragraph above, which matters because it is the version anyone will type: the fix is not verifiers==0.4.0. There is no such release. a298bcfe is v0.3.0-8-ga298bcfe; the vendored HEAD is v0.3.0-59-g4bcb48e5; the wheel that carries them is 0.3.1. Everything below this line about 600 seconds describes the old pin and is kept as the record of why the bump was worth doing, not as current behaviour. The max_tokens / wall-clock contradiction it describes — "the two limits point in opposite directions and there is no setting that satisfies both" — no longer holds: with read=None the only limit left is max_tokens, and §6 spends it.

Did the other six collapse too?

Only one of them, and only partly. Completion tokens per trace, and the mean weighted reward in each half of the length distribution:

environment	min	p25	median	p75	max	shape
grand-exchange	18	18	18	18	4954	collapsed; distribution {18: 27, 19: 1, 24: 3, 4954: 1} — 27/32 at 18 tokens, and 31/32 planned no orders at all
drop-table-inference	0	159	5347	7205	11034	bimodal
fault-localisation	34	37	37	39	191	short by design
schema-migration	105	145	168	271	1687	unimodal
redaction-pressure	153	240	310	369	2014	unimodal
canary-trap	56	139	148	626	3128	unimodal
bot-detection	1156	2508	3457	4603	8146	unimodal

⚠️ drop-table-inference 0.4174 is a mixture of two regimes, not a capability estimate. Twelve of its 31 scored rollouts answered in 116–174 tokens and averaged 0.0894; the other nineteen wrote 4,585–11,034 tokens and averaged 0.6465. The same prompt, the same sampling settings, a 7× gap in outcome depending on whether the model chose to work. Report it with that split or not at all.

The other five are unimodal and their reward does not track length — fault-localisation answers three fields in 37 tokens and scores 0.9531, canary-trap's shortest half scores 0.6141 against its longest half's 0.6437. Those five numbers stand as measured. They are still thinking-off numbers, which is a property of the run nobody chose and nothing recorded.

What to do before the next run

• Put the sampling configuration in the run's own name or its notes. A number that changes by two orders of magnitude with one server-side default is not a capability score unless the configuration travels with it.
• Ask the reference node what it defaults to (/get_server_info) rather than assuming a model's own default applies.
• If reasoning_tokens matters to you, read it off the provider — the trace drops it.

---

6. The thinking-ON re-measurement, n=32

Run 2026-08-21 evening, after the verifiers==0.3.1 bump of §2. This section exists because two of §4's seven numbers were artefacts of a sampling configuration nobody chose and nothing recorded, and because §5's remedy — "an n=32 thinking-on run is achievable today" — was owed a run.

Sampling configuration for every number in this section. verifiers==0.3.1 (read=None); chat_template_kwargs = {enable_thinking = true}; max_tokens = 49152; temperature, top_p, reasoning_effort unset; -c 8; harness null; agent runtime subprocess; model brain-qwen38-dspark on the reference node. The configs are outputs/thinking-n32/*.toml, the sweep is outputs/thinking-n32/run_all.sh, the runs are outputs/thinking-n32/run/<env>/.

Thinking is verified from the traces, not from the flag. traces_with_reasoning is 32 of 32 for grand-exchange, mean 56,560 characters of reasoning_content per trace, against 0 of 224 for run-20260821-1401.

grand-exchange — 0.0055 → 0.4338

	thinking off (§4)	6 rollouts, 16k (§5)	thinking on, n=32
reward_mean_attempted	0.0055	0.1667	0.4338
traces scored	32/32	6/6	32/32
traces with reasoning	0/32	6/6	32/32
finish_reason=length	0	5/6	2/32
median completion tokens	18	16,384	29,376
gate mean (max)	0.0000 (0.000)	—	0.2188 (1.000)

probe.py rates the crude maximiser at 0.082 and the plausible strategy at 0.608. 0.4338 sits between them, which is the first time this environment has produced a number that means anything: 0.0055 was below crude, and 0.1667 was one perfect run and five truncations.

Wall clock 18:28 → 20:33, 2 h 5 min, 965,217 completion tokens for 32 rollouts. Zero ProviderErrors. Under the old pin every one of these rollouts would have died: the median call ran ~20 minutes against a 600-second ceiling, and the median completion is 29,376 tokens against the 16,384 that section 5 could afford.

The distribution is properly graded — not the two-spike shape thinking-off produced:

 total  gate  profit   disc  tokens finish
   0.0   0.0     0.0    0.0   20305 stop     <- 7 genuine zeros: finished, planned nothing usable
   ...
   0.0   0.0     0.0    0.0   49152 length   <- 2 truncations, both scored 0
 0.102   0.0   0.203  0.038   38913 stop
 0.162   0.0   0.302  0.087   30042 stop
 0.413   0.0   0.653  0.397   37978 stop
  0.56   0.0   0.746  0.746   22000 stop
 0.683   0.0    0.91   0.91   20886 stop
 0.747   0.0   0.996  0.996   32545 stop
 0.993   1.0     1.0  0.977   23531 stop
   1.0   1.0     1.0    1.0   20579 stop     <- 6 perfect rollouts

Nine zeros, six 1.000s, seventeen strictly between. Median 0.4165. Dropping the two truncations gives 0.4627 over the thirty that finished — quote 0.4338 and say why, rather than quoting the higher number.

⚠️ The dead gate in grand-exchange was partly a sampling artefact. docs/GATE_DIAGNOSIS.md records it at exactly 0.000, max 0.000, across all 32 thinking-off rollouts. With thinking on it fires on 7 of 32, mean 0.2188, max 1.000. That does not repair the shape — it is still exact equality, still all-or-nothing, and it still only opens for a rollout that is already at profit ≈ 1.0, so it adds no gradient anywhere below the top. But the diagnosis's evidence for this environment was measured under a configuration in which the model answered in 18 tokens, and the three other dead gates were measured under the same one. Re-read GATE_DIAGNOSIS against this column before acting on it.

drop-table-inference — thinking on does not fit in 49,152 tokens

The question §5 left was whether thinking on collapses this environment's two regimes (twelve short rollouts at 0.0894, nineteen long ones at 0.6465). It does not collapse them; it pushes the whole distribution off the end of the token budget. The first eight rollouts of the n=32 run, all eight carrying reasoning_content:

weighted  fit    gate   rare    tokens  finish
  0.4082  0.692  0.0    0.277    31649  stop
  0.4082  0.692  0.0    0.277    33803  stop
  0.0000  0.0    0.0    0.0      49152  length
  0.0000  0.0    0.0    0.0      49152  length
  0.0000  0.0    0.0    0.0      49152  length
  0.0000  0.0    0.0    0.0      49152  length
  0.0000  0.0    0.0    0.0      49152  length
  0.0000  0.0    0.0    0.0      49152  length

Six of eight hit finish_reason=length at 49,152 completion tokens — three times the budget §5 could afford, on a model whose reasoning here runs 74–86 thousand characters. The two that finished scored 0.408 each, above §4's short regime (0.0894) and below its long one (0.6465).

⚠️ Whatever mean this run finishes with is a truncation artefact, exactly like grand-exchange's 0.1667 was. Do not publish it as a capability score, and do not read it as evidence about the mixture. The comparison grand-exchange earned — a distribution where 30 of 32 finish — this environment has not: it needs a larger max_tokens (the reference node's context is 262,144, so there is room) and a correspondingly larger wall-clock budget. One 49,152-token rollout costs roughly 25 minutes at -c 8 on this box.

What is still running, and how to pick it up

outputs/thinking-n32/run_all.sh was launched detached (setsid) and works down the priority order in §6: grand-exchange (done), drop-table-inference, bot-detection, redaction-pressure, schema-migration, canary-trap, fault-localisation. It holds a flock on outputs/thinking-n32/.lock, so a second launch refuses rather than writing two evals into one traces.jsonl.

tail -f outputs/thinking-n32/sweep.log                  # stage transitions + per-rollout
grep 'exit=' outputs/thinking-n32/sweep.log             # which stages finished, with counts
uv run --project environments/<pkg> python outputs/summarise_run.py \
  outputs/thinking-n32/run/<env>                        # the numbers, with the sampling config

pkill -f 'thinking-n32/run_all.sh'                      # stop it after the current stage,
pkill -f 'bin/eval @ outputs/thinking-n32'              # then kill the eval in flight

⚠️ Read every stage's finish_reasons before its mean. length in that histogram means the model was cut off mid-answer, and a cut-off answer scores what an empty one scores. A stage whose histogram is mostly length has measured max_tokens, not the model — that is the single mistake this section exists to stop repeating.

Every number here is comparable only to the thinking-on column

grand-exchange moves by 79× between the two sampling configurations. Nothing in §4 and nothing in §6 may be tabulated beside the other without the configuration attached, and no number from either belongs on /evals or in eval-results.json without it. That is the whole lesson of the last two phases, stated as a rule:

A reward mean is not a number. It is a number plus a sampling configuration, and the pair travels or neither does.