arena/docs/FIRST_EVAL.md

Running an eval — the runbook

Written 2026-08-21, the day Arena first measured anything end to end. Before this, outputs/ held one config.toml and zero episodes: every number downstream of an eval — the dashboard, eval-results.json, the leaderboard, the before/after training table — was waiting on a run that had never completed.

This is that run, written down so the next one is a copy-paste.

---

1. The command that works

cd ~/repos/gitea/arena
export SPARK_API_KEY=dummy

uv run --project environments/canary_trap eval @ configs/canary_trap.toml \
  --model brain-qwen38-dspark \
  --client.base-url http://100.127.247.67:8001/v1 \
  --client.api-key-var SPARK_API_KEY \
  --no-push --no-rich \
  -c 8 -o outputs/run-<stamp>/canary-trap

Swap environments/<pkg> and configs/<pkg>.toml for any of the seven. The sweep script that ran all of them is outputs/run_all.sh (gitignored); the counter is outputs/count_rewards.py.

Why each flag is there

flag	why
--model brain-qwen38-dspark	The served id, pinned. --model brain does not 404. SGLang does not validate the model field on the chat route: it returns 200, Qwen answers, and the trace records brain. The old outputs/canary-trap--brain--null/ dir is exactly that mistake. Pin the id GET /v1/models reports.
--client.base-url	The default is https://api.pinference.ai/api/v1, which needs a PRIME_API_KEY this box does not have.
--client.api-key-var SPARK_API_KEY	spark-1 requires no key — a bare POST /v1/chat/completions returns 200. But the client still resolves some env var, so point it at a dummy one rather than leaving it on PRIME_API_KEY (empty → the failure mode of the one earlier attempt).
--no-push	Hub push is parked (PLAN.md). Push is on by default and needs $PRIME_API_KEY.
--no-rich	The live dashboard is in-process only and unreadable in a log. Without it you get one rollout done: … reward=… line per rollout.
-c 8	spark-1 is shared with Chatterbox, vox and ASR. The config's own max_concurrent = 128 will hammer it.
-o <dir>	Otherwise the run lands in outputs/<env>--<model>--<harness>/<uuid>/ and you have to go hunting for the uuid.

Flags our own docs get wrong

• README.md's "Running one" line omits --client.base-url / --client.api-key-var entirely, so copy-pasting it points at Prime's inference API with an empty key. That is the one attempt that was ever made, and why outputs/ was empty.
• MULTI_TURN.md's preconditions say the spark-1 model id is brain. The served id is brain-qwen38-dspark; brain silently mislabels the trace.
• EXECUTION.md is right that --harness.id null is not valid — EvalConfig has no top-level harness. The configs already carry [env.agent.harness] id = "null", so nothing needs passing on the command line. The CLI form, if you ever need it, is --env.agent.harness.id null.
• The @ config.toml form is a positional argument, not a flag: eval @ file.toml. Inside a shell script quote it as "@" so it is not glob-expanded.

---

2. The output layout verifiers 0.3.0 actually produces

verifiers/v1/cli/output.py:38 — output_path():

outputs/<env>--<model>--<harness>/<uuid>/
├── config.toml     # the RESOLVED config, re-runnable as `eval @ config.toml`
├── traces.jsonl    # one JSON *Episode* per line, appended as rollouts land
└── eval.log        # only when you redirect it there yourself

--output-dir replaces the whole <name>/<uuid> pair, so an explicit -o gives you a flat dir with no uuid leaf. Slashes in the model id become --.

One Episode per line, not one Trace. Shape:

{"id": "...", "env": {...}, "ok": true, "errors": [],
 "traces": [ { "id": "...", "agent": "...", "rewards": {...},
               "metrics": {...}, "stop_condition": "agent_completed",
               "errors": [], "timing": {...} } ] }

Two traps in that file, both of which bite an ingester:

1. Reward.value is a @property and never appears in the JSON. Each entry in rewards is {"score": float, "weight": float}. The episode total is sum(score * weight). The reward=0.525 in the log line is computed, not stored.
2. write_episode dumps with exclude_none=True. A None reward is dropped from the file entirely, so "not measured" and "key absent" are indistinguishable on the wire. Carry the run's declared reward-name set from config.toml if you need to tell them apart.

⚠️ 0.4.0 moves this

Upstream HEAD (~/vendor/prime-intellect/verifiers) produces one flat <env>--<model>--<harness>--<short-id> directory with no nested uuid, and the resolved config moves to configs/<cli>.json (JSON, because JSON keeps nulls). RunConfig.id is a PrivateAttr and is absent from that file — derive the run id from the directory basename, not from the config.

---

3. Counting what actually scored

eval exits 0 even when every rollout errors. An errored trace is written with rewards: {}. Never report a run on its exit code.

uv run python outputs/count_rewards.py outputs/run-<stamp>/<env>

It prints traces_scored (non-empty rewards), traces_errored (rewards: {}), provider_errors, the stop-condition histogram, and two means:

• reward_mean_scored — the mean over what survived. This is Arena's own "score what survived" trap in reporting form: a run where 30 of 32 rollouts errored and the 2 survivors scored 0.9 reads as 0.900.
• reward_mean_attempted — errors counted as 0. Rank on this one, and refuse to rank at all below a scored-fraction floor.

A quick eyeball without the script:

jq -r '.traces[] | if (.rewards|length)>0 then "scored" else "errored" end' \
  outputs/run-<stamp>/<env>/traces.jsonl | sort | uniq -c

---

4. What it cost, and what it scored

The run: outputs/run-20260821-1401/. Seven data environments, 32 rollouts each, 224 episodes, 223 scored. Model brain-qwen38-dspark (Qwen3.8-27B) on spark-1. Wall-clock 14:03 → 14:58, 55 minutes, -c 8.

⚠️ Compute the mean as sum(score × weight) per trace. An unweighted mean over the reward components is a different number and it is wrong — it disagrees with what the harness itself prints. Cross-check against the reward= field on each env's 32 rollout done lines in eval.log; the two must agree exactly. The first transcription of this run got six of seven means wrong by taking the unweighted mean.

environment	traces	scored	errored	reward_mean_attempted	reward_mean_scored
fault-localisation	32	32	0	0.9531	0.9531
canary-trap	32	32	0	0.6289	0.6289
schema-migration	32	32	0	0.4432	0.4432
drop-table-inference	32	31	1	0.4174	0.4309
redaction-pressure	32	32	0	0.4040	0.4040
bot-detection	32	32	0	0.3488	0.3488
grand-exchange	32	32	0	0.0055	0.0055

The single errored episode is a genuine spark-1 gateway timeout (ProviderError, 504) in drop-table-inference — not a scoring bug. Rank on _attempted (errors as 0); the _scored column is Arena's own "score what survived" trap in reporting form.

Two numbers not to publish as capability scores

⚠️ grand-exchange 0.0055 measures a sampling configuration, not a capability. In 31 of 32 rollouts the model returned a valid, parseable, degenerate answer (27 of them in exactly 18 completion tokens; the full distribution is {18: 27, 19: 1, 24: 3, 4954: 1}) — {"expected_profit": 0, "orders": []} in 18 completion tokens, finish_reason=stop, never entering a thinking block. The one rollout that did reason (4,954 completion tokens) placed 4 real orders and scored 0.176 weighted, with filled=1233 against planned=2100. The parser, the execution engine and the reward all work end to end — the environment is not broken. But probe.py rates the crude baseline here at 0.082, so 0.0055 sits below crude and barely above inaction. Re-run with thinking forced on before this number goes anywhere near /evals. That re-run happened — see §5. With thinking on the same model scored 1.000 on every component of the one rollout that finished inside its token budget.

⚠️ drop-table-inference 0.4174 is two populations, not one. Twelve of its 31 scored rollouts answered in 116–174 tokens and averaged 0.0894; the other nineteen wrote 4,585–11,034 tokens and averaged 0.6465. §5 has the distribution.

⚠️ fault-localisation 0.9531 has two of four components pegged at maximum. evidence mean 1.000 (max 1.000) and fault mean 1.000 (max 1.000) across all 32 rollouts; only gate (0.9062) and service (0.9062) still move. Half the reward mass no longer discriminates at this capability level. Nemotron scored 0.352 here on 19 August — a 2.7× swing that needs separating into a stronger model, a prompting difference, or an environment that has become too easy, before it appears on a scoreboard.

The gate component carried no gradient in four of seven environments

Across all 32 rollouts each, gate scored exactly 0.000, max 0.000, in bot-detection (weight 0.25), grand-exchange (0.25), redaction-pressure (0.30) and schema-migration (0.25). It fired elsewhere — canary-trap 0.2188, drop-table 0.2258, fault-localisation 0.9062.

This is not a house-rule-3 violation: probe.py's oracle row is 1.000 for every one of those environments, which requires gate = 1, so it is reachable. But a quarter to a third of the reward mass produced no gradient in this run, and any ranking or before/after training table over those four environments is really a ranking over their two soft components. Understand that before training against them.

---

5. The run was measured with thinking OFF, and nothing in the run says so

Appended 2026-08-21, after re-measuring grand-exchange.

The cause

spark-1's SGLang is started with a default that overrides the chat template:

curl -s http://100.127.247.67:8001/get_server_info | jq '.default_chat_template_kwargs, .reasoning_parser'
# {"enable_thinking": false}
# "qwen3"

Every one of the 224 episodes in run-20260821-1401 was sampled with Qwen3's thinking mode disabled, and not one of them requested otherwise. Confirmed from the traces: no assistant message in any of the seven environments carries a reasoning_content field, and none contains a <think> block — 0 of 223 scored traces.

Nothing in the run's own artefacts records this. The resolved config.toml says sampling = {}, and ChatDialect.parse_sampling keeps only an allow-list of known keys, so even a run that does pass chat_template_kwargs writes a calls[].sampling that does not mention it. usage.reasoning_tokens is null too: SGLang reports reasoning_tokens at the top level of usage, while Usage.from_openai reads it out of completion_tokens_details. A trace in this repository cannot tell you whether the model was allowed to think.

Turning it on

SamplingConfig is extra="allow" and ChatDialect.apply_overrides merges the whole dump into the request body, so an untyped key rides through to the provider untouched. There is no CLI flag — --sampling.* is typed — so it goes in a config file:

[sampling]
chat_template_kwargs = { enable_thinking = true }
max_tokens = 16384

outputs/thinking/grand_exchange_thinking.toml is that file. Verify it worked from the trace, not from the flag: the assistant message gains a reasoning_content field.

What thinking on actually did to grand-exchange

outputs/thinking/ge-think-n6/ — 6 tasks × 1 rollout, -c 2, max_tokens = 16384, thinking on. All 6 traces carry reasoning_content, so the flag reached the model.

tokens	finish_reason	content	weighted reward
16384	length	0 chars	0.000
16384	length	0 chars	0.000
16384	length	0 chars	0.000
16384	length	0 chars	0.000
16384	length	0 chars	0.000
8335	stop	1089 chars	1.000

Mean 0.1667 against 0.0055 with thinking off — a 30× move, and above probe.py's crude baseline of 0.082 for the first time. But read the column, not the mean: it is one perfect run and five truncations.

The run that finished scored 1.000 on every component — profit 1.000, discipline 1.000, gate 1.000 — with realised 40,081 gp against a reference_realised of 36,157, committed 0.994 of the purse and conversion 0.835. Four orders, a written rationale, orders_dropped 0:

{"expected_profit": 15000, "orders": [
  {"item": "Chipped bone charm",        "quantity": 400, "buy":  120, "sell":   134},
  {"item": "Coarse fletching feather",  "quantity": 700, "buy":   97, "sell":   114},
  {"item": "Emberglass shard",          "quantity":  80, "buy": 1420, "sell":  1640},
  {"item": "Heart of the sunken cairn", "quantity":   2, "buy": 9500, "sell": 13500}]}

⚠️ So the environment is not merely unbroken — it is fully solvable by this model, gate included, and the thinking-off number measured none of that. arith_ok was still 0.0 on that rollout (expected_profit 15,000 against an implied 40,184, arith_error 0.627), which is exactly the line the taskset docstring says it records for this reason.

The five truncations are not junk either. The reasoning is the analysis the environment asks for, item by item — counting crossings, applying the 25%-of-volume cap, sizing against the buy limit — it simply does not converge inside 16k:

**Heart of the sunken cairn**:
  Buy ticks (<=10000): 9267, 8846, 9656, 9541, 9703, 8853, 9945 = 7 ticks
  Volumes: 2, 2, 3, 6, 2, 3, 3   25%: 0.5, 0.5, 0.75, 1.5, 0.5, 0.75, 0.75
  Total: 5.25 in 56 ticks. In 30 ticks: ~2.8.   So I can only buy about 2-3 units!

A single rollout at -c 1 (outputs/thinking/smoke/) does the same: 486 seconds, completion_tokens 16384, reasoning_content 30,121 characters, content empty, reward 0.000. So does the same prompt sent straight at spark-1 with curl, no verifiers in the path — reasoning_tokens 16390, finish_reason length, content empty. This is not a harness bug, and it is not fixed by raising max_tokens — see the next section for why.

⚠️ grand-exchange 0.0055 was a sampling artefact and 0.1667 is a truncation artefact. Neither is the environment's number. The number is somewhere at or above 0.1667, taken with a budget large enough that the model finishes, and this box cannot currently run that.

⚠️ The harness has a hard 600-second ceiling on one model call

verifiers/v1/clients/base.py:11:

DEFAULT_TIMEOUT = httpx.Timeout(connect=5.0, read=600.0, write=600.0, pool=600.0)

BaseClientConfig exposes base_url, api_key_var and headers and nothing else. There is no config key and no CLI flag for this timeout. At spark-1's ~34 tok/s single-stream it caps one call at roughly 20,000 completion tokens, and far fewer under concurrency.

That is not theoretical. outputs/thinking/ge-think-n8.log is 8 tasks at -c 8 with max_tokens = 32768:

16:09:27 INFO rollout start: ... (×8)
16:19:30 WARNING model call failed: id=... ProviderError:      (×8)

Eight failures, all at 16:19:30, exactly 603 seconds after the starts, with an empty ProviderError message and traces.jsonl left at zero lines. eval still exited 0. The -c 1 smoke run survived at 486 seconds; the same work at -c 8 did not.

-c 2 is not safe either. ge-think-n6.log at max_tokens = 16384 still logged 4 ProviderErrors in 6 rollouts, each exactly 601–603 seconds after its rollout started — 16:36:12 → 16:46:15, 16:46:15 → 16:56:16. Those rollouts survived only because the null harness's program retries the call itself (tenacity), which is why their traces carry two calls entries, the first with a null finish_reason. A retry that also lands at 16k tokens costs another ten minutes and scores zero anyway.

Raising max_tokens therefore does not fix the truncations. The five rollouts that ran out at 16,384 tokens want more budget; the wall says they cannot have it, because 16,384 tokens is already 486–603 seconds of wall clock on this box. The two limits point in opposite directions and there is no setting that satisfies both.

And the thinking block cannot be bounded from the request either. SGLang's thinking_budget is accepted (HTTP 200) and ignored by this build: the same prompt with {"max_tokens": 8192, "thinking_budget": 3000} came back with reasoning_tokens 8195, finish_reason length, content empty. The server also reports enable_strict_thinking: false. The only lever is max_tokens, and max_tokens cuts the answer, not the reasoning.

Three consequences for anyone re-running this suite against a reasoning model:

1. Keep -c low and max_tokens under the wall, or budget the run knowing calls will be cut at ten minutes. spark-1 is shared — another lane's eval was running against it during this measurement, and contention alone moves a rollout across the line.
2. A timed-out run leaves no evidence. Zero traces, exit code 0. count_rewards.py reports on a file that does not exist. This is the same trap as §3 with a bigger blast radius: check the log for ProviderError and check wc -l traces.jsonl before believing any run.
3. ⚠️ CORRECTED: the 600-second ceiling is the INSTALLED WHEEL, not upstream. The remedy stated here first — "the 600-second default gets a config key upstream" — is wrong, and it is wrong in the expensive direction, because it reads as blocked on someone else. Upstream already removed it: verifiers/v1/clients/base.py:12 in ~/vendor/prime-intellect/verifiers reads httpx.Timeout(connect=5.0, read=None, ...), commit a298bcfe fix(v1): restore the unbounded model-call timeout (#2304), 2026-08-08. Every environments/*/.venv here pins read=600.0 because it resolves verifiers 0.3.0. The fix is a dependency bump, and an n=32 thinking-on run is achievable today. That makes the "grand-exchange is solvable" claim — currently resting on six rollouts — cheap to settle properly.

Did the other six collapse too?

Only one of them, and only partly. Completion tokens per trace, and the mean weighted reward in each half of the length distribution:

environment	min	p25	median	p75	max	shape
grand-exchange	18	18	18	18	4954	collapsed; distribution {18: 27, 19: 1, 24: 3, 4954: 1} — 27/32 at 18 tokens, and 31/32 planned no orders at all
drop-table-inference	0	159	5347	7205	11034	bimodal
fault-localisation	34	37	37	39	191	short by design
schema-migration	105	145	168	271	1687	unimodal
redaction-pressure	153	240	310	369	2014	unimodal
canary-trap	56	139	148	626	3128	unimodal
bot-detection	1156	2508	3457	4603	8146	unimodal

⚠️ drop-table-inference 0.4174 is a mixture of two regimes, not a capability estimate. Twelve of its 31 scored rollouts answered in 116–174 tokens and averaged 0.0894; the other nineteen wrote 4,585–11,034 tokens and averaged 0.6465. The same prompt, the same sampling settings, a 7× gap in outcome depending on whether the model chose to work. Report it with that split or not at all.

The other five are unimodal and their reward does not track length — fault-localisation answers three fields in 37 tokens and scores 0.9531, canary-trap's shortest half scores 0.6141 against its longest half's 0.6437. Those five numbers stand as measured. They are still thinking-off numbers, which is a property of the run nobody chose and nothing recorded.

What to do before the next run

• Put the sampling configuration in the run's own name or its notes. A number that changes by two orders of magnitude with one server-side default is not a capability score unless the configuration travels with it.
• Ask spark-1 what it defaults to (/get_server_info) rather than assuming a model's own default applies.
• If reasoning_tokens matters to you, read it off the provider — the trace drops it.