# Running an eval — the runbook Written 2026-08-21, the day Arena first measured anything end to end. Before this, `outputs/` held one `config.toml` and zero episodes: every number downstream of an eval — the dashboard, `eval-results.json`, the leaderboard, the before/after training table — was waiting on a run that had never completed. This is that run, written down so the next one is a copy-paste. --- ## 1. The command that works ```bash cd ~/repos/gitea/arena export SPARK_API_KEY=dummy uv run --project environments/canary_trap eval @ configs/canary_trap.toml \ --model brain-qwen38-dspark \ --client.base-url http://100.127.247.67:8001/v1 \ --client.api-key-var SPARK_API_KEY \ --no-push --no-rich \ -c 8 -o outputs/run-/canary-trap ``` Swap `environments/` and `configs/.toml` for any of the seven. The sweep script that ran all of them is `outputs/run_all.sh` (gitignored); the counter is `outputs/count_rewards.py`. ### Why each flag is there | flag | why | |---|---| | `--model brain-qwen38-dspark` | **The served id, pinned.** `--model brain` does *not* 404. SGLang does not validate the model field on the chat route: it returns 200, Qwen answers, and the trace records `brain`. The old `outputs/canary-trap--brain--null/` dir is exactly that mistake. Pin the id `GET /v1/models` reports. | | `--client.base-url` | The default is `https://api.pinference.ai/api/v1`, which needs a `PRIME_API_KEY` this box does not have. | | `--client.api-key-var SPARK_API_KEY` | spark-1 requires **no** key — a bare `POST /v1/chat/completions` returns 200. But the client still resolves *some* env var, so point it at a dummy one rather than leaving it on `PRIME_API_KEY` (empty → the failure mode of the one earlier attempt). | | `--no-push` | Hub push is parked (PLAN.md). Push is **on by default** and needs `$PRIME_API_KEY`. | | `--no-rich` | The live dashboard is in-process only and unreadable in a log. Without it you get one `rollout done: … reward=…` line per rollout. | | `-c 8` | spark-1 is shared with Chatterbox, vox and ASR. The config's own `max_concurrent = 128` will hammer it. | | `-o ` | Otherwise the run lands in `outputs/----//` and you have to go hunting for the uuid. | ### Flags our own docs get wrong - `README.md`'s "Running one" line omits `--client.base-url` / `--client.api-key-var` entirely, so copy-pasting it points at Prime's inference API with an empty key. That is the one attempt that was ever made, and why `outputs/` was empty. - `MULTI_TURN.md`'s preconditions say the spark-1 model id is `brain`. The served id is `brain-qwen38-dspark`; `brain` silently mislabels the trace. - `EXECUTION.md` is right that `--harness.id null` is not valid — `EvalConfig` has no top-level `harness`. The configs already carry `[env.agent.harness] id = "null"`, so nothing needs passing on the command line. The CLI form, if you ever need it, is `--env.agent.harness.id null`. - The `@ config.toml` form is a *positional* argument, not a flag: `eval @ file.toml`. Inside a shell script quote it as `"@"` so it is not glob-expanded. --- ## 2. The output layout verifiers 0.3.0 actually produces `verifiers/v1/cli/output.py:38` — `output_path()`: ``` outputs/----// ├── config.toml # the RESOLVED config, re-runnable as `eval @ config.toml` ├── traces.jsonl # one JSON *Episode* per line, appended as rollouts land └── eval.log # only when you redirect it there yourself ``` `--output-dir` replaces the whole `/` pair, so an explicit `-o` gives you a flat dir with no uuid leaf. Slashes in the model id become `--`. **One Episode per line, not one Trace.** Shape: ```jsonc {"id": "...", "env": {...}, "ok": true, "errors": [], "traces": [ { "id": "...", "agent": "...", "rewards": {...}, "metrics": {...}, "stop_condition": "agent_completed", "errors": [], "timing": {...} } ] } ``` Two traps in that file, both of which bite an ingester: 1. **`Reward.value` is a `@property` and never appears in the JSON.** Each entry in `rewards` is `{"score": float, "weight": float}`. The episode total is `sum(score * weight)`. The `reward=0.525` in the log line is computed, not stored. 2. **`write_episode` dumps with `exclude_none=True`.** A `None` reward is dropped from the file entirely, so "not measured" and "key absent" are indistinguishable on the wire. Carry the run's declared reward-name set from `config.toml` if you need to tell them apart. ### ⚠️ 0.4.0 moves this Upstream HEAD (`~/vendor/prime-intellect/verifiers`) produces one **flat** `------` directory with no nested uuid, and the resolved config moves to `configs/.json` (JSON, because JSON keeps nulls). `RunConfig.id` is a `PrivateAttr` and is absent from that file — derive the run id from the directory basename, not from the config. --- ## 3. Counting what actually scored **`eval` exits 0 even when every rollout errors.** An errored trace is written with `rewards: {}`. Never report a run on its exit code. ```bash uv run python outputs/count_rewards.py outputs/run-/ ``` It prints `traces_scored` (non-empty `rewards`), `traces_errored` (`rewards: {}`), `provider_errors`, the stop-condition histogram, and two means: - `reward_mean_scored` — the mean over what survived. This is Arena's own "score what survived" trap in reporting form: a run where 30 of 32 rollouts errored and the 2 survivors scored 0.9 reads as 0.900. - `reward_mean_attempted` — errors counted as 0. **Rank on this one**, and refuse to rank at all below a scored-fraction floor. A quick eyeball without the script: ```bash jq -r '.traces[] | if (.rewards|length)>0 then "scored" else "errored" end' \ outputs/run-//traces.jsonl | sort | uniq -c ``` --- ## 4. What it cost, and what it scored **The run: `outputs/run-20260821-1401/`.** Seven data environments, 32 rollouts each, 224 episodes, **223 scored**. Model `brain-qwen38-dspark` (Qwen3.8-27B) on spark-1. Wall-clock 14:03 → 14:58, 55 minutes, `-c 8`. ⚠️ **Compute the mean as `sum(score × weight)` per trace.** An unweighted mean over the reward components is a different number and it is wrong — it disagrees with what the harness itself prints. Cross-check against the `reward=` field on each env's 32 `rollout done` lines in `eval.log`; the two must agree exactly. The first transcription of this run got six of seven means wrong by taking the unweighted mean. | environment | traces | scored | errored | reward_mean_attempted | reward_mean_scored | |---|---|---|---|---|---| | `fault-localisation` | 32 | 32 | 0 | **0.9531** | 0.9531 | | `canary-trap` | 32 | 32 | 0 | **0.6289** | 0.6289 | | `schema-migration` | 32 | 32 | 0 | **0.4432** | 0.4432 | | `drop-table-inference` | 32 | 31 | 1 | **0.4174** | 0.4309 | | `redaction-pressure` | 32 | 32 | 0 | **0.4040** | 0.4040 | | `bot-detection` | 32 | 32 | 0 | **0.3488** | 0.3488 | | `grand-exchange` | 32 | 32 | 0 | **0.0055** | 0.0055 | The single errored episode is a genuine spark-1 gateway timeout (`ProviderError`, 504) in `drop-table-inference` — not a scoring bug. **Rank on `_attempted`** (errors as 0); the `_scored` column is Arena's own "score what survived" trap in reporting form. ### Two numbers not to publish as capability scores ⚠️ **`grand-exchange` 0.0055 measures a sampling configuration, not a capability.** In **31 of 32 rollouts** the model returned a valid, parseable, degenerate answer — `{"expected_profit": 0, "orders": []}` in 18 completion tokens, `finish_reason=stop`, never entering a thinking block. The one rollout that did reason (4,954 completion tokens) placed 4 real orders and scored 0.176 weighted, with `filled=1233` against `planned=2100`. **The parser, the execution engine and the reward all work end to end** — the environment is not broken. But `probe.py` rates the *crude* baseline here at 0.082, so 0.0055 sits **below crude** and barely above inaction. Re-run with thinking forced on before this number goes anywhere near `/evals`. ⚠️ **`fault-localisation` 0.9531 has two of four components pegged at maximum.** `evidence` mean 1.000 (max 1.000) and `fault` mean 1.000 (max 1.000) across all 32 rollouts; only `gate` (0.9062) and `service` (0.9062) still move. Half the reward mass no longer discriminates at this capability level. Nemotron scored 0.352 here on 19 August — a 2.7× swing that needs separating into a stronger model, a prompting difference, or an environment that has become too easy, before it appears on a scoreboard. ### The `gate` component carried no gradient in four of seven environments Across all 32 rollouts each, `gate` scored **exactly 0.000, max 0.000**, in `bot-detection` (weight 0.25), `grand-exchange` (0.25), `redaction-pressure` (0.30) and `schema-migration` (0.25). It fired elsewhere — `canary-trap` 0.2188, `drop-table` 0.2258, `fault-localisation` 0.9062. This is **not** a house-rule-3 violation: `probe.py`'s oracle row is 1.000 for every one of those environments, which requires `gate = 1`, so it is reachable. But a quarter to a third of the reward mass produced no gradient in this run, and any ranking or before/after training table over those four environments is really a ranking over their two soft components. Understand that before training against them.