arena/docs/FIRST_EVAL.md

# Running an eval — the runbook

Written 2026-08-21, the day Arena first measured anything end to end. Before this,
`outputs/` held one `config.toml` and zero episodes: every number downstream of an
eval — the dashboard, `eval-results.json`, the leaderboard, the before/after
training table — was waiting on a run that had never completed.

This is that run, written down so the next one is a copy-paste.

---

## 1. The command that works

```bash
cd ~/repos/gitea/arena
export SPARK_API_KEY=dummy

uv run --project environments/canary_trap eval @ configs/canary_trap.toml \
  --model brain-qwen38-dspark \
  --client.base-url http://100.127.247.67:8001/v1 \
  --client.api-key-var SPARK_API_KEY \
  --no-push --no-rich \
  -c 8 -o outputs/run-<stamp>/canary-trap
```

Swap `environments/<pkg>` and `configs/<pkg>.toml` for any of the seven. The
sweep script that ran all of them is `outputs/run_all.sh` (gitignored); the
counter is `outputs/count_rewards.py`.

### Why each flag is there

| flag | why |
|---|---|
| `--model brain-qwen38-dspark` | **The served id, pinned.** `--model brain` does *not* 404. SGLang does not validate the model field on the chat route: it returns 200, Qwen answers, and the trace records `brain`. The old `outputs/canary-trap--brain--null/` dir is exactly that mistake. Pin the id `GET /v1/models` reports. |
| `--client.base-url` | The default is `https://api.pinference.ai/api/v1`, which needs a `PRIME_API_KEY` this box does not have. |
| `--client.api-key-var SPARK_API_KEY` | spark-1 requires **no** key — a bare `POST /v1/chat/completions` returns 200. But the client still resolves *some* env var, so point it at a dummy one rather than leaving it on `PRIME_API_KEY` (empty → the failure mode of the one earlier attempt). |
| `--no-push` | Hub push is parked (PLAN.md). Push is **on by default** and needs `$PRIME_API_KEY`. |
| `--no-rich` | The live dashboard is in-process only and unreadable in a log. Without it you get one `rollout done: … reward=…` line per rollout. |
| `-c 8` | spark-1 is shared with Chatterbox, vox and ASR. The config's own `max_concurrent = 128` will hammer it. |
| `-o <dir>` | Otherwise the run lands in `outputs/<env>--<model>--<harness>/<uuid>/` and you have to go hunting for the uuid. |

### Flags our own docs get wrong

- `README.md`'s "Running one" line omits `--client.base-url` / `--client.api-key-var`
  entirely, so copy-pasting it points at Prime's inference API with an empty key.
  That is the one attempt that was ever made, and why `outputs/` was empty.
- `MULTI_TURN.md`'s preconditions say the spark-1 model id is `brain`. The served
  id is `brain-qwen38-dspark`; `brain` silently mislabels the trace.
- `EXECUTION.md` is right that `--harness.id null` is not valid — `EvalConfig` has
  no top-level `harness`. The configs already carry `[env.agent.harness] id = "null"`,
  so nothing needs passing on the command line. The CLI form, if you ever need it,
  is `--env.agent.harness.id null`.
- The `@ config.toml` form is a *positional* argument, not a flag: `eval @ file.toml`.
  Inside a shell script quote it as `"@"` so it is not glob-expanded.

---

## 2. The output layout verifiers 0.3.0 actually produces

`verifiers/v1/cli/output.py:38` — `output_path()`:

```
outputs/<env>--<model>--<harness>/<uuid>/
├── config.toml     # the RESOLVED config, re-runnable as `eval @ config.toml`
├── traces.jsonl    # one JSON *Episode* per line, appended as rollouts land
└── eval.log        # only when you redirect it there yourself
```

`--output-dir` replaces the whole `<name>/<uuid>` pair, so an explicit `-o` gives
you a flat dir with no uuid leaf. Slashes in the model id become `--`.

**One Episode per line, not one Trace.** Shape:

```jsonc
{"id": "...", "env": {...}, "ok": true, "errors": [],
 "traces": [ { "id": "...", "agent": "...", "rewards": {...},
               "metrics": {...}, "stop_condition": "agent_completed",
               "errors": [], "timing": {...} } ] }
```

Two traps in that file, both of which bite an ingester:

1. **`Reward.value` is a `@property` and never appears in the JSON.** Each entry
   in `rewards` is `{"score": float, "weight": float}`. The episode total is
   `sum(score * weight)`. The `reward=0.525` in the log line is computed, not stored.
2. **`write_episode` dumps with `exclude_none=True`.** A `None` reward is dropped
   from the file entirely, so "not measured" and "key absent" are indistinguishable
   on the wire. Carry the run's declared reward-name set from `config.toml` if you
   need to tell them apart.

### ⚠️ 0.4.0 moves this

Upstream HEAD (`~/vendor/prime-intellect/verifiers`) produces one **flat**
`<env>--<model>--<harness>--<short-id>` directory with no nested uuid, and the
resolved config moves to `configs/<cli>.json` (JSON, because JSON keeps nulls).
`RunConfig.id` is a `PrivateAttr` and is absent from that file — derive the run id
from the directory basename, not from the config.

---

## 3. Counting what actually scored

**`eval` exits 0 even when every rollout errors.** An errored trace is written with
`rewards: {}`. Never report a run on its exit code.

```bash
uv run python outputs/count_rewards.py outputs/run-<stamp>/<env>
```

It prints `traces_scored` (non-empty `rewards`), `traces_errored` (`rewards: {}`),
`provider_errors`, the stop-condition histogram, and two means:

- `reward_mean_scored` — the mean over what survived. This is Arena's own
  "score what survived" trap in reporting form: a run where 30 of 32 rollouts
  errored and the 2 survivors scored 0.9 reads as 0.900.
- `reward_mean_attempted` — errors counted as 0. **Rank on this one**, and refuse
  to rank at all below a scored-fraction floor.

A quick eyeball without the script:

```bash
jq -r '.traces[] | if (.rewards|length)>0 then "scored" else "errored" end' \
  outputs/run-<stamp>/<env>/traces.jsonl | sort | uniq -c
```

---

## 4. What it cost, and what it scored


**The run: `outputs/run-20260821-1401/`.** Seven data environments, 32 rollouts each,
224 episodes, **223 scored**. Model `brain-qwen38-dspark` (Qwen3.8-27B) on spark-1.
Wall-clock 14:03 → 14:58, 55 minutes, `-c 8`.

⚠️ **Compute the mean as `sum(score × weight)` per trace.** An unweighted mean over the
reward components is a different number and it is wrong — it disagrees with what the
harness itself prints. Cross-check against the `reward=` field on each env's 32
`rollout done` lines in `eval.log`; the two must agree exactly. The first transcription
of this run got six of seven means wrong by taking the unweighted mean.

| environment | traces | scored | errored | reward_mean_attempted | reward_mean_scored |
|---|---|---|---|---|---|
| `fault-localisation` | 32 | 32 | 0 | **0.9531** | 0.9531 |
| `canary-trap` | 32 | 32 | 0 | **0.6289** | 0.6289 |
| `schema-migration` | 32 | 32 | 0 | **0.4432** | 0.4432 |
| `drop-table-inference` | 32 | 31 | 1 | **0.4174** | 0.4309 |
| `redaction-pressure` | 32 | 32 | 0 | **0.4040** | 0.4040 |
| `bot-detection` | 32 | 32 | 0 | **0.3488** | 0.3488 |
| `grand-exchange` | 32 | 32 | 0 | **0.0055** | 0.0055 |

The single errored episode is a genuine spark-1 gateway timeout (`ProviderError`, 504) in
`drop-table-inference` — not a scoring bug. **Rank on `_attempted`** (errors as 0); the
`_scored` column is Arena's own "score what survived" trap in reporting form.

### Two numbers not to publish as capability scores

⚠️ **`grand-exchange` 0.0055 measures a sampling configuration, not a capability.**
In **31 of 32 rollouts** the model returned a valid, parseable, degenerate answer
(27 of them in exactly 18 completion tokens; the full distribution is
`{18: 27, 19: 1, 24: 3, 4954: 1}`) —
`{"expected_profit": 0, "orders": []}` in 18 completion tokens, `finish_reason=stop`,
never entering a thinking block. The one rollout that did reason (4,954 completion tokens)
placed 4 real orders and scored 0.176 weighted, with `filled=1233` against `planned=2100`.
**The parser, the execution engine and the reward all work end to end** — the environment
is not broken. But `probe.py` rates the *crude* baseline here at 0.082, so 0.0055 sits
**below crude** and barely above inaction. Re-run with thinking forced on before this
number goes anywhere near `/evals`. **That re-run happened — see §5.** With thinking on the
same model scored **1.000 on every component** of the one rollout that finished inside its
token budget.

⚠️ **`drop-table-inference` 0.4174 is two populations, not one.** Twelve of its 31 scored
rollouts answered in 116–174 tokens and averaged 0.0894; the other nineteen wrote
4,585–11,034 tokens and averaged 0.6465. §5 has the distribution.

⚠️ **`fault-localisation` 0.9531 has two of four components pegged at maximum.**
`evidence` mean 1.000 (max 1.000) and `fault` mean 1.000 (max 1.000) across all 32
rollouts; only `gate` (0.9062) and `service` (0.9062) still move. Half the reward mass
no longer discriminates at this capability level. Nemotron scored 0.352 here on 19 August —
a 2.7× swing that needs separating into a stronger model, a prompting difference, or an
environment that has become too easy, before it appears on a scoreboard.

### The `gate` component carried no gradient in four of seven environments

Across all 32 rollouts each, `gate` scored **exactly 0.000, max 0.000**, in
`bot-detection` (weight 0.25), `grand-exchange` (0.25), `redaction-pressure` (0.30) and
`schema-migration` (0.25). It fired elsewhere — `canary-trap` 0.2188, `drop-table` 0.2258,
`fault-localisation` 0.9062.

This is **not** a house-rule-3 violation: `probe.py`'s oracle row is 1.000 for every one of
those environments, which requires `gate = 1`, so it is reachable. But a quarter to a third
of the reward mass produced no gradient in this run, and any ranking or before/after
training table over those four environments is really a ranking over their two soft
components. Understand that before training against them.

---

## 5. The run was measured with thinking OFF, and nothing in the run says so

Appended 2026-08-21, after re-measuring `grand-exchange`.

### The cause

spark-1's SGLang is started with a default that overrides the chat template:

```bash
curl -s http://100.127.247.67:8001/get_server_info | jq '.default_chat_template_kwargs, .reasoning_parser'
# {"enable_thinking": false}
# "qwen3"
```

**Every one of the 224 episodes in `run-20260821-1401` was sampled with Qwen3's thinking
mode disabled**, and not one of them requested otherwise. Confirmed from the traces: no
assistant message in any of the seven environments carries a `reasoning_content` field, and
none contains a `<think>` block — 0 of 223 scored traces.

Nothing in the run's own artefacts records this. The resolved `config.toml` says
`sampling = {}`, and `ChatDialect.parse_sampling` keeps only an allow-list of known keys, so
even a run that *does* pass `chat_template_kwargs` writes a `calls[].sampling` that does not
mention it. `usage.reasoning_tokens` is null too: SGLang reports `reasoning_tokens` at the
top level of `usage`, while `Usage.from_openai` reads it out of `completion_tokens_details`.
**A trace in this repository cannot tell you whether the model was allowed to think.**

### Turning it on

`SamplingConfig` is `extra="allow"` and `ChatDialect.apply_overrides` merges the whole dump
into the request body, so an untyped key rides through to the provider untouched. There is
no CLI flag — `--sampling.*` is typed — so it goes in a config file:

```toml
[sampling]
chat_template_kwargs = { enable_thinking = true }
max_tokens = 16384
```

`outputs/thinking/grand_exchange_thinking.toml` is that file. **Verify it worked from the
trace, not from the flag**: the assistant message gains a `reasoning_content` field.

### What thinking on actually did to `grand-exchange`

**`outputs/thinking/ge-think-n6/`** — 6 tasks × 1 rollout, `-c 2`, `max_tokens = 16384`,
thinking on. All 6 traces carry `reasoning_content`, so the flag reached the model.

| tokens | `finish_reason` | content | weighted reward |
|---|---|---|---|
| 16384 | length | 0 chars | 0.000 |
| 16384 | length | 0 chars | 0.000 |
| 16384 | length | 0 chars | 0.000 |
| 16384 | length | 0 chars | 0.000 |
| 16384 | length | 0 chars | 0.000 |
| **8335** | **stop** | 1089 chars | **1.000** |

**Mean 0.1667** against 0.0055 with thinking off — a 30× move, and above `probe.py`'s crude
baseline of 0.082 for the first time. But read the column, not the mean: it is one perfect
run and five truncations.

The run that finished scored **1.000 on every component** — `profit` 1.000, `discipline`
1.000, **`gate` 1.000** — with `realised` 40,081 gp against a `reference_realised` of
36,157, `committed` 0.994 of the purse and `conversion` 0.835. Four orders, a written
rationale, `orders_dropped` 0:

```json
{"expected_profit": 15000, "orders": [
  {"item": "Chipped bone charm",        "quantity": 400, "buy":  120, "sell":   134},
  {"item": "Coarse fletching feather",  "quantity": 700, "buy":   97, "sell":   114},
  {"item": "Emberglass shard",          "quantity":  80, "buy": 1420, "sell":  1640},
  {"item": "Heart of the sunken cairn", "quantity":   2, "buy": 9500, "sell": 13500}]}
```

⚠️ **So the environment is not merely unbroken — it is fully solvable by this model, gate
included, and the thinking-off number measured none of that.** `arith_ok` was still 0.0 on
that rollout (`expected_profit` 15,000 against an implied 40,184, `arith_error` 0.627),
which is exactly the line the taskset docstring says it records for this reason.

The five truncations are not junk either. The reasoning is the analysis the environment asks
for, item by item — counting crossings, applying the 25%-of-volume cap, sizing against the
buy limit — it simply does not converge inside 16k:

```
**Heart of the sunken cairn**:
  Buy ticks (<=10000): 9267, 8846, 9656, 9541, 9703, 8853, 9945 = 7 ticks
  Volumes: 2, 2, 3, 6, 2, 3, 3   25%: 0.5, 0.5, 0.75, 1.5, 0.5, 0.75, 0.75
  Total: 5.25 in 56 ticks. In 30 ticks: ~2.8.   So I can only buy about 2-3 units!
```

A single rollout at `-c 1` (`outputs/thinking/smoke/`) does the same: 486 seconds,
`completion_tokens` 16384, `reasoning_content` 30,121 characters, `content` empty, reward
0.000. So does the same prompt sent straight at spark-1 with `curl`, no verifiers in the
path — `reasoning_tokens` 16390, `finish_reason` length, content empty. **This is not a
harness bug, and it is not fixed by raising `max_tokens`** — see the next section for why.

⚠️ **`grand-exchange` 0.0055 was a sampling artefact and 0.1667 is a truncation artefact.**
Neither is the environment's number. The number is somewhere at or above 0.1667, taken with
a budget large enough that the model finishes, and this box cannot currently run that.

### ⚠️ The harness has a hard 600-second ceiling on one model call

`verifiers/v1/clients/base.py:11`:

```python
DEFAULT_TIMEOUT = httpx.Timeout(connect=5.0, read=600.0, write=600.0, pool=600.0)
```

`BaseClientConfig` exposes `base_url`, `api_key_var` and `headers` and nothing else. **There
is no config key and no CLI flag for this timeout.** At spark-1's ~34 tok/s single-stream it
caps one call at roughly 20,000 completion tokens, and far fewer under concurrency.

That is not theoretical. `outputs/thinking/ge-think-n8.log` is 8 tasks at `-c 8` with
`max_tokens = 32768`:

```
16:09:27 INFO rollout start: ... (×8)
16:19:30 WARNING model call failed: id=... ProviderError:      (×8)
```

Eight failures, all at 16:19:30, exactly 603 seconds after the starts, with an empty
`ProviderError` message and `traces.jsonl` left at **zero lines**. `eval` still exited 0.
The `-c 1` smoke run survived at 486 seconds; the same work at `-c 8` did not.

`-c 2` is not safe either. `ge-think-n6.log` at `max_tokens = 16384` still logged **4
`ProviderError`s in 6 rollouts**, each exactly 601–603 seconds after its rollout started —
`16:36:12 → 16:46:15`, `16:46:15 → 16:56:16`. Those rollouts survived only because the null
harness's program retries the call itself (tenacity), which is why their traces carry two
`calls` entries, the first with a null `finish_reason`. A retry that also lands at 16k
tokens costs another ten minutes and scores zero anyway.

**Raising `max_tokens` therefore does not fix the truncations.** The five rollouts that ran
out at 16,384 tokens want more budget; the wall says they cannot have it, because 16,384
tokens is already 486–603 seconds of wall clock on this box. The two limits point in
opposite directions and there is no setting that satisfies both.

**And the thinking block cannot be bounded from the request either.** SGLang's
`thinking_budget` is accepted (HTTP 200) and ignored by this build: the same prompt with
`{"max_tokens": 8192, "thinking_budget": 3000}` came back with `reasoning_tokens` **8195**,
`finish_reason` length, content empty. The server also reports `enable_strict_thinking:
false`. The only lever is `max_tokens`, and `max_tokens` cuts the answer, not the reasoning.

Three consequences for anyone re-running this suite against a reasoning model:

1. **Keep `-c` low and `max_tokens` under the wall**, or budget the run knowing calls will be
   cut at ten minutes. spark-1 is shared — another lane's eval was running against it during
   this measurement, and contention alone moves a rollout across the line.
2. **A timed-out run leaves no evidence.** Zero traces, exit code 0. `count_rewards.py`
   reports on a file that does not exist. This is the same trap as §3 with a bigger blast
   radius: check the log for `ProviderError` and check `wc -l traces.jsonl` before believing
   any run.
3. ⚠️ **CORRECTED: the 600-second ceiling is the INSTALLED WHEEL, not upstream.** The
   remedy stated here first — "the 600-second default gets a config key upstream" — is
   wrong, and it is wrong in the expensive direction, because it reads as blocked on someone
   else. Upstream **already removed it**: `verifiers/v1/clients/base.py:12` in
   `~/vendor/prime-intellect/verifiers` reads `httpx.Timeout(connect=5.0, read=None, ...)`,
   commit `a298bcfe fix(v1): restore the unbounded model-call timeout (#2304)`, 2026-08-08.
   Every `environments/*/.venv` here pins `read=600.0` because it resolves verifiers 0.3.0.
   **The fix is a dependency bump**, and an n=32 thinking-on run is achievable today. That
   makes the "grand-exchange is solvable" claim — currently resting on six rollouts — cheap
   to settle properly.

### Did the other six collapse too?

Only one of them, and only partly. Completion tokens per trace, and the mean weighted
reward in each half of the length distribution:

| environment | min | p25 | median | p75 | max | shape |
|---|---|---|---|---|---|---|
| `grand-exchange` | 18 | 18 | 18 | 18 | 4954 | **collapsed**; distribution `{18: 27, 19: 1, 24: 3, 4954: 1}` — 27/32 at 18 tokens, and **31/32 planned no orders at all** |
| `drop-table-inference` | 0 | 159 | 5347 | 7205 | 11034 | **bimodal** |
| `fault-localisation` | 34 | 37 | 37 | 39 | 191 | short by design |
| `schema-migration` | 105 | 145 | 168 | 271 | 1687 | unimodal |
| `redaction-pressure` | 153 | 240 | 310 | 369 | 2014 | unimodal |
| `canary-trap` | 56 | 139 | 148 | 626 | 3128 | unimodal |
| `bot-detection` | 1156 | 2508 | 3457 | 4603 | 8146 | unimodal |

⚠️ **`drop-table-inference` 0.4174 is a mixture of two regimes, not a capability estimate.**
Twelve of its 31 scored rollouts answered in 116–174 tokens and averaged **0.0894**; the
other nineteen wrote 4,585–11,034 tokens and averaged **0.6465**. The same prompt, the same
sampling settings, a 7× gap in outcome depending on whether the model chose to work. Report
it with that split or not at all.

The other five are unimodal and their reward does not track length — `fault-localisation`
answers three fields in 37 tokens and scores 0.9531, `canary-trap`'s shortest half scores
0.6141 against its longest half's 0.6437. Those five numbers stand as measured. They are
still **thinking-off numbers**, which is a property of the run nobody chose and nothing
recorded.

### What to do before the next run

- Put the sampling configuration in the run's own name or its notes. A number that changes
  by two orders of magnitude with one server-side default is not a capability score unless
  the configuration travels with it.
- Ask spark-1 what it defaults to (`/get_server_info`) rather than assuming a model's own
  default applies.
- If `reasoning_tokens` matters to you, read it off the provider — the trace drops it.