lumbridge-public/arena
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: run-20260821-1401 was sampled with thinking off, and four gates are dead

Browse Source 
Two measurement findings, both larger than the lanes that found them.

FIRST_EVAL.md: every number in the first run was sampled with thinking OFF —
0 of 224 traces carry reasoning_content or a <think> block, and spark-1 serves
with enable_thinking False. No artefact of the run recorded that. Five of the
seven numbers still stand; two do not. grand-exchange 0.0055 measured nothing
about the environment — with thinking on it scores 0.1667 on six rollouts and
one of them scored a clean 1.000, so it is fully solvable. drop-table-inference
0.4174 is a mixture of two regimes (0.0894 short, 0.6465 long) and should be
reported split or not at all. The defect is the missing sampling footnote, not
the values.

The 600-second per-call ceiling that blocked an n=32 thinking-on run is the
INSTALLED wheel, not upstream: verifiers fixed it in a298bcfe on 2026-08-08 and
0.3.0 predates it. The remedy is a dependency bump, not a wait.

GATE_DIAGNOSIS.md: the `gate` component scored exactly 0.000, max 0.000, across
all 32 rollouts in four environments. One shape explains three of them — exact
equality. Nothing below the oracle earns a fraction: not the plausible strategy,
not six of redaction's seven rules, not the oracle bot list minus one account.
bot-detection and schema-migration are a threshold set at the ceiling;
redaction-pressure is that and genuinely hard; grand-exchange's zero was the
sampling artefact above. The repo already contains the fix and already uses it
twice — drop-table's GATE_MARGIN and grand-exchange's TARGET_SHARE are margins,
not equalities. Recommended, not yet measured.

Consequence worth stating plainly: 25-30% of the reward mass on three
environments carries identically zero gradient, so they train against a
0.70-0.75 objective while being scored out of 1.00. The mirror failure exists
too — fault-localisation's evidence and fault components are pinned at exactly
1.0 on all 32 rollouts, so half its headline 0.9531 is constant.

tools/gate_probe.py is the reproduction, moved out of the gitignored outputs/
so the finding survives.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
karti
2026-08-21 17:44:46 -07:00
co-authored by Claude Opus 5
parent 4147de0282
commit 8048a61a1b
3 changed files with 600 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/FIRST_EVAL.md
+208 -2
View File
@@ -155,14 +155,22 @@ The single errored episode is a genuine spark-1 gateway timeout (`ProviderError`
### Two numbers not to publish as capability scores
⚠️ **`grand-exchange` 0.0055 measures a sampling configuration, not a capability.**
In **31 of 32 rollouts** the model returned a valid, parseable, degenerate answer —
In **31 of 32 rollouts** the model returned a valid, parseable, degenerate answer
(27 of them in exactly 18 completion tokens; the full distribution is
`{18: 27, 19: 1, 24: 3, 4954: 1}`) —
`{"expected_profit": 0, "orders": []}` in 18 completion tokens, `finish_reason=stop`,
never entering a thinking block. The one rollout that did reason (4,954 completion tokens)
placed 4 real orders and scored 0.176 weighted, with `filled=1233` against `planned=2100`.
**The parser, the execution engine and the reward all work end to end** — the environment
is not broken. But `probe.py` rates the *crude* baseline here at 0.082, so 0.0055 sits
**below crude** and barely above inaction. Re-run with thinking forced on before this
number goes anywhere near `/evals`.
number goes anywhere near `/evals`. **That re-run happened — see §5.** With thinking on the
same model scored **1.000 on every component** of the one rollout that finished inside its
token budget.
⚠️ **`drop-table-inference` 0.4174 is two populations, not one.** Twelve of its 31 scored
rollouts answered in 116–174 tokens and averaged 0.0894; the other nineteen wrote
4,585–11,034 tokens and averaged 0.6465. §5 has the distribution.
⚠️ **`fault-localisation` 0.9531 has two of four components pegged at maximum.**
`evidence` mean 1.000 (max 1.000) and `fault` mean 1.000 (max 1.000) across all 32
@@ -183,3 +191,201 @@ those environments, which requires `gate = 1`, so it is reachable. But a quarter
of the reward mass produced no gradient in this run, and any ranking or before/after
training table over those four environments is really a ranking over their two soft
components. Understand that before training against them.
---
## 5. The run was measured with thinking OFF, and nothing in the run says so
Appended 2026-08-21, after re-measuring `grand-exchange`.
### The cause
spark-1's SGLang is started with a default that overrides the chat template:
```bash
curl -s http://100.127.247.67:8001/get_server_info | jq '.default_chat_template_kwargs, .reasoning_parser'
# {"enable_thinking": false}
# "qwen3"
```
**Every one of the 224 episodes in `run-20260821-1401` was sampled with Qwen3's thinking
mode disabled**, and not one of them requested otherwise. Confirmed from the traces: no
assistant message in any of the seven environments carries a `reasoning_content` field, and
none contains a `<think>` block — 0 of 223 scored traces.
Nothing in the run's own artefacts records this. The resolved `config.toml` says
`sampling = {}`, and `ChatDialect.parse_sampling` keeps only an allow-list of known keys, so
even a run that *does* pass `chat_template_kwargs` writes a `calls[].sampling` that does not
mention it. `usage.reasoning_tokens` is null too: SGLang reports `reasoning_tokens` at the
top level of `usage`, while `Usage.from_openai` reads it out of `completion_tokens_details`.
**A trace in this repository cannot tell you whether the model was allowed to think.**
### Turning it on
`SamplingConfig` is `extra="allow"` and `ChatDialect.apply_overrides` merges the whole dump
into the request body, so an untyped key rides through to the provider untouched. There is
no CLI flag — `--sampling.*` is typed — so it goes in a config file:
```toml
[sampling]
chat_template_kwargs = { enable_thinking = true }
max_tokens = 16384
```
`outputs/thinking/grand_exchange_thinking.toml` is that file. **Verify it worked from the
trace, not from the flag**: the assistant message gains a `reasoning_content` field.
### What thinking on actually did to `grand-exchange`
**`outputs/thinking/ge-think-n6/`** — 6 tasks × 1 rollout, `-c 2`, `max_tokens = 16384`,
thinking on. All 6 traces carry `reasoning_content`, so the flag reached the model.
| tokens | `finish_reason` | content | weighted reward |
|---|---|---|---|
| 16384 | length | 0 chars | 0.000 |
| 16384 | length | 0 chars | 0.000 |
| 16384 | length | 0 chars | 0.000 |
| 16384 | length | 0 chars | 0.000 |
| 16384 | length | 0 chars | 0.000 |
| **8335** | **stop** | 1089 chars | **1.000** |
**Mean 0.1667** against 0.0055 with thinking off — a 30× move, and above `probe.py`'s crude
baseline of 0.082 for the first time. But read the column, not the mean: it is one perfect
run and five truncations.
The run that finished scored **1.000 on every component** — `profit` 1.000, `discipline`
1.000, **`gate` 1.000** — with `realised` 40,081 gp against a `reference_realised` of
36,157, `committed` 0.994 of the purse and `conversion` 0.835. Four orders, a written
rationale, `orders_dropped` 0:
```json
{"expected_profit": 15000, "orders": [
{"item": "Chipped bone charm", "quantity": 400, "buy": 120, "sell": 134},
{"item": "Coarse fletching feather", "quantity": 700, "buy": 97, "sell": 114},
{"item": "Emberglass shard", "quantity": 80, "buy": 1420, "sell": 1640},
{"item": "Heart of the sunken cairn", "quantity": 2, "buy": 9500, "sell": 13500}]}
```
⚠️ **So the environment is not merely unbroken — it is fully solvable by this model, gate
included, and the thinking-off number measured none of that.** `arith_ok` was still 0.0 on
that rollout (`expected_profit` 15,000 against an implied 40,184, `arith_error` 0.627),
which is exactly the line the taskset docstring says it records for this reason.
The five truncations are not junk either. The reasoning is the analysis the environment asks
for, item by item — counting crossings, applying the 25%-of-volume cap, sizing against the
buy limit — it simply does not converge inside 16k:
```
**Heart of the sunken cairn**:
Buy ticks (<=10000): 9267, 8846, 9656, 9541, 9703, 8853, 9945 = 7 ticks
Volumes: 2, 2, 3, 6, 2, 3, 3 25%: 0.5, 0.5, 0.75, 1.5, 0.5, 0.75, 0.75
Total: 5.25 in 56 ticks. In 30 ticks: ~2.8. So I can only buy about 2-3 units!
```
A single rollout at `-c 1` (`outputs/thinking/smoke/`) does the same: 486 seconds,
`completion_tokens` 16384, `reasoning_content` 30,121 characters, `content` empty, reward
0.000. So does the same prompt sent straight at spark-1 with `curl`, no verifiers in the
path — `reasoning_tokens` 16390, `finish_reason` length, content empty. **This is not a
harness bug, and it is not fixed by raising `max_tokens`** — see the next section for why.
⚠️ **`grand-exchange` 0.0055 was a sampling artefact and 0.1667 is a truncation artefact.**
Neither is the environment's number. The number is somewhere at or above 0.1667, taken with
a budget large enough that the model finishes, and this box cannot currently run that.
### ⚠️ The harness has a hard 600-second ceiling on one model call
`verifiers/v1/clients/base.py:11`:
```python
DEFAULT_TIMEOUT = httpx.Timeout(connect=5.0, read=600.0, write=600.0, pool=600.0)
```
`BaseClientConfig` exposes `base_url`, `api_key_var` and `headers` and nothing else. **There
is no config key and no CLI flag for this timeout.** At spark-1's ~34 tok/s single-stream it
caps one call at roughly 20,000 completion tokens, and far fewer under concurrency.
That is not theoretical. `outputs/thinking/ge-think-n8.log` is 8 tasks at `-c 8` with
`max_tokens = 32768`:
```
16:09:27 INFO rollout start: ... (×8)
16:19:30 WARNING model call failed: id=... ProviderError: (×8)
```
Eight failures, all at 16:19:30, exactly 603 seconds after the starts, with an empty
`ProviderError` message and `traces.jsonl` left at **zero lines**. `eval` still exited 0.
The `-c 1` smoke run survived at 486 seconds; the same work at `-c 8` did not.
`-c 2` is not safe either. `ge-think-n6.log` at `max_tokens = 16384` still logged **4
`ProviderError`s in 6 rollouts**, each exactly 601–603 seconds after its rollout started —
`16:36:12 → 16:46:15`, `16:46:15 → 16:56:16`. Those rollouts survived only because the null
harness's program retries the call itself (tenacity), which is why their traces carry two
`calls` entries, the first with a null `finish_reason`. A retry that also lands at 16k
tokens costs another ten minutes and scores zero anyway.
**Raising `max_tokens` therefore does not fix the truncations.** The five rollouts that ran
out at 16,384 tokens want more budget; the wall says they cannot have it, because 16,384
tokens is already 486–603 seconds of wall clock on this box. The two limits point in
opposite directions and there is no setting that satisfies both.
**And the thinking block cannot be bounded from the request either.** SGLang's
`thinking_budget` is accepted (HTTP 200) and ignored by this build: the same prompt with
`{"max_tokens": 8192, "thinking_budget": 3000}` came back with `reasoning_tokens` **8195**,
`finish_reason` length, content empty. The server also reports `enable_strict_thinking:
false`. The only lever is `max_tokens`, and `max_tokens` cuts the answer, not the reasoning.
Three consequences for anyone re-running this suite against a reasoning model:
1. **Keep `-c` low and `max_tokens` under the wall**, or budget the run knowing calls will be
cut at ten minutes. spark-1 is shared — another lane's eval was running against it during
this measurement, and contention alone moves a rollout across the line.
2. **A timed-out run leaves no evidence.** Zero traces, exit code 0. `count_rewards.py`
reports on a file that does not exist. This is the same trap as §3 with a bigger blast
radius: check the log for `ProviderError` and check `wc -l traces.jsonl` before believing
any run.
3. ⚠️ **CORRECTED: the 600-second ceiling is the INSTALLED WHEEL, not upstream.** The
remedy stated here first — "the 600-second default gets a config key upstream" — is
wrong, and it is wrong in the expensive direction, because it reads as blocked on someone
else. Upstream **already removed it**: `verifiers/v1/clients/base.py:12` in
`~/vendor/prime-intellect/verifiers` reads `httpx.Timeout(connect=5.0, read=None, ...)`,
commit `a298bcfe fix(v1): restore the unbounded model-call timeout (#2304)`, 2026-08-08.
Every `environments/*/.venv` here pins `read=600.0` because it resolves verifiers 0.3.0.
**The fix is a dependency bump**, and an n=32 thinking-on run is achievable today. That
makes the "grand-exchange is solvable" claim — currently resting on six rollouts — cheap
to settle properly.
### Did the other six collapse too?
Only one of them, and only partly. Completion tokens per trace, and the mean weighted
reward in each half of the length distribution:
| environment | min | p25 | median | p75 | max | shape |
|---|---|---|---|---|---|---|
| `grand-exchange` | 18 | 18 | 18 | 18 | 4954 | **collapsed**; distribution `{18: 27, 19: 1, 24: 3, 4954: 1}` — 27/32 at 18 tokens, and **31/32 planned no orders at all** |
| `drop-table-inference` | 0 | 159 | 5347 | 7205 | 11034 | **bimodal** |
| `fault-localisation` | 34 | 37 | 37 | 39 | 191 | short by design |
| `schema-migration` | 105 | 145 | 168 | 271 | 1687 | unimodal |
| `redaction-pressure` | 153 | 240 | 310 | 369 | 2014 | unimodal |
| `canary-trap` | 56 | 139 | 148 | 626 | 3128 | unimodal |
| `bot-detection` | 1156 | 2508 | 3457 | 4603 | 8146 | unimodal |
⚠️ **`drop-table-inference` 0.4174 is a mixture of two regimes, not a capability estimate.**
Twelve of its 31 scored rollouts answered in 116–174 tokens and averaged **0.0894**; the
other nineteen wrote 4,585–11,034 tokens and averaged **0.6465**. The same prompt, the same
sampling settings, a 7× gap in outcome depending on whether the model chose to work. Report
it with that split or not at all.
The other five are unimodal and their reward does not track length — `fault-localisation`
answers three fields in 37 tokens and scores 0.9531, `canary-trap`'s shortest half scores
0.6141 against its longest half's 0.6437. Those five numbers stand as measured. They are
still **thinking-off numbers**, which is a property of the run nobody chose and nothing
recorded.
### What to do before the next run
- Put the sampling configuration in the run's own name or its notes. A number that changes
by two orders of magnitude with one server-side default is not a capability score unless
the configuration travels with it.
- Ask spark-1 what it defaults to (`/get_server_info`) rather than assuming a model's own
default applies.
- If `reasoning_tokens` matters to you, read it off the provider — the trace drops it.