lumbridge-public/arena
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: run-20260821-1401 was sampled with thinking off, and four gates are dead

Browse Source 
Two measurement findings, both larger than the lanes that found them.

FIRST_EVAL.md: every number in the first run was sampled with thinking OFF —
0 of 224 traces carry reasoning_content or a <think> block, and spark-1 serves
with enable_thinking False. No artefact of the run recorded that. Five of the
seven numbers still stand; two do not. grand-exchange 0.0055 measured nothing
about the environment — with thinking on it scores 0.1667 on six rollouts and
one of them scored a clean 1.000, so it is fully solvable. drop-table-inference
0.4174 is a mixture of two regimes (0.0894 short, 0.6465 long) and should be
reported split or not at all. The defect is the missing sampling footnote, not
the values.

The 600-second per-call ceiling that blocked an n=32 thinking-on run is the
INSTALLED wheel, not upstream: verifiers fixed it in a298bcfe on 2026-08-08 and
0.3.0 predates it. The remedy is a dependency bump, not a wait.

GATE_DIAGNOSIS.md: the `gate` component scored exactly 0.000, max 0.000, across
all 32 rollouts in four environments. One shape explains three of them — exact
equality. Nothing below the oracle earns a fraction: not the plausible strategy,
not six of redaction's seven rules, not the oracle bot list minus one account.
bot-detection and schema-migration are a threshold set at the ceiling;
redaction-pressure is that and genuinely hard; grand-exchange's zero was the
sampling artefact above. The repo already contains the fix and already uses it
twice — drop-table's GATE_MARGIN and grand-exchange's TARGET_SHARE are margins,
not equalities. Recommended, not yet measured.

Consequence worth stating plainly: 25-30% of the reward mass on three
environments carries identically zero gradient, so they train against a
0.70-0.75 objective while being scored out of 1.00. The mirror failure exists
too — fault-localisation's evidence and fault components are pinned at exactly
1.0 on all 32 rollouts, so half its headline 0.9531 is constant.

tools/gate_probe.py is the reproduction, moved out of the gitignored outputs/
so the finding survives.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
karti
2026-08-21 17:44:46 -07:00
co-authored by Claude Opus 5
parent 4147de0282
commit 8048a61a1b
3 changed files with 600 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/FIRST_EVAL.md
+208 -2
View File
@@ -155,14 +155,22 @@ The single errored episode is a genuine spark-1 gateway timeout (`ProviderError`
### Two numbers not to publish as capability scores
⚠️ **`grand-exchange` 0.0055 measures a sampling configuration, not a capability.**
In **31 of 32 rollouts** the model returned a valid, parseable, degenerate answer —
In **31 of 32 rollouts** the model returned a valid, parseable, degenerate answer
(27 of them in exactly 18 completion tokens; the full distribution is
`{18: 27, 19: 1, 24: 3, 4954: 1}`) —
`{"expected_profit": 0, "orders": []}` in 18 completion tokens, `finish_reason=stop`,
never entering a thinking block. The one rollout that did reason (4,954 completion tokens)
placed 4 real orders and scored 0.176 weighted, with `filled=1233` against `planned=2100`.
**The parser, the execution engine and the reward all work end to end** — the environment
is not broken. But `probe.py` rates the *crude* baseline here at 0.082, so 0.0055 sits
**below crude** and barely above inaction. Re-run with thinking forced on before this
number goes anywhere near `/evals`.
number goes anywhere near `/evals`. **That re-run happened — see §5.** With thinking on the
same model scored **1.000 on every component** of the one rollout that finished inside its
token budget.
⚠️ **`drop-table-inference` 0.4174 is two populations, not one.** Twelve of its 31 scored
rollouts answered in 116–174 tokens and averaged 0.0894; the other nineteen wrote
4,585–11,034 tokens and averaged 0.6465. §5 has the distribution.
⚠️ **`fault-localisation` 0.9531 has two of four components pegged at maximum.**
`evidence` mean 1.000 (max 1.000) and `fault` mean 1.000 (max 1.000) across all 32
@@ -183,3 +191,201 @@ those environments, which requires `gate = 1`, so it is reachable. But a quarter
of the reward mass produced no gradient in this run, and any ranking or before/after
training table over those four environments is really a ranking over their two soft
components. Understand that before training against them.
---
## 5. The run was measured with thinking OFF, and nothing in the run says so
Appended 2026-08-21, after re-measuring `grand-exchange`.
### The cause
spark-1's SGLang is started with a default that overrides the chat template:
```bash
curl -s http://100.127.247.67:8001/get_server_info | jq '.default_chat_template_kwargs, .reasoning_parser'
# {"enable_thinking": false}
# "qwen3"
```
**Every one of the 224 episodes in `run-20260821-1401` was sampled with Qwen3's thinking
mode disabled**, and not one of them requested otherwise. Confirmed from the traces: no
assistant message in any of the seven environments carries a `reasoning_content` field, and
none contains a `<think>` block — 0 of 223 scored traces.
Nothing in the run's own artefacts records this. The resolved `config.toml` says
`sampling = {}`, and `ChatDialect.parse_sampling` keeps only an allow-list of known keys, so
even a run that *does* pass `chat_template_kwargs` writes a `calls[].sampling` that does not
mention it. `usage.reasoning_tokens` is null too: SGLang reports `reasoning_tokens` at the
top level of `usage`, while `Usage.from_openai` reads it out of `completion_tokens_details`.
**A trace in this repository cannot tell you whether the model was allowed to think.**
### Turning it on
`SamplingConfig` is `extra="allow"` and `ChatDialect.apply_overrides` merges the whole dump
into the request body, so an untyped key rides through to the provider untouched. There is
no CLI flag — `--sampling.*` is typed — so it goes in a config file:
```toml
[sampling]
chat_template_kwargs = { enable_thinking = true }
max_tokens = 16384
```
`outputs/thinking/grand_exchange_thinking.toml` is that file. **Verify it worked from the
trace, not from the flag**: the assistant message gains a `reasoning_content` field.
### What thinking on actually did to `grand-exchange`
**`outputs/thinking/ge-think-n6/`** — 6 tasks × 1 rollout, `-c 2`, `max_tokens = 16384`,
thinking on. All 6 traces carry `reasoning_content`, so the flag reached the model.
| tokens | `finish_reason` | content | weighted reward |
|---|---|---|---|
| 16384 | length | 0 chars | 0.000 |
| 16384 | length | 0 chars | 0.000 |
| 16384 | length | 0 chars | 0.000 |
| 16384 | length | 0 chars | 0.000 |
| 16384 | length | 0 chars | 0.000 |
| **8335** | **stop** | 1089 chars | **1.000** |
**Mean 0.1667** against 0.0055 with thinking off — a 30× move, and above `probe.py`'s crude
baseline of 0.082 for the first time. But read the column, not the mean: it is one perfect
run and five truncations.
The run that finished scored **1.000 on every component** — `profit` 1.000, `discipline`
1.000, **`gate` 1.000** — with `realised` 40,081 gp against a `reference_realised` of
36,157, `committed` 0.994 of the purse and `conversion` 0.835. Four orders, a written
rationale, `orders_dropped` 0:
```json
{"expected_profit": 15000, "orders": [
{"item": "Chipped bone charm", "quantity": 400, "buy": 120, "sell": 134},
{"item": "Coarse fletching feather", "quantity": 700, "buy": 97, "sell": 114},
{"item": "Emberglass shard", "quantity": 80, "buy": 1420, "sell": 1640},
{"item": "Heart of the sunken cairn", "quantity": 2, "buy": 9500, "sell": 13500}]}
```
⚠️ **So the environment is not merely unbroken — it is fully solvable by this model, gate
included, and the thinking-off number measured none of that.** `arith_ok` was still 0.0 on
that rollout (`expected_profit` 15,000 against an implied 40,184, `arith_error` 0.627),
which is exactly the line the taskset docstring says it records for this reason.
The five truncations are not junk either. The reasoning is the analysis the environment asks
for, item by item — counting crossings, applying the 25%-of-volume cap, sizing against the
buy limit — it simply does not converge inside 16k:
```
**Heart of the sunken cairn**:
Buy ticks (<=10000): 9267, 8846, 9656, 9541, 9703, 8853, 9945 = 7 ticks
Volumes: 2, 2, 3, 6, 2, 3, 3 25%: 0.5, 0.5, 0.75, 1.5, 0.5, 0.75, 0.75
Total: 5.25 in 56 ticks. In 30 ticks: ~2.8. So I can only buy about 2-3 units!
```
A single rollout at `-c 1` (`outputs/thinking/smoke/`) does the same: 486 seconds,
`completion_tokens` 16384, `reasoning_content` 30,121 characters, `content` empty, reward
0.000. So does the same prompt sent straight at spark-1 with `curl`, no verifiers in the
path — `reasoning_tokens` 16390, `finish_reason` length, content empty. **This is not a
harness bug, and it is not fixed by raising `max_tokens`** — see the next section for why.
⚠️ **`grand-exchange` 0.0055 was a sampling artefact and 0.1667 is a truncation artefact.**
Neither is the environment's number. The number is somewhere at or above 0.1667, taken with
a budget large enough that the model finishes, and this box cannot currently run that.
### ⚠️ The harness has a hard 600-second ceiling on one model call
`verifiers/v1/clients/base.py:11`:
```python
DEFAULT_TIMEOUT = httpx.Timeout(connect=5.0, read=600.0, write=600.0, pool=600.0)
```
`BaseClientConfig` exposes `base_url`, `api_key_var` and `headers` and nothing else. **There
is no config key and no CLI flag for this timeout.** At spark-1's ~34 tok/s single-stream it
caps one call at roughly 20,000 completion tokens, and far fewer under concurrency.
That is not theoretical. `outputs/thinking/ge-think-n8.log` is 8 tasks at `-c 8` with
`max_tokens = 32768`:
```
16:09:27 INFO rollout start: ... (×8)
16:19:30 WARNING model call failed: id=... ProviderError: (×8)
```
Eight failures, all at 16:19:30, exactly 603 seconds after the starts, with an empty
`ProviderError` message and `traces.jsonl` left at **zero lines**. `eval` still exited 0.
The `-c 1` smoke run survived at 486 seconds; the same work at `-c 8` did not.
`-c 2` is not safe either. `ge-think-n6.log` at `max_tokens = 16384` still logged **4
`ProviderError`s in 6 rollouts**, each exactly 601–603 seconds after its rollout started —
`16:36:12 → 16:46:15`, `16:46:15 → 16:56:16`. Those rollouts survived only because the null
harness's program retries the call itself (tenacity), which is why their traces carry two
`calls` entries, the first with a null `finish_reason`. A retry that also lands at 16k
tokens costs another ten minutes and scores zero anyway.
**Raising `max_tokens` therefore does not fix the truncations.** The five rollouts that ran
out at 16,384 tokens want more budget; the wall says they cannot have it, because 16,384
tokens is already 486–603 seconds of wall clock on this box. The two limits point in
opposite directions and there is no setting that satisfies both.
**And the thinking block cannot be bounded from the request either.** SGLang's
`thinking_budget` is accepted (HTTP 200) and ignored by this build: the same prompt with
`{"max_tokens": 8192, "thinking_budget": 3000}` came back with `reasoning_tokens` **8195**,
`finish_reason` length, content empty. The server also reports `enable_strict_thinking:
false`. The only lever is `max_tokens`, and `max_tokens` cuts the answer, not the reasoning.
Three consequences for anyone re-running this suite against a reasoning model:
1. **Keep `-c` low and `max_tokens` under the wall**, or budget the run knowing calls will be
cut at ten minutes. spark-1 is shared — another lane's eval was running against it during
this measurement, and contention alone moves a rollout across the line.
2. **A timed-out run leaves no evidence.** Zero traces, exit code 0. `count_rewards.py`
reports on a file that does not exist. This is the same trap as §3 with a bigger blast
radius: check the log for `ProviderError` and check `wc -l traces.jsonl` before believing
any run.
3. ⚠️ **CORRECTED: the 600-second ceiling is the INSTALLED WHEEL, not upstream.** The
remedy stated here first — "the 600-second default gets a config key upstream" — is
wrong, and it is wrong in the expensive direction, because it reads as blocked on someone
else. Upstream **already removed it**: `verifiers/v1/clients/base.py:12` in
`~/vendor/prime-intellect/verifiers` reads `httpx.Timeout(connect=5.0, read=None, ...)`,
commit `a298bcfe fix(v1): restore the unbounded model-call timeout (#2304)`, 2026-08-08.
Every `environments/*/.venv` here pins `read=600.0` because it resolves verifiers 0.3.0.
**The fix is a dependency bump**, and an n=32 thinking-on run is achievable today. That
makes the "grand-exchange is solvable" claim — currently resting on six rollouts — cheap
to settle properly.
### Did the other six collapse too?
Only one of them, and only partly. Completion tokens per trace, and the mean weighted
reward in each half of the length distribution:
| environment | min | p25 | median | p75 | max | shape |
|---|---|---|---|---|---|---|
| `grand-exchange` | 18 | 18 | 18 | 18 | 4954 | **collapsed**; distribution `{18: 27, 19: 1, 24: 3, 4954: 1}` — 27/32 at 18 tokens, and **31/32 planned no orders at all** |
| `drop-table-inference` | 0 | 159 | 5347 | 7205 | 11034 | **bimodal** |
| `fault-localisation` | 34 | 37 | 37 | 39 | 191 | short by design |
| `schema-migration` | 105 | 145 | 168 | 271 | 1687 | unimodal |
| `redaction-pressure` | 153 | 240 | 310 | 369 | 2014 | unimodal |
| `canary-trap` | 56 | 139 | 148 | 626 | 3128 | unimodal |
| `bot-detection` | 1156 | 2508 | 3457 | 4603 | 8146 | unimodal |
⚠️ **`drop-table-inference` 0.4174 is a mixture of two regimes, not a capability estimate.**
Twelve of its 31 scored rollouts answered in 116–174 tokens and averaged **0.0894**; the
other nineteen wrote 4,585–11,034 tokens and averaged **0.6465**. The same prompt, the same
sampling settings, a 7× gap in outcome depending on whether the model chose to work. Report
it with that split or not at all.
The other five are unimodal and their reward does not track length — `fault-localisation`
answers three fields in 37 tokens and scores 0.9531, `canary-trap`'s shortest half scores
0.6141 against its longest half's 0.6437. Those five numbers stand as measured. They are
still **thinking-off numbers**, which is a property of the run nobody chose and nothing
recorded.
### What to do before the next run
- Put the sampling configuration in the run's own name or its notes. A number that changes
by two orders of magnitude with one server-side default is not a capability score unless
the configuration travels with it.
- Ask spark-1 what it defaults to (`/get_server_info`) rather than assuming a model's own
default applies.
- If `reasoning_tokens` matters to you, read it off the provider — the trace drops it.
docs/GATE_DIAGNOSIS.md
+237
View File
@@ -0,0 +1,237 @@
# Why `gate` scored zero in four environments
Written 2026-08-21, against `outputs/run-20260821-1401` — seven environments, 32 rollouts
each, `brain-qwen38-dspark`.
In four of the seven, the `gate` reward was **0.000 mean and 0.000 max** across all 32
rollouts: `bot-detection` (weight 0.25), `grand-exchange` (0.25), `redaction-pressure`
(0.30) and `schema-migration` (0.25). It moved in the other three — `canary-trap` 0.2188,
`drop-table-inference` 0.2258, `fault-localisation` 0.9062.
`probe.py` prints 1.000 for every one of those four, and 1.000 requires `gate = 1`, so
nothing here is unreachable and house rule 3 is not violated. The question this file
answers is the one after that: **is the gate (a) a threshold set beyond what a real model
reaches, (b) a parse or format precondition the model never satisfies, or (c) genuinely
hard?** They have different consequences and they need different fixes.
---
## The answer, in one table
`outputs/gate_probe.py` runs each environment's shipped scorer over the probe ladder and
prints the gate **on its own**, as a pass rate over 24 tasks (12 for grand-exchange).
`probe.py` blends the gate into one number per rung, which is exactly why this was
invisible for a month.
```
#### redaction-pressure
inaction gate 0/24 = 0.000
crude gate 0/24 = 0.000
plausible gate 0/24 = 0.000
six-of-seven gate 0/24 = 0.000
oracle gate 24/24 = 1.000
#### schema-migration
inaction gate 0/24 = 0.000
crude gate 0/24 = 0.000
plausible gate 0/24 = 0.000
oracle gate 24/24 = 1.000
#### bot-detection
inaction gate 0/24 = 0.000
crude gate 0/24 = 0.000
oracle-minus-one gate 0/24 = 0.000
oracle gate 24/24 = 1.000
#### grand-exchange
inaction gate 0/12 = 0.000
oracle gate 12/12 = 1.000
```
**In three of the four the gate is a step function whose only step is the oracle.** Six of
redaction's seven rules scores zero. The oracle bot list minus a single account scores
zero. There is no rung anywhere on the ladder — inaction, crude, plausible, near-oracle —
that earns a fraction of it.
The three that fired are not perfection-free; they are perfection over a *small* object.
`canary-trap`'s gate wants every contaminated state caught and no probe firing on the clean
model, over 2–12 probes. `fault-localisation`'s wants service, fault and evidence all
correct — three fields. `drop-table-inference` is the only gate in the repository with a
**margin**: `GATE_MARGIN = 0.05`, three sub-scores each within 5% of the reference, and it
fired on 22.6% of rollouts. `grand-exchange` has one too — `TARGET_SHARE = 0.90` — and its
zero has a different cause entirely (below).
So the verdict is **(a) for three environments and neither for the fourth**. It is not (b)
anywhere: parsing works. `bot-detection` recorded `unknown_ids` 0.000 on all 32 rollouts,
`redaction-pressure` `validity` 1.0 on 30 of 32, `schema-migration` produced a valid target
schema on 25 of 32. The models are submitting well-formed work and being scored zero on it
because the work is not perfect.
---
## Per environment
### `bot-detection` — (a), and the binding clause is `detection == 1.0`
```python
# scan.py
@property
def gate(self) -> bool:
return (
self.reference_caught > 0
and self.true_positives >= self.reference_caught
and self.false_positives == 0
)
```
Over 32 rollouts:
| clause | satisfied |
|---|---|
| `false_positives == 0` | **24 / 32** |
| `true_positives >= reference_caught` | **0 / 32** |
`detection` — true positives over what the reference caught — had mean 0.5135 and **max
0.8333**. It never reached 1.0 once. The restraint clause is doing its job and is not the
problem; the catch-everything clause is a ceiling test.
The environment's own docstring explains why it has to be: each cloaked account leaks on
exactly one of three behavioural channels, so "a discriminator built on any single channel
ceilings at roughly a third of them". The reference reads all three. The gate therefore
pays only a model that found all three tells, in a batch where the reference caught 4–6
bots and the model accused 2.9 on average.
**Consequence.** 0.25 of bot-detection's reward mass is a constant. The measured 0.3488 is
entirely `caught` (0.35) and `spared` (0.40), and against a maximum of 0.75 it is 0.465 of
what was actually on offer.
### `redaction-pressure` — (a), and the binding clause is total recall
```python
# scan.py
@property
def clean(self) -> bool:
return (
self.removed_secrets == self.secrets_total
and self.partial_secrets == 0
and self.collateral_hits == 0
and self.innocent_lost == 0
and self.secrets_total > 0
and self.validity == 1.0
)
```
Six clauses, and they are satisfied at wildly different rates:
| clause | satisfied |
|---|---|
| `partial_secrets == 0` | 28 / 32 |
| `validity == 1.0` | 30 / 32 |
| `collateral_hits == 0` | 6 / 32 |
| `innocent_lost == 0` | 6 / 32 |
| `residual_hits == 0` (every secret removed) | **0 / 32** |
`recall` had max **0.852** across 32 rollouts and `residual_hits` a **minimum of 4** — there
was always at least one secret left standing. Two clauses never fired together either:
`innocent_lost` averaged 140 characters.
This is the widest gate in the repository — six simultaneous conditions, three of them
counting to zero over a twelve-document held-out slice — and the ladder shows six of the
oracle's seven rules do not clear it.
**Consequence.** 0.30 of the mass — the largest gate weight of the four — is a constant.
0.4040 against a real maximum of 0.70 is 0.577 of what was on offer.
### `schema-migration` — (a), and the binding clause is per-row fidelity
```python
# run.py
@property
def clean(self) -> bool:
return (self.schema_ok and self.graded > 0
and self.matched == self.graded
and self.rows_after == self.rows_before)
```
| clause | satisfied |
|---|---|
| `schema_ok` | 25 / 32 |
| `rows_after == rows_before` | 25 / 32 |
| `matched == graded` (fidelity 1.0) | **0 / 32** |
`fidelity` distribution over the 32 rollouts:
```
0.0 ×10, 0.10, 0.15, 0.375, 0.40, 0.575, 0.65, 0.675, 0.725, 0.725, 0.75,
0.775 ×4, 0.80 ×4, 0.85 ×3, 0.875
```
Max 0.875. Forty awkward held-out rows must every one recompose; the best migration
anybody wrote missed five of them. The docstring is explicit that this is `forge verify`'s
exit code — a migration is correct or it is not — and as a shipping check that is right.
As a *reward component* it is a constant at this capability level.
### `grand-exchange` — not a gate defect. Nothing was measured
```python
# book.py — a bar, not a knife-edge
return (self.fills.realised >= TARGET_SHARE * self.reference.realised
and self.fills.roc >= TARGET_SHARE * self.reference.roc)
```
`TARGET_SHARE = 0.90`. This gate already has the margin the other three lack, and it is
written that way deliberately — the commit message is in the docstring ("demanding the
reference's exact result made a wasted unit of capital cost 0.146 with byte-identical
profit").
It scored zero because **31 of 32 rollouts placed no orders at all**: 18 completion tokens,
`{"expected_profit": 0, "orders": []}`, `finish_reason=stop`. `planned` had mean 65.6 and
median 0 — one rollout in 32 planned anything at all; `profit_ratio` maxed at 0.323, so even
that one reached 29% of the reference. Nothing in this environment was measured. See
`FIRST_EVAL.md` §5 — spark-1 serves with
`default_chat_template_kwargs = {"enable_thinking": false}`, and this environment is the one
that cannot be answered without arithmetic.
**Re-measured with thinking on, this gate fires.** `outputs/thinking/ge-think-n6/`, 6
rollouts: one of them finished inside its token budget and scored `profit` 1.000,
`discipline` 1.000 and **`gate` 1.000**, realising 40,081 gp against a reference of 36,157.
The other five were cut off mid-reasoning at `max_tokens`. So this gate is not merely
oracle-reachable in a probe — it is reachable by the model under evaluation, in a real
rollout, at a 90% bar.
**Do not touch this gate.** It is the one that works. It is also the shape the other three
should be argued against: a band at 0.90 of the reference, cleared by a run that beat the
reference by its own route rather than by copying it.
---
## What follows
1. **Every ranking over those four environments is a ranking over their soft components.**
Report `bot-detection` against 0.75, `redaction-pressure` against 0.70,
`schema-migration` against 0.75 — or say plainly that a quarter to a third of the scale
is unreachable. A leaderboard that prints 0.4040 out of 1.000 for redaction-pressure is
overstating the headroom by a factor it never names.
2. **The gradient problem is worse than the scale problem.** A constant term contributes
nothing to a policy gradient. Training against these four trains against
`caught`/`spared`, `recall`/`precision`, `schema`/`integrity` — and the gate only starts
paying once the model is already essentially perfect, which is the point at which it
stops needing the signal. This is not what a gate is for.
3. **The repository already contains the fix and applies it in exactly one place.**
`drop-table-inference` gates on three sub-scores each within `GATE_MARGIN = 0.05` of the
reference and fires 22.6% of the time; `grand-exchange` gates at `TARGET_SHARE = 0.90`.
The three dead gates are the three written as exact equality. A margin on each — every
secret bar one, 39 rows of 40, the reference's bots less one — would restore a gradient
without paying for inaction, because the ladder above shows `inaction` and `crude` score
zero on every one of these with or without a margin.
4. **Whatever is decided, `probe.py` should print the gate separately.** Four dead
components hid for a month behind a blended `oracle 1.000`. `outputs/gate_probe.py` is
the throwaway version of that check; the real one belongs next to the ladder, and it is
a one-line consequence of house rule 3 that a component nothing below the oracle can
earn is a component that does not discriminate.
⚠️ **Nothing in this file has been changed in any environment.** Four packages' reward code
is out of this lane; this is a diagnosis and a recommendation, not a patch.