docs: run-20260821-1401 was sampled with thinking off, and four gates are dead
Two measurement findings, both larger than the lanes that found them. FIRST_EVAL.md: every number in the first run was sampled with thinking OFF — 0 of 224 traces carry reasoning_content or a <think> block, and spark-1 serves with enable_thinking False. No artefact of the run recorded that. Five of the seven numbers still stand; two do not. grand-exchange 0.0055 measured nothing about the environment — with thinking on it scores 0.1667 on six rollouts and one of them scored a clean 1.000, so it is fully solvable. drop-table-inference 0.4174 is a mixture of two regimes (0.0894 short, 0.6465 long) and should be reported split or not at all. The defect is the missing sampling footnote, not the values. The 600-second per-call ceiling that blocked an n=32 thinking-on run is the INSTALLED wheel, not upstream: verifiers fixed it in a298bcfe on 2026-08-08 and 0.3.0 predates it. The remedy is a dependency bump, not a wait. GATE_DIAGNOSIS.md: the `gate` component scored exactly 0.000, max 0.000, across all 32 rollouts in four environments. One shape explains three of them — exact equality. Nothing below the oracle earns a fraction: not the plausible strategy, not six of redaction's seven rules, not the oracle bot list minus one account. bot-detection and schema-migration are a threshold set at the ceiling; redaction-pressure is that and genuinely hard; grand-exchange's zero was the sampling artefact above. The repo already contains the fix and already uses it twice — drop-table's GATE_MARGIN and grand-exchange's TARGET_SHARE are margins, not equalities. Recommended, not yet measured. Consequence worth stating plainly: 25-30% of the reward mass on three environments carries identically zero gradient, so they train against a 0.70-0.75 objective while being scored out of 1.00. The mirror failure exists too — fault-localisation's evidence and fault components are pinned at exactly 1.0 on all 32 rollouts, so half its headline 0.9531 is constant. tools/gate_probe.py is the reproduction, moved out of the gitignored outputs/ so the finding survives. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
+208
-2
@@ -155,14 +155,22 @@ The single errored episode is a genuine spark-1 gateway timeout (`ProviderError`
|
|||||||
### Two numbers not to publish as capability scores
|
### Two numbers not to publish as capability scores
|
||||||
|
|
||||||
⚠️ **`grand-exchange` 0.0055 measures a sampling configuration, not a capability.**
|
⚠️ **`grand-exchange` 0.0055 measures a sampling configuration, not a capability.**
|
||||||
In **31 of 32 rollouts** the model returned a valid, parseable, degenerate answer —
|
In **31 of 32 rollouts** the model returned a valid, parseable, degenerate answer
|
||||||
|
(27 of them in exactly 18 completion tokens; the full distribution is
|
||||||
|
`{18: 27, 19: 1, 24: 3, 4954: 1}`) —
|
||||||
`{"expected_profit": 0, "orders": []}` in 18 completion tokens, `finish_reason=stop`,
|
`{"expected_profit": 0, "orders": []}` in 18 completion tokens, `finish_reason=stop`,
|
||||||
never entering a thinking block. The one rollout that did reason (4,954 completion tokens)
|
never entering a thinking block. The one rollout that did reason (4,954 completion tokens)
|
||||||
placed 4 real orders and scored 0.176 weighted, with `filled=1233` against `planned=2100`.
|
placed 4 real orders and scored 0.176 weighted, with `filled=1233` against `planned=2100`.
|
||||||
**The parser, the execution engine and the reward all work end to end** — the environment
|
**The parser, the execution engine and the reward all work end to end** — the environment
|
||||||
is not broken. But `probe.py` rates the *crude* baseline here at 0.082, so 0.0055 sits
|
is not broken. But `probe.py` rates the *crude* baseline here at 0.082, so 0.0055 sits
|
||||||
**below crude** and barely above inaction. Re-run with thinking forced on before this
|
**below crude** and barely above inaction. Re-run with thinking forced on before this
|
||||||
number goes anywhere near `/evals`.
|
number goes anywhere near `/evals`. **That re-run happened — see §5.** With thinking on the
|
||||||
|
same model scored **1.000 on every component** of the one rollout that finished inside its
|
||||||
|
token budget.
|
||||||
|
|
||||||
|
⚠️ **`drop-table-inference` 0.4174 is two populations, not one.** Twelve of its 31 scored
|
||||||
|
rollouts answered in 116–174 tokens and averaged 0.0894; the other nineteen wrote
|
||||||
|
4,585–11,034 tokens and averaged 0.6465. §5 has the distribution.
|
||||||
|
|
||||||
⚠️ **`fault-localisation` 0.9531 has two of four components pegged at maximum.**
|
⚠️ **`fault-localisation` 0.9531 has two of four components pegged at maximum.**
|
||||||
`evidence` mean 1.000 (max 1.000) and `fault` mean 1.000 (max 1.000) across all 32
|
`evidence` mean 1.000 (max 1.000) and `fault` mean 1.000 (max 1.000) across all 32
|
||||||
@@ -183,3 +191,201 @@ those environments, which requires `gate = 1`, so it is reachable. But a quarter
|
|||||||
of the reward mass produced no gradient in this run, and any ranking or before/after
|
of the reward mass produced no gradient in this run, and any ranking or before/after
|
||||||
training table over those four environments is really a ranking over their two soft
|
training table over those four environments is really a ranking over their two soft
|
||||||
components. Understand that before training against them.
|
components. Understand that before training against them.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. The run was measured with thinking OFF, and nothing in the run says so
|
||||||
|
|
||||||
|
Appended 2026-08-21, after re-measuring `grand-exchange`.
|
||||||
|
|
||||||
|
### The cause
|
||||||
|
|
||||||
|
spark-1's SGLang is started with a default that overrides the chat template:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s http://100.127.247.67:8001/get_server_info | jq '.default_chat_template_kwargs, .reasoning_parser'
|
||||||
|
# {"enable_thinking": false}
|
||||||
|
# "qwen3"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Every one of the 224 episodes in `run-20260821-1401` was sampled with Qwen3's thinking
|
||||||
|
mode disabled**, and not one of them requested otherwise. Confirmed from the traces: no
|
||||||
|
assistant message in any of the seven environments carries a `reasoning_content` field, and
|
||||||
|
none contains a `<think>` block — 0 of 223 scored traces.
|
||||||
|
|
||||||
|
Nothing in the run's own artefacts records this. The resolved `config.toml` says
|
||||||
|
`sampling = {}`, and `ChatDialect.parse_sampling` keeps only an allow-list of known keys, so
|
||||||
|
even a run that *does* pass `chat_template_kwargs` writes a `calls[].sampling` that does not
|
||||||
|
mention it. `usage.reasoning_tokens` is null too: SGLang reports `reasoning_tokens` at the
|
||||||
|
top level of `usage`, while `Usage.from_openai` reads it out of `completion_tokens_details`.
|
||||||
|
**A trace in this repository cannot tell you whether the model was allowed to think.**
|
||||||
|
|
||||||
|
### Turning it on
|
||||||
|
|
||||||
|
`SamplingConfig` is `extra="allow"` and `ChatDialect.apply_overrides` merges the whole dump
|
||||||
|
into the request body, so an untyped key rides through to the provider untouched. There is
|
||||||
|
no CLI flag — `--sampling.*` is typed — so it goes in a config file:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[sampling]
|
||||||
|
chat_template_kwargs = { enable_thinking = true }
|
||||||
|
max_tokens = 16384
|
||||||
|
```
|
||||||
|
|
||||||
|
`outputs/thinking/grand_exchange_thinking.toml` is that file. **Verify it worked from the
|
||||||
|
trace, not from the flag**: the assistant message gains a `reasoning_content` field.
|
||||||
|
|
||||||
|
### What thinking on actually did to `grand-exchange`
|
||||||
|
|
||||||
|
**`outputs/thinking/ge-think-n6/`** — 6 tasks × 1 rollout, `-c 2`, `max_tokens = 16384`,
|
||||||
|
thinking on. All 6 traces carry `reasoning_content`, so the flag reached the model.
|
||||||
|
|
||||||
|
| tokens | `finish_reason` | content | weighted reward |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 16384 | length | 0 chars | 0.000 |
|
||||||
|
| 16384 | length | 0 chars | 0.000 |
|
||||||
|
| 16384 | length | 0 chars | 0.000 |
|
||||||
|
| 16384 | length | 0 chars | 0.000 |
|
||||||
|
| 16384 | length | 0 chars | 0.000 |
|
||||||
|
| **8335** | **stop** | 1089 chars | **1.000** |
|
||||||
|
|
||||||
|
**Mean 0.1667** against 0.0055 with thinking off — a 30× move, and above `probe.py`'s crude
|
||||||
|
baseline of 0.082 for the first time. But read the column, not the mean: it is one perfect
|
||||||
|
run and five truncations.
|
||||||
|
|
||||||
|
The run that finished scored **1.000 on every component** — `profit` 1.000, `discipline`
|
||||||
|
1.000, **`gate` 1.000** — with `realised` 40,081 gp against a `reference_realised` of
|
||||||
|
36,157, `committed` 0.994 of the purse and `conversion` 0.835. Four orders, a written
|
||||||
|
rationale, `orders_dropped` 0:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{"expected_profit": 15000, "orders": [
|
||||||
|
{"item": "Chipped bone charm", "quantity": 400, "buy": 120, "sell": 134},
|
||||||
|
{"item": "Coarse fletching feather", "quantity": 700, "buy": 97, "sell": 114},
|
||||||
|
{"item": "Emberglass shard", "quantity": 80, "buy": 1420, "sell": 1640},
|
||||||
|
{"item": "Heart of the sunken cairn", "quantity": 2, "buy": 9500, "sell": 13500}]}
|
||||||
|
```
|
||||||
|
|
||||||
|
⚠️ **So the environment is not merely unbroken — it is fully solvable by this model, gate
|
||||||
|
included, and the thinking-off number measured none of that.** `arith_ok` was still 0.0 on
|
||||||
|
that rollout (`expected_profit` 15,000 against an implied 40,184, `arith_error` 0.627),
|
||||||
|
which is exactly the line the taskset docstring says it records for this reason.
|
||||||
|
|
||||||
|
The five truncations are not junk either. The reasoning is the analysis the environment asks
|
||||||
|
for, item by item — counting crossings, applying the 25%-of-volume cap, sizing against the
|
||||||
|
buy limit — it simply does not converge inside 16k:
|
||||||
|
|
||||||
|
```
|
||||||
|
**Heart of the sunken cairn**:
|
||||||
|
Buy ticks (<=10000): 9267, 8846, 9656, 9541, 9703, 8853, 9945 = 7 ticks
|
||||||
|
Volumes: 2, 2, 3, 6, 2, 3, 3 25%: 0.5, 0.5, 0.75, 1.5, 0.5, 0.75, 0.75
|
||||||
|
Total: 5.25 in 56 ticks. In 30 ticks: ~2.8. So I can only buy about 2-3 units!
|
||||||
|
```
|
||||||
|
|
||||||
|
A single rollout at `-c 1` (`outputs/thinking/smoke/`) does the same: 486 seconds,
|
||||||
|
`completion_tokens` 16384, `reasoning_content` 30,121 characters, `content` empty, reward
|
||||||
|
0.000. So does the same prompt sent straight at spark-1 with `curl`, no verifiers in the
|
||||||
|
path — `reasoning_tokens` 16390, `finish_reason` length, content empty. **This is not a
|
||||||
|
harness bug, and it is not fixed by raising `max_tokens`** — see the next section for why.
|
||||||
|
|
||||||
|
⚠️ **`grand-exchange` 0.0055 was a sampling artefact and 0.1667 is a truncation artefact.**
|
||||||
|
Neither is the environment's number. The number is somewhere at or above 0.1667, taken with
|
||||||
|
a budget large enough that the model finishes, and this box cannot currently run that.
|
||||||
|
|
||||||
|
### ⚠️ The harness has a hard 600-second ceiling on one model call
|
||||||
|
|
||||||
|
`verifiers/v1/clients/base.py:11`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
DEFAULT_TIMEOUT = httpx.Timeout(connect=5.0, read=600.0, write=600.0, pool=600.0)
|
||||||
|
```
|
||||||
|
|
||||||
|
`BaseClientConfig` exposes `base_url`, `api_key_var` and `headers` and nothing else. **There
|
||||||
|
is no config key and no CLI flag for this timeout.** At spark-1's ~34 tok/s single-stream it
|
||||||
|
caps one call at roughly 20,000 completion tokens, and far fewer under concurrency.
|
||||||
|
|
||||||
|
That is not theoretical. `outputs/thinking/ge-think-n8.log` is 8 tasks at `-c 8` with
|
||||||
|
`max_tokens = 32768`:
|
||||||
|
|
||||||
|
```
|
||||||
|
16:09:27 INFO rollout start: ... (×8)
|
||||||
|
16:19:30 WARNING model call failed: id=... ProviderError: (×8)
|
||||||
|
```
|
||||||
|
|
||||||
|
Eight failures, all at 16:19:30, exactly 603 seconds after the starts, with an empty
|
||||||
|
`ProviderError` message and `traces.jsonl` left at **zero lines**. `eval` still exited 0.
|
||||||
|
The `-c 1` smoke run survived at 486 seconds; the same work at `-c 8` did not.
|
||||||
|
|
||||||
|
`-c 2` is not safe either. `ge-think-n6.log` at `max_tokens = 16384` still logged **4
|
||||||
|
`ProviderError`s in 6 rollouts**, each exactly 601–603 seconds after its rollout started —
|
||||||
|
`16:36:12 → 16:46:15`, `16:46:15 → 16:56:16`. Those rollouts survived only because the null
|
||||||
|
harness's program retries the call itself (tenacity), which is why their traces carry two
|
||||||
|
`calls` entries, the first with a null `finish_reason`. A retry that also lands at 16k
|
||||||
|
tokens costs another ten minutes and scores zero anyway.
|
||||||
|
|
||||||
|
**Raising `max_tokens` therefore does not fix the truncations.** The five rollouts that ran
|
||||||
|
out at 16,384 tokens want more budget; the wall says they cannot have it, because 16,384
|
||||||
|
tokens is already 486–603 seconds of wall clock on this box. The two limits point in
|
||||||
|
opposite directions and there is no setting that satisfies both.
|
||||||
|
|
||||||
|
**And the thinking block cannot be bounded from the request either.** SGLang's
|
||||||
|
`thinking_budget` is accepted (HTTP 200) and ignored by this build: the same prompt with
|
||||||
|
`{"max_tokens": 8192, "thinking_budget": 3000}` came back with `reasoning_tokens` **8195**,
|
||||||
|
`finish_reason` length, content empty. The server also reports `enable_strict_thinking:
|
||||||
|
false`. The only lever is `max_tokens`, and `max_tokens` cuts the answer, not the reasoning.
|
||||||
|
|
||||||
|
Three consequences for anyone re-running this suite against a reasoning model:
|
||||||
|
|
||||||
|
1. **Keep `-c` low and `max_tokens` under the wall**, or budget the run knowing calls will be
|
||||||
|
cut at ten minutes. spark-1 is shared — another lane's eval was running against it during
|
||||||
|
this measurement, and contention alone moves a rollout across the line.
|
||||||
|
2. **A timed-out run leaves no evidence.** Zero traces, exit code 0. `count_rewards.py`
|
||||||
|
reports on a file that does not exist. This is the same trap as §3 with a bigger blast
|
||||||
|
radius: check the log for `ProviderError` and check `wc -l traces.jsonl` before believing
|
||||||
|
any run.
|
||||||
|
3. ⚠️ **CORRECTED: the 600-second ceiling is the INSTALLED WHEEL, not upstream.** The
|
||||||
|
remedy stated here first — "the 600-second default gets a config key upstream" — is
|
||||||
|
wrong, and it is wrong in the expensive direction, because it reads as blocked on someone
|
||||||
|
else. Upstream **already removed it**: `verifiers/v1/clients/base.py:12` in
|
||||||
|
`~/vendor/prime-intellect/verifiers` reads `httpx.Timeout(connect=5.0, read=None, ...)`,
|
||||||
|
commit `a298bcfe fix(v1): restore the unbounded model-call timeout (#2304)`, 2026-08-08.
|
||||||
|
Every `environments/*/.venv` here pins `read=600.0` because it resolves verifiers 0.3.0.
|
||||||
|
**The fix is a dependency bump**, and an n=32 thinking-on run is achievable today. That
|
||||||
|
makes the "grand-exchange is solvable" claim — currently resting on six rollouts — cheap
|
||||||
|
to settle properly.
|
||||||
|
|
||||||
|
### Did the other six collapse too?
|
||||||
|
|
||||||
|
Only one of them, and only partly. Completion tokens per trace, and the mean weighted
|
||||||
|
reward in each half of the length distribution:
|
||||||
|
|
||||||
|
| environment | min | p25 | median | p75 | max | shape |
|
||||||
|
|---|---|---|---|---|---|---|
|
||||||
|
| `grand-exchange` | 18 | 18 | 18 | 18 | 4954 | **collapsed**; distribution `{18: 27, 19: 1, 24: 3, 4954: 1}` — 27/32 at 18 tokens, and **31/32 planned no orders at all** |
|
||||||
|
| `drop-table-inference` | 0 | 159 | 5347 | 7205 | 11034 | **bimodal** |
|
||||||
|
| `fault-localisation` | 34 | 37 | 37 | 39 | 191 | short by design |
|
||||||
|
| `schema-migration` | 105 | 145 | 168 | 271 | 1687 | unimodal |
|
||||||
|
| `redaction-pressure` | 153 | 240 | 310 | 369 | 2014 | unimodal |
|
||||||
|
| `canary-trap` | 56 | 139 | 148 | 626 | 3128 | unimodal |
|
||||||
|
| `bot-detection` | 1156 | 2508 | 3457 | 4603 | 8146 | unimodal |
|
||||||
|
|
||||||
|
⚠️ **`drop-table-inference` 0.4174 is a mixture of two regimes, not a capability estimate.**
|
||||||
|
Twelve of its 31 scored rollouts answered in 116–174 tokens and averaged **0.0894**; the
|
||||||
|
other nineteen wrote 4,585–11,034 tokens and averaged **0.6465**. The same prompt, the same
|
||||||
|
sampling settings, a 7× gap in outcome depending on whether the model chose to work. Report
|
||||||
|
it with that split or not at all.
|
||||||
|
|
||||||
|
The other five are unimodal and their reward does not track length — `fault-localisation`
|
||||||
|
answers three fields in 37 tokens and scores 0.9531, `canary-trap`'s shortest half scores
|
||||||
|
0.6141 against its longest half's 0.6437. Those five numbers stand as measured. They are
|
||||||
|
still **thinking-off numbers**, which is a property of the run nobody chose and nothing
|
||||||
|
recorded.
|
||||||
|
|
||||||
|
### What to do before the next run
|
||||||
|
|
||||||
|
- Put the sampling configuration in the run's own name or its notes. A number that changes
|
||||||
|
by two orders of magnitude with one server-side default is not a capability score unless
|
||||||
|
the configuration travels with it.
|
||||||
|
- Ask spark-1 what it defaults to (`/get_server_info`) rather than assuming a model's own
|
||||||
|
default applies.
|
||||||
|
- If `reasoning_tokens` matters to you, read it off the provider — the trace drops it.
|
||||||
|
|||||||
@@ -0,0 +1,237 @@
|
|||||||
|
# Why `gate` scored zero in four environments
|
||||||
|
|
||||||
|
Written 2026-08-21, against `outputs/run-20260821-1401` — seven environments, 32 rollouts
|
||||||
|
each, `brain-qwen38-dspark`.
|
||||||
|
|
||||||
|
In four of the seven, the `gate` reward was **0.000 mean and 0.000 max** across all 32
|
||||||
|
rollouts: `bot-detection` (weight 0.25), `grand-exchange` (0.25), `redaction-pressure`
|
||||||
|
(0.30) and `schema-migration` (0.25). It moved in the other three — `canary-trap` 0.2188,
|
||||||
|
`drop-table-inference` 0.2258, `fault-localisation` 0.9062.
|
||||||
|
|
||||||
|
`probe.py` prints 1.000 for every one of those four, and 1.000 requires `gate = 1`, so
|
||||||
|
nothing here is unreachable and house rule 3 is not violated. The question this file
|
||||||
|
answers is the one after that: **is the gate (a) a threshold set beyond what a real model
|
||||||
|
reaches, (b) a parse or format precondition the model never satisfies, or (c) genuinely
|
||||||
|
hard?** They have different consequences and they need different fixes.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The answer, in one table
|
||||||
|
|
||||||
|
`outputs/gate_probe.py` runs each environment's shipped scorer over the probe ladder and
|
||||||
|
prints the gate **on its own**, as a pass rate over 24 tasks (12 for grand-exchange).
|
||||||
|
`probe.py` blends the gate into one number per rung, which is exactly why this was
|
||||||
|
invisible for a month.
|
||||||
|
|
||||||
|
```
|
||||||
|
#### redaction-pressure
|
||||||
|
inaction gate 0/24 = 0.000
|
||||||
|
crude gate 0/24 = 0.000
|
||||||
|
plausible gate 0/24 = 0.000
|
||||||
|
six-of-seven gate 0/24 = 0.000
|
||||||
|
oracle gate 24/24 = 1.000
|
||||||
|
|
||||||
|
#### schema-migration
|
||||||
|
inaction gate 0/24 = 0.000
|
||||||
|
crude gate 0/24 = 0.000
|
||||||
|
plausible gate 0/24 = 0.000
|
||||||
|
oracle gate 24/24 = 1.000
|
||||||
|
|
||||||
|
#### bot-detection
|
||||||
|
inaction gate 0/24 = 0.000
|
||||||
|
crude gate 0/24 = 0.000
|
||||||
|
oracle-minus-one gate 0/24 = 0.000
|
||||||
|
oracle gate 24/24 = 1.000
|
||||||
|
|
||||||
|
#### grand-exchange
|
||||||
|
inaction gate 0/12 = 0.000
|
||||||
|
oracle gate 12/12 = 1.000
|
||||||
|
```
|
||||||
|
|
||||||
|
**In three of the four the gate is a step function whose only step is the oracle.** Six of
|
||||||
|
redaction's seven rules scores zero. The oracle bot list minus a single account scores
|
||||||
|
zero. There is no rung anywhere on the ladder — inaction, crude, plausible, near-oracle —
|
||||||
|
that earns a fraction of it.
|
||||||
|
|
||||||
|
The three that fired are not perfection-free; they are perfection over a *small* object.
|
||||||
|
`canary-trap`'s gate wants every contaminated state caught and no probe firing on the clean
|
||||||
|
model, over 2–12 probes. `fault-localisation`'s wants service, fault and evidence all
|
||||||
|
correct — three fields. `drop-table-inference` is the only gate in the repository with a
|
||||||
|
**margin**: `GATE_MARGIN = 0.05`, three sub-scores each within 5% of the reference, and it
|
||||||
|
fired on 22.6% of rollouts. `grand-exchange` has one too — `TARGET_SHARE = 0.90` — and its
|
||||||
|
zero has a different cause entirely (below).
|
||||||
|
|
||||||
|
So the verdict is **(a) for three environments and neither for the fourth**. It is not (b)
|
||||||
|
anywhere: parsing works. `bot-detection` recorded `unknown_ids` 0.000 on all 32 rollouts,
|
||||||
|
`redaction-pressure` `validity` 1.0 on 30 of 32, `schema-migration` produced a valid target
|
||||||
|
schema on 25 of 32. The models are submitting well-formed work and being scored zero on it
|
||||||
|
because the work is not perfect.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Per environment
|
||||||
|
|
||||||
|
### `bot-detection` — (a), and the binding clause is `detection == 1.0`
|
||||||
|
|
||||||
|
```python
|
||||||
|
# scan.py
|
||||||
|
@property
|
||||||
|
def gate(self) -> bool:
|
||||||
|
return (
|
||||||
|
self.reference_caught > 0
|
||||||
|
and self.true_positives >= self.reference_caught
|
||||||
|
and self.false_positives == 0
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
Over 32 rollouts:
|
||||||
|
|
||||||
|
| clause | satisfied |
|
||||||
|
|---|---|
|
||||||
|
| `false_positives == 0` | **24 / 32** |
|
||||||
|
| `true_positives >= reference_caught` | **0 / 32** |
|
||||||
|
|
||||||
|
`detection` — true positives over what the reference caught — had mean 0.5135 and **max
|
||||||
|
0.8333**. It never reached 1.0 once. The restraint clause is doing its job and is not the
|
||||||
|
problem; the catch-everything clause is a ceiling test.
|
||||||
|
|
||||||
|
The environment's own docstring explains why it has to be: each cloaked account leaks on
|
||||||
|
exactly one of three behavioural channels, so "a discriminator built on any single channel
|
||||||
|
ceilings at roughly a third of them". The reference reads all three. The gate therefore
|
||||||
|
pays only a model that found all three tells, in a batch where the reference caught 4–6
|
||||||
|
bots and the model accused 2.9 on average.
|
||||||
|
|
||||||
|
**Consequence.** 0.25 of bot-detection's reward mass is a constant. The measured 0.3488 is
|
||||||
|
entirely `caught` (0.35) and `spared` (0.40), and against a maximum of 0.75 it is 0.465 of
|
||||||
|
what was actually on offer.
|
||||||
|
|
||||||
|
### `redaction-pressure` — (a), and the binding clause is total recall
|
||||||
|
|
||||||
|
```python
|
||||||
|
# scan.py
|
||||||
|
@property
|
||||||
|
def clean(self) -> bool:
|
||||||
|
return (
|
||||||
|
self.removed_secrets == self.secrets_total
|
||||||
|
and self.partial_secrets == 0
|
||||||
|
and self.collateral_hits == 0
|
||||||
|
and self.innocent_lost == 0
|
||||||
|
and self.secrets_total > 0
|
||||||
|
and self.validity == 1.0
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
Six clauses, and they are satisfied at wildly different rates:
|
||||||
|
|
||||||
|
| clause | satisfied |
|
||||||
|
|---|---|
|
||||||
|
| `partial_secrets == 0` | 28 / 32 |
|
||||||
|
| `validity == 1.0` | 30 / 32 |
|
||||||
|
| `collateral_hits == 0` | 6 / 32 |
|
||||||
|
| `innocent_lost == 0` | 6 / 32 |
|
||||||
|
| `residual_hits == 0` (every secret removed) | **0 / 32** |
|
||||||
|
|
||||||
|
`recall` had max **0.852** across 32 rollouts and `residual_hits` a **minimum of 4** — there
|
||||||
|
was always at least one secret left standing. Two clauses never fired together either:
|
||||||
|
`innocent_lost` averaged 140 characters.
|
||||||
|
|
||||||
|
This is the widest gate in the repository — six simultaneous conditions, three of them
|
||||||
|
counting to zero over a twelve-document held-out slice — and the ladder shows six of the
|
||||||
|
oracle's seven rules do not clear it.
|
||||||
|
|
||||||
|
**Consequence.** 0.30 of the mass — the largest gate weight of the four — is a constant.
|
||||||
|
0.4040 against a real maximum of 0.70 is 0.577 of what was on offer.
|
||||||
|
|
||||||
|
### `schema-migration` — (a), and the binding clause is per-row fidelity
|
||||||
|
|
||||||
|
```python
|
||||||
|
# run.py
|
||||||
|
@property
|
||||||
|
def clean(self) -> bool:
|
||||||
|
return (self.schema_ok and self.graded > 0
|
||||||
|
and self.matched == self.graded
|
||||||
|
and self.rows_after == self.rows_before)
|
||||||
|
```
|
||||||
|
|
||||||
|
| clause | satisfied |
|
||||||
|
|---|---|
|
||||||
|
| `schema_ok` | 25 / 32 |
|
||||||
|
| `rows_after == rows_before` | 25 / 32 |
|
||||||
|
| `matched == graded` (fidelity 1.0) | **0 / 32** |
|
||||||
|
|
||||||
|
`fidelity` distribution over the 32 rollouts:
|
||||||
|
|
||||||
|
```
|
||||||
|
0.0 ×10, 0.10, 0.15, 0.375, 0.40, 0.575, 0.65, 0.675, 0.725, 0.725, 0.75,
|
||||||
|
0.775 ×4, 0.80 ×4, 0.85 ×3, 0.875
|
||||||
|
```
|
||||||
|
|
||||||
|
Max 0.875. Forty awkward held-out rows must every one recompose; the best migration
|
||||||
|
anybody wrote missed five of them. The docstring is explicit that this is `forge verify`'s
|
||||||
|
exit code — a migration is correct or it is not — and as a shipping check that is right.
|
||||||
|
As a *reward component* it is a constant at this capability level.
|
||||||
|
|
||||||
|
### `grand-exchange` — not a gate defect. Nothing was measured
|
||||||
|
|
||||||
|
```python
|
||||||
|
# book.py — a bar, not a knife-edge
|
||||||
|
return (self.fills.realised >= TARGET_SHARE * self.reference.realised
|
||||||
|
and self.fills.roc >= TARGET_SHARE * self.reference.roc)
|
||||||
|
```
|
||||||
|
|
||||||
|
`TARGET_SHARE = 0.90`. This gate already has the margin the other three lack, and it is
|
||||||
|
written that way deliberately — the commit message is in the docstring ("demanding the
|
||||||
|
reference's exact result made a wasted unit of capital cost 0.146 with byte-identical
|
||||||
|
profit").
|
||||||
|
|
||||||
|
It scored zero because **31 of 32 rollouts placed no orders at all**: 18 completion tokens,
|
||||||
|
`{"expected_profit": 0, "orders": []}`, `finish_reason=stop`. `planned` had mean 65.6 and
|
||||||
|
median 0 — one rollout in 32 planned anything at all; `profit_ratio` maxed at 0.323, so even
|
||||||
|
that one reached 29% of the reference. Nothing in this environment was measured. See
|
||||||
|
`FIRST_EVAL.md` §5 — spark-1 serves with
|
||||||
|
`default_chat_template_kwargs = {"enable_thinking": false}`, and this environment is the one
|
||||||
|
that cannot be answered without arithmetic.
|
||||||
|
|
||||||
|
**Re-measured with thinking on, this gate fires.** `outputs/thinking/ge-think-n6/`, 6
|
||||||
|
rollouts: one of them finished inside its token budget and scored `profit` 1.000,
|
||||||
|
`discipline` 1.000 and **`gate` 1.000**, realising 40,081 gp against a reference of 36,157.
|
||||||
|
The other five were cut off mid-reasoning at `max_tokens`. So this gate is not merely
|
||||||
|
oracle-reachable in a probe — it is reachable by the model under evaluation, in a real
|
||||||
|
rollout, at a 90% bar.
|
||||||
|
|
||||||
|
**Do not touch this gate.** It is the one that works. It is also the shape the other three
|
||||||
|
should be argued against: a band at 0.90 of the reference, cleared by a run that beat the
|
||||||
|
reference by its own route rather than by copying it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What follows
|
||||||
|
|
||||||
|
1. **Every ranking over those four environments is a ranking over their soft components.**
|
||||||
|
Report `bot-detection` against 0.75, `redaction-pressure` against 0.70,
|
||||||
|
`schema-migration` against 0.75 — or say plainly that a quarter to a third of the scale
|
||||||
|
is unreachable. A leaderboard that prints 0.4040 out of 1.000 for redaction-pressure is
|
||||||
|
overstating the headroom by a factor it never names.
|
||||||
|
|
||||||
|
2. **The gradient problem is worse than the scale problem.** A constant term contributes
|
||||||
|
nothing to a policy gradient. Training against these four trains against
|
||||||
|
`caught`/`spared`, `recall`/`precision`, `schema`/`integrity` — and the gate only starts
|
||||||
|
paying once the model is already essentially perfect, which is the point at which it
|
||||||
|
stops needing the signal. This is not what a gate is for.
|
||||||
|
|
||||||
|
3. **The repository already contains the fix and applies it in exactly one place.**
|
||||||
|
`drop-table-inference` gates on three sub-scores each within `GATE_MARGIN = 0.05` of the
|
||||||
|
reference and fires 22.6% of the time; `grand-exchange` gates at `TARGET_SHARE = 0.90`.
|
||||||
|
The three dead gates are the three written as exact equality. A margin on each — every
|
||||||
|
secret bar one, 39 rows of 40, the reference's bots less one — would restore a gradient
|
||||||
|
without paying for inaction, because the ladder above shows `inaction` and `crude` score
|
||||||
|
zero on every one of these with or without a margin.
|
||||||
|
|
||||||
|
4. **Whatever is decided, `probe.py` should print the gate separately.** Four dead
|
||||||
|
components hid for a month behind a blended `oracle 1.000`. `outputs/gate_probe.py` is
|
||||||
|
the throwaway version of that check; the real one belongs next to the ladder, and it is
|
||||||
|
a one-line consequence of house rule 3 that a component nothing below the oracle can
|
||||||
|
earn is a component that does not discriminate.
|
||||||
|
|
||||||
|
⚠️ **Nothing in this file has been changed in any environment.** Four packages' reward code
|
||||||
|
is out of this lane; this is a diagnosis and a recommendation, not a patch.
|
||||||
@@ -0,0 +1,155 @@
|
|||||||
|
"""What the `gate` component pays the probe ladder, rung by rung.
|
||||||
|
|
||||||
|
`probe.py` reports one blended number per rung, so a gate that only ever fires for
|
||||||
|
the oracle is invisible in its table. This prints the gate on its own — the share of
|
||||||
|
tasks each rung clears it on — for the four environments where the first eval measured
|
||||||
|
`gate` 0.000 mean, 0.000 max over 32 rollouts.
|
||||||
|
|
||||||
|
Read-only: it imports the shipped scorers and changes nothing.
|
||||||
|
|
||||||
|
uv run --project environments/grand_exchange python outputs/gate_probe.py
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
import types
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parent.parent
|
||||||
|
ENVS = ROOT / "environments"
|
||||||
|
for pkg in ("redaction_pressure", "schema_migration", "bot_detection", "grand_exchange"):
|
||||||
|
shim = types.ModuleType(pkg)
|
||||||
|
shim.__path__ = [str(ENVS / pkg / pkg)]
|
||||||
|
sys.modules[pkg] = shim
|
||||||
|
|
||||||
|
TASKS = 24
|
||||||
|
|
||||||
|
|
||||||
|
def rate(name: str, rows: dict[str, list[bool]]) -> None:
|
||||||
|
print(f"\n#### {name}")
|
||||||
|
for rung, hits in rows.items():
|
||||||
|
print(f" {rung:12s} gate {sum(hits)}/{len(hits)} = {sum(hits) / len(hits):.3f}")
|
||||||
|
|
||||||
|
|
||||||
|
def redaction() -> None:
|
||||||
|
from redaction_pressure.corpus import build_slices
|
||||||
|
from redaction_pressure.scan import measure, parse_rules
|
||||||
|
|
||||||
|
def run(rules: list[dict]) -> list[bool]:
|
||||||
|
out = []
|
||||||
|
for i in range(TASKS):
|
||||||
|
_, held = build_slices(200_000 + i, 4, 12)
|
||||||
|
o = measure(held, parse_rules("```json\n" + json.dumps(rules) + "\n```"))
|
||||||
|
out.append(bool(o.clean) and o.validity == 1.0)
|
||||||
|
return out
|
||||||
|
|
||||||
|
oracle = [
|
||||||
|
{"pattern": r"(?<=Customer pasted their key )sk-live-[a-z0-9]{24}", "replacement": "[REDACTED]"},
|
||||||
|
{"pattern": r"(?<=Reply-to was )[a-z]+[.][a-z]+@[a-z]{6}[.]com", "replacement": "[REDACTED]"},
|
||||||
|
{"pattern": r"(?<=Callback number )[+]1-[0-9]{3}-[0-9]{3}-[0-9]{4}", "replacement": "[REDACTED]"},
|
||||||
|
{"pattern": r"(?<=They read the card )[0-9]{4}(?:-[0-9]{4}){3}", "replacement": "[REDACTED]"},
|
||||||
|
{"pattern": r"(?<=Trace pointed at )[a-z]{5}-db-[1-9][.]internal[.]lumbridge", "replacement": "[REDACTED]"},
|
||||||
|
{"pattern": r"(?<=Session )[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}", "replacement": "[REDACTED]"},
|
||||||
|
{"pattern": r"(?<=Escalated by )(?:Dana|Priya|Marcus|Ines|Tomas|Rui|Nadia|Owen) (?:Okafor|Lindqvist|Baptiste|Moreau|Sato|Ahmed|Vance|Duarte)", "replacement": "[REDACTED]"},
|
||||||
|
]
|
||||||
|
rate("redaction-pressure", {
|
||||||
|
"inaction": run([]),
|
||||||
|
"crude": run([{"pattern": r"\S+", "replacement": "[REDACTED]"}]),
|
||||||
|
"plausible": run(oracle[:4]),
|
||||||
|
"six-of-seven": run(oracle[:6]),
|
||||||
|
"oracle": run(oracle),
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
def schema() -> None:
|
||||||
|
from schema_migration.run import measure
|
||||||
|
|
||||||
|
def run(sql: str) -> list[bool]:
|
||||||
|
return [bool(measure(30_000 + i, 40, sql).clean) for i in range(TASKS)]
|
||||||
|
|
||||||
|
naive = ("ALTER TABLE readings ADD COLUMN value_num REAL;"
|
||||||
|
"ALTER TABLE readings ADD COLUMN unit TEXT;"
|
||||||
|
"UPDATE readings SET value_num=CAST(substr(value_text,1,instr(value_text,' ')-1) AS REAL),"
|
||||||
|
" unit=substr(value_text,instr(value_text,' ')+1);"
|
||||||
|
"ALTER TABLE readings DROP COLUMN value_text;")
|
||||||
|
clean = "replace(value_text,',','')"
|
||||||
|
oracle = (f"ALTER TABLE readings ADD COLUMN value_num REAL;"
|
||||||
|
f"ALTER TABLE readings ADD COLUMN unit TEXT;"
|
||||||
|
f"UPDATE readings SET"
|
||||||
|
f" value_num=CAST(CASE WHEN instr({clean},' ')>0"
|
||||||
|
f" THEN substr({clean},1,instr({clean},' ')-1) ELSE {clean} END AS REAL),"
|
||||||
|
f" unit=CASE WHEN instr({clean},' ')>0"
|
||||||
|
f" THEN trim(substr({clean},instr({clean},' ')+1)) ELSE '' END;"
|
||||||
|
f"ALTER TABLE readings DROP COLUMN value_text;")
|
||||||
|
rate("schema-migration", {
|
||||||
|
"inaction": run(""),
|
||||||
|
"crude": run("ALTER TABLE readings ADD COLUMN value_num REAL; ALTER TABLE readings ADD COLUMN unit TEXT;"),
|
||||||
|
"plausible": run(naive),
|
||||||
|
"oracle": run(oracle),
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
def bots() -> None:
|
||||||
|
from bot_detection.accounts import build_slices
|
||||||
|
from bot_detection.scan import measure, parse_accusations, reference_bots
|
||||||
|
|
||||||
|
def run(fn) -> list[bool]:
|
||||||
|
out = []
|
||||||
|
for i in range(TASKS):
|
||||||
|
_, batch = build_slices(80_000 + i, 12)
|
||||||
|
names = fn(batch)
|
||||||
|
o = measure(batch, parse_accusations("```json\n" + json.dumps(names) + "\n```"))
|
||||||
|
out.append(bool(o.gate))
|
||||||
|
return out
|
||||||
|
|
||||||
|
def drop_one(batch):
|
||||||
|
found = sorted(reference_bots(batch))
|
||||||
|
return found[:-1]
|
||||||
|
|
||||||
|
rate("bot-detection", {
|
||||||
|
"inaction": run(lambda b: []),
|
||||||
|
"crude": run(lambda b: [a.acc_id for a in b]),
|
||||||
|
"oracle-minus-one": run(drop_one),
|
||||||
|
"oracle": run(lambda b: sorted(reference_bots(b))),
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
def exchange() -> None:
|
||||||
|
from grand_exchange.book import measure, parse_orders, reference_orders, viable_market
|
||||||
|
from grand_exchange.market import SEED_BASE
|
||||||
|
|
||||||
|
markets, seed = [], SEED_BASE
|
||||||
|
while len(markets) < 12:
|
||||||
|
m = viable_market(seed, 5, 56, 30)
|
||||||
|
markets.append(m)
|
||||||
|
seed = m.seed + 1
|
||||||
|
|
||||||
|
def reply(orders) -> str:
|
||||||
|
return "```json\n" + json.dumps({
|
||||||
|
"expected_profit": 0,
|
||||||
|
"orders": [
|
||||||
|
{"item": o.item, "quantity": o.quantity, "buy": o.buy, "sell": o.sell}
|
||||||
|
for o in orders
|
||||||
|
],
|
||||||
|
}) + "\n```"
|
||||||
|
|
||||||
|
def run(fn) -> list[bool]:
|
||||||
|
out = []
|
||||||
|
for m in markets:
|
||||||
|
orders, stated = parse_orders(fn(m))
|
||||||
|
out.append(bool(measure(m, orders, stated).clean))
|
||||||
|
return out
|
||||||
|
|
||||||
|
rate("grand-exchange", {
|
||||||
|
"inaction": run(lambda m: "No trades today."),
|
||||||
|
"oracle": run(lambda m: reply(reference_orders(m))),
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
redaction()
|
||||||
|
schema()
|
||||||
|
bots()
|
||||||
|
exchange()
|
||||||
Reference in New Issue
Block a user