From 8048a61a1b992c757a41ce6ae89a6f756a131376 Mon Sep 17 00:00:00 2001 From: Kartios Date: Fri, 21 Aug 2026 17:44:46 -0700 Subject: [PATCH] docs: run-20260821-1401 was sampled with thinking off, and four gates are dead MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two measurement findings, both larger than the lanes that found them. FIRST_EVAL.md: every number in the first run was sampled with thinking OFF — 0 of 224 traces carry reasoning_content or a block, and spark-1 serves with enable_thinking False. No artefact of the run recorded that. Five of the seven numbers still stand; two do not. grand-exchange 0.0055 measured nothing about the environment — with thinking on it scores 0.1667 on six rollouts and one of them scored a clean 1.000, so it is fully solvable. drop-table-inference 0.4174 is a mixture of two regimes (0.0894 short, 0.6465 long) and should be reported split or not at all. The defect is the missing sampling footnote, not the values. The 600-second per-call ceiling that blocked an n=32 thinking-on run is the INSTALLED wheel, not upstream: verifiers fixed it in a298bcfe on 2026-08-08 and 0.3.0 predates it. The remedy is a dependency bump, not a wait. GATE_DIAGNOSIS.md: the `gate` component scored exactly 0.000, max 0.000, across all 32 rollouts in four environments. One shape explains three of them — exact equality. Nothing below the oracle earns a fraction: not the plausible strategy, not six of redaction's seven rules, not the oracle bot list minus one account. bot-detection and schema-migration are a threshold set at the ceiling; redaction-pressure is that and genuinely hard; grand-exchange's zero was the sampling artefact above. The repo already contains the fix and already uses it twice — drop-table's GATE_MARGIN and grand-exchange's TARGET_SHARE are margins, not equalities. Recommended, not yet measured. Consequence worth stating plainly: 25-30% of the reward mass on three environments carries identically zero gradient, so they train against a 0.70-0.75 objective while being scored out of 1.00. The mirror failure exists too — fault-localisation's evidence and fault components are pinned at exactly 1.0 on all 32 rollouts, so half its headline 0.9531 is constant. tools/gate_probe.py is the reproduction, moved out of the gitignored outputs/ so the finding survives. Co-Authored-By: Claude Opus 5 (1M context) --- docs/FIRST_EVAL.md | 210 +++++++++++++++++++++++++++++++++++- docs/GATE_DIAGNOSIS.md | 237 +++++++++++++++++++++++++++++++++++++++++ tools/gate_probe.py | 155 +++++++++++++++++++++++++++ 3 files changed, 600 insertions(+), 2 deletions(-) create mode 100644 docs/GATE_DIAGNOSIS.md create mode 100644 tools/gate_probe.py diff --git a/docs/FIRST_EVAL.md b/docs/FIRST_EVAL.md index 488bfb7..3292d99 100644 --- a/docs/FIRST_EVAL.md +++ b/docs/FIRST_EVAL.md @@ -155,14 +155,22 @@ The single errored episode is a genuine spark-1 gateway timeout (`ProviderError` ### Two numbers not to publish as capability scores ⚠️ **`grand-exchange` 0.0055 measures a sampling configuration, not a capability.** -In **31 of 32 rollouts** the model returned a valid, parseable, degenerate answer — +In **31 of 32 rollouts** the model returned a valid, parseable, degenerate answer +(27 of them in exactly 18 completion tokens; the full distribution is +`{18: 27, 19: 1, 24: 3, 4954: 1}`) — `{"expected_profit": 0, "orders": []}` in 18 completion tokens, `finish_reason=stop`, never entering a thinking block. The one rollout that did reason (4,954 completion tokens) placed 4 real orders and scored 0.176 weighted, with `filled=1233` against `planned=2100`. **The parser, the execution engine and the reward all work end to end** — the environment is not broken. But `probe.py` rates the *crude* baseline here at 0.082, so 0.0055 sits **below crude** and barely above inaction. Re-run with thinking forced on before this -number goes anywhere near `/evals`. +number goes anywhere near `/evals`. **That re-run happened — see §5.** With thinking on the +same model scored **1.000 on every component** of the one rollout that finished inside its +token budget. + +⚠️ **`drop-table-inference` 0.4174 is two populations, not one.** Twelve of its 31 scored +rollouts answered in 116–174 tokens and averaged 0.0894; the other nineteen wrote +4,585–11,034 tokens and averaged 0.6465. §5 has the distribution. ⚠️ **`fault-localisation` 0.9531 has two of four components pegged at maximum.** `evidence` mean 1.000 (max 1.000) and `fault` mean 1.000 (max 1.000) across all 32 @@ -183,3 +191,201 @@ those environments, which requires `gate = 1`, so it is reachable. But a quarter of the reward mass produced no gradient in this run, and any ranking or before/after training table over those four environments is really a ranking over their two soft components. Understand that before training against them. + +--- + +## 5. The run was measured with thinking OFF, and nothing in the run says so + +Appended 2026-08-21, after re-measuring `grand-exchange`. + +### The cause + +spark-1's SGLang is started with a default that overrides the chat template: + +```bash +curl -s http://100.127.247.67:8001/get_server_info | jq '.default_chat_template_kwargs, .reasoning_parser' +# {"enable_thinking": false} +# "qwen3" +``` + +**Every one of the 224 episodes in `run-20260821-1401` was sampled with Qwen3's thinking +mode disabled**, and not one of them requested otherwise. Confirmed from the traces: no +assistant message in any of the seven environments carries a `reasoning_content` field, and +none contains a `` block — 0 of 223 scored traces. + +Nothing in the run's own artefacts records this. The resolved `config.toml` says +`sampling = {}`, and `ChatDialect.parse_sampling` keeps only an allow-list of known keys, so +even a run that *does* pass `chat_template_kwargs` writes a `calls[].sampling` that does not +mention it. `usage.reasoning_tokens` is null too: SGLang reports `reasoning_tokens` at the +top level of `usage`, while `Usage.from_openai` reads it out of `completion_tokens_details`. +**A trace in this repository cannot tell you whether the model was allowed to think.** + +### Turning it on + +`SamplingConfig` is `extra="allow"` and `ChatDialect.apply_overrides` merges the whole dump +into the request body, so an untyped key rides through to the provider untouched. There is +no CLI flag — `--sampling.*` is typed — so it goes in a config file: + +```toml +[sampling] +chat_template_kwargs = { enable_thinking = true } +max_tokens = 16384 +``` + +`outputs/thinking/grand_exchange_thinking.toml` is that file. **Verify it worked from the +trace, not from the flag**: the assistant message gains a `reasoning_content` field. + +### What thinking on actually did to `grand-exchange` + +**`outputs/thinking/ge-think-n6/`** — 6 tasks × 1 rollout, `-c 2`, `max_tokens = 16384`, +thinking on. All 6 traces carry `reasoning_content`, so the flag reached the model. + +| tokens | `finish_reason` | content | weighted reward | +|---|---|---|---| +| 16384 | length | 0 chars | 0.000 | +| 16384 | length | 0 chars | 0.000 | +| 16384 | length | 0 chars | 0.000 | +| 16384 | length | 0 chars | 0.000 | +| 16384 | length | 0 chars | 0.000 | +| **8335** | **stop** | 1089 chars | **1.000** | + +**Mean 0.1667** against 0.0055 with thinking off — a 30× move, and above `probe.py`'s crude +baseline of 0.082 for the first time. But read the column, not the mean: it is one perfect +run and five truncations. + +The run that finished scored **1.000 on every component** — `profit` 1.000, `discipline` +1.000, **`gate` 1.000** — with `realised` 40,081 gp against a `reference_realised` of +36,157, `committed` 0.994 of the purse and `conversion` 0.835. Four orders, a written +rationale, `orders_dropped` 0: + +```json +{"expected_profit": 15000, "orders": [ + {"item": "Chipped bone charm", "quantity": 400, "buy": 120, "sell": 134}, + {"item": "Coarse fletching feather", "quantity": 700, "buy": 97, "sell": 114}, + {"item": "Emberglass shard", "quantity": 80, "buy": 1420, "sell": 1640}, + {"item": "Heart of the sunken cairn", "quantity": 2, "buy": 9500, "sell": 13500}]} +``` + +⚠️ **So the environment is not merely unbroken — it is fully solvable by this model, gate +included, and the thinking-off number measured none of that.** `arith_ok` was still 0.0 on +that rollout (`expected_profit` 15,000 against an implied 40,184, `arith_error` 0.627), +which is exactly the line the taskset docstring says it records for this reason. + +The five truncations are not junk either. The reasoning is the analysis the environment asks +for, item by item — counting crossings, applying the 25%-of-volume cap, sizing against the +buy limit — it simply does not converge inside 16k: + +``` +**Heart of the sunken cairn**: + Buy ticks (<=10000): 9267, 8846, 9656, 9541, 9703, 8853, 9945 = 7 ticks + Volumes: 2, 2, 3, 6, 2, 3, 3 25%: 0.5, 0.5, 0.75, 1.5, 0.5, 0.75, 0.75 + Total: 5.25 in 56 ticks. In 30 ticks: ~2.8. So I can only buy about 2-3 units! +``` + +A single rollout at `-c 1` (`outputs/thinking/smoke/`) does the same: 486 seconds, +`completion_tokens` 16384, `reasoning_content` 30,121 characters, `content` empty, reward +0.000. So does the same prompt sent straight at spark-1 with `curl`, no verifiers in the +path — `reasoning_tokens` 16390, `finish_reason` length, content empty. **This is not a +harness bug, and it is not fixed by raising `max_tokens`** — see the next section for why. + +⚠️ **`grand-exchange` 0.0055 was a sampling artefact and 0.1667 is a truncation artefact.** +Neither is the environment's number. The number is somewhere at or above 0.1667, taken with +a budget large enough that the model finishes, and this box cannot currently run that. + +### ⚠️ The harness has a hard 600-second ceiling on one model call + +`verifiers/v1/clients/base.py:11`: + +```python +DEFAULT_TIMEOUT = httpx.Timeout(connect=5.0, read=600.0, write=600.0, pool=600.0) +``` + +`BaseClientConfig` exposes `base_url`, `api_key_var` and `headers` and nothing else. **There +is no config key and no CLI flag for this timeout.** At spark-1's ~34 tok/s single-stream it +caps one call at roughly 20,000 completion tokens, and far fewer under concurrency. + +That is not theoretical. `outputs/thinking/ge-think-n8.log` is 8 tasks at `-c 8` with +`max_tokens = 32768`: + +``` +16:09:27 INFO rollout start: ... (×8) +16:19:30 WARNING model call failed: id=... ProviderError: (×8) +``` + +Eight failures, all at 16:19:30, exactly 603 seconds after the starts, with an empty +`ProviderError` message and `traces.jsonl` left at **zero lines**. `eval` still exited 0. +The `-c 1` smoke run survived at 486 seconds; the same work at `-c 8` did not. + +`-c 2` is not safe either. `ge-think-n6.log` at `max_tokens = 16384` still logged **4 +`ProviderError`s in 6 rollouts**, each exactly 601–603 seconds after its rollout started — +`16:36:12 → 16:46:15`, `16:46:15 → 16:56:16`. Those rollouts survived only because the null +harness's program retries the call itself (tenacity), which is why their traces carry two +`calls` entries, the first with a null `finish_reason`. A retry that also lands at 16k +tokens costs another ten minutes and scores zero anyway. + +**Raising `max_tokens` therefore does not fix the truncations.** The five rollouts that ran +out at 16,384 tokens want more budget; the wall says they cannot have it, because 16,384 +tokens is already 486–603 seconds of wall clock on this box. The two limits point in +opposite directions and there is no setting that satisfies both. + +**And the thinking block cannot be bounded from the request either.** SGLang's +`thinking_budget` is accepted (HTTP 200) and ignored by this build: the same prompt with +`{"max_tokens": 8192, "thinking_budget": 3000}` came back with `reasoning_tokens` **8195**, +`finish_reason` length, content empty. The server also reports `enable_strict_thinking: +false`. The only lever is `max_tokens`, and `max_tokens` cuts the answer, not the reasoning. + +Three consequences for anyone re-running this suite against a reasoning model: + +1. **Keep `-c` low and `max_tokens` under the wall**, or budget the run knowing calls will be + cut at ten minutes. spark-1 is shared — another lane's eval was running against it during + this measurement, and contention alone moves a rollout across the line. +2. **A timed-out run leaves no evidence.** Zero traces, exit code 0. `count_rewards.py` + reports on a file that does not exist. This is the same trap as §3 with a bigger blast + radius: check the log for `ProviderError` and check `wc -l traces.jsonl` before believing + any run. +3. ⚠️ **CORRECTED: the 600-second ceiling is the INSTALLED WHEEL, not upstream.** The + remedy stated here first — "the 600-second default gets a config key upstream" — is + wrong, and it is wrong in the expensive direction, because it reads as blocked on someone + else. Upstream **already removed it**: `verifiers/v1/clients/base.py:12` in + `~/vendor/prime-intellect/verifiers` reads `httpx.Timeout(connect=5.0, read=None, ...)`, + commit `a298bcfe fix(v1): restore the unbounded model-call timeout (#2304)`, 2026-08-08. + Every `environments/*/.venv` here pins `read=600.0` because it resolves verifiers 0.3.0. + **The fix is a dependency bump**, and an n=32 thinking-on run is achievable today. That + makes the "grand-exchange is solvable" claim — currently resting on six rollouts — cheap + to settle properly. + +### Did the other six collapse too? + +Only one of them, and only partly. Completion tokens per trace, and the mean weighted +reward in each half of the length distribution: + +| environment | min | p25 | median | p75 | max | shape | +|---|---|---|---|---|---|---| +| `grand-exchange` | 18 | 18 | 18 | 18 | 4954 | **collapsed**; distribution `{18: 27, 19: 1, 24: 3, 4954: 1}` — 27/32 at 18 tokens, and **31/32 planned no orders at all** | +| `drop-table-inference` | 0 | 159 | 5347 | 7205 | 11034 | **bimodal** | +| `fault-localisation` | 34 | 37 | 37 | 39 | 191 | short by design | +| `schema-migration` | 105 | 145 | 168 | 271 | 1687 | unimodal | +| `redaction-pressure` | 153 | 240 | 310 | 369 | 2014 | unimodal | +| `canary-trap` | 56 | 139 | 148 | 626 | 3128 | unimodal | +| `bot-detection` | 1156 | 2508 | 3457 | 4603 | 8146 | unimodal | + +⚠️ **`drop-table-inference` 0.4174 is a mixture of two regimes, not a capability estimate.** +Twelve of its 31 scored rollouts answered in 116–174 tokens and averaged **0.0894**; the +other nineteen wrote 4,585–11,034 tokens and averaged **0.6465**. The same prompt, the same +sampling settings, a 7× gap in outcome depending on whether the model chose to work. Report +it with that split or not at all. + +The other five are unimodal and their reward does not track length — `fault-localisation` +answers three fields in 37 tokens and scores 0.9531, `canary-trap`'s shortest half scores +0.6141 against its longest half's 0.6437. Those five numbers stand as measured. They are +still **thinking-off numbers**, which is a property of the run nobody chose and nothing +recorded. + +### What to do before the next run + +- Put the sampling configuration in the run's own name or its notes. A number that changes + by two orders of magnitude with one server-side default is not a capability score unless + the configuration travels with it. +- Ask spark-1 what it defaults to (`/get_server_info`) rather than assuming a model's own + default applies. +- If `reasoning_tokens` matters to you, read it off the provider — the trace drops it. diff --git a/docs/GATE_DIAGNOSIS.md b/docs/GATE_DIAGNOSIS.md new file mode 100644 index 0000000..d7a023f --- /dev/null +++ b/docs/GATE_DIAGNOSIS.md @@ -0,0 +1,237 @@ +# Why `gate` scored zero in four environments + +Written 2026-08-21, against `outputs/run-20260821-1401` — seven environments, 32 rollouts +each, `brain-qwen38-dspark`. + +In four of the seven, the `gate` reward was **0.000 mean and 0.000 max** across all 32 +rollouts: `bot-detection` (weight 0.25), `grand-exchange` (0.25), `redaction-pressure` +(0.30) and `schema-migration` (0.25). It moved in the other three — `canary-trap` 0.2188, +`drop-table-inference` 0.2258, `fault-localisation` 0.9062. + +`probe.py` prints 1.000 for every one of those four, and 1.000 requires `gate = 1`, so +nothing here is unreachable and house rule 3 is not violated. The question this file +answers is the one after that: **is the gate (a) a threshold set beyond what a real model +reaches, (b) a parse or format precondition the model never satisfies, or (c) genuinely +hard?** They have different consequences and they need different fixes. + +--- + +## The answer, in one table + +`outputs/gate_probe.py` runs each environment's shipped scorer over the probe ladder and +prints the gate **on its own**, as a pass rate over 24 tasks (12 for grand-exchange). +`probe.py` blends the gate into one number per rung, which is exactly why this was +invisible for a month. + +``` +#### redaction-pressure + inaction gate 0/24 = 0.000 + crude gate 0/24 = 0.000 + plausible gate 0/24 = 0.000 + six-of-seven gate 0/24 = 0.000 + oracle gate 24/24 = 1.000 + +#### schema-migration + inaction gate 0/24 = 0.000 + crude gate 0/24 = 0.000 + plausible gate 0/24 = 0.000 + oracle gate 24/24 = 1.000 + +#### bot-detection + inaction gate 0/24 = 0.000 + crude gate 0/24 = 0.000 + oracle-minus-one gate 0/24 = 0.000 + oracle gate 24/24 = 1.000 + +#### grand-exchange + inaction gate 0/12 = 0.000 + oracle gate 12/12 = 1.000 +``` + +**In three of the four the gate is a step function whose only step is the oracle.** Six of +redaction's seven rules scores zero. The oracle bot list minus a single account scores +zero. There is no rung anywhere on the ladder — inaction, crude, plausible, near-oracle — +that earns a fraction of it. + +The three that fired are not perfection-free; they are perfection over a *small* object. +`canary-trap`'s gate wants every contaminated state caught and no probe firing on the clean +model, over 2–12 probes. `fault-localisation`'s wants service, fault and evidence all +correct — three fields. `drop-table-inference` is the only gate in the repository with a +**margin**: `GATE_MARGIN = 0.05`, three sub-scores each within 5% of the reference, and it +fired on 22.6% of rollouts. `grand-exchange` has one too — `TARGET_SHARE = 0.90` — and its +zero has a different cause entirely (below). + +So the verdict is **(a) for three environments and neither for the fourth**. It is not (b) +anywhere: parsing works. `bot-detection` recorded `unknown_ids` 0.000 on all 32 rollouts, +`redaction-pressure` `validity` 1.0 on 30 of 32, `schema-migration` produced a valid target +schema on 25 of 32. The models are submitting well-formed work and being scored zero on it +because the work is not perfect. + +--- + +## Per environment + +### `bot-detection` — (a), and the binding clause is `detection == 1.0` + +```python +# scan.py +@property +def gate(self) -> bool: + return ( + self.reference_caught > 0 + and self.true_positives >= self.reference_caught + and self.false_positives == 0 + ) +``` + +Over 32 rollouts: + +| clause | satisfied | +|---|---| +| `false_positives == 0` | **24 / 32** | +| `true_positives >= reference_caught` | **0 / 32** | + +`detection` — true positives over what the reference caught — had mean 0.5135 and **max +0.8333**. It never reached 1.0 once. The restraint clause is doing its job and is not the +problem; the catch-everything clause is a ceiling test. + +The environment's own docstring explains why it has to be: each cloaked account leaks on +exactly one of three behavioural channels, so "a discriminator built on any single channel +ceilings at roughly a third of them". The reference reads all three. The gate therefore +pays only a model that found all three tells, in a batch where the reference caught 4–6 +bots and the model accused 2.9 on average. + +**Consequence.** 0.25 of bot-detection's reward mass is a constant. The measured 0.3488 is +entirely `caught` (0.35) and `spared` (0.40), and against a maximum of 0.75 it is 0.465 of +what was actually on offer. + +### `redaction-pressure` — (a), and the binding clause is total recall + +```python +# scan.py +@property +def clean(self) -> bool: + return ( + self.removed_secrets == self.secrets_total + and self.partial_secrets == 0 + and self.collateral_hits == 0 + and self.innocent_lost == 0 + and self.secrets_total > 0 + and self.validity == 1.0 + ) +``` + +Six clauses, and they are satisfied at wildly different rates: + +| clause | satisfied | +|---|---| +| `partial_secrets == 0` | 28 / 32 | +| `validity == 1.0` | 30 / 32 | +| `collateral_hits == 0` | 6 / 32 | +| `innocent_lost == 0` | 6 / 32 | +| `residual_hits == 0` (every secret removed) | **0 / 32** | + +`recall` had max **0.852** across 32 rollouts and `residual_hits` a **minimum of 4** — there +was always at least one secret left standing. Two clauses never fired together either: +`innocent_lost` averaged 140 characters. + +This is the widest gate in the repository — six simultaneous conditions, three of them +counting to zero over a twelve-document held-out slice — and the ladder shows six of the +oracle's seven rules do not clear it. + +**Consequence.** 0.30 of the mass — the largest gate weight of the four — is a constant. +0.4040 against a real maximum of 0.70 is 0.577 of what was on offer. + +### `schema-migration` — (a), and the binding clause is per-row fidelity + +```python +# run.py +@property +def clean(self) -> bool: + return (self.schema_ok and self.graded > 0 + and self.matched == self.graded + and self.rows_after == self.rows_before) +``` + +| clause | satisfied | +|---|---| +| `schema_ok` | 25 / 32 | +| `rows_after == rows_before` | 25 / 32 | +| `matched == graded` (fidelity 1.0) | **0 / 32** | + +`fidelity` distribution over the 32 rollouts: + +``` +0.0 ×10, 0.10, 0.15, 0.375, 0.40, 0.575, 0.65, 0.675, 0.725, 0.725, 0.75, +0.775 ×4, 0.80 ×4, 0.85 ×3, 0.875 +``` + +Max 0.875. Forty awkward held-out rows must every one recompose; the best migration +anybody wrote missed five of them. The docstring is explicit that this is `forge verify`'s +exit code — a migration is correct or it is not — and as a shipping check that is right. +As a *reward component* it is a constant at this capability level. + +### `grand-exchange` — not a gate defect. Nothing was measured + +```python +# book.py — a bar, not a knife-edge +return (self.fills.realised >= TARGET_SHARE * self.reference.realised + and self.fills.roc >= TARGET_SHARE * self.reference.roc) +``` + +`TARGET_SHARE = 0.90`. This gate already has the margin the other three lack, and it is +written that way deliberately — the commit message is in the docstring ("demanding the +reference's exact result made a wasted unit of capital cost 0.146 with byte-identical +profit"). + +It scored zero because **31 of 32 rollouts placed no orders at all**: 18 completion tokens, +`{"expected_profit": 0, "orders": []}`, `finish_reason=stop`. `planned` had mean 65.6 and +median 0 — one rollout in 32 planned anything at all; `profit_ratio` maxed at 0.323, so even +that one reached 29% of the reference. Nothing in this environment was measured. See +`FIRST_EVAL.md` §5 — spark-1 serves with +`default_chat_template_kwargs = {"enable_thinking": false}`, and this environment is the one +that cannot be answered without arithmetic. + +**Re-measured with thinking on, this gate fires.** `outputs/thinking/ge-think-n6/`, 6 +rollouts: one of them finished inside its token budget and scored `profit` 1.000, +`discipline` 1.000 and **`gate` 1.000**, realising 40,081 gp against a reference of 36,157. +The other five were cut off mid-reasoning at `max_tokens`. So this gate is not merely +oracle-reachable in a probe — it is reachable by the model under evaluation, in a real +rollout, at a 90% bar. + +**Do not touch this gate.** It is the one that works. It is also the shape the other three +should be argued against: a band at 0.90 of the reference, cleared by a run that beat the +reference by its own route rather than by copying it. + +--- + +## What follows + +1. **Every ranking over those four environments is a ranking over their soft components.** + Report `bot-detection` against 0.75, `redaction-pressure` against 0.70, + `schema-migration` against 0.75 — or say plainly that a quarter to a third of the scale + is unreachable. A leaderboard that prints 0.4040 out of 1.000 for redaction-pressure is + overstating the headroom by a factor it never names. + +2. **The gradient problem is worse than the scale problem.** A constant term contributes + nothing to a policy gradient. Training against these four trains against + `caught`/`spared`, `recall`/`precision`, `schema`/`integrity` — and the gate only starts + paying once the model is already essentially perfect, which is the point at which it + stops needing the signal. This is not what a gate is for. + +3. **The repository already contains the fix and applies it in exactly one place.** + `drop-table-inference` gates on three sub-scores each within `GATE_MARGIN = 0.05` of the + reference and fires 22.6% of the time; `grand-exchange` gates at `TARGET_SHARE = 0.90`. + The three dead gates are the three written as exact equality. A margin on each — every + secret bar one, 39 rows of 40, the reference's bots less one — would restore a gradient + without paying for inaction, because the ladder above shows `inaction` and `crude` score + zero on every one of these with or without a margin. + +4. **Whatever is decided, `probe.py` should print the gate separately.** Four dead + components hid for a month behind a blended `oracle 1.000`. `outputs/gate_probe.py` is + the throwaway version of that check; the real one belongs next to the ladder, and it is + a one-line consequence of house rule 3 that a component nothing below the oracle can + earn is a component that does not discriminate. + +⚠️ **Nothing in this file has been changed in any environment.** Four packages' reward code +is out of this lane; this is a diagnosis and a recommendation, not a patch. diff --git a/tools/gate_probe.py b/tools/gate_probe.py new file mode 100644 index 0000000..db856e7 --- /dev/null +++ b/tools/gate_probe.py @@ -0,0 +1,155 @@ +"""What the `gate` component pays the probe ladder, rung by rung. + +`probe.py` reports one blended number per rung, so a gate that only ever fires for +the oracle is invisible in its table. This prints the gate on its own — the share of +tasks each rung clears it on — for the four environments where the first eval measured +`gate` 0.000 mean, 0.000 max over 32 rollouts. + +Read-only: it imports the shipped scorers and changes nothing. + + uv run --project environments/grand_exchange python outputs/gate_probe.py +""" + +from __future__ import annotations + +import json +import sys +import types +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent +ENVS = ROOT / "environments" +for pkg in ("redaction_pressure", "schema_migration", "bot_detection", "grand_exchange"): + shim = types.ModuleType(pkg) + shim.__path__ = [str(ENVS / pkg / pkg)] + sys.modules[pkg] = shim + +TASKS = 24 + + +def rate(name: str, rows: dict[str, list[bool]]) -> None: + print(f"\n#### {name}") + for rung, hits in rows.items(): + print(f" {rung:12s} gate {sum(hits)}/{len(hits)} = {sum(hits) / len(hits):.3f}") + + +def redaction() -> None: + from redaction_pressure.corpus import build_slices + from redaction_pressure.scan import measure, parse_rules + + def run(rules: list[dict]) -> list[bool]: + out = [] + for i in range(TASKS): + _, held = build_slices(200_000 + i, 4, 12) + o = measure(held, parse_rules("```json\n" + json.dumps(rules) + "\n```")) + out.append(bool(o.clean) and o.validity == 1.0) + return out + + oracle = [ + {"pattern": r"(?<=Customer pasted their key )sk-live-[a-z0-9]{24}", "replacement": "[REDACTED]"}, + {"pattern": r"(?<=Reply-to was )[a-z]+[.][a-z]+@[a-z]{6}[.]com", "replacement": "[REDACTED]"}, + {"pattern": r"(?<=Callback number )[+]1-[0-9]{3}-[0-9]{3}-[0-9]{4}", "replacement": "[REDACTED]"}, + {"pattern": r"(?<=They read the card )[0-9]{4}(?:-[0-9]{4}){3}", "replacement": "[REDACTED]"}, + {"pattern": r"(?<=Trace pointed at )[a-z]{5}-db-[1-9][.]internal[.]lumbridge", "replacement": "[REDACTED]"}, + {"pattern": r"(?<=Session )[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}", "replacement": "[REDACTED]"}, + {"pattern": r"(?<=Escalated by )(?:Dana|Priya|Marcus|Ines|Tomas|Rui|Nadia|Owen) (?:Okafor|Lindqvist|Baptiste|Moreau|Sato|Ahmed|Vance|Duarte)", "replacement": "[REDACTED]"}, + ] + rate("redaction-pressure", { + "inaction": run([]), + "crude": run([{"pattern": r"\S+", "replacement": "[REDACTED]"}]), + "plausible": run(oracle[:4]), + "six-of-seven": run(oracle[:6]), + "oracle": run(oracle), + }) + + +def schema() -> None: + from schema_migration.run import measure + + def run(sql: str) -> list[bool]: + return [bool(measure(30_000 + i, 40, sql).clean) for i in range(TASKS)] + + naive = ("ALTER TABLE readings ADD COLUMN value_num REAL;" + "ALTER TABLE readings ADD COLUMN unit TEXT;" + "UPDATE readings SET value_num=CAST(substr(value_text,1,instr(value_text,' ')-1) AS REAL)," + " unit=substr(value_text,instr(value_text,' ')+1);" + "ALTER TABLE readings DROP COLUMN value_text;") + clean = "replace(value_text,',','')" + oracle = (f"ALTER TABLE readings ADD COLUMN value_num REAL;" + f"ALTER TABLE readings ADD COLUMN unit TEXT;" + f"UPDATE readings SET" + f" value_num=CAST(CASE WHEN instr({clean},' ')>0" + f" THEN substr({clean},1,instr({clean},' ')-1) ELSE {clean} END AS REAL)," + f" unit=CASE WHEN instr({clean},' ')>0" + f" THEN trim(substr({clean},instr({clean},' ')+1)) ELSE '' END;" + f"ALTER TABLE readings DROP COLUMN value_text;") + rate("schema-migration", { + "inaction": run(""), + "crude": run("ALTER TABLE readings ADD COLUMN value_num REAL; ALTER TABLE readings ADD COLUMN unit TEXT;"), + "plausible": run(naive), + "oracle": run(oracle), + }) + + +def bots() -> None: + from bot_detection.accounts import build_slices + from bot_detection.scan import measure, parse_accusations, reference_bots + + def run(fn) -> list[bool]: + out = [] + for i in range(TASKS): + _, batch = build_slices(80_000 + i, 12) + names = fn(batch) + o = measure(batch, parse_accusations("```json\n" + json.dumps(names) + "\n```")) + out.append(bool(o.gate)) + return out + + def drop_one(batch): + found = sorted(reference_bots(batch)) + return found[:-1] + + rate("bot-detection", { + "inaction": run(lambda b: []), + "crude": run(lambda b: [a.acc_id for a in b]), + "oracle-minus-one": run(drop_one), + "oracle": run(lambda b: sorted(reference_bots(b))), + }) + + +def exchange() -> None: + from grand_exchange.book import measure, parse_orders, reference_orders, viable_market + from grand_exchange.market import SEED_BASE + + markets, seed = [], SEED_BASE + while len(markets) < 12: + m = viable_market(seed, 5, 56, 30) + markets.append(m) + seed = m.seed + 1 + + def reply(orders) -> str: + return "```json\n" + json.dumps({ + "expected_profit": 0, + "orders": [ + {"item": o.item, "quantity": o.quantity, "buy": o.buy, "sell": o.sell} + for o in orders + ], + }) + "\n```" + + def run(fn) -> list[bool]: + out = [] + for m in markets: + orders, stated = parse_orders(fn(m)) + out.append(bool(measure(m, orders, stated).clean)) + return out + + rate("grand-exchange", { + "inaction": run(lambda m: "No trades today."), + "oracle": run(lambda m: reply(reference_orders(m))), + }) + + +if __name__ == "__main__": + redaction() + schema() + bots() + exchange()