lumbridge-public/arena
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

gates: give two of them a margin, and make the probe see components

Browse Source 
The probe printed one blended number per policy, so a component pinned at 0.000
across every rung was invisible. Four gates hid there for a month. It now reports
floor / best-below-oracle / oracle / ceiling for 25 components across 8
environments, and a component flat across every rung is fatal.

bot-detection GATE_SLACK = 1 — fires 5/32 on the real rollouts, was 0/32. The
max(1, reference_caught - SLACK) guard is verified by construction, not by
sampling: without it the required count reaches 0 and an EMPTY accusation list
clears the gate. Observed reference_caught is 4-6, so no amount of sampling would
have found that hole.

schema-migration GATE_MARGIN = 0.15 — fires 4/32, was 0/32. The cost is disclosed
and bounded: the naive split clears it on 5.5% of 1,000 unseen seeds, fenced by
an assert at 10%. Margin 0.10 keeps the leak at zero and fires 0/32, i.e. stays
dead. A live gradient with a bounded leak beats a clean corpse.

redaction-pressure is NOT given a margin, and that is the result rather than a
failure. The only setting that fires at all leaves half the secrets standing and
pays a four-of-seven ruleset on five seeds in six — a margin that pays for
inaction is strictly worse than a dead gate. Recall maxes at 0.852 and no rollout
ever cleared both clauses in one episode. It is genuinely hard, not miscalibrated.

⚠️ The per-component check did not catch the defect it was built for. Reverting
schema-migration's margin to 0.0 — restoring the exact dead gate — printed ok and
exited 0, because the near-oracle rung scrapes the unmargined gate on ~2 seeds in
24 and that kept best<oracle non-zero. Every assertion bounded how much a margin
may PAY; none noticed if it stopped existing. migration() now carries the mirror
of bot-detection's guard, and reverting the margin fails with "the margin is dead
and the component carries no gradient between the crude answer and the exact one".

canary-trap's oracle-minus-one rung is documented as degenerate rather than
quietly relied on: it is identical to the oracle to four decimals, so it measures
specificity and gate at the ceiling, not mid-ladder as its comment claimed.

The CI lock policy asks git instead of the disk. It was checking the working
tree, where a lock file is a normal by-product of uv sync, so it passed in a clean
checkout and failed on every machine that had run an eval.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
karti
2026-08-21 23:14:05 -07:00
co-authored by Claude Opus 5
parent 122a858b61
commit 3305be4ff7
12 changed files with 1107 additions and 95 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+94 -4
View File
@@ -23,9 +23,14 @@ An interactive one is held to a fourth.
environment does not measure generalisation and does not ship.
2. **No single-sided reward.** Every reward has a counterweight in the same episode. A reward
you can max by doing the crude thing is a reward that teaches the crude thing.
3. **A zero floor and a reachable ceiling, demonstrated.** Inaction scores 0.0 and an oracle
scores 1.0, measured, in `probe.py`. An unreachable component is dead weight in the
gradient; a floor above zero pays for doing nothing.
3. **A zero floor and a reachable ceiling, demonstrated — per component, not per
environment.** Inaction scores 0.0 and an oracle scores 1.0, measured, in `probe.py`. An
unreachable component is dead weight in the gradient; a floor above zero pays for doing
nothing. The qualifier was added on 2026-08-21 and it was expensive: for a month this
file printed `oracle 1.000` and `ok` for four environments whose `gate` component scored
exactly 0.000 mean and 0.000 max over 32 real rollouts each. The blend was true and the
thing inside it was a constant. `probe.py` reports every weighted component's floor and
ceiling now, and exits 1 on one that never moves.
4. **A turn budget that binds.** In an environment the model acts in over several turns, a
policy that spends every turn must score measurably below one that spends the turns worth
spending. Every reward here saturates, so without a cost for looking the budget is free
@@ -58,12 +63,89 @@ Reward for the degenerate strategies and for an oracle, from `uv run python prob
| `redaction-pressure` | 0.000 | 0.387 *(redact everything)* | 0.549 | **1.000** |
| `canary-trap` | 0.000 | 0.000 *(public-knowledge probes)* | 0.525 *(GUID recall)* | **1.000** |
| `fault-localisation` | 0.000 | 0.250 *(blame the loudest)* | 0.500 | **1.000** |
| `schema-migration` | 0.000 | 0.000 *(add columns, touch nothing)* | 0.630 | **1.000** |
| `schema-migration` | 0.000 | 0.000 *(add columns, touch nothing)* | 0.641 | **1.000** |
| `bot-detection` | 0.000 | 0.000 *(ban everyone)* | 0.400 *(one tell of three)* | **1.000** |
| `drop-table-inference` | 0.000 | 0.179 *(copy the frequencies, floor the rest at a memorised constant)* | 0.476 *(per-item posterior over the grid)* | **1.000** |
| `grand-exchange` | 0.000 | 0.082 *(buy and sell at market)* | 0.608 *(anchor on the mean and size to the volume, trap included)* | **1.000** |
| `grand-exchange-live` | 0.000 | 0.126 *(buy and sell at market, on one look)* | 0.406 *(anchor on the mean, no trap filter, one look and no re-quote)* | **1.000** |
`schema-migration`'s plausible rung moved from 0.630 to 0.641 on 2026-08-21 when its gate
gained a margin — the naive split clears the loosened gate on one seed in twenty-four. Every
other cell is unchanged.
### And the same table one reward component at a time
The four columns above are blends, and a blend cannot show a constant inside it. Below each
component's **weighted** floor, the best any rung below the oracle manages, and its ceiling.
The interesting column is the middle one: where it equals the floor, nothing short of the
oracle earns a fraction of that term.
| environment | component | floor | best below oracle | ceiling |
|---|---|---|---|---|
| `bot-detection` | caught | 0.000 | 0.278 | 0.350 |
| `bot-detection` | spared | 0.000 | 0.317 | 0.400 |
| `bot-detection` | gate | 0.000 | 0.250 | 0.250 |
| `canary-trap` | detection | 0.000 | 0.350 | 0.350 |
| `canary-trap` | specificity | 0.000 | 0.350 | 0.350 |
| `canary-trap` | gate | 0.000 | 0.300 | 0.300 |
| `drop-table-inference` | fit | 0.000 | 0.242 | 0.450 |
| `drop-table-inference` | rare | 0.000 | 0.326 | 0.350 |
| `drop-table-inference` | gate | 0.000 | 0.025 | 0.200 |
| `fault-localisation` | service | 0.000 | 0.300 | 0.300 |
| `fault-localisation` | fault | 0.000 | 0.250 | 0.250 |
| `fault-localisation` | evidence | 0.000 | 0.250 | 0.250 |
| `fault-localisation` | gate | 0.000 | **0.000** | 0.200 |
| `grand-exchange` | profit | 0.000 | 0.450 | 0.450 |
| `grand-exchange` | discipline | 0.000 | 0.298 | 0.300 |
| `grand-exchange` | gate | 0.000 | 0.244 | 0.250 |
| `grand-exchange-live` | profit | 0.000 | 0.382 | 0.450 |
| `grand-exchange-live` | discipline | 0.000 | 0.201 | 0.300 |
| `grand-exchange-live` | gate | 0.000 | 0.048 | 0.250 |
| `redaction-pressure` | recall | 0.000 | 0.304 | 0.350 |
| `redaction-pressure` | precision | 0.000 | 0.350 | 0.350 |
| `redaction-pressure` | gate | 0.000 | **0.000** | 0.300 |
| `schema-migration` | schema | 0.000 | 0.300 | 0.300 |
| `schema-migration` | integrity | 0.000 | 0.422 | 0.450 |
| `schema-migration` | gate | 0.000 | 0.240 | 0.250 |
Two components are bold, and `probe.py` names both of them on the way past. They are not the
same problem. `fault-localisation`'s gate wants three fields correct at once and fires on
**29 of 32** real rollouts — the healthiest gate here; the ladder simply has no rung standing
at two-of-three. `redaction-pressure`'s fires on **0 of 32**, and unlike the two that were
fixed below, no margin rescues it: see the sweep in its scanner.
The probe cannot tell those apart, which is why it warns rather than fails there. What tells
them apart is `tools/regate.py` — it replays the real traces in `outputs/` through a
candidate gate and reports the fire rate, holding the model's behaviour fixed and varying
only the reward.
### The margins, and what they were measured against
Two gates demanded exact equality with the reference and paid **nothing at all** for anything
short of it. Both were re-scored over the 32 real rollouts in `outputs/run-20260821-1401`
before being changed, and both floors were re-measured after:
| environment | gate | before | after | inaction | crude | plausible |
|---|---|---|---|---|---|---|
| `bot-detection` | `GATE_SLACK = 1` — the reference's bots bar one, still zero wrongful bans | 0/32 | **5/32 = 0.156** | 0.000 | 0.000 | 0.000 |
| `schema-migration` | `GATE_MARGIN = 0.15` — 34 of 40 rows recompose; schema and row count still exact | 0/32 | **4/32 = 0.125** | 0.000 | 0.000 | 0.042 |
| `redaction-pressure` | unchanged, and deliberately | 0/32 | 0/32 | 0.000 | 0.000 | 0.000 |
The slack is on the catching clause only in both cases. A single wrongful ban still shuts
bot-detection's gate outright, and a migration that leaves `value_text` standing or loses a
row still fails schema-migration's — those are format preconditions, not matters of degree.
`schema-migration`'s 0.042 is the price of the margin, named rather than hidden: the naive
parser's luckiest draw of forty awkward rows recomposes exactly 34 of them, and `probe.py`
asserts that number stays under a tenth.
`redaction-pressure` was swept and left alone, which is the honest outcome rather than a
gap. Every margin loose enough to fire on the measured population — half the secrets left
standing — also pays a four-of-seven ruleset on five seeds in six; every margin tight enough
to keep that at zero fires on none of the 32. The failure is joint rather than a threshold:
recall maxed at 0.852 with never fewer than four secrets left, `collateral_hits == 0` held
on 6 of 32, and no rollout managed both at once. That gate is hard, not dead, and forcing a
margin onto it would trade the meaning of "redacted" for nothing measurable.
## Running one
```bash
@@ -111,5 +193,13 @@ on one DGX Spark. 16 tasks x 2 rollouts per environment, 32 rollouts each.
`probe.py` shows none is unsolvable. That gap is the whole point: an environment a model
already passes has no gradient left in it, and one nothing can pass has none yet.
⚠️ Four of those zeros were **0.000 mean and 0.000 max on all 32 rollouts**, which is a
different claim: not a hard gate, a constant. `docs/GATE_DIAGNOSIS.md` is the post-mortem.
Two have since been given measured margins, so re-scoring the same traces under the shipped
scorers moves `bot-detection` from 0.3488 to **0.3879** and `schema-migration` from 0.4432
to **0.4745**. Those two rows are stale above and will be replaced by a fresh run, not
patched — and every number in that table was sampled with thinking OFF, which is its own
retraction and is documented in `docs/FIRST_EVAL.md`.
The replies parse: 32/32 rollouts produced non-zero metrics in every environment, so these are
model scores rather than a harness failing to read its own output.