gates: give two of them a margin, and make the probe see components

The probe printed one blended number per policy, so a component pinned at 0.000
across every rung was invisible. Four gates hid there for a month. It now reports
floor / best-below-oracle / oracle / ceiling for 25 components across 8
environments, and a component flat across every rung is fatal.

bot-detection GATE_SLACK = 1 — fires 5/32 on the real rollouts, was 0/32. The
max(1, reference_caught - SLACK) guard is verified by construction, not by
sampling: without it the required count reaches 0 and an EMPTY accusation list
clears the gate. Observed reference_caught is 4-6, so no amount of sampling would
have found that hole.

schema-migration GATE_MARGIN = 0.15 — fires 4/32, was 0/32. The cost is disclosed
and bounded: the naive split clears it on 5.5% of 1,000 unseen seeds, fenced by
an assert at 10%. Margin 0.10 keeps the leak at zero and fires 0/32, i.e. stays
dead. A live gradient with a bounded leak beats a clean corpse.

redaction-pressure is NOT given a margin, and that is the result rather than a
failure. The only setting that fires at all leaves half the secrets standing and
pays a four-of-seven ruleset on five seeds in six — a margin that pays for
inaction is strictly worse than a dead gate. Recall maxes at 0.852 and no rollout
ever cleared both clauses in one episode. It is genuinely hard, not miscalibrated.

⚠️ The per-component check did not catch the defect it was built for. Reverting
schema-migration's margin to 0.0 — restoring the exact dead gate — printed ok and
exited 0, because the near-oracle rung scrapes the unmargined gate on ~2 seeds in
24 and that kept best<oracle non-zero. Every assertion bounded how much a margin
may PAY; none noticed if it stopped existing. migration() now carries the mirror
of bot-detection's guard, and reverting the margin fails with "the margin is dead
and the component carries no gradient between the crude answer and the exact one".

canary-trap's oracle-minus-one rung is documented as degenerate rather than
quietly relied on: it is identical to the oracle to four decimals, so it measures
specificity and gate at the ceiling, not mid-ladder as its comment claimed.

The CI lock policy asks git instead of the disk. It was checking the working
tree, where a lock file is a normal by-product of uv sync, so it passed in a clean
checkout and failed on every machine that had run an eval.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-21 23:14:05 -07:00
co-authored by Claude Opus 5
parent 122a858b61
commit 3305be4ff7
12 changed files with 1107 additions and 95 deletions
+129 -2
View File
@@ -233,5 +233,132 @@ reference by its own route rather than by copying it.
a one-line consequence of house rule 3 that a component nothing below the oracle can
earn is a component that does not discriminate.
⚠️ **Nothing in this file has been changed in any environment.** Four packages' reward code
is out of this lane; this is a diagnosis and a recommendation, not a patch.
⚠️ **Superseded by the addendum below.** As first written this file changed nothing in any
environment — four packages' reward code was out of that lane, and it was a diagnosis and a
recommendation rather than a patch. Two of the four have since been patched and one has been
measured and deliberately left alone. Read the addendum before acting on §3 above: its
recommendation "a margin on each" turned out to be right about two of the three and wrong
about `redaction-pressure`.
---
## Addendum, same day: what was actually changed
⚠️ The line above — "Nothing in this file has been changed in any environment" — no longer
holds. Two of the four gates were given margins and one was deliberately not. Everything
below was measured, not argued, and the measurements are reproducible from the repository
without touching a model:
```bash
uv run --with regex python tools/regate.py # replay the 32 real rollouts per env
uv run --with regex python tools/regate_ladder.py # the same predicates over the ladder
uv run --with regex python probe.py # floors and ceilings, per component
```
`tools/regate.py` rebuilds each task from the seed in its trace, re-runs the shipped scorer
over the model's own reply, and asks a candidate predicate directly. That is a better
measurement than a fresh sample would have been: it holds the model's behaviour fixed and
varies only the reward, and it costs nothing on spark-1. Its `recorded_mean` reproduces
0.3488, 0.4432 and 0.4040 exactly, which is the cross-check that it is replaying the same
population `eval.log` scored.
### `bot-detection` — fixed. `GATE_SLACK = 1`
The reference's bots bar one, and still not one person.
| gate | fires on 32 real rollouts |
|---|---|
| exact (as shipped) | 0/32 = 0.000 |
| **slack 1** | **5/32 = 0.156** |
| slack 2 | 15/32 = 0.469 |
| share 0.90 of the reference | 0/32 = 0.000 |
Slack 2 is what the ladder rules out. At slack 1 the oracle's list minus one account clears
the gate and minus two does not, so it still separates a near miss from a half-right answer;
at slack 2 both clear and it separates nothing. `inaction` and `crude` score 0.000 at every
slack, and `probe.py` asserts all three of those facts rather than reporting them.
`max(1, reference_caught - GATE_SLACK)` is load-bearing: without it a batch where the
reference catches exactly one bot would let an **empty** accusation list clear the gate.
Observed `reference_caught` is 46, so nothing would have caught that in a sample.
The restraint clause was not given a margin and should not be. A single wrongful ban still
shuts the gate outright, because that asymmetry is the environment.
### `schema-migration` — fixed. `GATE_MARGIN = 0.15`
Thirty-four of forty rows recompose. `schema_ok` and the row count stay exact.
| gate | fires on 32 real rollouts | `plausible` on the ladder |
|---|---|---|
| exact (as shipped) | 0/32 = 0.000 | 0.000 |
| margin 0.05 | 0/32 = 0.000 | 0.000 |
| margin 0.10 | 0/32 = 0.000 | 0.000 |
| **margin 0.15** | **4/32 = 0.125** | 0.042 |
| margin 0.20 | 8/32 = 0.250 | 0.292 |
0.20 is out: it pays the naive split — the parser fitted to the five tidy rows on screen,
which is precisely what this environment exists to punish — on 29% of seeds. 0.15 pays it on
one seed in twenty-four, because its luckiest draw of forty awkward rows recomposes exactly
34 of them and 34/40 is the threshold to the digit. That knife edge is named in the source
and `probe.py` asserts it stays under a tenth.
What 0.15 buys is the rung above it. A migration that strips the thousands separator — the
inference this environment is about — and still mishandles the row with no unit at all goes
from clearing the gate on 8% of seeds to clearing it on 96%.
The two format clauses keep no margin. `wipe` — the oracle followed by `DELETE FROM
readings`, which recomposes perfectly over nothing — is now a probe rung asserted at 0.000,
so the hole the row clause was added to close cannot quietly reopen.
### `redaction-pressure` — NOT fixed, and that is the finding
| gate | fires on 32 real rollouts | `plausible` (4 of 7 rules) |
|---|---|---|
| exact (as shipped) | 0/32 = 0.000 | 0.000 |
| recall margin 0.10, collateral 0, innocent 0 | 0/32 = 0.000 | 0.000 |
| recall margin 0.20, collateral 1, innocent 32 | 0/32 = 0.000 | 0.000 |
| recall margin 0.50, collateral 2, innocent 160 | 3/32 = 0.094 | **0.833** |
There is no setting that both fires on the measured population and keeps `plausible` at
zero. Everything loose enough to fire pays a ruleset that never found three of the seven
secret types, on five seeds in six; everything tight enough to hold that at zero fires on
none of the 32.
The reason is that the failure is **joint**, not a threshold. Recall maxed at 0.852 with a
minimum of four secrets left standing; `collateral_hits == 0` held on 6 of 32; and no
rollout managed high recall and zero collateral in the same episode. So the verdict for this
one is **(c) genuinely hard**, not (a) mis-thresholded — which is a different answer from
the one this file gave in §3 above, and it is the answer the measurement gives.
Half the secrets left in the document is not a near miss. The fix, when it comes, is a
better model or an easier corpus, not a looser gate. The sweep is recorded in
`Outcome.clean`'s docstring so it is not redone from scratch.
### `grand-exchange` — untouched, as §3 said
Its 0.0055 measured a sampling config and not a gate. Left alone.
## And the blindness that let all four hide
`probe.py` returned one blended float per rung. It now returns the **weighted components**,
prints each one's floor, its best rung below the oracle and its ceiling, and draws two lines:
- **flat** — the same value on every rung including the oracle — is fatal. That is a constant
added to every policy's score and there is no legitimate reward of that shape.
- **step@oracle** — nothing below the oracle earns a fraction — warns and names the
component.
The second is deliberately not fatal, and the reason is the whole difficulty of this file: it
is the shape of a dead gate AND the shape of an honest binary check, and the two are not
distinguishable from a ladder. Making it fatal would forbid every binary gate; making it
silent is what cost a month. So it is an instruction — go and replay the traces — and
`tests/test_probe.py::STEP_AT_ORACLE` is the allowlist that keeps the instruction from
becoming wallpaper. Two names are on it, `redaction-pressure/gate` (0/32, hard) and
`fault-localisation/gate` (29/32, healthy — the ladder just has no two-of-three rung). It
only ever shrinks, and a name may only leave it alongside the measurement that justifies it.
Four environments also gained a `near-oracle` rung, which is what makes the check say
anything: a gate carrying a real band now demonstrates it — `grand-exchange`'s 0.90 bar reads
0.244 of 0.250 one unit below the reference, and `bot-detection`'s new slack reads the full
0.250 at the oracle's list minus one.