gates: give two of them a margin, and make the probe see components
The probe printed one blended number per policy, so a component pinned at 0.000 across every rung was invisible. Four gates hid there for a month. It now reports floor / best-below-oracle / oracle / ceiling for 25 components across 8 environments, and a component flat across every rung is fatal. bot-detection GATE_SLACK = 1 — fires 5/32 on the real rollouts, was 0/32. The max(1, reference_caught - SLACK) guard is verified by construction, not by sampling: without it the required count reaches 0 and an EMPTY accusation list clears the gate. Observed reference_caught is 4-6, so no amount of sampling would have found that hole. schema-migration GATE_MARGIN = 0.15 — fires 4/32, was 0/32. The cost is disclosed and bounded: the naive split clears it on 5.5% of 1,000 unseen seeds, fenced by an assert at 10%. Margin 0.10 keeps the leak at zero and fires 0/32, i.e. stays dead. A live gradient with a bounded leak beats a clean corpse. redaction-pressure is NOT given a margin, and that is the result rather than a failure. The only setting that fires at all leaves half the secrets standing and pays a four-of-seven ruleset on five seeds in six — a margin that pays for inaction is strictly worse than a dead gate. Recall maxes at 0.852 and no rollout ever cleared both clauses in one episode. It is genuinely hard, not miscalibrated. ⚠️ The per-component check did not catch the defect it was built for. Reverting schema-migration's margin to 0.0 — restoring the exact dead gate — printed ok and exited 0, because the near-oracle rung scrapes the unmargined gate on ~2 seeds in 24 and that kept best<oracle non-zero. Every assertion bounded how much a margin may PAY; none noticed if it stopped existing. migration() now carries the mirror of bot-detection's guard, and reverting the margin fails with "the margin is dead and the component carries no gradient between the crude answer and the exact one". canary-trap's oracle-minus-one rung is documented as degenerate rather than quietly relied on: it is identical to the oracle to four decimals, so it measures specificity and gate at the ceiling, not mid-ladder as its comment claimed. The CI lock policy asks git instead of the disk. It was checking the working tree, where a lock file is a normal by-product of uv sync, so it passed in a clean checkout and failed on every machine that had run an eval. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
+129
-2
@@ -233,5 +233,132 @@ reference by its own route rather than by copying it.
|
||||
a one-line consequence of house rule 3 that a component nothing below the oracle can
|
||||
earn is a component that does not discriminate.
|
||||
|
||||
⚠️ **Nothing in this file has been changed in any environment.** Four packages' reward code
|
||||
is out of this lane; this is a diagnosis and a recommendation, not a patch.
|
||||
⚠️ **Superseded by the addendum below.** As first written this file changed nothing in any
|
||||
environment — four packages' reward code was out of that lane, and it was a diagnosis and a
|
||||
recommendation rather than a patch. Two of the four have since been patched and one has been
|
||||
measured and deliberately left alone. Read the addendum before acting on §3 above: its
|
||||
recommendation "a margin on each" turned out to be right about two of the three and wrong
|
||||
about `redaction-pressure`.
|
||||
|
||||
---
|
||||
|
||||
## Addendum, same day: what was actually changed
|
||||
|
||||
⚠️ The line above — "Nothing in this file has been changed in any environment" — no longer
|
||||
holds. Two of the four gates were given margins and one was deliberately not. Everything
|
||||
below was measured, not argued, and the measurements are reproducible from the repository
|
||||
without touching a model:
|
||||
|
||||
```bash
|
||||
uv run --with regex python tools/regate.py # replay the 32 real rollouts per env
|
||||
uv run --with regex python tools/regate_ladder.py # the same predicates over the ladder
|
||||
uv run --with regex python probe.py # floors and ceilings, per component
|
||||
```
|
||||
|
||||
`tools/regate.py` rebuilds each task from the seed in its trace, re-runs the shipped scorer
|
||||
over the model's own reply, and asks a candidate predicate directly. That is a better
|
||||
measurement than a fresh sample would have been: it holds the model's behaviour fixed and
|
||||
varies only the reward, and it costs nothing on spark-1. Its `recorded_mean` reproduces
|
||||
0.3488, 0.4432 and 0.4040 exactly, which is the cross-check that it is replaying the same
|
||||
population `eval.log` scored.
|
||||
|
||||
### `bot-detection` — fixed. `GATE_SLACK = 1`
|
||||
|
||||
The reference's bots bar one, and still not one person.
|
||||
|
||||
| gate | fires on 32 real rollouts |
|
||||
|---|---|
|
||||
| exact (as shipped) | 0/32 = 0.000 |
|
||||
| **slack 1** | **5/32 = 0.156** |
|
||||
| slack 2 | 15/32 = 0.469 |
|
||||
| share 0.90 of the reference | 0/32 = 0.000 |
|
||||
|
||||
Slack 2 is what the ladder rules out. At slack 1 the oracle's list minus one account clears
|
||||
the gate and minus two does not, so it still separates a near miss from a half-right answer;
|
||||
at slack 2 both clear and it separates nothing. `inaction` and `crude` score 0.000 at every
|
||||
slack, and `probe.py` asserts all three of those facts rather than reporting them.
|
||||
|
||||
`max(1, reference_caught - GATE_SLACK)` is load-bearing: without it a batch where the
|
||||
reference catches exactly one bot would let an **empty** accusation list clear the gate.
|
||||
Observed `reference_caught` is 4–6, so nothing would have caught that in a sample.
|
||||
|
||||
The restraint clause was not given a margin and should not be. A single wrongful ban still
|
||||
shuts the gate outright, because that asymmetry is the environment.
|
||||
|
||||
### `schema-migration` — fixed. `GATE_MARGIN = 0.15`
|
||||
|
||||
Thirty-four of forty rows recompose. `schema_ok` and the row count stay exact.
|
||||
|
||||
| gate | fires on 32 real rollouts | `plausible` on the ladder |
|
||||
|---|---|---|
|
||||
| exact (as shipped) | 0/32 = 0.000 | 0.000 |
|
||||
| margin 0.05 | 0/32 = 0.000 | 0.000 |
|
||||
| margin 0.10 | 0/32 = 0.000 | 0.000 |
|
||||
| **margin 0.15** | **4/32 = 0.125** | 0.042 |
|
||||
| margin 0.20 | 8/32 = 0.250 | 0.292 |
|
||||
|
||||
0.20 is out: it pays the naive split — the parser fitted to the five tidy rows on screen,
|
||||
which is precisely what this environment exists to punish — on 29% of seeds. 0.15 pays it on
|
||||
one seed in twenty-four, because its luckiest draw of forty awkward rows recomposes exactly
|
||||
34 of them and 34/40 is the threshold to the digit. That knife edge is named in the source
|
||||
and `probe.py` asserts it stays under a tenth.
|
||||
|
||||
What 0.15 buys is the rung above it. A migration that strips the thousands separator — the
|
||||
inference this environment is about — and still mishandles the row with no unit at all goes
|
||||
from clearing the gate on 8% of seeds to clearing it on 96%.
|
||||
|
||||
The two format clauses keep no margin. `wipe` — the oracle followed by `DELETE FROM
|
||||
readings`, which recomposes perfectly over nothing — is now a probe rung asserted at 0.000,
|
||||
so the hole the row clause was added to close cannot quietly reopen.
|
||||
|
||||
### `redaction-pressure` — NOT fixed, and that is the finding
|
||||
|
||||
| gate | fires on 32 real rollouts | `plausible` (4 of 7 rules) |
|
||||
|---|---|---|
|
||||
| exact (as shipped) | 0/32 = 0.000 | 0.000 |
|
||||
| recall margin 0.10, collateral 0, innocent 0 | 0/32 = 0.000 | 0.000 |
|
||||
| recall margin 0.20, collateral 1, innocent 32 | 0/32 = 0.000 | 0.000 |
|
||||
| recall margin 0.50, collateral 2, innocent 160 | 3/32 = 0.094 | **0.833** |
|
||||
|
||||
There is no setting that both fires on the measured population and keeps `plausible` at
|
||||
zero. Everything loose enough to fire pays a ruleset that never found three of the seven
|
||||
secret types, on five seeds in six; everything tight enough to hold that at zero fires on
|
||||
none of the 32.
|
||||
|
||||
The reason is that the failure is **joint**, not a threshold. Recall maxed at 0.852 with a
|
||||
minimum of four secrets left standing; `collateral_hits == 0` held on 6 of 32; and no
|
||||
rollout managed high recall and zero collateral in the same episode. So the verdict for this
|
||||
one is **(c) genuinely hard**, not (a) mis-thresholded — which is a different answer from
|
||||
the one this file gave in §3 above, and it is the answer the measurement gives.
|
||||
|
||||
Half the secrets left in the document is not a near miss. The fix, when it comes, is a
|
||||
better model or an easier corpus, not a looser gate. The sweep is recorded in
|
||||
`Outcome.clean`'s docstring so it is not redone from scratch.
|
||||
|
||||
### `grand-exchange` — untouched, as §3 said
|
||||
|
||||
Its 0.0055 measured a sampling config and not a gate. Left alone.
|
||||
|
||||
## And the blindness that let all four hide
|
||||
|
||||
`probe.py` returned one blended float per rung. It now returns the **weighted components**,
|
||||
prints each one's floor, its best rung below the oracle and its ceiling, and draws two lines:
|
||||
|
||||
- **flat** — the same value on every rung including the oracle — is fatal. That is a constant
|
||||
added to every policy's score and there is no legitimate reward of that shape.
|
||||
- **step@oracle** — nothing below the oracle earns a fraction — warns and names the
|
||||
component.
|
||||
|
||||
The second is deliberately not fatal, and the reason is the whole difficulty of this file: it
|
||||
is the shape of a dead gate AND the shape of an honest binary check, and the two are not
|
||||
distinguishable from a ladder. Making it fatal would forbid every binary gate; making it
|
||||
silent is what cost a month. So it is an instruction — go and replay the traces — and
|
||||
`tests/test_probe.py::STEP_AT_ORACLE` is the allowlist that keeps the instruction from
|
||||
becoming wallpaper. Two names are on it, `redaction-pressure/gate` (0/32, hard) and
|
||||
`fault-localisation/gate` (29/32, healthy — the ladder just has no two-of-three rung). It
|
||||
only ever shrinks, and a name may only leave it alongside the measurement that justifies it.
|
||||
|
||||
Four environments also gained a `near-oracle` rung, which is what makes the check say
|
||||
anything: a gate carrying a real band now demonstrates it — `grand-exchange`'s 0.90 bar reads
|
||||
0.244 of 0.250 one unit below the reference, and `bot-detection`'s new slack reads the full
|
||||
0.250 at the oracle's list minus one.
|
||||
|
||||
Reference in New Issue
Block a user