gates: give two of them a margin, and make the probe see components

The probe printed one blended number per policy, so a component pinned at 0.000
across every rung was invisible. Four gates hid there for a month. It now reports
floor / best-below-oracle / oracle / ceiling for 25 components across 8
environments, and a component flat across every rung is fatal.

bot-detection GATE_SLACK = 1 — fires 5/32 on the real rollouts, was 0/32. The
max(1, reference_caught - SLACK) guard is verified by construction, not by
sampling: without it the required count reaches 0 and an EMPTY accusation list
clears the gate. Observed reference_caught is 4-6, so no amount of sampling would
have found that hole.

schema-migration GATE_MARGIN = 0.15 — fires 4/32, was 0/32. The cost is disclosed
and bounded: the naive split clears it on 5.5% of 1,000 unseen seeds, fenced by
an assert at 10%. Margin 0.10 keeps the leak at zero and fires 0/32, i.e. stays
dead. A live gradient with a bounded leak beats a clean corpse.

redaction-pressure is NOT given a margin, and that is the result rather than a
failure. The only setting that fires at all leaves half the secrets standing and
pays a four-of-seven ruleset on five seeds in six — a margin that pays for
inaction is strictly worse than a dead gate. Recall maxes at 0.852 and no rollout
ever cleared both clauses in one episode. It is genuinely hard, not miscalibrated.

⚠️ The per-component check did not catch the defect it was built for. Reverting
schema-migration's margin to 0.0 — restoring the exact dead gate — printed ok and
exited 0, because the near-oracle rung scrapes the unmargined gate on ~2 seeds in
24 and that kept best<oracle non-zero. Every assertion bounded how much a margin
may PAY; none noticed if it stopped existing. migration() now carries the mirror
of bot-detection's guard, and reverting the margin fails with "the margin is dead
and the component carries no gradient between the crude answer and the exact one".

canary-trap's oracle-minus-one rung is documented as degenerate rather than
quietly relied on: it is identical to the oracle to four decimals, so it measures
specificity and gate at the ceiling, not mid-ladder as its comment claimed.

The CI lock policy asks git instead of the disk. It was checking the working
tree, where a lock file is a normal by-product of uv sync, so it passed in a clean
checkout and failed on every machine that had run an eval.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-21 23:14:05 -07:00
co-authored by Claude Opus 5
parent 122a858b61
commit 3305be4ff7
12 changed files with 1107 additions and 95 deletions
@@ -10,8 +10,11 @@ Four things are checked, and they are in tension by construction:
rows the row count is unchanged. Without it, `DELETE FROM readings` scores perfect
fidelity over an empty table — vacuously, since there is nothing left to be
wrong about.
gate all three, exactly. `forge verify`'s exit code: a migration is correct or it
is not run in production.
gate schema and rows exactly, fidelity within `GATE_MARGIN`. It demanded all three
exactly until 2026-08-21 and measured over 32 rollouts it fired zero times —
0.25 of the reward mass with no gradient in it. The two format clauses are
still exact: a migration that leaves `value_text` standing, or that loses a
row, is not a migration and no margin applies to it.
"""
from __future__ import annotations
@@ -27,6 +30,27 @@ STATEMENT_LIMIT = 40
FORBIDDEN = ("attach", "pragma", "vacuum")
TOLERANCE = 1e-9
# How far below perfect per-row fidelity the gate still fires. Six rows of forty.
#
# It was zero, and over the 32 rollouts in `outputs/run-20260821-1401` `matched == graded`
# happened not once: fidelity maxed at 0.875, so a quarter of this environment's reward was
# a constant. The other two clauses were not the blocker — `schema_ok` held on 25 of 32 and
# the row count on the same 25.
#
# Swept over those 32 rollouts by replaying them (`tools/regate.py`): exact 0/32,
# margin 0.05 0/32, 0.10 0/32, **0.15 4/32 = 0.125**, 0.20 8/32 = 0.250. And over the probe
# ladder (`tools/regate_ladder.py`), which is what rules 0.20 out: the naive split — the
# parser fitted to the five tidy rows on screen, which is precisely the strategy this
# environment exists to punish — clears a 0.20 gate on 29% of seeds. At 0.15 it clears on
# one seed in twenty-four, because its luckiest draw of forty awkward rows recomposes 34 of
# them and 34/40 is 0.850 exactly. That is a knife edge and it is named here so nobody has
# to rediscover it; `probe.py` asserts the plausible rung stays under a tenth.
#
# What 0.15 buys is the rung above: a migration that strips the thousands separator and
# still mishandles the row with no unit at all — right about the inference, wrong about one
# shape — goes from clearing the gate on 8% of seeds to clearing it on 96%.
GATE_MARGIN = 0.15
@dataclass
class Outcome:
@@ -50,7 +74,16 @@ class Outcome:
@property
def clean(self) -> bool:
return self.schema_ok and self.graded > 0 and self.matched == self.graded and self.rows_after == self.rows_before
"""Schema and row count exactly; fidelity within `GATE_MARGIN` of perfect.
The margin is on fidelity alone. `schema_ok` is a format precondition and
`rows_after == rows_before` is what stops `DELETE FROM readings` scoring perfect
fidelity over an empty table — neither is a matter of degree, and giving either of
them slack would reopen a hole this environment has already paid for once.
"""
return (self.schema_ok and self.graded > 0
and self.fidelity >= 1.0 - GATE_MARGIN
and self.rows_after == self.rows_before)
def _columns(con: sqlite3.Connection) -> set[str]: