gates: give two of them a margin, and make the probe see components
The probe printed one blended number per policy, so a component pinned at 0.000 across every rung was invisible. Four gates hid there for a month. It now reports floor / best-below-oracle / oracle / ceiling for 25 components across 8 environments, and a component flat across every rung is fatal. bot-detection GATE_SLACK = 1 — fires 5/32 on the real rollouts, was 0/32. The max(1, reference_caught - SLACK) guard is verified by construction, not by sampling: without it the required count reaches 0 and an EMPTY accusation list clears the gate. Observed reference_caught is 4-6, so no amount of sampling would have found that hole. schema-migration GATE_MARGIN = 0.15 — fires 4/32, was 0/32. The cost is disclosed and bounded: the naive split clears it on 5.5% of 1,000 unseen seeds, fenced by an assert at 10%. Margin 0.10 keeps the leak at zero and fires 0/32, i.e. stays dead. A live gradient with a bounded leak beats a clean corpse. redaction-pressure is NOT given a margin, and that is the result rather than a failure. The only setting that fires at all leaves half the secrets standing and pays a four-of-seven ruleset on five seeds in six — a margin that pays for inaction is strictly worse than a dead gate. Recall maxes at 0.852 and no rollout ever cleared both clauses in one episode. It is genuinely hard, not miscalibrated. ⚠️ The per-component check did not catch the defect it was built for. Reverting schema-migration's margin to 0.0 — restoring the exact dead gate — printed ok and exited 0, because the near-oracle rung scrapes the unmargined gate on ~2 seeds in 24 and that kept best<oracle non-zero. Every assertion bounded how much a margin may PAY; none noticed if it stopped existing. migration() now carries the mirror of bot-detection's guard, and reverting the margin fails with "the margin is dead and the component carries no gradient between the crude answer and the exact one". canary-trap's oracle-minus-one rung is documented as degenerate rather than quietly relied on: it is identical to the oracle to four decimals, so it measures specificity and gate at the ceiling, not mid-ladder as its comment claimed. The CI lock policy asks git instead of the disk. It was checking the working tree, where a lock file is a normal by-product of uv sync, so it passed in a clean checkout and failed on every machine that had run an eval. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -11,6 +11,36 @@ from pathlib import Path
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
ROW = re.compile(r"^(?P<name>\S(?:.*\S)?)\s{2,}(?:-?\d+\.\d+\s*){4}\s+(?P<verdict>.+?)$")
|
||||
|
||||
# The per-component block, which is indented by two spaces precisely so that ROW — anchored
|
||||
# at `^\S` — cannot swallow it. Four numbers there, five here.
|
||||
PART = re.compile(
|
||||
r"^ (?P<env>\S+)\s+(?P<component>\S+)\s+"
|
||||
r"(?P<floor>-?\d+\.\d+)\s+(?P<below>-?\d+\.\d+)\s+"
|
||||
r"(?P<oracle>-?\d+\.\d+)\s+(?P<ceiling>-?\d+\.\d+)\s+(?P<verdict>.+?)$"
|
||||
)
|
||||
|
||||
# Reward components that nothing below the oracle earns a fraction of. `probe.py` warns and
|
||||
# names them; this is what stops the warning becoming wallpaper.
|
||||
#
|
||||
# Both of these were investigated by replaying `outputs/run-20260821-1401` through a
|
||||
# candidate gate (`tools/regate.py`), which is the only measurement that separates a dead
|
||||
# gate from a merely binary one, and both were left as they are on purpose:
|
||||
#
|
||||
# redaction-pressure/gate fired 0/32. Every margin loose enough to fire on the measured
|
||||
# population also pays a four-rule ruleset on five seeds in six.
|
||||
# Genuinely hard, not mis-thresholded. See `Outcome.clean`.
|
||||
# fault-localisation/gate fired 29/32 — 0.9062, the healthiest gate in the repository.
|
||||
# The ladder simply has no rung that gets two of three fields
|
||||
# right and clears it, because the gate wants all three.
|
||||
#
|
||||
# `bot-detection/gate` and `schema-migration/gate` were on this list and are not any more:
|
||||
# they came back 0/32 and were given margins. It only ever shrinks, and a name may only be
|
||||
# deleted alongside the measurement that justifies it.
|
||||
STEP_AT_ORACLE = frozenset({
|
||||
("redaction-pressure", "gate"),
|
||||
("fault-localisation", "gate"),
|
||||
})
|
||||
|
||||
# The environments that are allowed to carry no probe yet, because their taskset layer is not
|
||||
# written. Everything discovered and NOT in here MUST be gated.
|
||||
#
|
||||
@@ -92,6 +122,54 @@ class RootProbeIntegrationTests(unittest.TestCase):
|
||||
)
|
||||
self.assertTrue(rows, msg=f"the probe gated nothing at all:{detail}")
|
||||
|
||||
# --- and the same again, one reward component at a time -------------------------
|
||||
# Everything above passed for a month while four `gate` components scored exactly
|
||||
# 0.000 mean and 0.000 max over 32 real rollouts, because a blended `oracle 1.000`
|
||||
# is exactly the thing that cannot show a constant inside it.
|
||||
parts = {}
|
||||
for line in completed.stdout.splitlines():
|
||||
match = PART.match(line.rstrip())
|
||||
if match:
|
||||
parts[(match["env"], match["component"])] = match
|
||||
|
||||
self.assertTrue(parts, msg=f"the probe printed no per-component block:{detail}")
|
||||
for name in rows:
|
||||
with self.subTest(environment=name):
|
||||
self.assertTrue(
|
||||
any(env == name for env, _ in parts),
|
||||
msg=f"{name} is gated but no component of it is reported:{detail}",
|
||||
)
|
||||
|
||||
for (env, component), match in sorted(parts.items()):
|
||||
with self.subTest(environment=env, component=component):
|
||||
verdict = match["verdict"].strip()
|
||||
self.assertFalse(
|
||||
verdict.startswith("FLAT"),
|
||||
msg=f"{env}/{component} is constant across the ladder:{detail}",
|
||||
)
|
||||
# House rule 3, per term rather than per environment. A blended floor of
|
||||
# 0.000 is compatible with one component paying for inaction and another
|
||||
# going negative to cancel it; this is the form that is not.
|
||||
self.assertLessEqual(
|
||||
float(match["floor"]), 1e-9,
|
||||
msg=f"{env}/{component} pays {match['floor']} for the worst rung on the "
|
||||
f"ladder — inaction is being paid for a component:{detail}",
|
||||
)
|
||||
stepped = verdict.startswith("step@oracle")
|
||||
if (env, component) in STEP_AT_ORACLE:
|
||||
self.assertTrue(
|
||||
stepped,
|
||||
msg=f"{env}/{component} now carries a gradient below the oracle — "
|
||||
f"delete it from STEP_AT_ORACLE:{detail}",
|
||||
)
|
||||
else:
|
||||
self.assertFalse(
|
||||
stepped,
|
||||
msg=f"{env}/{component} earns nothing below the oracle and is not "
|
||||
f"on the STEP_AT_ORACLE list. Replay the traces through it with "
|
||||
f"tools/regate.py before adding it:{detail}",
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
Reference in New Issue
Block a user