The probe printed one blended number per policy, so a component pinned at 0.000
across every rung was invisible. Four gates hid there for a month. It now reports
floor / best-below-oracle / oracle / ceiling for 25 components across 8
environments, and a component flat across every rung is fatal.
bot-detection GATE_SLACK = 1 — fires 5/32 on the real rollouts, was 0/32. The
max(1, reference_caught - SLACK) guard is verified by construction, not by
sampling: without it the required count reaches 0 and an EMPTY accusation list
clears the gate. Observed reference_caught is 4-6, so no amount of sampling would
have found that hole.
schema-migration GATE_MARGIN = 0.15 — fires 4/32, was 0/32. The cost is disclosed
and bounded: the naive split clears it on 5.5% of 1,000 unseen seeds, fenced by
an assert at 10%. Margin 0.10 keeps the leak at zero and fires 0/32, i.e. stays
dead. A live gradient with a bounded leak beats a clean corpse.
redaction-pressure is NOT given a margin, and that is the result rather than a
failure. The only setting that fires at all leaves half the secrets standing and
pays a four-of-seven ruleset on five seeds in six — a margin that pays for
inaction is strictly worse than a dead gate. Recall maxes at 0.852 and no rollout
ever cleared both clauses in one episode. It is genuinely hard, not miscalibrated.
⚠️ The per-component check did not catch the defect it was built for. Reverting
schema-migration's margin to 0.0 — restoring the exact dead gate — printed ok and
exited 0, because the near-oracle rung scrapes the unmargined gate on ~2 seeds in
24 and that kept best<oracle non-zero. Every assertion bounded how much a margin
may PAY; none noticed if it stopped existing. migration() now carries the mirror
of bot-detection's guard, and reverting the margin fails with "the margin is dead
and the component carries no gradient between the crude answer and the exact one".
canary-trap's oracle-minus-one rung is documented as degenerate rather than
quietly relied on: it is identical to the oracle to four decimals, so it measures
specificity and gate at the ceiling, not mid-ladder as its comment claimed.
The CI lock policy asks git instead of the disk. It was checking the working
tree, where a lock file is a normal by-product of uv sync, so it passed in a clean
checkout and failed on every machine that had run an eval.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The environment list was written out four times — probe.py's shim tuple and its
results dict, tests/test_probe.py's ENVIRONMENTS, and ci.yml's wheel loop — so
adding an environment meant four edits and five workstreams collided on three
files. Discovery is now a tomllib scan of environments/*/pyproject.toml plus a
@probes(name) registry, with an optional [tool.arena] tasksets key for one
package that ships several (tera_spatial declares four).
An environment discovered without a registered probe WARNS rather than exits 1,
because interactive environments land their engine before their probe. That is
temporary; it becomes a hard failure with the last registration.
The warning is not a licence to stop being gated. test_probe.py pins the
ungated set to an explicit PENDING_PROBES allowlist that only ever shrinks —
without it, removing a @probes decorator moved the name from the table into a
warning line that already carries five, and every remaining assertion stayed
vacuously true while the environment silently stopped being gated.
Also untracks three environments/*/uv.lock files, which had been failing the
lock-policy step on main.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The redaction-pressure hardening landed a test asserting probe.py gates every
environment and a CI step building every wheel, and both carried a hardcoded
list of four. Merging three more environments in left the assertion counting
four "ok" lines against seven, and left the new wheels unbuilt by CI.
Both lists now name all seven. The count assertion is kept rather than relaxed:
a new environment missing from the list fails the build instead of quietly
narrowing what the test covers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kartiandClaude Opus 5