gates: give two of them a margin, and make the probe see components

The probe printed one blended number per policy, so a component pinned at 0.000
across every rung was invisible. Four gates hid there for a month. It now reports
floor / best-below-oracle / oracle / ceiling for 25 components across 8
environments, and a component flat across every rung is fatal.

bot-detection GATE_SLACK = 1 — fires 5/32 on the real rollouts, was 0/32. The
max(1, reference_caught - SLACK) guard is verified by construction, not by
sampling: without it the required count reaches 0 and an EMPTY accusation list
clears the gate. Observed reference_caught is 4-6, so no amount of sampling would
have found that hole.

schema-migration GATE_MARGIN = 0.15 — fires 4/32, was 0/32. The cost is disclosed
and bounded: the naive split clears it on 5.5% of 1,000 unseen seeds, fenced by
an assert at 10%. Margin 0.10 keeps the leak at zero and fires 0/32, i.e. stays
dead. A live gradient with a bounded leak beats a clean corpse.

redaction-pressure is NOT given a margin, and that is the result rather than a
failure. The only setting that fires at all leaves half the secrets standing and
pays a four-of-seven ruleset on five seeds in six — a margin that pays for
inaction is strictly worse than a dead gate. Recall maxes at 0.852 and no rollout
ever cleared both clauses in one episode. It is genuinely hard, not miscalibrated.

⚠️ The per-component check did not catch the defect it was built for. Reverting
schema-migration's margin to 0.0 — restoring the exact dead gate — printed ok and
exited 0, because the near-oracle rung scrapes the unmargined gate on ~2 seeds in
24 and that kept best<oracle non-zero. Every assertion bounded how much a margin
may PAY; none noticed if it stopped existing. migration() now carries the mirror
of bot-detection's guard, and reverting the margin fails with "the margin is dead
and the component carries no gradient between the crude answer and the exact one".

canary-trap's oracle-minus-one rung is documented as degenerate rather than
quietly relied on: it is identical to the oracle to four decimals, so it measures
specificity and gate at the ceiling, not mid-ladder as its comment claimed.

The CI lock policy asks git instead of the disk. It was checking the working
tree, where a lock file is a normal by-product of uv sync, so it passed in a clean
checkout and failed on every machine that had run an eval.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-21 23:14:05 -07:00
co-authored by Claude Opus 5
parent 122a858b61
commit 3305be4ff7
12 changed files with 1107 additions and 95 deletions
+1 -1
View File
@@ -87,7 +87,7 @@ Corrected, blocking:
6. **`portal_host_samples` and `portal_container_samples` have no node identity** — no `node_id`, no host column, no PK. Per-node charts need `ALTER TABLE … ADD COLUMN IF NOT EXISTS node_id text` plus an index first.
7. **`scored = ok AND rewards non-empty` is wrong.** `write_episode` dumps with `exclude_none=True`, so a `None` reward is dropped from the file entirely — "not measured" and "key absent" are indistinguishable on the wire. Require at least one non-null `Reward`, and carry the run's declared reward-name set from the config.
8. **`Reward.value` is a `@property`** and never appears in dumped JSON. Compute `score × weight` in the ingester.
9. **0.3.0 → 0.4.0 changes the directory layout, not just the config filename** — HEAD produces one flat `<env>--<model>--<harness>--<hex8>` dir with no nested uuid, and `RunConfig.id` is a `PrivateAttr` absent from `configs/eval.json`. Derive the run id from the directory basename.
9. ⚠️ **CORRECTED 2026-08-21: there is NO verifiers 0.4.0.** The `version = "0.4.0"` at `~/vendor/prime-intellect/verifiers/pyproject.toml:136` belongs to a `[[tool.uv.dependency-metadata]]` block for **nemo-gym**, not to verifiers — read as verifiers' own version it sends you hunting a release that does not exist and concluding the bump is blocked. The vendored checkout is `v0.3.0-59-g4bcb48e5`, published to PyPI as **`0.3.1.dev59`**, byte-identical to the vendored tree, and that is what all nine manifests now pin. The layout warning itself is CORRECT: **0.3.0 → dev59 changes the directory layout, not just the config filename** — HEAD produces one flat `<env>--<model>--<harness>--<hex8>` dir with no nested uuid, and `RunConfig.id` is a `PrivateAttr` absent from `configs/eval.json`. Derive the run id from the directory basename.
10. **Name the private console `/nodes`**, not `/compute``/compute` is already a public marketing section with its own nav.
11. Put `isAppHost` in `packages/shared` — two independent string checks that must agree, with a silent asymmetric failure mode (dashboard rendered under a 404 status).
12. `sendShell(url, reply)` takes no request; the signature and both call sites change. `spa-fallback.test.ts` needs a case. `@radix-ui/react-select` is **not** installed.