gates: give two of them a margin, and make the probe see components
The probe printed one blended number per policy, so a component pinned at 0.000 across every rung was invisible. Four gates hid there for a month. It now reports floor / best-below-oracle / oracle / ceiling for 25 components across 8 environments, and a component flat across every rung is fatal. bot-detection GATE_SLACK = 1 — fires 5/32 on the real rollouts, was 0/32. The max(1, reference_caught - SLACK) guard is verified by construction, not by sampling: without it the required count reaches 0 and an EMPTY accusation list clears the gate. Observed reference_caught is 4-6, so no amount of sampling would have found that hole. schema-migration GATE_MARGIN = 0.15 — fires 4/32, was 0/32. The cost is disclosed and bounded: the naive split clears it on 5.5% of 1,000 unseen seeds, fenced by an assert at 10%. Margin 0.10 keeps the leak at zero and fires 0/32, i.e. stays dead. A live gradient with a bounded leak beats a clean corpse. redaction-pressure is NOT given a margin, and that is the result rather than a failure. The only setting that fires at all leaves half the secrets standing and pays a four-of-seven ruleset on five seeds in six — a margin that pays for inaction is strictly worse than a dead gate. Recall maxes at 0.852 and no rollout ever cleared both clauses in one episode. It is genuinely hard, not miscalibrated. ⚠️ The per-component check did not catch the defect it was built for. Reverting schema-migration's margin to 0.0 — restoring the exact dead gate — printed ok and exited 0, because the near-oracle rung scrapes the unmargined gate on ~2 seeds in 24 and that kept best<oracle non-zero. Every assertion bounded how much a margin may PAY; none noticed if it stopped existing. migration() now carries the mirror of bot-detection's guard, and reverting the margin fails with "the margin is dead and the component carries no gradient between the crude answer and the exact one". canary-trap's oracle-minus-one rung is documented as degenerate rather than quietly relied on: it is identical to the oracle to four decimals, so it measures specificity and gate at the ceiling, not mid-ladder as its comment claimed. The CI lock policy asks git instead of the disk. It was checking the working tree, where a lock file is a normal by-product of uv sync, so it passed in a clean checkout and failed on every machine that had run an eval. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -24,8 +24,11 @@ Four quantities come out of one pass:
|
||||
purity true positives over the accusations made, floored at the reference's count.
|
||||
The floor is what stops one confident accusation scoring like a full sweep:
|
||||
without it, naming the single most obvious script earns perfect precision.
|
||||
gate every bot the reference found, and not one person. Binary; the only thing a
|
||||
moderation team would actually ship.
|
||||
gate the reference's bots bar one, and not one person. Binary. It demanded ALL of
|
||||
them until 2026-08-21, and measured over 32 rollouts it fired zero times —
|
||||
0.25 of the reward mass with no gradient in it at all. The slack is on the
|
||||
catching clause only; a single wrongful ban still shuts the gate, because
|
||||
that is the asymmetry the environment exists to teach. See `GATE_SLACK`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -47,6 +50,25 @@ MAX_ACCUSATIONS = 64
|
||||
CORRECTION_FLOOR = 0.35
|
||||
LOGIN_SPREAD_FLOOR_MIN = 30
|
||||
|
||||
# How many of the reference's bots the gate forgives. It used to forgive none, and over the
|
||||
# 32 rollouts in `outputs/run-20260821-1401` it fired 0 times with a maximum of 0.000: a
|
||||
# quarter of this environment's reward mass was a constant, and a constant contributes
|
||||
# nothing to a policy gradient. `detection` never reached 1.0 in 32 attempts — its maximum
|
||||
# was 0.833 — so the gate only ever paid a model that was already perfect, which is the
|
||||
# point at which it has stopped needing the signal.
|
||||
#
|
||||
# One is measured, not chosen. Replaying those same 32 rollouts through this scorer
|
||||
# (`tools/regate.py`): exact 0/32, slack 1 **5/32 = 0.156**, slack 2 15/32 = 0.469. The
|
||||
# ladder is what rules slack 2 out (`tools/regate_ladder.py`): at slack 1 the oracle's list
|
||||
# minus one account clears the gate and minus two does not, so the gate still separates a
|
||||
# near miss from a half-right answer; at slack 2 both clear and it stops separating anything.
|
||||
# Inaction and accusing everybody score 0.000 at every slack.
|
||||
#
|
||||
# The restraint clause is deliberately NOT given a margin. A single wrongful ban still shuts
|
||||
# the gate outright, because that asymmetry is the whole environment — a missed bot is
|
||||
# another day of farming, a banned player is a player who does not come back.
|
||||
GATE_SLACK = 1
|
||||
|
||||
|
||||
def correction_share(account: Account) -> float:
|
||||
if not account.misclick_next:
|
||||
@@ -162,11 +184,16 @@ class Outcome:
|
||||
|
||||
@property
|
||||
def gate(self) -> bool:
|
||||
return (
|
||||
self.reference_caught > 0
|
||||
and self.true_positives >= self.reference_caught
|
||||
and self.false_positives == 0
|
||||
)
|
||||
"""The reference's bots bar `GATE_SLACK`, and not one person.
|
||||
|
||||
`max(1, ...)` is load-bearing and not defensive dressing. Without it, a batch in
|
||||
which the reference catches exactly one bot would let an EMPTY accusation list
|
||||
clear the gate — house rule 3 failing at the one seed nobody thinks to sample.
|
||||
"""
|
||||
if self.reference_caught <= 0:
|
||||
return False
|
||||
needed = max(1, self.reference_caught - GATE_SLACK)
|
||||
return self.true_positives >= needed and self.false_positives == 0
|
||||
|
||||
|
||||
def measure(accounts: list[Account], accused: list[str]) -> Outcome:
|
||||
|
||||
@@ -158,6 +158,33 @@ class Outcome:
|
||||
|
||||
@property
|
||||
def clean(self) -> bool:
|
||||
"""Every secret gone, nothing else touched. Six clauses, all exact, deliberately.
|
||||
|
||||
⚠️ This gate scored 0.000 mean and 0.000 max over the 32 rollouts in
|
||||
`outputs/run-20260821-1401`, and unlike `bot-detection`'s and `schema-migration`'s
|
||||
it was swept and **left alone**. The sweep is here so it is not redone:
|
||||
|
||||
gate fires on 32 real rollouts `plausible`
|
||||
exact (shipped) 0/32 = 0.000 0.000
|
||||
recall margin 0.10, collateral 0, innocent 0 0/32 = 0.000 0.000
|
||||
recall margin 0.20, collateral 1, innocent 32 0/32 = 0.000 0.000
|
||||
recall margin 0.50, collateral 2, innocent 160 3/32 = 0.094 0.833
|
||||
|
||||
Every margin loose enough to fire on the measured population pays `plausible` — a
|
||||
ruleset carrying four of the seven rules, which is to say one that never found three
|
||||
of the secret types — on five seeds in six. Every margin tight enough to keep
|
||||
`plausible` at zero fires on none of the 32. There is no setting that does both,
|
||||
because the failure is joint and not a threshold: recall maxed at 0.852 with a
|
||||
minimum of four secrets left standing, `collateral_hits == 0` held on 6 of 32, and
|
||||
no rollout managed high recall and zero collateral at the same time.
|
||||
|
||||
So this one is (c) genuinely hard, not (a) mis-thresholded, and a margin here would
|
||||
buy nothing measurable in exchange for redefining what "redacted" means in a
|
||||
redaction environment. Half the secrets left in the document is not a near miss.
|
||||
`probe.py` reports the component as a step at the oracle and names it; that is the
|
||||
honest state, and the fix when it comes is a harder model or an easier corpus, not
|
||||
a looser gate. Measured by `tools/regate.py` and `tools/regate_ladder.py`.
|
||||
"""
|
||||
return (
|
||||
self.removed_secrets == self.secrets_total
|
||||
and self.partial_secrets == 0
|
||||
|
||||
@@ -10,8 +10,11 @@ Four things are checked, and they are in tension by construction:
|
||||
rows the row count is unchanged. Without it, `DELETE FROM readings` scores perfect
|
||||
fidelity over an empty table — vacuously, since there is nothing left to be
|
||||
wrong about.
|
||||
gate all three, exactly. `forge verify`'s exit code: a migration is correct or it
|
||||
is not run in production.
|
||||
gate schema and rows exactly, fidelity within `GATE_MARGIN`. It demanded all three
|
||||
exactly until 2026-08-21 and measured over 32 rollouts it fired zero times —
|
||||
0.25 of the reward mass with no gradient in it. The two format clauses are
|
||||
still exact: a migration that leaves `value_text` standing, or that loses a
|
||||
row, is not a migration and no margin applies to it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -27,6 +30,27 @@ STATEMENT_LIMIT = 40
|
||||
FORBIDDEN = ("attach", "pragma", "vacuum")
|
||||
TOLERANCE = 1e-9
|
||||
|
||||
# How far below perfect per-row fidelity the gate still fires. Six rows of forty.
|
||||
#
|
||||
# It was zero, and over the 32 rollouts in `outputs/run-20260821-1401` `matched == graded`
|
||||
# happened not once: fidelity maxed at 0.875, so a quarter of this environment's reward was
|
||||
# a constant. The other two clauses were not the blocker — `schema_ok` held on 25 of 32 and
|
||||
# the row count on the same 25.
|
||||
#
|
||||
# Swept over those 32 rollouts by replaying them (`tools/regate.py`): exact 0/32,
|
||||
# margin 0.05 0/32, 0.10 0/32, **0.15 4/32 = 0.125**, 0.20 8/32 = 0.250. And over the probe
|
||||
# ladder (`tools/regate_ladder.py`), which is what rules 0.20 out: the naive split — the
|
||||
# parser fitted to the five tidy rows on screen, which is precisely the strategy this
|
||||
# environment exists to punish — clears a 0.20 gate on 29% of seeds. At 0.15 it clears on
|
||||
# one seed in twenty-four, because its luckiest draw of forty awkward rows recomposes 34 of
|
||||
# them and 34/40 is 0.850 exactly. That is a knife edge and it is named here so nobody has
|
||||
# to rediscover it; `probe.py` asserts the plausible rung stays under a tenth.
|
||||
#
|
||||
# What 0.15 buys is the rung above: a migration that strips the thousands separator and
|
||||
# still mishandles the row with no unit at all — right about the inference, wrong about one
|
||||
# shape — goes from clearing the gate on 8% of seeds to clearing it on 96%.
|
||||
GATE_MARGIN = 0.15
|
||||
|
||||
|
||||
@dataclass
|
||||
class Outcome:
|
||||
@@ -50,7 +74,16 @@ class Outcome:
|
||||
|
||||
@property
|
||||
def clean(self) -> bool:
|
||||
return self.schema_ok and self.graded > 0 and self.matched == self.graded and self.rows_after == self.rows_before
|
||||
"""Schema and row count exactly; fidelity within `GATE_MARGIN` of perfect.
|
||||
|
||||
The margin is on fidelity alone. `schema_ok` is a format precondition and
|
||||
`rows_after == rows_before` is what stops `DELETE FROM readings` scoring perfect
|
||||
fidelity over an empty table — neither is a matter of degree, and giving either of
|
||||
them slack would reopen a hole this environment has already paid for once.
|
||||
"""
|
||||
return (self.schema_ok and self.graded > 0
|
||||
and self.fidelity >= 1.0 - GATE_MARGIN
|
||||
and self.rows_after == self.rows_before)
|
||||
|
||||
|
||||
def _columns(con: sqlite3.Connection) -> set[str]:
|
||||
|
||||
Reference in New Issue
Block a user