arena/docs/GATE_DIAGNOSIS.md

Why gate scored zero in four environments

Written 2026-08-21, against outputs/run-20260821-1401 — seven environments, 32 rollouts each, brain-qwen38-dspark.

In four of the seven, the gate reward was 0.000 mean and 0.000 max across all 32 rollouts: bot-detection (weight 0.25), grand-exchange (0.25), redaction-pressure (0.30) and schema-migration (0.25). It moved in the other three — canary-trap 0.2188, drop-table-inference 0.2258, fault-localisation 0.9062.

probe.py prints 1.000 for every one of those four, and 1.000 requires gate = 1, so nothing here is unreachable and house rule 3 is not violated. The question this file answers is the one after that: is the gate (a) a threshold set beyond what a real model reaches, (b) a parse or format precondition the model never satisfies, or (c) genuinely hard? They have different consequences and they need different fixes.

---

The answer, in one table

outputs/gate_probe.py runs each environment's shipped scorer over the probe ladder and prints the gate on its own, as a pass rate over 24 tasks (12 for grand-exchange). probe.py blends the gate into one number per rung, which is exactly why this was invisible for a month.

#### redaction-pressure
   inaction     gate 0/24 = 0.000
   crude        gate 0/24 = 0.000
   plausible    gate 0/24 = 0.000
   six-of-seven gate 0/24 = 0.000
   oracle       gate 24/24 = 1.000

#### schema-migration
   inaction     gate 0/24 = 0.000
   crude        gate 0/24 = 0.000
   plausible    gate 0/24 = 0.000
   oracle       gate 24/24 = 1.000

#### bot-detection
   inaction     gate 0/24 = 0.000
   crude        gate 0/24 = 0.000
   oracle-minus-one gate 0/24 = 0.000
   oracle       gate 24/24 = 1.000

#### grand-exchange
   inaction     gate 0/12 = 0.000
   oracle       gate 12/12 = 1.000

In three of the four the gate is a step function whose only step is the oracle. Six of redaction's seven rules scores zero. The oracle bot list minus a single account scores zero. There is no rung anywhere on the ladder — inaction, crude, plausible, near-oracle — that earns a fraction of it.

The three that fired are not perfection-free; they are perfection over a small object. canary-trap's gate wants every contaminated state caught and no probe firing on the clean model, over 2–12 probes. fault-localisation's wants service, fault and evidence all correct — three fields. drop-table-inference is the only gate in the repository with a margin: GATE_MARGIN = 0.05, three sub-scores each within 5% of the reference, and it fired on 22.6% of rollouts. grand-exchange has one too — TARGET_SHARE = 0.90 — and its zero has a different cause entirely (below).

So the verdict is (a) for three environments and neither for the fourth. It is not (b) anywhere: parsing works. bot-detection recorded unknown_ids 0.000 on all 32 rollouts, redaction-pressure validity 1.0 on 30 of 32, schema-migration produced a valid target schema on 25 of 32. The models are submitting well-formed work and being scored zero on it because the work is not perfect.

---

Per environment

bot-detection — (a), and the binding clause is detection == 1.0

# scan.py
@property
def gate(self) -> bool:
    return (
        self.reference_caught > 0
        and self.true_positives >= self.reference_caught
        and self.false_positives == 0
    )

Over 32 rollouts:

clause	satisfied
false_positives == 0	24 / 32
true_positives >= reference_caught	0 / 32

detection — true positives over what the reference caught — had mean 0.5135 and max 0.8333. It never reached 1.0 once. The restraint clause is doing its job and is not the problem; the catch-everything clause is a ceiling test.

The environment's own docstring explains why it has to be: each cloaked account leaks on exactly one of three behavioural channels, so "a discriminator built on any single channel ceilings at roughly a third of them". The reference reads all three. The gate therefore pays only a model that found all three tells, in a batch where the reference caught 4–6 bots and the model accused 2.9 on average.

Consequence. 0.25 of bot-detection's reward mass is a constant. The measured 0.3488 is entirely caught (0.35) and spared (0.40), and against a maximum of 0.75 it is 0.465 of what was actually on offer.

redaction-pressure — (a), and the binding clause is total recall

# scan.py
@property
def clean(self) -> bool:
    return (
        self.removed_secrets == self.secrets_total
        and self.partial_secrets == 0
        and self.collateral_hits == 0
        and self.innocent_lost == 0
        and self.secrets_total > 0
        and self.validity == 1.0
    )

Six clauses, and they are satisfied at wildly different rates:

clause	satisfied
partial_secrets == 0	28 / 32
validity == 1.0	30 / 32
collateral_hits == 0	6 / 32
innocent_lost == 0	6 / 32
residual_hits == 0 (every secret removed)	0 / 32

recall had max 0.852 across 32 rollouts and residual_hits a minimum of 4 — there was always at least one secret left standing. Two clauses never fired together either: innocent_lost averaged 140 characters.

This is the widest gate in the repository — six simultaneous conditions, three of them counting to zero over a twelve-document held-out slice — and the ladder shows six of the oracle's seven rules do not clear it.

Consequence. 0.30 of the mass — the largest gate weight of the four — is a constant. 0.4040 against a real maximum of 0.70 is 0.577 of what was on offer.

schema-migration — (a), and the binding clause is per-row fidelity

# run.py
@property
def clean(self) -> bool:
    return (self.schema_ok and self.graded > 0
            and self.matched == self.graded
            and self.rows_after == self.rows_before)

clause	satisfied
schema_ok	25 / 32
rows_after == rows_before	25 / 32
matched == graded (fidelity 1.0)	0 / 32

fidelity distribution over the 32 rollouts:

0.0 ×10, 0.10, 0.15, 0.375, 0.40, 0.575, 0.65, 0.675, 0.725, 0.725, 0.75,
0.775 ×4, 0.80 ×4, 0.85 ×3, 0.875

Max 0.875. Forty awkward held-out rows must every one recompose; the best migration anybody wrote missed five of them. The docstring is explicit that this is forge verify's exit code — a migration is correct or it is not — and as a shipping check that is right. As a reward component it is a constant at this capability level.

grand-exchange — not a gate defect. Nothing was measured

# book.py — a bar, not a knife-edge
return (self.fills.realised >= TARGET_SHARE * self.reference.realised
        and self.fills.roc      >= TARGET_SHARE * self.reference.roc)

TARGET_SHARE = 0.90. This gate already has the margin the other three lack, and it is written that way deliberately — the commit message is in the docstring ("demanding the reference's exact result made a wasted unit of capital cost 0.146 with byte-identical profit").

It scored zero because 31 of 32 rollouts placed no orders at all: 18 completion tokens, {"expected_profit": 0, "orders": []}, finish_reason=stop. planned had mean 65.6 and median 0 — one rollout in 32 planned anything at all; profit_ratio maxed at 0.323, so even that one reached 29% of the reference. Nothing in this environment was measured. See FIRST_EVAL.md §5 — the reference node serves with default_chat_template_kwargs = {"enable_thinking": false}, and this environment is the one that cannot be answered without arithmetic.

Re-measured with thinking on, this gate fires. outputs/thinking/ge-think-n6/, 6 rollouts: one of them finished inside its token budget and scored profit 1.000, discipline 1.000 and gate 1.000, realising 40,081 gp against a reference of 36,157. The other five were cut off mid-reasoning at max_tokens. So this gate is not merely oracle-reachable in a probe — it is reachable by the model under evaluation, in a real rollout, at a 90% bar.

Do not touch this gate. It is the one that works. It is also the shape the other three should be argued against: a band at 0.90 of the reference, cleared by a run that beat the reference by its own route rather than by copying it.

---

What follows

1. Every ranking over those four environments is a ranking over their soft components. Report bot-detection against 0.75, redaction-pressure against 0.70, schema-migration against 0.75 — or say plainly that a quarter to a third of the scale is unreachable. A leaderboard that prints 0.4040 out of 1.000 for redaction-pressure is overstating the headroom by a factor it never names.

2. The gradient problem is worse than the scale problem. A constant term contributes nothing to a policy gradient. Training against these four trains against caught/spared, recall/precision, schema/integrity — and the gate only starts paying once the model is already essentially perfect, which is the point at which it stops needing the signal. This is not what a gate is for.

3. The repository already contains the fix and applies it in exactly one place. drop-table-inference gates on three sub-scores each within GATE_MARGIN = 0.05 of the reference and fires 22.6% of the time; grand-exchange gates at TARGET_SHARE = 0.90. The three dead gates are the three written as exact equality. A margin on each — every secret bar one, 39 rows of 40, the reference's bots less one — would restore a gradient without paying for inaction, because the ladder above shows inaction and crude score zero on every one of these with or without a margin.

4. Whatever is decided, probe.py should print the gate separately. Four dead components hid for a month behind a blended oracle 1.000. outputs/gate_probe.py is the throwaway version of that check; the real one belongs next to the ladder, and it is a one-line consequence of house rule 3 that a component nothing below the oracle can earn is a component that does not discriminate.

⚠️ Superseded by the addendum below. As first written this file changed nothing in any environment — four packages' reward code was out of that lane, and it was a diagnosis and a recommendation rather than a patch. Two of the four have since been patched and one has been measured and deliberately left alone. Read the addendum before acting on §3 above: its recommendation "a margin on each" turned out to be right about two of the three and wrong about redaction-pressure.

---

Addendum, same day: what was actually changed

⚠️ The line above — "Nothing in this file has been changed in any environment" — no longer holds. Two of the four gates were given margins and one was deliberately not. Everything below was measured, not argued, and the measurements are reproducible from the repository without touching a model:

uv run --with regex python tools/regate.py          # replay the 32 real rollouts per env
uv run --with regex python tools/regate_ladder.py   # the same predicates over the ladder
uv run --with regex python probe.py                 # floors and ceilings, per component

tools/regate.py rebuilds each task from the seed in its trace, re-runs the shipped scorer over the model's own reply, and asks a candidate predicate directly. That is a better measurement than a fresh sample would have been: it holds the model's behaviour fixed and varies only the reward, and it costs nothing on the reference node. Its recorded_mean reproduces 0.3488, 0.4432 and 0.4040 exactly, which is the cross-check that it is replaying the same population eval.log scored.

bot-detection — fixed. GATE_SLACK = 1

The reference's bots bar one, and still not one person.

gate	fires on 32 real rollouts
exact (as shipped)	0/32 = 0.000
slack 1	5/32 = 0.156
slack 2	15/32 = 0.469
share 0.90 of the reference	0/32 = 0.000

Slack 2 is what the ladder rules out. At slack 1 the oracle's list minus one account clears the gate and minus two does not, so it still separates a near miss from a half-right answer; at slack 2 both clear and it separates nothing. inaction and crude score 0.000 at every slack, and probe.py asserts all three of those facts rather than reporting them.

max(1, reference_caught - GATE_SLACK) is load-bearing: without it a batch where the reference catches exactly one bot would let an empty accusation list clear the gate. Observed reference_caught is 4–6, so nothing would have caught that in a sample.

The restraint clause was not given a margin and should not be. A single wrongful ban still shuts the gate outright, because that asymmetry is the environment.

schema-migration — fixed. GATE_MARGIN = 0.15

Thirty-four of forty rows recompose. schema_ok and the row count stay exact.

gate	fires on 32 real rollouts	plausible on the ladder
exact (as shipped)	0/32 = 0.000	0.000
margin 0.05	0/32 = 0.000	0.000
margin 0.10	0/32 = 0.000	0.000
margin 0.15	4/32 = 0.125	0.042
margin 0.20	8/32 = 0.250	0.292

0.20 is out: it pays the naive split — the parser fitted to the five tidy rows on screen, which is precisely what this environment exists to punish — on 29% of seeds. 0.15 pays it on one seed in twenty-four, because its luckiest draw of forty awkward rows recomposes exactly 34 of them and 34/40 is the threshold to the digit. That knife edge is named in the source and probe.py asserts it stays under a tenth.

What 0.15 buys is the rung above it. A migration that strips the thousands separator — the inference this environment is about — and still mishandles the row with no unit at all goes from clearing the gate on 8% of seeds to clearing it on 96%.

The two format clauses keep no margin. wipe — the oracle followed by DELETE FROM readings, which recomposes perfectly over nothing — is now a probe rung asserted at 0.000, so the hole the row clause was added to close cannot quietly reopen.

redaction-pressure — NOT fixed, and that is the finding

gate	fires on 32 real rollouts	plausible (4 of 7 rules)
exact (as shipped)	0/32 = 0.000	0.000
recall margin 0.10, collateral 0, innocent 0	0/32 = 0.000	0.000
recall margin 0.20, collateral 1, innocent 32	0/32 = 0.000	0.000
recall margin 0.50, collateral 2, innocent 160	3/32 = 0.094	0.833

There is no setting that both fires on the measured population and keeps plausible at zero. Everything loose enough to fire pays a ruleset that never found three of the seven secret types, on five seeds in six; everything tight enough to hold that at zero fires on none of the 32.

The reason is that the failure is joint, not a threshold. Recall maxed at 0.852 with a minimum of four secrets left standing; collateral_hits == 0 held on 6 of 32; and no rollout managed high recall and zero collateral in the same episode. So the verdict for this one is (c) genuinely hard, not (a) mis-thresholded — which is a different answer from the one this file gave in §3 above, and it is the answer the measurement gives.

Half the secrets left in the document is not a near miss. The fix, when it comes, is a better model or an easier corpus, not a looser gate. The sweep is recorded in Outcome.clean's docstring so it is not redone from scratch.

grand-exchange — untouched, as §3 said

Its 0.0055 measured a sampling config and not a gate. Left alone.

And the blindness that let all four hide

probe.py returned one blended float per rung. It now returns the weighted components, prints each one's floor, its best rung below the oracle and its ceiling, and draws two lines:

• flat — the same value on every rung including the oracle — is fatal. That is a constant added to every policy's score and there is no legitimate reward of that shape.
• step@oracle — nothing below the oracle earns a fraction — warns and names the component.

The second is deliberately not fatal, and the reason is the whole difficulty of this file: it is the shape of a dead gate AND the shape of an honest binary check, and the two are not distinguishable from a ladder. Making it fatal would forbid every binary gate; making it silent is what cost a month. So it is an instruction — go and replay the traces — and tests/test_probe.py::STEP_AT_ORACLE is the allowlist that keeps the instruction from becoming wallpaper. Two names are on it, redaction-pressure/gate (0/32, hard) and fault-localisation/gate (29/32, healthy — the ladder just has no two-of-three rung). It only ever shrinks, and a name may only leave it alongside the measurement that justifies it.

Four environments also gained a near-oracle rung, which is what makes the check say anything: a gate carrying a real band now demonstrates it — grand-exchange's 0.90 bar reads 0.244 of 0.250 one unit below the reference, and bot-detection's new slack reads the full 0.250 at the oracle's list minus one.