arena/docs/GATE_DIAGNOSIS.md

Why gate scored zero in four environments

Written 2026-08-21, against outputs/run-20260821-1401 — seven environments, 32 rollouts each, brain-qwen38-dspark.

In four of the seven, the gate reward was 0.000 mean and 0.000 max across all 32 rollouts: bot-detection (weight 0.25), grand-exchange (0.25), redaction-pressure (0.30) and schema-migration (0.25). It moved in the other three — canary-trap 0.2188, drop-table-inference 0.2258, fault-localisation 0.9062.

probe.py prints 1.000 for every one of those four, and 1.000 requires gate = 1, so nothing here is unreachable and house rule 3 is not violated. The question this file answers is the one after that: is the gate (a) a threshold set beyond what a real model reaches, (b) a parse or format precondition the model never satisfies, or (c) genuinely hard? They have different consequences and they need different fixes.

---

The answer, in one table

outputs/gate_probe.py runs each environment's shipped scorer over the probe ladder and prints the gate on its own, as a pass rate over 24 tasks (12 for grand-exchange). probe.py blends the gate into one number per rung, which is exactly why this was invisible for a month.

#### redaction-pressure
   inaction     gate 0/24 = 0.000
   crude        gate 0/24 = 0.000
   plausible    gate 0/24 = 0.000
   six-of-seven gate 0/24 = 0.000
   oracle       gate 24/24 = 1.000

#### schema-migration
   inaction     gate 0/24 = 0.000
   crude        gate 0/24 = 0.000
   plausible    gate 0/24 = 0.000
   oracle       gate 24/24 = 1.000

#### bot-detection
   inaction     gate 0/24 = 0.000
   crude        gate 0/24 = 0.000
   oracle-minus-one gate 0/24 = 0.000
   oracle       gate 24/24 = 1.000

#### grand-exchange
   inaction     gate 0/12 = 0.000
   oracle       gate 12/12 = 1.000

In three of the four the gate is a step function whose only step is the oracle. Six of redaction's seven rules scores zero. The oracle bot list minus a single account scores zero. There is no rung anywhere on the ladder — inaction, crude, plausible, near-oracle — that earns a fraction of it.

The three that fired are not perfection-free; they are perfection over a small object. canary-trap's gate wants every contaminated state caught and no probe firing on the clean model, over 2–12 probes. fault-localisation's wants service, fault and evidence all correct — three fields. drop-table-inference is the only gate in the repository with a margin: GATE_MARGIN = 0.05, three sub-scores each within 5% of the reference, and it fired on 22.6% of rollouts. grand-exchange has one too — TARGET_SHARE = 0.90 — and its zero has a different cause entirely (below).

So the verdict is (a) for three environments and neither for the fourth. It is not (b) anywhere: parsing works. bot-detection recorded unknown_ids 0.000 on all 32 rollouts, redaction-pressure validity 1.0 on 30 of 32, schema-migration produced a valid target schema on 25 of 32. The models are submitting well-formed work and being scored zero on it because the work is not perfect.

---

Per environment

bot-detection — (a), and the binding clause is detection == 1.0

# scan.py
@property
def gate(self) -> bool:
    return (
        self.reference_caught > 0
        and self.true_positives >= self.reference_caught
        and self.false_positives == 0
    )

Over 32 rollouts:

clause	satisfied
false_positives == 0	24 / 32
true_positives >= reference_caught	0 / 32

detection — true positives over what the reference caught — had mean 0.5135 and max 0.8333. It never reached 1.0 once. The restraint clause is doing its job and is not the problem; the catch-everything clause is a ceiling test.

The environment's own docstring explains why it has to be: each cloaked account leaks on exactly one of three behavioural channels, so "a discriminator built on any single channel ceilings at roughly a third of them". The reference reads all three. The gate therefore pays only a model that found all three tells, in a batch where the reference caught 4–6 bots and the model accused 2.9 on average.

Consequence. 0.25 of bot-detection's reward mass is a constant. The measured 0.3488 is entirely caught (0.35) and spared (0.40), and against a maximum of 0.75 it is 0.465 of what was actually on offer.

redaction-pressure — (a), and the binding clause is total recall

# scan.py
@property
def clean(self) -> bool:
    return (
        self.removed_secrets == self.secrets_total
        and self.partial_secrets == 0
        and self.collateral_hits == 0
        and self.innocent_lost == 0
        and self.secrets_total > 0
        and self.validity == 1.0
    )

Six clauses, and they are satisfied at wildly different rates:

clause	satisfied
partial_secrets == 0	28 / 32
validity == 1.0	30 / 32
collateral_hits == 0	6 / 32
innocent_lost == 0	6 / 32
residual_hits == 0 (every secret removed)	0 / 32

recall had max 0.852 across 32 rollouts and residual_hits a minimum of 4 — there was always at least one secret left standing. Two clauses never fired together either: innocent_lost averaged 140 characters.

This is the widest gate in the repository — six simultaneous conditions, three of them counting to zero over a twelve-document held-out slice — and the ladder shows six of the oracle's seven rules do not clear it.

Consequence. 0.30 of the mass — the largest gate weight of the four — is a constant. 0.4040 against a real maximum of 0.70 is 0.577 of what was on offer.

schema-migration — (a), and the binding clause is per-row fidelity

# run.py
@property
def clean(self) -> bool:
    return (self.schema_ok and self.graded > 0
            and self.matched == self.graded
            and self.rows_after == self.rows_before)

clause	satisfied
schema_ok	25 / 32
rows_after == rows_before	25 / 32
matched == graded (fidelity 1.0)	0 / 32

fidelity distribution over the 32 rollouts:

0.0 ×10, 0.10, 0.15, 0.375, 0.40, 0.575, 0.65, 0.675, 0.725, 0.725, 0.75,
0.775 ×4, 0.80 ×4, 0.85 ×3, 0.875

Max 0.875. Forty awkward held-out rows must every one recompose; the best migration anybody wrote missed five of them. The docstring is explicit that this is forge verify's exit code — a migration is correct or it is not — and as a shipping check that is right. As a reward component it is a constant at this capability level.

grand-exchange — not a gate defect. Nothing was measured

# book.py — a bar, not a knife-edge
return (self.fills.realised >= TARGET_SHARE * self.reference.realised
        and self.fills.roc      >= TARGET_SHARE * self.reference.roc)

TARGET_SHARE = 0.90. This gate already has the margin the other three lack, and it is written that way deliberately — the commit message is in the docstring ("demanding the reference's exact result made a wasted unit of capital cost 0.146 with byte-identical profit").

It scored zero because 31 of 32 rollouts placed no orders at all: 18 completion tokens, {"expected_profit": 0, "orders": []}, finish_reason=stop. planned had mean 65.6 and median 0 — one rollout in 32 planned anything at all; profit_ratio maxed at 0.323, so even that one reached 29% of the reference. Nothing in this environment was measured. See FIRST_EVAL.md §5 — spark-1 serves with default_chat_template_kwargs = {"enable_thinking": false}, and this environment is the one that cannot be answered without arithmetic.

Re-measured with thinking on, this gate fires. outputs/thinking/ge-think-n6/, 6 rollouts: one of them finished inside its token budget and scored profit 1.000, discipline 1.000 and gate 1.000, realising 40,081 gp against a reference of 36,157. The other five were cut off mid-reasoning at max_tokens. So this gate is not merely oracle-reachable in a probe — it is reachable by the model under evaluation, in a real rollout, at a 90% bar.

Do not touch this gate. It is the one that works. It is also the shape the other three should be argued against: a band at 0.90 of the reference, cleared by a run that beat the reference by its own route rather than by copying it.

---

What follows

1. Every ranking over those four environments is a ranking over their soft components. Report bot-detection against 0.75, redaction-pressure against 0.70, schema-migration against 0.75 — or say plainly that a quarter to a third of the scale is unreachable. A leaderboard that prints 0.4040 out of 1.000 for redaction-pressure is overstating the headroom by a factor it never names.

2. The gradient problem is worse than the scale problem. A constant term contributes nothing to a policy gradient. Training against these four trains against caught/spared, recall/precision, schema/integrity — and the gate only starts paying once the model is already essentially perfect, which is the point at which it stops needing the signal. This is not what a gate is for.

3. The repository already contains the fix and applies it in exactly one place. drop-table-inference gates on three sub-scores each within GATE_MARGIN = 0.05 of the reference and fires 22.6% of the time; grand-exchange gates at TARGET_SHARE = 0.90. The three dead gates are the three written as exact equality. A margin on each — every secret bar one, 39 rows of 40, the reference's bots less one — would restore a gradient without paying for inaction, because the ladder above shows inaction and crude score zero on every one of these with or without a margin.

4. Whatever is decided, probe.py should print the gate separately. Four dead components hid for a month behind a blended oracle 1.000. outputs/gate_probe.py is the throwaway version of that check; the real one belongs next to the ladder, and it is a one-line consequence of house rule 3 that a component nothing below the oracle can earn is a component that does not discriminate.

⚠️ Nothing in this file has been changed in any environment. Four packages' reward code is out of this lane; this is a diagnosis and a recommendation, not a patch.