# Why `gate` scored zero in four environments Written 2026-08-21, against `outputs/run-20260821-1401` — seven environments, 32 rollouts each, `brain-qwen38-dspark`. In four of the seven, the `gate` reward was **0.000 mean and 0.000 max** across all 32 rollouts: `bot-detection` (weight 0.25), `grand-exchange` (0.25), `redaction-pressure` (0.30) and `schema-migration` (0.25). It moved in the other three — `canary-trap` 0.2188, `drop-table-inference` 0.2258, `fault-localisation` 0.9062. `probe.py` prints 1.000 for every one of those four, and 1.000 requires `gate = 1`, so nothing here is unreachable and house rule 3 is not violated. The question this file answers is the one after that: **is the gate (a) a threshold set beyond what a real model reaches, (b) a parse or format precondition the model never satisfies, or (c) genuinely hard?** They have different consequences and they need different fixes. --- ## The answer, in one table `outputs/gate_probe.py` runs each environment's shipped scorer over the probe ladder and prints the gate **on its own**, as a pass rate over 24 tasks (12 for grand-exchange). `probe.py` blends the gate into one number per rung, which is exactly why this was invisible for a month. ``` #### redaction-pressure inaction gate 0/24 = 0.000 crude gate 0/24 = 0.000 plausible gate 0/24 = 0.000 six-of-seven gate 0/24 = 0.000 oracle gate 24/24 = 1.000 #### schema-migration inaction gate 0/24 = 0.000 crude gate 0/24 = 0.000 plausible gate 0/24 = 0.000 oracle gate 24/24 = 1.000 #### bot-detection inaction gate 0/24 = 0.000 crude gate 0/24 = 0.000 oracle-minus-one gate 0/24 = 0.000 oracle gate 24/24 = 1.000 #### grand-exchange inaction gate 0/12 = 0.000 oracle gate 12/12 = 1.000 ``` **In three of the four the gate is a step function whose only step is the oracle.** Six of redaction's seven rules scores zero. The oracle bot list minus a single account scores zero. There is no rung anywhere on the ladder — inaction, crude, plausible, near-oracle — that earns a fraction of it. The three that fired are not perfection-free; they are perfection over a *small* object. `canary-trap`'s gate wants every contaminated state caught and no probe firing on the clean model, over 2–12 probes. `fault-localisation`'s wants service, fault and evidence all correct — three fields. `drop-table-inference` is the only gate in the repository with a **margin**: `GATE_MARGIN = 0.05`, three sub-scores each within 5% of the reference, and it fired on 22.6% of rollouts. `grand-exchange` has one too — `TARGET_SHARE = 0.90` — and its zero has a different cause entirely (below). So the verdict is **(a) for three environments and neither for the fourth**. It is not (b) anywhere: parsing works. `bot-detection` recorded `unknown_ids` 0.000 on all 32 rollouts, `redaction-pressure` `validity` 1.0 on 30 of 32, `schema-migration` produced a valid target schema on 25 of 32. The models are submitting well-formed work and being scored zero on it because the work is not perfect. --- ## Per environment ### `bot-detection` — (a), and the binding clause is `detection == 1.0` ```python # scan.py @property def gate(self) -> bool: return ( self.reference_caught > 0 and self.true_positives >= self.reference_caught and self.false_positives == 0 ) ``` Over 32 rollouts: | clause | satisfied | |---|---| | `false_positives == 0` | **24 / 32** | | `true_positives >= reference_caught` | **0 / 32** | `detection` — true positives over what the reference caught — had mean 0.5135 and **max 0.8333**. It never reached 1.0 once. The restraint clause is doing its job and is not the problem; the catch-everything clause is a ceiling test. The environment's own docstring explains why it has to be: each cloaked account leaks on exactly one of three behavioural channels, so "a discriminator built on any single channel ceilings at roughly a third of them". The reference reads all three. The gate therefore pays only a model that found all three tells, in a batch where the reference caught 4–6 bots and the model accused 2.9 on average. **Consequence.** 0.25 of bot-detection's reward mass is a constant. The measured 0.3488 is entirely `caught` (0.35) and `spared` (0.40), and against a maximum of 0.75 it is 0.465 of what was actually on offer. ### `redaction-pressure` — (a), and the binding clause is total recall ```python # scan.py @property def clean(self) -> bool: return ( self.removed_secrets == self.secrets_total and self.partial_secrets == 0 and self.collateral_hits == 0 and self.innocent_lost == 0 and self.secrets_total > 0 and self.validity == 1.0 ) ``` Six clauses, and they are satisfied at wildly different rates: | clause | satisfied | |---|---| | `partial_secrets == 0` | 28 / 32 | | `validity == 1.0` | 30 / 32 | | `collateral_hits == 0` | 6 / 32 | | `innocent_lost == 0` | 6 / 32 | | `residual_hits == 0` (every secret removed) | **0 / 32** | `recall` had max **0.852** across 32 rollouts and `residual_hits` a **minimum of 4** — there was always at least one secret left standing. Two clauses never fired together either: `innocent_lost` averaged 140 characters. This is the widest gate in the repository — six simultaneous conditions, three of them counting to zero over a twelve-document held-out slice — and the ladder shows six of the oracle's seven rules do not clear it. **Consequence.** 0.30 of the mass — the largest gate weight of the four — is a constant. 0.4040 against a real maximum of 0.70 is 0.577 of what was on offer. ### `schema-migration` — (a), and the binding clause is per-row fidelity ```python # run.py @property def clean(self) -> bool: return (self.schema_ok and self.graded > 0 and self.matched == self.graded and self.rows_after == self.rows_before) ``` | clause | satisfied | |---|---| | `schema_ok` | 25 / 32 | | `rows_after == rows_before` | 25 / 32 | | `matched == graded` (fidelity 1.0) | **0 / 32** | `fidelity` distribution over the 32 rollouts: ``` 0.0 ×10, 0.10, 0.15, 0.375, 0.40, 0.575, 0.65, 0.675, 0.725, 0.725, 0.75, 0.775 ×4, 0.80 ×4, 0.85 ×3, 0.875 ``` Max 0.875. Forty awkward held-out rows must every one recompose; the best migration anybody wrote missed five of them. The docstring is explicit that this is `forge verify`'s exit code — a migration is correct or it is not — and as a shipping check that is right. As a *reward component* it is a constant at this capability level. ### `grand-exchange` — not a gate defect. Nothing was measured ```python # book.py — a bar, not a knife-edge return (self.fills.realised >= TARGET_SHARE * self.reference.realised and self.fills.roc >= TARGET_SHARE * self.reference.roc) ``` `TARGET_SHARE = 0.90`. This gate already has the margin the other three lack, and it is written that way deliberately — the commit message is in the docstring ("demanding the reference's exact result made a wasted unit of capital cost 0.146 with byte-identical profit"). It scored zero because **31 of 32 rollouts placed no orders at all**: 18 completion tokens, `{"expected_profit": 0, "orders": []}`, `finish_reason=stop`. `planned` had mean 65.6 and median 0 — one rollout in 32 planned anything at all; `profit_ratio` maxed at 0.323, so even that one reached 29% of the reference. Nothing in this environment was measured. See `FIRST_EVAL.md` §5 — spark-1 serves with `default_chat_template_kwargs = {"enable_thinking": false}`, and this environment is the one that cannot be answered without arithmetic. **Re-measured with thinking on, this gate fires.** `outputs/thinking/ge-think-n6/`, 6 rollouts: one of them finished inside its token budget and scored `profit` 1.000, `discipline` 1.000 and **`gate` 1.000**, realising 40,081 gp against a reference of 36,157. The other five were cut off mid-reasoning at `max_tokens`. So this gate is not merely oracle-reachable in a probe — it is reachable by the model under evaluation, in a real rollout, at a 90% bar. **Do not touch this gate.** It is the one that works. It is also the shape the other three should be argued against: a band at 0.90 of the reference, cleared by a run that beat the reference by its own route rather than by copying it. --- ## What follows 1. **Every ranking over those four environments is a ranking over their soft components.** Report `bot-detection` against 0.75, `redaction-pressure` against 0.70, `schema-migration` against 0.75 — or say plainly that a quarter to a third of the scale is unreachable. A leaderboard that prints 0.4040 out of 1.000 for redaction-pressure is overstating the headroom by a factor it never names. 2. **The gradient problem is worse than the scale problem.** A constant term contributes nothing to a policy gradient. Training against these four trains against `caught`/`spared`, `recall`/`precision`, `schema`/`integrity` — and the gate only starts paying once the model is already essentially perfect, which is the point at which it stops needing the signal. This is not what a gate is for. 3. **The repository already contains the fix and applies it in exactly one place.** `drop-table-inference` gates on three sub-scores each within `GATE_MARGIN = 0.05` of the reference and fires 22.6% of the time; `grand-exchange` gates at `TARGET_SHARE = 0.90`. The three dead gates are the three written as exact equality. A margin on each — every secret bar one, 39 rows of 40, the reference's bots less one — would restore a gradient without paying for inaction, because the ladder above shows `inaction` and `crude` score zero on every one of these with or without a margin. 4. **Whatever is decided, `probe.py` should print the gate separately.** Four dead components hid for a month behind a blended `oracle 1.000`. `outputs/gate_probe.py` is the throwaway version of that check; the real one belongs next to the ladder, and it is a one-line consequence of house rule 3 that a component nothing below the oracle can earn is a component that does not discriminate. ⚠️ **Nothing in this file has been changed in any environment.** Four packages' reward code is out of this lane; this is a diagnosis and a recommendation, not a patch.