arena/docs/GATE_DIAGNOSIS.md

# Why `gate` scored zero in four environments

Written 2026-08-21, against `outputs/run-20260821-1401` — seven environments, 32 rollouts
each, `brain-qwen38-dspark`.

In four of the seven, the `gate` reward was **0.000 mean and 0.000 max** across all 32
rollouts: `bot-detection` (weight 0.25), `grand-exchange` (0.25), `redaction-pressure`
(0.30) and `schema-migration` (0.25). It moved in the other three — `canary-trap` 0.2188,
`drop-table-inference` 0.2258, `fault-localisation` 0.9062.

`probe.py` prints 1.000 for every one of those four, and 1.000 requires `gate = 1`, so
nothing here is unreachable and house rule 3 is not violated. The question this file
answers is the one after that: **is the gate (a) a threshold set beyond what a real model
reaches, (b) a parse or format precondition the model never satisfies, or (c) genuinely
hard?** They have different consequences and they need different fixes.

---

## The answer, in one table

`outputs/gate_probe.py` runs each environment's shipped scorer over the probe ladder and
prints the gate **on its own**, as a pass rate over 24 tasks (12 for grand-exchange).
`probe.py` blends the gate into one number per rung, which is exactly why this was
invisible for a month.

```
#### redaction-pressure
   inaction     gate 0/24 = 0.000
   crude        gate 0/24 = 0.000
   plausible    gate 0/24 = 0.000
   six-of-seven gate 0/24 = 0.000
   oracle       gate 24/24 = 1.000

#### schema-migration
   inaction     gate 0/24 = 0.000
   crude        gate 0/24 = 0.000
   plausible    gate 0/24 = 0.000
   oracle       gate 24/24 = 1.000

#### bot-detection
   inaction     gate 0/24 = 0.000
   crude        gate 0/24 = 0.000
   oracle-minus-one gate 0/24 = 0.000
   oracle       gate 24/24 = 1.000

#### grand-exchange
   inaction     gate 0/12 = 0.000
   oracle       gate 12/12 = 1.000
```

**In three of the four the gate is a step function whose only step is the oracle.** Six of
redaction's seven rules scores zero. The oracle bot list minus a single account scores
zero. There is no rung anywhere on the ladder — inaction, crude, plausible, near-oracle —
that earns a fraction of it.

The three that fired are not perfection-free; they are perfection over a *small* object.
`canary-trap`'s gate wants every contaminated state caught and no probe firing on the clean
model, over 2–12 probes. `fault-localisation`'s wants service, fault and evidence all
correct — three fields. `drop-table-inference` is the only gate in the repository with a
**margin**: `GATE_MARGIN = 0.05`, three sub-scores each within 5% of the reference, and it
fired on 22.6% of rollouts. `grand-exchange` has one too — `TARGET_SHARE = 0.90` — and its
zero has a different cause entirely (below).

So the verdict is **(a) for three environments and neither for the fourth**. It is not (b)
anywhere: parsing works. `bot-detection` recorded `unknown_ids` 0.000 on all 32 rollouts,
`redaction-pressure` `validity` 1.0 on 30 of 32, `schema-migration` produced a valid target
schema on 25 of 32. The models are submitting well-formed work and being scored zero on it
because the work is not perfect.

---

## Per environment

### `bot-detection` — (a), and the binding clause is `detection == 1.0`

```python
# scan.py
@property
def gate(self) -> bool:
    return (
        self.reference_caught > 0
        and self.true_positives >= self.reference_caught
        and self.false_positives == 0
    )
```

Over 32 rollouts:

| clause | satisfied |
|---|---|
| `false_positives == 0` | **24 / 32** |
| `true_positives >= reference_caught` | **0 / 32** |

`detection` — true positives over what the reference caught — had mean 0.5135 and **max
0.8333**. It never reached 1.0 once. The restraint clause is doing its job and is not the
problem; the catch-everything clause is a ceiling test.

The environment's own docstring explains why it has to be: each cloaked account leaks on
exactly one of three behavioural channels, so "a discriminator built on any single channel
ceilings at roughly a third of them". The reference reads all three. The gate therefore
pays only a model that found all three tells, in a batch where the reference caught 4–6
bots and the model accused 2.9 on average.

**Consequence.** 0.25 of bot-detection's reward mass is a constant. The measured 0.3488 is
entirely `caught` (0.35) and `spared` (0.40), and against a maximum of 0.75 it is 0.465 of
what was actually on offer.

### `redaction-pressure` — (a), and the binding clause is total recall

```python
# scan.py
@property
def clean(self) -> bool:
    return (
        self.removed_secrets == self.secrets_total
        and self.partial_secrets == 0
        and self.collateral_hits == 0
        and self.innocent_lost == 0
        and self.secrets_total > 0
        and self.validity == 1.0
    )
```

Six clauses, and they are satisfied at wildly different rates:

| clause | satisfied |
|---|---|
| `partial_secrets == 0` | 28 / 32 |
| `validity == 1.0` | 30 / 32 |
| `collateral_hits == 0` | 6 / 32 |
| `innocent_lost == 0` | 6 / 32 |
| `residual_hits == 0` (every secret removed) | **0 / 32** |

`recall` had max **0.852** across 32 rollouts and `residual_hits` a **minimum of 4** — there
was always at least one secret left standing. Two clauses never fired together either:
`innocent_lost` averaged 140 characters.

This is the widest gate in the repository — six simultaneous conditions, three of them
counting to zero over a twelve-document held-out slice — and the ladder shows six of the
oracle's seven rules do not clear it.

**Consequence.** 0.30 of the mass — the largest gate weight of the four — is a constant.
0.4040 against a real maximum of 0.70 is 0.577 of what was on offer.

### `schema-migration` — (a), and the binding clause is per-row fidelity

```python
# run.py
@property
def clean(self) -> bool:
    return (self.schema_ok and self.graded > 0
            and self.matched == self.graded
            and self.rows_after == self.rows_before)
```

| clause | satisfied |
|---|---|
| `schema_ok` | 25 / 32 |
| `rows_after == rows_before` | 25 / 32 |
| `matched == graded` (fidelity 1.0) | **0 / 32** |

`fidelity` distribution over the 32 rollouts:

```
0.0 ×10, 0.10, 0.15, 0.375, 0.40, 0.575, 0.65, 0.675, 0.725, 0.725, 0.75,
0.775 ×4, 0.80 ×4, 0.85 ×3, 0.875
```

Max 0.875. Forty awkward held-out rows must every one recompose; the best migration
anybody wrote missed five of them. The docstring is explicit that this is `forge verify`'s
exit code — a migration is correct or it is not — and as a shipping check that is right.
As a *reward component* it is a constant at this capability level.

### `grand-exchange` — not a gate defect. Nothing was measured

```python
# book.py — a bar, not a knife-edge
return (self.fills.realised >= TARGET_SHARE * self.reference.realised
        and self.fills.roc      >= TARGET_SHARE * self.reference.roc)
```

`TARGET_SHARE = 0.90`. This gate already has the margin the other three lack, and it is
written that way deliberately — the commit message is in the docstring ("demanding the
reference's exact result made a wasted unit of capital cost 0.146 with byte-identical
profit").

It scored zero because **31 of 32 rollouts placed no orders at all**: 18 completion tokens,
`{"expected_profit": 0, "orders": []}`, `finish_reason=stop`. `planned` had mean 65.6 and
median 0 — one rollout in 32 planned anything at all; `profit_ratio` maxed at 0.323, so even
that one reached 29% of the reference. Nothing in this environment was measured. See
`FIRST_EVAL.md` §5 — spark-1 serves with
`default_chat_template_kwargs = {"enable_thinking": false}`, and this environment is the one
that cannot be answered without arithmetic.

**Re-measured with thinking on, this gate fires.** `outputs/thinking/ge-think-n6/`, 6
rollouts: one of them finished inside its token budget and scored `profit` 1.000,
`discipline` 1.000 and **`gate` 1.000**, realising 40,081 gp against a reference of 36,157.
The other five were cut off mid-reasoning at `max_tokens`. So this gate is not merely
oracle-reachable in a probe — it is reachable by the model under evaluation, in a real
rollout, at a 90% bar.

**Do not touch this gate.** It is the one that works. It is also the shape the other three
should be argued against: a band at 0.90 of the reference, cleared by a run that beat the
reference by its own route rather than by copying it.

---

## What follows

1. **Every ranking over those four environments is a ranking over their soft components.**
   Report `bot-detection` against 0.75, `redaction-pressure` against 0.70,
   `schema-migration` against 0.75 — or say plainly that a quarter to a third of the scale
   is unreachable. A leaderboard that prints 0.4040 out of 1.000 for redaction-pressure is
   overstating the headroom by a factor it never names.

2. **The gradient problem is worse than the scale problem.** A constant term contributes
   nothing to a policy gradient. Training against these four trains against
   `caught`/`spared`, `recall`/`precision`, `schema`/`integrity` — and the gate only starts
   paying once the model is already essentially perfect, which is the point at which it
   stops needing the signal. This is not what a gate is for.

3. **The repository already contains the fix and applies it in exactly one place.**
   `drop-table-inference` gates on three sub-scores each within `GATE_MARGIN = 0.05` of the
   reference and fires 22.6% of the time; `grand-exchange` gates at `TARGET_SHARE = 0.90`.
   The three dead gates are the three written as exact equality. A margin on each — every
   secret bar one, 39 rows of 40, the reference's bots less one — would restore a gradient
   without paying for inaction, because the ladder above shows `inaction` and `crude` score
   zero on every one of these with or without a margin.

4. **Whatever is decided, `probe.py` should print the gate separately.** Four dead
   components hid for a month behind a blended `oracle 1.000`. `outputs/gate_probe.py` is
   the throwaway version of that check; the real one belongs next to the ladder, and it is
   a one-line consequence of house rule 3 that a component nothing below the oracle can
   earn is a component that does not discriminate.

⚠️ **Nothing in this file has been changed in any environment.** Four packages' reward code
is out of this lane; this is a diagnosis and a recommendation, not a patch.