arena/README.md

# Arena

RL environments from Lumbridge. Apache-2.0, built to the [`verifiers`](https://github.com/PrimeIntellect-ai/verifiers)
v1 spec so the same wheel installs from the Prime Intellect Environments Hub and runs here
unchanged.

An environment is an eval you take the gradient of. That is not a rename — it changes what
the artifact has to carry. A benchmark's score is its output, so a benchmark can be a number
in a table. An environment's score is the input to the next weight update, so a defect in one
is inherited by the model rather than printed beside it.

Most of what is on the Hub today is a benchmark that was ported into the spec. A port cannot
generate a held-out slice, because its dataset is fixed — so it is contaminated the moment
anyone trains on it. And an exact-match reward is single-sided by construction, so it teaches
whatever maxes the string comparison. Nothing here is a port.

## The house rules

Every environment in this repository is held to three, and ships the probe that proves it.

1. **Graded on what the model did not see.** If a held-out slice cannot be generated, the
   environment does not measure generalisation and does not ship.
2. **No single-sided reward.** Every reward has a counterweight in the same episode. A reward
   you can max by doing the crude thing is a reward that teaches the crude thing.
3. **A zero floor and a reachable ceiling, demonstrated.** Inaction scores 0.0 and an oracle
   scores 1.0, measured, in `probe.py`. An unreachable component is dead weight in the
   gradient; a floor above zero pays for doing nothing.

Rule 3 is not decoration. Every environment here was wrong the first time and the probe is
what caught it — see the table below, and the git history.

## The environments

| | What it measures | The trap |
|---|---|---|
| [`redaction-pressure`](environments/redaction_pressure) | Write provenance-safe redaction rules for a ticket corpus, run against the next records the generator would have produced. | Partial edits remain residual secrets, every non-secret source character is protected, and hostile regexes share one episode budget. |
| [`canary-trap`](environments/canary_trap) | Write contamination probes that catch a model trained on your eval set. | Probes are run against a model that memorised the corpus **and** one that learned the same facts from a reworded copy. GUID-recall probes are sound and see only half of what is there. |
| [`fault-localisation`](environments/fault_localisation) | Read an incident log and name the root cause, its class, and the line proving it. | The service that broke emits one line and goes quiet; its dependents emit a dozen timeouts. Ranking by error volume answers the victim, every time. |
| [`schema-migration`](environments/schema_migration) | Split a text column into a number and a unit, executed against held-out rows. | The visible rows are tidy. The graded ones carry thousands separators, negatives, a unit containing a slash, and a value with no unit at all. |

## What the probes measure

Reward for the degenerate strategies and for an oracle, from `uv run python probe.py`:

| environment | inaction | crude maximiser | plausible attempt | oracle |
|---|---|---|---|---|
| `redaction-pressure` | 0.000 | 0.387 *(redact everything)* | 0.549 | **1.000** |
| `canary-trap` | 0.000 | 0.000 *(public-knowledge probes)* | 0.525 *(GUID recall)* | **1.000** |
| `fault-localisation` | 0.000 | 0.250 *(blame the loudest)* | 0.500 | **1.000** |
| `schema-migration` | 0.000 | 0.000 *(add columns, touch nothing)* | 0.630 | **1.000** |

## Running one

```bash
uv run --project environments/redaction_pressure eval @ configs/redaction_pressure.toml --model <model-id>
```

Run its release gate from the repository root:

```bash
uv sync --project environments/redaction_pressure
uv run --project environments/redaction_pressure \
  python -m unittest discover -s environments/redaction_pressure/tests -v
uv run --with regex python probe.py
```

Arena contains independently published environment libraries, so per-environment
`uv.lock` files are intentionally not committed. Redaction v0.2 pins its runtime contract
in `pyproject.toml`; CI resolves it on Python 3.11 and 3.12, builds every environment, and
runs the shared four-environment probe.

## Publishing

The layout mirrors `verifiers`' own, so an environment goes to the Hub without a fork:

```bash
prime env push redaction-pressure -v PUBLIC
```

No model has been evaluated in any of these yet. That is a different sentence from having a
leaderboard, and this file will say so until it is not true.