4.5 KiB
Arena
RL environments from Lumbridge. Apache-2.0, built to the verifiers
v1 spec so the same wheel installs from the Prime Intellect Environments Hub and runs here
unchanged.
An environment is an eval you take the gradient of. That is not a rename — it changes what the artifact has to carry. A benchmark's score is its output, so a benchmark can be a number in a table. An environment's score is the input to the next weight update, so a defect in one is inherited by the model rather than printed beside it.
Most of what is on the Hub today is a benchmark that was ported into the spec. A port cannot generate a held-out slice, because its dataset is fixed — so it is contaminated the moment anyone trains on it. And an exact-match reward is single-sided by construction, so it teaches whatever maxes the string comparison. Nothing here is a port.
The house rules
Every environment in this repository is held to three, and ships the probe that proves it.
- Graded on what the model did not see. If a held-out slice cannot be generated, the environment does not measure generalisation and does not ship.
- No single-sided reward. Every reward has a counterweight in the same episode. A reward you can max by doing the crude thing is a reward that teaches the crude thing.
- A zero floor and a reachable ceiling, demonstrated. Inaction scores 0.0 and an oracle
scores 1.0, measured, in
probe.py. An unreachable component is dead weight in the gradient; a floor above zero pays for doing nothing.
Rule 3 is not decoration. Every environment here was wrong the first time and the probe is what caught it — see the table below, and the git history.
The environments
| What it measures | The trap | |
|---|---|---|
redaction-pressure |
Write provenance-safe redaction rules for a ticket corpus, run against the next records the generator would have produced. | Partial edits remain residual secrets, every non-secret source character is protected, and hostile regexes share one episode budget. |
canary-trap |
Write contamination probes that catch a model trained on your eval set. | Probes are run against a model that memorised the corpus and one that learned the same facts from a reworded copy. GUID-recall probes are sound and see only half of what is there. |
fault-localisation |
Read an incident log and name the root cause, its class, and the line proving it. | The service that broke emits one line and goes quiet; its dependents emit a dozen timeouts. Ranking by error volume answers the victim, every time. |
schema-migration |
Split a text column into a number and a unit, executed against held-out rows. | The visible rows are tidy. The graded ones carry thousands separators, negatives, a unit containing a slash, and a value with no unit at all. |
What the probes measure
Reward for the degenerate strategies and for an oracle, from uv run python probe.py:
| environment | inaction | crude maximiser | plausible attempt | oracle |
|---|---|---|---|---|
redaction-pressure |
0.000 | 0.387 (redact everything) | 0.549 | 1.000 |
canary-trap |
0.000 | 0.000 (public-knowledge probes) | 0.525 (GUID recall) | 1.000 |
fault-localisation |
0.000 | 0.250 (blame the loudest) | 0.500 | 1.000 |
schema-migration |
0.000 | 0.000 (add columns, touch nothing) | 0.630 | 1.000 |
Running one
uv run --project environments/redaction_pressure eval @ configs/redaction_pressure.toml --model <model-id>
Run its release gate from the repository root:
uv sync --project environments/redaction_pressure
uv run --project environments/redaction_pressure \
python -m unittest discover -s environments/redaction_pressure/tests -v
uv run --with regex python probe.py
Arena contains independently published environment libraries, so per-environment
uv.lock files are intentionally not committed. Redaction v0.2 pins its runtime contract
in pyproject.toml; CI resolves it on Python 3.11 and 3.12, builds every environment, and
runs the shared four-environment probe.
Publishing
The layout mirrors verifiers' own, so an environment goes to the Hub without a fork:
prime env push redaction-pressure -v PUBLIC
No model has been evaluated in any of these yet. That is a different sentence from having a leaderboard, and this file will say so until it is not true.