lumbridge-public/arena
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Three more environments, and probe.py to gate all four

Browse Source 
canary-trap, fault-localisation and schema-migration, in the three domains the
Environments Hub has nothing in: contamination, operations, data engineering.
None is a port. Each is graded on a slice the model never read.

canary-trap runs a probe set against a model that memorised the corpus AND one
that learned the same facts from a reworded copy with every identifier
regenerated. The obvious probe -- plant a GUID, ask for it back -- is sound and
caps at 0.525 because it cannot see the paraphrase. It scored higher than that
at first: facts with no distinct rewording had the paraphrase restating the
corpus verbatim, so GUID probes caught it by accident. Every fact now carries a
"core" token present in both wordings and in no public one, asserted at import,
which is also what makes the ceiling reachable.

fault-localisation puts the fault upstream and the noise downstream, the way
incidents actually present. The broken service logs one line and goes quiet
while its dependents log a dozen timeouts, so ranking by error volume answers
the victim -- 0/64 incidents have the loudest service as the root, and probe.py
asserts it rather than trusting the generator to stay that way.

schema-migration executes the model's SQL against forty held-out rows carrying
thousands separators, negatives, a unit with a slash in it and a value with no
unit. Row preservation started as a reward beside fidelity and paid 0.15 for
adding two empty columns and touching nothing; it is a multiplier now, so
losing rows still zeroes a perfect migration and keeping them earns nothing on
its own.

probe.py is house rule 3 as an exit code: 0.000 for inaction and 1.000 for an
oracle, or it returns 1. It loads the leaf modules through a namespace shim
rather than the packages, so it runs without verifiers installed -- a self-check
that needs the training stack is a self-check nobody runs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
karti
2026-08-19 01:03:05 -07:00
committed by karti
co-authored by Claude Opus 5
parent c28d864766
commit 5a99eb86a5
22 changed files with 1168 additions and 3539 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+50 -24
View File
@@ -1,30 +1,53 @@
# Arena
RL environments from Lumbridge. Apache-2.0, built to the [`verifiers`](https://github.com/PrimeIntellect-ai/verifiers)
v1 spec so they install and run on the Prime Intellect Environments Hub unchanged.
v1 spec so the same wheel installs from the Prime Intellect Environments Hub and runs here
unchanged.
An environment here is an eval you take the gradient of. That is not a rename — it changes
what the artifact has to carry. A benchmark's score is its output, so a benchmark can be a
number in a table. An environment's score is the input to the next weight update, so it has
to be reproducible, gameable-proof, and traceable back to the code that produced it, or the
model that trained on it inherits whatever was wrong with it silently.
An environment is an eval you take the gradient of. That is not a rename — it changes what
the artifact has to carry. A benchmark's score is its output, so a benchmark can be a number
in a table. An environment's score is the input to the next weight update, so a defect in one
is inherited by the model rather than printed beside it.
Three rules follow, and every environment in this repository is held to them:
Most of what is on the Hub today is a benchmark that was ported into the spec. A port cannot
generate a held-out slice, because its dataset is fixed — so it is contaminated the moment
anyone trains on it. And an exact-match reward is single-sided by construction, so it teaches
whatever maxes the string comparison. Nothing here is a port.
1. **Graded on what the model did not see.** Rules, plans and policies fitted to visible
examples pass on visible examples. If a held-out slice cannot be generated, the
## The house rules
Every environment in this repository is held to three, and ships the probe that proves it.
1. **Graded on what the model did not see.** If a held-out slice cannot be generated, the
environment does not measure generalisation and does not ship.
2. **No single-sided reward.** Every reward has a counterweight in the same episode. A
reward you can max by doing the crude thing is a reward that teaches the crude thing.
3. **A reachable ceiling and a zero floor, demonstrated.** Each environment ships the check
that shows an oracle scores 1.0 and inaction scores 0.0. An unreachable component is
dead weight in the gradient; a floor above zero pays for doing nothing.
2. **No single-sided reward.** Every reward has a counterweight in the same episode. A reward
you can max by doing the crude thing is a reward that teaches the crude thing.
3. **A zero floor and a reachable ceiling, demonstrated.** Inaction scores 0.0 and an oracle
scores 1.0, measured, in `probe.py`. An unreachable component is dead weight in the
gradient; a floor above zero pays for doing nothing.
## Environments
Rule 3 is not decoration. Every environment here was wrong the first time and the probe is
what caught it — see the table below, and the git history.
| | |
|---|---|
| [`redaction-pressure`](environments/redaction_pressure) | Write redaction rules for a ticket corpus. Graded on the next records the generator would have produced. Every secret class is shape-matched to a decoy, so a rule keyed on shape earns on one and pays on the other. Reward is a scanner — no judge model, no runtime, milliseconds per rollout. |
## The environments
| | What it measures | The trap |
|---|---|---|
| [`redaction-pressure`](environments/redaction_pressure) | Write redaction rules for a ticket corpus, run against the next records the generator would have produced. | Every secret class is shape-matched to a decoy: a live key beside a published test key, a card number beside a build id. A rule keyed on shape earns on one and pays on the other. |
| [`canary-trap`](environments/canary_trap) | Write contamination probes that catch a model trained on your eval set. | Probes are run against a model that memorised the corpus **and** one that learned the same facts from a reworded copy. GUID-recall probes are sound and see only half of what is there. |
| [`fault-localisation`](environments/fault_localisation) | Read an incident log and name the root cause, its class, and the line proving it. | The service that broke emits one line and goes quiet; its dependents emit a dozen timeouts. Ranking by error volume answers the victim, every time. |
| [`schema-migration`](environments/schema_migration) | Split a text column into a number and a unit, executed against held-out rows. | The visible rows are tidy. The graded ones carry thousands separators, negatives, a unit containing a slash, and a value with no unit at all. |
## What the probes measure
Reward for the degenerate strategies and for an oracle, from `uv run python probe.py`:
| environment | inaction | crude maximiser | plausible attempt | oracle |
|---|---|---|---|---|
| `redaction-pressure` | 0.000 | 0.380 *(redact everything)* | 0.552 | **1.000** |
| `canary-trap` | 0.000 | 0.000 *(public-knowledge probes)* | 0.525 *(GUID recall)* | **1.000** |
| `fault-localisation` | 0.000 | 0.250 *(blame the loudest)* | 0.500 | **1.000** |
| `schema-migration` | 0.000 | 0.000 *(add columns, touch nothing)* | 0.630 | **1.000** |
## Running one
@@ -32,10 +55,13 @@ Three rules follow, and every environment in this repository is held to them:
uv run eval @ configs/redaction_pressure.toml --model <model-id>
```
## Why these, and not benchmark reimplementations
## Publishing
The Hub is filling with reimplementations, which are worth doing and are not what we have.
What Lumbridge has is a set of tools that already do verifiable work — Bench's
contamination gate, Forge's signed build chain — and those gates make good rewards
precisely because they were built to be un-arguable about a machine. `redaction-pressure`
is Forge's `redact` stage with the exit code turned into a gradient.
The layout mirrors `verifiers`' own, so an environment goes to the Hub without a fork:
```bash
prime env push redaction-pressure -v PUBLIC
```
No model has been evaluated in any of these yet. That is a different sentence from having a
leaderboard, and this file will say so until it is not true.