kartiandClaude Opus 5
5a99eb86a5
canary-trap, fault-localisation and schema-migration, in the three domains the Environments Hub has nothing in: contamination, operations, data engineering. None is a port. Each is graded on a slice the model never read. canary-trap runs a probe set against a model that memorised the corpus AND one that learned the same facts from a reworded copy with every identifier regenerated. The obvious probe -- plant a GUID, ask for it back -- is sound and caps at 0.525 because it cannot see the paraphrase. It scored higher than that at first: facts with no distinct rewording had the paraphrase restating the corpus verbatim, so GUID probes caught it by accident. Every fact now carries a "core" token present in both wordings and in no public one, asserted at import, which is also what makes the ceiling reachable. fault-localisation puts the fault upstream and the noise downstream, the way incidents actually present. The broken service logs one line and goes quiet while its dependents log a dozen timeouts, so ranking by error volume answers the victim -- 0/64 incidents have the loudest service as the root, and probe.py asserts it rather than trusting the generator to stay that way. schema-migration executes the model's SQL against forty held-out rows carrying thousands separators, negatives, a unit with a slash in it and a value with no unit. Row preservation started as a reward beside fidelity and paid 0.15 for adding two empty columns and touching nothing; it is a multiplier now, so losing rows still zeroes a perfect migration and keeping them earns nothing on its own. probe.py is house rule 3 as an exit code: 0.000 for inaction and 1.000 for an oracle, or it returns 1. It loads the leaf modules through a namespace shim rather than the packages, so it runs without verifiers installed -- a self-check that needs the training stack is a self-check nobody runs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3.9 KiB
Arena
RL environments from Lumbridge. Apache-2.0, built to the verifiers
v1 spec so the same wheel installs from the Prime Intellect Environments Hub and runs here
unchanged.
An environment is an eval you take the gradient of. That is not a rename — it changes what the artifact has to carry. A benchmark's score is its output, so a benchmark can be a number in a table. An environment's score is the input to the next weight update, so a defect in one is inherited by the model rather than printed beside it.
Most of what is on the Hub today is a benchmark that was ported into the spec. A port cannot generate a held-out slice, because its dataset is fixed — so it is contaminated the moment anyone trains on it. And an exact-match reward is single-sided by construction, so it teaches whatever maxes the string comparison. Nothing here is a port.
The house rules
Every environment in this repository is held to three, and ships the probe that proves it.
- Graded on what the model did not see. If a held-out slice cannot be generated, the environment does not measure generalisation and does not ship.
- No single-sided reward. Every reward has a counterweight in the same episode. A reward you can max by doing the crude thing is a reward that teaches the crude thing.
- A zero floor and a reachable ceiling, demonstrated. Inaction scores 0.0 and an oracle
scores 1.0, measured, in
probe.py. An unreachable component is dead weight in the gradient; a floor above zero pays for doing nothing.
Rule 3 is not decoration. Every environment here was wrong the first time and the probe is what caught it — see the table below, and the git history.
The environments
| What it measures | The trap | |
|---|---|---|
redaction-pressure |
Write redaction rules for a ticket corpus, run against the next records the generator would have produced. | Every secret class is shape-matched to a decoy: a live key beside a published test key, a card number beside a build id. A rule keyed on shape earns on one and pays on the other. |
canary-trap |
Write contamination probes that catch a model trained on your eval set. | Probes are run against a model that memorised the corpus and one that learned the same facts from a reworded copy. GUID-recall probes are sound and see only half of what is there. |
fault-localisation |
Read an incident log and name the root cause, its class, and the line proving it. | The service that broke emits one line and goes quiet; its dependents emit a dozen timeouts. Ranking by error volume answers the victim, every time. |
schema-migration |
Split a text column into a number and a unit, executed against held-out rows. | The visible rows are tidy. The graded ones carry thousands separators, negatives, a unit containing a slash, and a value with no unit at all. |
What the probes measure
Reward for the degenerate strategies and for an oracle, from uv run python probe.py:
| environment | inaction | crude maximiser | plausible attempt | oracle |
|---|---|---|---|---|
redaction-pressure |
0.000 | 0.380 (redact everything) | 0.552 | 1.000 |
canary-trap |
0.000 | 0.000 (public-knowledge probes) | 0.525 (GUID recall) | 1.000 |
fault-localisation |
0.000 | 0.250 (blame the loudest) | 0.500 | 1.000 |
schema-migration |
0.000 | 0.000 (add columns, touch nothing) | 0.630 | 1.000 |
Running one
uv run eval @ configs/redaction_pressure.toml --model <model-id>
Publishing
The layout mirrors verifiers' own, so an environment goes to the Hub without a fork:
prime env push redaction-pressure -v PUBLIC
No model has been evaluated in any of these yet. That is a different sentence from having a leaderboard, and this file will say so until it is not true.