kartiandClaude Opus 5
5a99eb86a5
canary-trap, fault-localisation and schema-migration, in the three domains the Environments Hub has nothing in: contamination, operations, data engineering. None is a port. Each is graded on a slice the model never read. canary-trap runs a probe set against a model that memorised the corpus AND one that learned the same facts from a reworded copy with every identifier regenerated. The obvious probe -- plant a GUID, ask for it back -- is sound and caps at 0.525 because it cannot see the paraphrase. It scored higher than that at first: facts with no distinct rewording had the paraphrase restating the corpus verbatim, so GUID probes caught it by accident. Every fact now carries a "core" token present in both wordings and in no public one, asserted at import, which is also what makes the ceiling reachable. fault-localisation puts the fault upstream and the noise downstream, the way incidents actually present. The broken service logs one line and goes quiet while its dependents log a dozen timeouts, so ranking by error volume answers the victim -- 0/64 incidents have the loudest service as the root, and probe.py asserts it rather than trusting the generator to stay that way. schema-migration executes the model's SQL against forty held-out rows carrying thousands separators, negatives, a unit with a slash in it and a value with no unit. Row preservation started as a reward beside fidelity and paid 0.15 for adding two empty columns and touching nothing; it is a multiplier now, so losing rows still zeroes a perfect migration and keeping them earns nothing on its own. probe.py is house rule 3 as an exit code: 0.000 for inaction and 1.000 for an oracle, or it returns 1. It loads the leaf modules through a namespace shim rather than the packages, so it runs without verifiers installed -- a self-check that needs the training stack is a self-check nobody runs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
68 lines
3.9 KiB
Markdown
68 lines
3.9 KiB
Markdown
# Arena
|
|
|
|
RL environments from Lumbridge. Apache-2.0, built to the [`verifiers`](https://github.com/PrimeIntellect-ai/verifiers)
|
|
v1 spec so the same wheel installs from the Prime Intellect Environments Hub and runs here
|
|
unchanged.
|
|
|
|
An environment is an eval you take the gradient of. That is not a rename — it changes what
|
|
the artifact has to carry. A benchmark's score is its output, so a benchmark can be a number
|
|
in a table. An environment's score is the input to the next weight update, so a defect in one
|
|
is inherited by the model rather than printed beside it.
|
|
|
|
Most of what is on the Hub today is a benchmark that was ported into the spec. A port cannot
|
|
generate a held-out slice, because its dataset is fixed — so it is contaminated the moment
|
|
anyone trains on it. And an exact-match reward is single-sided by construction, so it teaches
|
|
whatever maxes the string comparison. Nothing here is a port.
|
|
|
|
## The house rules
|
|
|
|
Every environment in this repository is held to three, and ships the probe that proves it.
|
|
|
|
1. **Graded on what the model did not see.** If a held-out slice cannot be generated, the
|
|
environment does not measure generalisation and does not ship.
|
|
2. **No single-sided reward.** Every reward has a counterweight in the same episode. A reward
|
|
you can max by doing the crude thing is a reward that teaches the crude thing.
|
|
3. **A zero floor and a reachable ceiling, demonstrated.** Inaction scores 0.0 and an oracle
|
|
scores 1.0, measured, in `probe.py`. An unreachable component is dead weight in the
|
|
gradient; a floor above zero pays for doing nothing.
|
|
|
|
Rule 3 is not decoration. Every environment here was wrong the first time and the probe is
|
|
what caught it — see the table below, and the git history.
|
|
|
|
## The environments
|
|
|
|
| | What it measures | The trap |
|
|
|---|---|---|
|
|
| [`redaction-pressure`](environments/redaction_pressure) | Write redaction rules for a ticket corpus, run against the next records the generator would have produced. | Every secret class is shape-matched to a decoy: a live key beside a published test key, a card number beside a build id. A rule keyed on shape earns on one and pays on the other. |
|
|
| [`canary-trap`](environments/canary_trap) | Write contamination probes that catch a model trained on your eval set. | Probes are run against a model that memorised the corpus **and** one that learned the same facts from a reworded copy. GUID-recall probes are sound and see only half of what is there. |
|
|
| [`fault-localisation`](environments/fault_localisation) | Read an incident log and name the root cause, its class, and the line proving it. | The service that broke emits one line and goes quiet; its dependents emit a dozen timeouts. Ranking by error volume answers the victim, every time. |
|
|
| [`schema-migration`](environments/schema_migration) | Split a text column into a number and a unit, executed against held-out rows. | The visible rows are tidy. The graded ones carry thousands separators, negatives, a unit containing a slash, and a value with no unit at all. |
|
|
|
|
## What the probes measure
|
|
|
|
Reward for the degenerate strategies and for an oracle, from `uv run python probe.py`:
|
|
|
|
| environment | inaction | crude maximiser | plausible attempt | oracle |
|
|
|---|---|---|---|---|
|
|
| `redaction-pressure` | 0.000 | 0.380 *(redact everything)* | 0.552 | **1.000** |
|
|
| `canary-trap` | 0.000 | 0.000 *(public-knowledge probes)* | 0.525 *(GUID recall)* | **1.000** |
|
|
| `fault-localisation` | 0.000 | 0.250 *(blame the loudest)* | 0.500 | **1.000** |
|
|
| `schema-migration` | 0.000 | 0.000 *(add columns, touch nothing)* | 0.630 | **1.000** |
|
|
|
|
## Running one
|
|
|
|
```bash
|
|
uv run eval @ configs/redaction_pressure.toml --model <model-id>
|
|
```
|
|
|
|
## Publishing
|
|
|
|
The layout mirrors `verifiers`' own, so an environment goes to the Hub without a fork:
|
|
|
|
```bash
|
|
prime env push redaction-pressure -v PUBLIC
|
|
```
|
|
|
|
No model has been evaluated in any of these yet. That is a different sentence from having a
|
|
leaderboard, and this file will say so until it is not true.
|