kartiandClaude Opus 5
c28d864766
An environment is an eval you take the gradient of, so it carries obligations a benchmark does not: a held-out slice, a counterweighted reward, and a demonstrated 0.0 floor and 1.0 ceiling. README states the three as house rules. redaction-pressure is Forge's redact stage as a task. The first reward design scored survivors — secrets caught, decoys kept, prose kept — and an empty ruleset tied the best real attempt at 0.500, because destroying nothing keeps everything. Counting removals instead puts inaction at 0.000, shredding at 0.380, and an oracle at 1.000. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Arena
RL environments from Lumbridge. Apache-2.0, built to the verifiers
v1 spec so they install and run on the Prime Intellect Environments Hub unchanged.
An environment here is an eval you take the gradient of. That is not a rename — it changes what the artifact has to carry. A benchmark's score is its output, so a benchmark can be a number in a table. An environment's score is the input to the next weight update, so it has to be reproducible, gameable-proof, and traceable back to the code that produced it, or the model that trained on it inherits whatever was wrong with it silently.
Three rules follow, and every environment in this repository is held to them:
- Graded on what the model did not see. Rules, plans and policies fitted to visible examples pass on visible examples. If a held-out slice cannot be generated, the environment does not measure generalisation and does not ship.
- No single-sided reward. Every reward has a counterweight in the same episode. A reward you can max by doing the crude thing is a reward that teaches the crude thing.
- A reachable ceiling and a zero floor, demonstrated. Each environment ships the check that shows an oracle scores 1.0 and inaction scores 0.0. An unreachable component is dead weight in the gradient; a floor above zero pays for doing nothing.
Environments
redaction-pressure |
Write redaction rules for a ticket corpus. Graded on the next records the generator would have produced. Every secret class is shape-matched to a decoy, so a rule keyed on shape earns on one and pays on the other. Reward is a scanner — no judge model, no runtime, milliseconds per rollout. |
Running one
uv run eval @ configs/redaction_pressure.toml --model <model-id>
Why these, and not benchmark reimplementations
The Hub is filling with reimplementations, which are worth doing and are not what we have.
What Lumbridge has is a set of tools that already do verifiable work — Bench's
contamination gate, Forge's signed build chain — and those gates make good rewards
precisely because they were built to be un-arguable about a machine. redaction-pressure
is Forge's redact stage with the exit code turned into a gradient.