lumbridge-public/arena
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Harden redaction-pressure reward and provenance
arena-environments / validate (3.11) (push) Successful in 1m8s
Details
arena-environments / validate (3.12) (push) Successful in 31s
Details

Browse Source
This commit is contained in:
karti
2026-08-19 01:26:21 -07:00
parent 5a99eb86a5
commit d691acf131
16 changed files with 897 additions and 163 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+17 -3
View File
@@ -33,7 +33,7 @@ what caught it — see the table below, and the git history.
| | What it measures | The trap |
|---|---|---|
| [`redaction-pressure`](environments/redaction_pressure) | Write redaction rules for a ticket corpus, run against the next records the generator would have produced. | Every secret class is shape-matched to a decoy: a live key beside a published test key, a card number beside a build id. A rule keyed on shape earns on one and pays on the other. |
| [`redaction-pressure`](environments/redaction_pressure) | Write provenance-safe redaction rules for a ticket corpus, run against the next records the generator would have produced. | Partial edits remain residual secrets, every non-secret source character is protected, and hostile regexes share one episode budget. |
| [`canary-trap`](environments/canary_trap) | Write contamination probes that catch a model trained on your eval set. | Probes are run against a model that memorised the corpus **and** one that learned the same facts from a reworded copy. GUID-recall probes are sound and see only half of what is there. |
| [`fault-localisation`](environments/fault_localisation) | Read an incident log and name the root cause, its class, and the line proving it. | The service that broke emits one line and goes quiet; its dependents emit a dozen timeouts. Ranking by error volume answers the victim, every time. |
| [`schema-migration`](environments/schema_migration) | Split a text column into a number and a unit, executed against held-out rows. | The visible rows are tidy. The graded ones carry thousands separators, negatives, a unit containing a slash, and a value with no unit at all. |
@@ -44,7 +44,7 @@ Reward for the degenerate strategies and for an oracle, from `uv run python prob
| environment | inaction | crude maximiser | plausible attempt | oracle |
|---|---|---|---|---|
| `redaction-pressure` | 0.000 | 0.380 *(redact everything)* | 0.552 | **1.000** |
| `redaction-pressure` | 0.000 | 0.387 *(redact everything)* | 0.549 | **1.000** |
| `canary-trap` | 0.000 | 0.000 *(public-knowledge probes)* | 0.525 *(GUID recall)* | **1.000** |
| `fault-localisation` | 0.000 | 0.250 *(blame the loudest)* | 0.500 | **1.000** |
| `schema-migration` | 0.000 | 0.000 *(add columns, touch nothing)* | 0.630 | **1.000** |
@@ -52,9 +52,23 @@ Reward for the degenerate strategies and for an oracle, from `uv run python prob
## Running one
```bash
uv run eval @ configs/redaction_pressure.toml --model <model-id>
uv run --project environments/redaction_pressure eval @ configs/redaction_pressure.toml --model <model-id>
```
Run its release gate from the repository root:
```bash
uv sync --project environments/redaction_pressure
uv run --project environments/redaction_pressure \
python -m unittest discover -s environments/redaction_pressure/tests -v
uv run --with regex python probe.py
```
Arena contains independently published environment libraries, so per-environment
`uv.lock` files are intentionally not committed. Redaction v0.2 pins its runtime contract
in `pyproject.toml`; CI resolves it on Python 3.11 and 3.12, builds every environment, and
runs the shared four-environment probe.
## Publishing
The layout mirrors `verifiers`' own, so an environment goes to the Hub without a fork: