Harden redaction-pressure reward and provenance
This commit is contained in:
@@ -0,0 +1,52 @@
|
||||
# redaction-pressure v0.2
|
||||
|
||||
A one-turn Verifiers v1 environment. The model returns regex rules; the scanner runs
|
||||
them against later records from the same deterministic corpus stream.
|
||||
|
||||
## Reward contract
|
||||
|
||||
- **Recall (0.35):** a labelled secret counts only when every original character in
|
||||
its span is gone. Partial edits earn zero span recall.
|
||||
- **Precision (0.35):** source characters belonging to fully removed secret spans,
|
||||
divided by all removed source characters. Partial damage stays in the denominator.
|
||||
- **Clean gate (0.30):** every secret fully removed, every non-secret character and
|
||||
decoy preserved, and no validation or timeout errors.
|
||||
- **Validity:** malformed JSON, bad entries, more than 24 rules, unsafe replacements,
|
||||
invalid regexes, excessive matches, timeouts, or an exhausted episode budget multiply
|
||||
every reward by zero.
|
||||
|
||||
Replacement is intentionally limited to `"[REDACTED]"` or `""`. That prevents a rule
|
||||
from matching a complete secret and then leaking it through a capture expansion.
|
||||
|
||||
## Splits
|
||||
|
||||
`train` and `dev` have separate, deterministic public seed ranges. `private_eval` derives
|
||||
seeds with HMAC from `REDACTION_PRESSURE_PRIVATE_SEED`, which must contain at least 32
|
||||
characters. The secret is never task data; traces carry its SHA-256 commitment and reveal
|
||||
the derived task seed only in the completed trace so the episode can be reproduced by an
|
||||
authorized evaluator.
|
||||
|
||||
Never present public `dev` scores as private evaluation.
|
||||
|
||||
## Trace identity
|
||||
|
||||
Each task records the environment, corpus, and reward-schema versions, split commitment,
|
||||
and a SHA-256 digest of the behavior-bearing installed Python sources. Scoring repeats
|
||||
the identity under `trace.info.environment` and records validation failures under
|
||||
`trace.info.redaction_errors`.
|
||||
|
||||
## Checks
|
||||
|
||||
From the Arena repository root:
|
||||
|
||||
```bash
|
||||
uv lock --project environments/redaction_pressure --check
|
||||
uv sync --project environments/redaction_pressure --frozen
|
||||
uv run --project environments/redaction_pressure \
|
||||
python -m unittest discover -s environments/redaction_pressure/tests -v
|
||||
```
|
||||
|
||||
The committed tests demonstrate oracle 1.0 and inaction 0.0, reject the historical
|
||||
first-character exploit across all 64 public development tasks, exercise parser and
|
||||
rule validation, prove deterministic/private split behavior, and interrupt hostile
|
||||
regexes under the shared episode budget.
|
||||
Reference in New Issue
Block a user