53 lines
2.3 KiB
Markdown
53 lines
2.3 KiB
Markdown
# redaction-pressure v0.2
|
|
|
|
A one-turn Verifiers v1 environment. The model returns regex rules; the scanner runs
|
|
them against later records from the same deterministic corpus stream.
|
|
|
|
## Reward contract
|
|
|
|
- **Recall (0.35):** a labelled secret counts only when every original character in
|
|
its span is gone. Partial edits earn zero span recall.
|
|
- **Precision (0.35):** source characters belonging to fully removed secret spans,
|
|
divided by all removed source characters. Partial damage stays in the denominator.
|
|
- **Clean gate (0.30):** every secret fully removed, every non-secret character and
|
|
decoy preserved, and no validation or timeout errors.
|
|
- **Validity:** malformed JSON, bad entries, more than 24 rules, unsafe replacements,
|
|
invalid regexes, excessive matches, timeouts, or an exhausted episode budget multiply
|
|
every reward by zero.
|
|
|
|
Replacement is intentionally limited to `"[REDACTED]"` or `""`. That prevents a rule
|
|
from matching a complete secret and then leaking it through a capture expansion.
|
|
|
|
## Splits
|
|
|
|
`train` and `dev` have separate, deterministic public seed ranges. `private_eval` derives
|
|
seeds with HMAC from `REDACTION_PRESSURE_PRIVATE_SEED`, which must contain at least 32
|
|
characters. The secret is never task data; traces carry its SHA-256 commitment and reveal
|
|
the derived task seed only in the completed trace so the episode can be reproduced by an
|
|
authorized evaluator.
|
|
|
|
Never present public `dev` scores as private evaluation.
|
|
|
|
## Trace identity
|
|
|
|
Each task records the environment, corpus, and reward-schema versions, split commitment,
|
|
and a SHA-256 digest of the behavior-bearing installed Python sources. Scoring repeats
|
|
the identity under `trace.info.environment` and records validation failures under
|
|
`trace.info.redaction_errors`.
|
|
|
|
## Checks
|
|
|
|
From the Arena repository root:
|
|
|
|
```bash
|
|
uv lock --project environments/redaction_pressure --check
|
|
uv sync --project environments/redaction_pressure --frozen
|
|
uv run --project environments/redaction_pressure \
|
|
python -m unittest discover -s environments/redaction_pressure/tests -v
|
|
```
|
|
|
|
The committed tests demonstrate oracle 1.0 and inaction 0.0, reject the historical
|
|
first-character exploit across all 64 public development tasks, exercise parser and
|
|
rule validation, prove deterministic/private split behavior, and interrupt hostile
|
|
regexes under the shared episode budget.
|