kartiandClaude Opus 5
122a858b61
The 600-second per-model-call ceiling that capped the thinking-on run at n=6 is gone: DEFAULT_TIMEOUT is read=None in all nine venvs. There is NO verifiers 0.4.0. The `version = "0.4.0"` in the vendored tree belongs to a [[tool.uv.dependency-metadata]] block for nemo-gym, and reading it as verifiers' own sends you hunting a release that does not exist and concluding the bump is blocked. The vendored checkout is v0.3.0-59-g4bcb48e5, published to PyPI as 0.3.1.dev59, and the installed wheel diffs clean against the whole vendored v1 tree. Pinned as an ordinary index dependency: a path source is reproducible only on this box and a git URL pins a commit no wheel matches. ⚠️ dev59 changes the default agent runtime from subprocess to `prime`, a REMOTE sandbox. Without an explicit runtime every eval here exits "not authenticated with prime" before a single rollout — having already created its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than an unauthenticated one. All nine configs now say subprocess. Nothing here needs a sandbox: the reward is a pure function of the trace and the harness is null. Output layout is flat now — <env>--<model>--<harness>--<hex8>/ with configs/, traces.jsonl and a logs/eval.log that 0.3.0 never wrote. `-o` is the output_dir, not the run dir; `--run.dir` pins the leaf. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
redaction-pressure v0.2
A one-turn Verifiers v1 environment. The model returns regex rules; the scanner runs them against later records from the same deterministic corpus stream.
Reward contract
- Recall (0.35): a labelled secret counts only when every original character in its span is gone. Partial edits earn zero span recall.
- Precision (0.35): source characters belonging to fully removed secret spans, divided by all removed source characters. Partial damage stays in the denominator.
- Clean gate (0.30): every secret fully removed, every non-secret character and decoy preserved, and no validation or timeout errors.
- Validity: malformed JSON, bad entries, more than 24 rules, unsafe replacements, invalid regexes, excessive matches, timeouts, or an exhausted episode budget multiply every reward by zero.
Replacement is intentionally limited to "[REDACTED]" or "". That prevents a rule
from matching a complete secret and then leaking it through a capture expansion.
Splits
train and dev have separate, deterministic public seed ranges. private_eval derives
seeds with HMAC from REDACTION_PRESSURE_PRIVATE_SEED, which must contain at least 32
characters. The secret is never task data; traces carry its SHA-256 commitment and reveal
the derived task seed only in the completed trace so the episode can be reproduced by an
authorized evaluator.
Never present public dev scores as private evaluation.
Trace identity
Each task records the environment, corpus, and reward-schema versions, split commitment,
and a SHA-256 digest of the behavior-bearing installed Python sources. Scoring repeats
the identity under trace.info.environment and records validation failures under
trace.info.redaction_errors.
Checks
From the Arena repository root:
uv sync --project environments/redaction_pressure
uv run --project environments/redaction_pressure \
python -m unittest discover -s environments/redaction_pressure/tests -v
uv run --with regex python probe.py
Arena's environments are independently published libraries, so their generated
uv.lock files are intentionally not committed. The runtime contract above is pinned in
this environment's pyproject.toml and exercised on Python 3.11 and 3.12 in CI.
The committed tests demonstrate oracle 1.0 and inaction 0.0, reject the historical first-character exploit across all 64 public development tasks, exercise parser and rule validation, prove deterministic/private split behavior, and interrupt hostile regexes under the shared episode budget.