kartiandClaude Opus 5
3305be4ff7
The probe printed one blended number per policy, so a component pinned at 0.000 across every rung was invisible. Four gates hid there for a month. It now reports floor / best-below-oracle / oracle / ceiling for 25 components across 8 environments, and a component flat across every rung is fatal. bot-detection GATE_SLACK = 1 — fires 5/32 on the real rollouts, was 0/32. The max(1, reference_caught - SLACK) guard is verified by construction, not by sampling: without it the required count reaches 0 and an EMPTY accusation list clears the gate. Observed reference_caught is 4-6, so no amount of sampling would have found that hole. schema-migration GATE_MARGIN = 0.15 — fires 4/32, was 0/32. The cost is disclosed and bounded: the naive split clears it on 5.5% of 1,000 unseen seeds, fenced by an assert at 10%. Margin 0.10 keeps the leak at zero and fires 0/32, i.e. stays dead. A live gradient with a bounded leak beats a clean corpse. redaction-pressure is NOT given a margin, and that is the result rather than a failure. The only setting that fires at all leaves half the secrets standing and pays a four-of-seven ruleset on five seeds in six — a margin that pays for inaction is strictly worse than a dead gate. Recall maxes at 0.852 and no rollout ever cleared both clauses in one episode. It is genuinely hard, not miscalibrated. ⚠️ The per-component check did not catch the defect it was built for. Reverting schema-migration's margin to 0.0 — restoring the exact dead gate — printed ok and exited 0, because the near-oracle rung scrapes the unmargined gate on ~2 seeds in 24 and that kept best<oracle non-zero. Every assertion bounded how much a margin may PAY; none noticed if it stopped existing. migration() now carries the mirror of bot-detection's guard, and reverting the margin fails with "the margin is dead and the component carries no gradient between the crude answer and the exact one". canary-trap's oracle-minus-one rung is documented as degenerate rather than quietly relied on: it is identical to the oracle to four decimals, so it measures specificity and gate at the ceiling, not mid-ladder as its comment claimed. The CI lock policy asks git instead of the disk. It was checking the working tree, where a lock file is a normal by-product of uv sync, so it passed in a clean checkout and failed on every machine that had run an eval. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
60 lines
2.5 KiB
YAML
60 lines
2.5 KiB
YAML
name: arena-environments
|
|
|
|
on:
|
|
push:
|
|
branches: [main]
|
|
pull_request:
|
|
workflow_dispatch:
|
|
|
|
jobs:
|
|
validate:
|
|
runs-on: ubuntu-latest
|
|
strategy:
|
|
matrix:
|
|
python: ["3.11", "3.12"]
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- uses: astral-sh/setup-uv@v6
|
|
with:
|
|
version: "0.12.5"
|
|
enable-cache: false
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
python-version: ${{ matrix.python }}
|
|
# Asks git, not the disk. `find` was checking the working tree, where a lock file is a
|
|
# normal by-product of `uv sync` — so the step passed in CI's clean checkout and failed
|
|
# on every developer machine that had ever run an eval. The policy is about what is
|
|
# TRACKED: an environment is a library and must not pin its consumers' resolution.
|
|
- name: Enforce library lock policy
|
|
run: |
|
|
tracked="$(git ls-files 'environments/*/uv.lock')"
|
|
test -z "$tracked" || { echo "tracked lock files: $tracked"; exit 1; }
|
|
# An environment is `environments/*/pyproject.toml` and nothing else — the same
|
|
# denominator probe.py's discover() uses. A half-created scaffold directory with no
|
|
# manifest is invisible to both, so the two can never disagree about what exists.
|
|
- name: Refuse an empty environment scan
|
|
run: |
|
|
set -euo pipefail
|
|
count=$(find environments -mindepth 2 -maxdepth 2 -name pyproject.toml | wc -l)
|
|
echo "discovered $count environment manifests"
|
|
test "$count" -gt 0
|
|
- name: Run the floor and ceiling probe for every discovered environment
|
|
run: uv run --with regex python probe.py
|
|
- name: Run root probe integration test
|
|
run: uv run --with regex python -m unittest discover -s tests -v
|
|
- name: Install Redaction Pressure v0.2
|
|
run: uv sync --project environments/redaction_pressure --python ${{ matrix.python }}
|
|
- name: Run Redaction Pressure tests
|
|
run: >-
|
|
uv run --project environments/redaction_pressure
|
|
python -m unittest discover -s environments/redaction_pressure/tests -v
|
|
- name: Build all environment distributions
|
|
run: |
|
|
set -euo pipefail
|
|
for manifest in environments/*/pyproject.toml; do
|
|
environment=$(basename "$(dirname "$manifest")")
|
|
uv build "environments/$environment" --out-dir "dist/$environment"
|
|
done
|
|
- name: Verify documented Redaction Pressure entrypoint
|
|
run: uv run --project environments/redaction_pressure eval --help >/dev/null
|