Three more environments, and probe.py to gate all four
canary-trap, fault-localisation and schema-migration, in the three domains the Environments Hub has nothing in: contamination, operations, data engineering. None is a port. Each is graded on a slice the model never read. canary-trap runs a probe set against a model that memorised the corpus AND one that learned the same facts from a reworded copy with every identifier regenerated. The obvious probe -- plant a GUID, ask for it back -- is sound and caps at 0.525 because it cannot see the paraphrase. It scored higher than that at first: facts with no distinct rewording had the paraphrase restating the corpus verbatim, so GUID probes caught it by accident. Every fact now carries a "core" token present in both wordings and in no public one, asserted at import, which is also what makes the ceiling reachable. fault-localisation puts the fault upstream and the noise downstream, the way incidents actually present. The broken service logs one line and goes quiet while its dependents log a dozen timeouts, so ranking by error volume answers the victim -- 0/64 incidents have the loudest service as the root, and probe.py asserts it rather than trusting the generator to stay that way. schema-migration executes the model's SQL against forty held-out rows carrying thousands separators, negatives, a unit with a slash in it and a value with no unit. Row preservation started as a reward beside fidelity and paid 0.15 for adding two empty columns and touching nothing; it is a multiplier now, so losing rows still zeroes a perfect migration and keeping them earns nothing on its own. probe.py is house rule 3 as an exit code: 0.000 for inaction and 1.000 for an oracle, or it returns 1. It loads the leaf modules through a namespace shim rather than the packages, so it runs without verifiers installed -- a self-check that needs the training stack is a self-check nobody runs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -23,6 +23,9 @@ otherwise hang a rollout until the harness killed it.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
import regex
|
||||
@@ -134,3 +137,24 @@ def measure(records: list[Record], rules: list[Rule]) -> Outcome:
|
||||
total += len(innocent)
|
||||
lost += sum(1 for w in innocent if w not in redacted)
|
||||
return Outcome(residual, secrets, collateral, decoys, lost, total, invalid)
|
||||
|
||||
|
||||
_BLOCK = re.compile(r"```(?:json)?\s*\n(.*?)```", re.DOTALL)
|
||||
|
||||
|
||||
def parse_rules(reply: str) -> list[Rule]:
|
||||
"""The last JSON array in the reply. A reply with no parsable ruleset is an empty
|
||||
ruleset, not an error — it scores what doing nothing scores."""
|
||||
candidates = _BLOCK.findall(reply or "")
|
||||
raw = candidates[-1] if candidates else (reply or "")
|
||||
try:
|
||||
parsed = json.loads(raw.strip())
|
||||
except json.JSONDecodeError:
|
||||
return []
|
||||
if not isinstance(parsed, list):
|
||||
return []
|
||||
rules = []
|
||||
for item in parsed:
|
||||
if isinstance(item, dict) and isinstance(item.get("pattern"), str):
|
||||
rules.append(Rule(item["pattern"], str(item.get("replacement", ""))))
|
||||
return rules
|
||||
|
||||
@@ -21,7 +21,6 @@ and grades in milliseconds. Deterministic from the task seed.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from typing import ClassVar
|
||||
|
||||
@@ -30,7 +29,7 @@ from pydantic import Field
|
||||
import verifiers.v1 as vf
|
||||
|
||||
from redaction_pressure.corpus import build_slices
|
||||
from redaction_pressure.scan import MAX_RULES, Rule, measure
|
||||
from redaction_pressure.scan import MAX_RULES, Rule, measure, parse_rules
|
||||
|
||||
SYSTEM = f"""You are writing redaction rules for a support-ticket corpus.
|
||||
|
||||
@@ -48,27 +47,6 @@ Example shape:
|
||||
[{{"pattern": "\\\\bfoo-\\\\d+\\\\b", "replacement": "[REF]"}}]
|
||||
```"""
|
||||
|
||||
_BLOCK = re.compile(r"```(?:json)?\s*\n(.*?)```", re.DOTALL)
|
||||
|
||||
|
||||
def parse_rules(reply: str) -> list[Rule]:
|
||||
"""The last JSON array in the reply. A reply with no parsable ruleset is an empty
|
||||
ruleset, not an error — it scores what doing nothing scores."""
|
||||
candidates = _BLOCK.findall(reply or "")
|
||||
raw = candidates[-1] if candidates else (reply or "")
|
||||
try:
|
||||
parsed = json.loads(raw.strip())
|
||||
except json.JSONDecodeError:
|
||||
return []
|
||||
if not isinstance(parsed, list):
|
||||
return []
|
||||
rules = []
|
||||
for item in parsed:
|
||||
if isinstance(item, dict) and isinstance(item.get("pattern"), str):
|
||||
rules.append(Rule(item["pattern"], str(item.get("replacement", ""))))
|
||||
return rules
|
||||
|
||||
|
||||
class RedactionData(vf.TaskData):
|
||||
seed: int
|
||||
"""Rebuilds both slices exactly; the held-out records are never serialized here."""
|
||||
|
||||
Generated
-3492
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user