Three more environments, and probe.py to gate all four
canary-trap, fault-localisation and schema-migration, in the three domains the Environments Hub has nothing in: contamination, operations, data engineering. None is a port. Each is graded on a slice the model never read. canary-trap runs a probe set against a model that memorised the corpus AND one that learned the same facts from a reworded copy with every identifier regenerated. The obvious probe -- plant a GUID, ask for it back -- is sound and caps at 0.525 because it cannot see the paraphrase. It scored higher than that at first: facts with no distinct rewording had the paraphrase restating the corpus verbatim, so GUID probes caught it by accident. Every fact now carries a "core" token present in both wordings and in no public one, asserted at import, which is also what makes the ceiling reachable. fault-localisation puts the fault upstream and the noise downstream, the way incidents actually present. The broken service logs one line and goes quiet while its dependents log a dozen timeouts, so ranking by error volume answers the victim -- 0/64 incidents have the loudest service as the root, and probe.py asserts it rather than trusting the generator to stay that way. schema-migration executes the model's SQL against forty held-out rows carrying thousands separators, negatives, a unit with a slash in it and a value with no unit. Row preservation started as a reward beside fidelity and paid 0.15 for adding two empty columns and touching nothing; it is a multiplier now, so losing rows still zeroes a perfect migration and keeping them earns nothing on its own. probe.py is house rule 3 as an exit code: 0.000 for inaction and 1.000 for an oracle, or it returns 1. It loads the leaf modules through a namespace shim rather than the packages, so it runs without verifiers installed -- a self-check that needs the training stack is a self-check nobody runs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
@@ -0,0 +1,3 @@
|
||||
from canary_trap.taskset import CanaryTaskset
|
||||
|
||||
__all__ = ["CanaryTaskset"]
|
||||
@@ -0,0 +1,121 @@
|
||||
"""The three knowledge states a probe has to tell apart.
|
||||
|
||||
A contamination probe is only useful if it separates a model that has seen your eval set
|
||||
from one that has not. The trap is that the obvious probe — carry a GUID into the corpus
|
||||
and ask for it back — separates only the *verbatim* case. Training data gets reworded,
|
||||
deduplicated and paraphrased on the way into a model, and a model that learned every fact
|
||||
in your eval set from a reworded copy answers no GUID and is contaminated anyway.
|
||||
|
||||
So a task builds three knowledge states from one seed:
|
||||
|
||||
clean public facts only. Answering from here is not evidence of anything.
|
||||
verbatim public + the corpus exactly as written, GUIDs and all.
|
||||
paraphrase public + the same FACTS, reworded, with every GUID regenerated.
|
||||
|
||||
A probe set is scored on catching both contaminated states without firing on the clean
|
||||
one. Verbatim-only probes cap at half the detection score, and that ceiling is the whole
|
||||
lesson of the environment.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import random
|
||||
import string
|
||||
from dataclasses import dataclass
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Fact:
|
||||
"""One fact in three parts.
|
||||
|
||||
`canonical` is how the corpus states it. `alias` is how the paraphrase states it, and
|
||||
it shares no wording with `canonical` beyond `core`. `core` is the token a probe has to
|
||||
key on to see BOTH — and it is deliberately absent from public knowledge, so keying on
|
||||
it is also sound. Every fact therefore has an answer that scores 1.0, and none of them
|
||||
is the sentence the corpus actually contains.
|
||||
"""
|
||||
|
||||
subject: str
|
||||
attribute: str
|
||||
canonical: str
|
||||
alias: str
|
||||
core: str
|
||||
|
||||
|
||||
SUBJECTS = [
|
||||
Fact("the Lisbon relay", "throughput", "412 requests per second", "412 rps", "412"),
|
||||
Fact("the Osaka mirror", "operator", "Ines Baptiste", "I. Baptiste", "Baptiste"),
|
||||
Fact("the Tallinn cache", "eviction policy", "least-recently-used (LRU)", "LRU ordering", "LRU"),
|
||||
Fact("the Bogota gateway", "timeout", "1400 milliseconds", "1400 ms", "1400"),
|
||||
Fact("the Leeds shard", "replica count", "seven replicas", "seven copies", "seven"),
|
||||
Fact("the Dakar queue", "retry budget", "nine retries", "nine attempts", "nine"),
|
||||
]
|
||||
|
||||
# Facts a model could know without ever seeing our corpus. A probe whose answer lives here
|
||||
# fires on a clean model, which is what makes it a false positive rather than a probe.
|
||||
PUBLIC = [
|
||||
"TCP uses a three-way handshake.",
|
||||
"Postgres defaults to port 5432.",
|
||||
"HTTP 429 means too many requests.",
|
||||
"UTC does not observe daylight saving.",
|
||||
"SHA-256 emits 32 bytes.",
|
||||
"The capital of Portugal is Lisbon.",
|
||||
]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Knowledge:
|
||||
"""What one model can answer from. Membership is a substring test on `text`."""
|
||||
|
||||
name: str
|
||||
text: str
|
||||
|
||||
|
||||
def _check_facts() -> None:
|
||||
"""The environment's own floor-and-ceiling guarantee, asserted at import.
|
||||
|
||||
Every fact must have an answer that a probe can key on to see both contaminated states
|
||||
(`core` in canonical AND in alias) without accusing a clean model (`core` absent from
|
||||
public). Without this the ceiling is unreachable for some seeds and the reward quietly
|
||||
stops being solvable — which is exactly the failure the house rules exist to catch.
|
||||
"""
|
||||
public = " ".join(PUBLIC).lower()
|
||||
for fact in SUBJECTS:
|
||||
assert fact.core.lower() in fact.canonical.lower(), fact
|
||||
assert fact.core.lower() in fact.alias.lower(), fact
|
||||
assert fact.core.lower() not in public, f"{fact.core} collides with public knowledge"
|
||||
# And the paraphrase must not simply restate the corpus, or a verbatim probe would
|
||||
# catch it for free and the environment would teach nothing.
|
||||
assert fact.alias.lower() != fact.canonical.lower(), fact
|
||||
|
||||
|
||||
_check_facts()
|
||||
|
||||
|
||||
def _guid(rng: random.Random) -> str:
|
||||
hexd = "0123456789abcdef"
|
||||
return "-".join("".join(rng.choice(hexd) for _ in range(n)) for n in (8, 4, 4, 12))
|
||||
|
||||
|
||||
def build(seed: int, facts: int) -> tuple[str, dict[str, Knowledge]]:
|
||||
"""One corpus, and the three knowledge states built from it."""
|
||||
rng = random.Random(seed)
|
||||
picked = rng.sample(SUBJECTS, min(facts, len(SUBJECTS)))
|
||||
public_text = " ".join(PUBLIC)
|
||||
|
||||
verbatim_lines, paraphrase_lines = [], []
|
||||
for fact in picked:
|
||||
verbatim_lines.append(
|
||||
f"[{_guid(rng)}] Record for {fact.subject}: {fact.attribute} is {fact.canonical}."
|
||||
)
|
||||
# Same fact, different sentence, and a REGENERATED tag — the paraphrase remembers
|
||||
# what is true and has forgotten every string that carried it.
|
||||
paraphrase_lines.append(
|
||||
f"[{_guid(rng)}] Concerning {fact.subject}, its {fact.attribute} sits at {fact.alias}."
|
||||
)
|
||||
|
||||
corpus = "\n".join(verbatim_lines)
|
||||
return corpus, {
|
||||
"clean": Knowledge("clean", public_text),
|
||||
"verbatim": Knowledge("verbatim", public_text + "\n" + corpus),
|
||||
"paraphrase": Knowledge("paraphrase", public_text + "\n" + "\n".join(paraphrase_lines)),
|
||||
}
|
||||
@@ -0,0 +1,105 @@
|
||||
"""Scoring a probe set against the three knowledge states.
|
||||
|
||||
Two properties, and they pull against each other:
|
||||
|
||||
sound the probe does NOT answer on the clean model. An unsound probe accuses a model
|
||||
that has seen nothing, which is worse than no probe at all — Bench VOIDS a
|
||||
score on a contamination verdict, so a false positive destroys a real result.
|
||||
catching a sound probe answers on a contaminated model. Both contaminated states have
|
||||
to be caught, and the paraphrase is the one that costs: it kept every fact and
|
||||
not one of the strings, so a GUID-recall probe cannot see it.
|
||||
|
||||
Detection is scored per state rather than per probe, because catching the same state with
|
||||
six probes is not six times the evidence. Specificity is per probe, so shotgunning a
|
||||
hundred guesses to land one is paid for.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from canary_trap.corpus import Knowledge
|
||||
|
||||
MAX_PROBES = 12
|
||||
MIN_ANSWER_CHARS = 3
|
||||
CONTAMINATED = ("verbatim", "paraphrase")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Probe:
|
||||
question: str
|
||||
answer: str
|
||||
|
||||
|
||||
@dataclass
|
||||
class Outcome:
|
||||
sound: int = 0
|
||||
total: int = 0
|
||||
caught: set[str] = field(default_factory=set)
|
||||
rejected: list[str] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def specificity(self) -> float:
|
||||
"""Share of probes that do not accuse a clean model. Zero probes scores zero: an
|
||||
empty set has no false positives and also no evidence, and rewarding it would make
|
||||
returning nothing the safe play."""
|
||||
return self.sound / self.total if self.total else 0.0
|
||||
|
||||
@property
|
||||
def detection(self) -> float:
|
||||
return len(self.caught) / len(CONTAMINATED)
|
||||
|
||||
@property
|
||||
def clean_gate(self) -> bool:
|
||||
"""Every contaminated state caught, and not one probe firing on the clean model."""
|
||||
return self.detection == 1.0 and self.total > 0 and self.sound == self.total
|
||||
|
||||
|
||||
def _answers(knowledge: Knowledge, probe: Probe) -> bool:
|
||||
"""The model stands in for itself: it can produce an answer exactly when that answer is
|
||||
present in what it knows. Crude, and the crudeness is load-bearing — it makes the whole
|
||||
environment deterministic and free, and it still distinguishes recall of a string from
|
||||
recall of a fact, which is the only distinction being taught."""
|
||||
return probe.answer.lower() in knowledge.text.lower()
|
||||
|
||||
|
||||
def measure(states: dict[str, Knowledge], probes: list[Probe]) -> Outcome:
|
||||
outcome = Outcome()
|
||||
for probe in probes[:MAX_PROBES]:
|
||||
# A blank or near-blank answer is a substring of everything, which would read as a
|
||||
# probe that catches all three states. Rejected rather than scored as unsound, so
|
||||
# the reason shows up in the trace.
|
||||
if len(probe.answer.strip()) < MIN_ANSWER_CHARS:
|
||||
outcome.rejected.append(probe.answer[:24])
|
||||
outcome.total += 1
|
||||
continue
|
||||
outcome.total += 1
|
||||
if _answers(states["clean"], probe):
|
||||
continue # unsound: fires on a model that has seen nothing
|
||||
outcome.sound += 1
|
||||
for name in CONTAMINATED:
|
||||
if _answers(states[name], probe):
|
||||
outcome.caught.add(name)
|
||||
return outcome
|
||||
|
||||
|
||||
_BLOCK = re.compile(r"```(?:json)?\s*\n(.*?)```", re.DOTALL)
|
||||
|
||||
|
||||
def parse_probes(reply: str) -> list[Probe]:
|
||||
blocks = _BLOCK.findall(reply or "")
|
||||
raw = blocks[-1] if blocks else (reply or "")
|
||||
try:
|
||||
parsed = json.loads(raw.strip())
|
||||
except json.JSONDecodeError:
|
||||
return []
|
||||
if not isinstance(parsed, list):
|
||||
return []
|
||||
return [
|
||||
Probe(str(p.get("question", "")), p["answer"])
|
||||
for p in parsed
|
||||
if isinstance(p, dict) and isinstance(p.get("answer"), str)
|
||||
]
|
||||
@@ -0,0 +1,110 @@
|
||||
"""canary-trap: write contamination probes that survive a paraphrase.
|
||||
|
||||
The agent reads a corpus destined to become an eval set and returns probes that would
|
||||
reveal a model trained on it. Probes are then run against three stand-in models: one that
|
||||
has seen only public facts, one that memorised the corpus verbatim, and one that learned
|
||||
the same facts from a reworded copy with every identifier regenerated.
|
||||
|
||||
The obvious probe — ask for a GUID that was planted in the corpus — is sound and catches
|
||||
exactly one of the two contaminated models, so it ceilings at 0.5 detection. The probe that
|
||||
scores is one keyed on a FACT the paraphrase kept, whose answer does not also appear in
|
||||
public knowledge. Learning that difference is the environment.
|
||||
|
||||
This is Bench's contamination gate as a task. There, a canary hit does not annotate a score,
|
||||
it voids it — which is why a probe that fires on a clean model is the expensive failure here
|
||||
and specificity carries as much weight as detection.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pydantic import Field
|
||||
|
||||
import verifiers.v1 as vf
|
||||
|
||||
from canary_trap.corpus import build
|
||||
from canary_trap.scan import MAX_PROBES, Probe, measure, parse_probes
|
||||
|
||||
SYSTEM = f"""You are designing contamination probes for an evaluation corpus.
|
||||
|
||||
Return a JSON array of at most {MAX_PROBES} probes in one ```json code block. Each probe is
|
||||
an object with "question" (what you would ask a model) and "answer" (the exact string a
|
||||
model that has seen this corpus would produce).
|
||||
|
||||
Your probes are run against three models: one that has seen only public knowledge, one that
|
||||
memorised this corpus word for word, and one that learned the same facts from a reworded
|
||||
copy in which every bracketed identifier was regenerated.
|
||||
|
||||
A probe that the public-knowledge model can answer is worse than no probe — it accuses an
|
||||
innocent model. A probe only the word-for-word model can answer catches half of what is
|
||||
there.
|
||||
|
||||
Example shape:
|
||||
```json
|
||||
[{{"question": "What is the retry budget for the Dakar queue?", "answer": "three retries"}}]
|
||||
```"""
|
||||
|
||||
class CanaryData(vf.TaskData):
|
||||
seed: int
|
||||
facts: int
|
||||
|
||||
|
||||
class CanaryTask(vf.Task[CanaryData]):
|
||||
@vf.stop
|
||||
async def single_turn(self, trace: vf.Trace) -> bool:
|
||||
return trace.num_turns >= 1
|
||||
|
||||
@vf.metric
|
||||
async def scan(self, trace: vf.Trace) -> dict[str, float]:
|
||||
_, states = build(self.data.seed, self.data.facts)
|
||||
outcome = measure(states, parse_probes(trace.last_reply))
|
||||
return {
|
||||
"detection": outcome.detection,
|
||||
"specificity": outcome.specificity,
|
||||
"clean": float(outcome.clean_gate),
|
||||
"probes": float(outcome.total),
|
||||
"rejected": float(len(outcome.rejected)),
|
||||
# Reported separately because it is the finding the environment exists to
|
||||
# teach: a probe set can be perfectly sound and still blind to a paraphrase.
|
||||
"caught_paraphrase": float("paraphrase" in outcome.caught),
|
||||
}
|
||||
|
||||
@vf.reward(weight=0.35)
|
||||
async def detection(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("detection", 0.0)
|
||||
|
||||
@vf.reward(weight=0.35)
|
||||
async def specificity(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("specificity", 0.0)
|
||||
|
||||
@vf.reward(weight=0.30)
|
||||
async def gate(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("clean", 0.0)
|
||||
|
||||
|
||||
class CanaryConfig(vf.TasksetConfig):
|
||||
num_tasks: int = Field(48, ge=1)
|
||||
facts: int = Field(4, ge=1, le=6)
|
||||
|
||||
|
||||
class CanaryTaskset(vf.Taskset[CanaryTask, CanaryConfig]):
|
||||
SEED_BASE = 70_000
|
||||
|
||||
def load(self) -> list[CanaryTask]:
|
||||
tasks = []
|
||||
for i in range(self.config.num_tasks):
|
||||
seed = self.SEED_BASE + i
|
||||
corpus, _ = build(seed, self.config.facts)
|
||||
tasks.append(
|
||||
CanaryTask(
|
||||
CanaryData(
|
||||
idx=i,
|
||||
name=f"corpus-{seed}",
|
||||
prompt=f"The corpus:\n\n{corpus}\n\nWrite the probes.",
|
||||
system_prompt=SYSTEM,
|
||||
seed=seed,
|
||||
facts=self.config.facts,
|
||||
),
|
||||
self.config.task,
|
||||
)
|
||||
)
|
||||
return tasks
|
||||
@@ -0,0 +1,13 @@
|
||||
[project]
|
||||
name = "canary-trap"
|
||||
version = "0.1.0"
|
||||
description = "canary-trap — write contamination probes that survive a paraphrase, without accusing a clean model."
|
||||
requires-python = ">=3.11"
|
||||
dependencies = ["verifiers"]
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["canary_trap"]
|
||||
@@ -0,0 +1,3 @@
|
||||
from fault_localisation.taskset import FaultTaskset
|
||||
|
||||
__all__ = ["FaultTaskset"]
|
||||
@@ -0,0 +1,94 @@
|
||||
"""Generating one incident: a service graph, an injected fault, and the logs it produced.
|
||||
|
||||
The fault is placed upstream and the noise is emitted downstream, because that is how real
|
||||
incidents present and it is the one thing a log-reading agent has to get right. A service
|
||||
whose dependency has stalled logs timeouts, retries and 503s — loudly, once per request —
|
||||
while the service that actually broke logs a single line about its connection pool and then
|
||||
goes quiet. Ranking services by error volume therefore points at the victim every time.
|
||||
|
||||
Everything is derived from one seed, so a task is reproducible from its integer id and the
|
||||
answer key is generated alongside the logs rather than written by hand.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import random
|
||||
from dataclasses import dataclass
|
||||
|
||||
SERVICES = ["edge", "checkout", "ledger", "inventory", "pricing", "vault"]
|
||||
|
||||
# (fault class, the single line the ROOT service emits, what its callers emit)
|
||||
FAULTS = [
|
||||
("pool_exhausted", "connection pool exhausted (size=16, waiters=88)", "upstream timeout after 3000ms"),
|
||||
("disk_full", "write failed: no space left on device", "upstream returned 503"),
|
||||
("cert_expired", "TLS handshake failed: certificate expired", "upstream connection reset"),
|
||||
("deadlock", "transaction aborted: deadlock detected", "upstream timeout after 3000ms"),
|
||||
("oom_kill", "worker killed: cgroup memory limit exceeded", "upstream returned 502"),
|
||||
("clock_skew", "rejecting request: token nbf 41s in the future", "upstream returned 401"),
|
||||
]
|
||||
|
||||
|
||||
@dataclass
|
||||
class Incident:
|
||||
lines: list[str]
|
||||
"""Log lines, each prefixed with a stable `L07` style id the answer can cite."""
|
||||
root: str
|
||||
fault: str
|
||||
evidence: str
|
||||
"""The id of the one line that names the cause. Every other line is a symptom."""
|
||||
|
||||
|
||||
def build(seed: int) -> Incident:
|
||||
rng = random.Random(seed)
|
||||
root, *rest = rng.sample(SERVICES, 4)
|
||||
# The callers: they depend on root, so they are the ones that will look broken.
|
||||
callers = rest[:2]
|
||||
bystander = rest[2]
|
||||
fault, root_line, caller_line = rng.choice(FAULTS)
|
||||
|
||||
raw: list[tuple[int, str, str]] = []
|
||||
clock = rng.randint(0, 40)
|
||||
|
||||
# Ordinary traffic first, from everyone, so "the service with any errors" is not free.
|
||||
for _ in range(6):
|
||||
clock += rng.randint(1, 4)
|
||||
svc = rng.choice([root, *callers, bystander])
|
||||
raw.append((clock, svc, rng.choice([
|
||||
"handled request in 24ms", "cache hit", "healthcheck ok", "handled request in 61ms",
|
||||
])))
|
||||
|
||||
# The cause: one line, once, and then the root goes quiet.
|
||||
clock += rng.randint(1, 3)
|
||||
cause_at = clock
|
||||
raw.append((clock, root, root_line))
|
||||
|
||||
# The symptoms: loud, repeated, and in the wrong place.
|
||||
for _ in range(rng.randint(9, 14)):
|
||||
clock += rng.randint(1, 3)
|
||||
raw.append((clock, rng.choice(callers), caller_line))
|
||||
# A red herring the bystander emits on its own schedule, unrelated to the fault.
|
||||
clock += rng.randint(1, 2)
|
||||
raw.append((clock, bystander, "retrying scheduled job (attempt 2)"))
|
||||
|
||||
raw.sort(key=lambda r: r[0])
|
||||
lines, evidence = [], ""
|
||||
for i, (at, svc, text) in enumerate(raw):
|
||||
line_id = f"L{i:02d}"
|
||||
if at == cause_at and svc == root and text == root_line:
|
||||
evidence = line_id
|
||||
lines.append(f"{line_id} t+{at:03d}s {svc:<10} {text}")
|
||||
return Incident(lines=lines, root=root, fault=fault, evidence=evidence)
|
||||
|
||||
|
||||
def loudest(incident: Incident) -> str:
|
||||
"""The service with the most error lines — what ranking by volume would answer.
|
||||
|
||||
Exported because the probe uses it: if this ever starts coinciding with the root, the
|
||||
environment has stopped testing the thing it was built to test.
|
||||
"""
|
||||
counts: dict[str, int] = {}
|
||||
for line in incident.lines:
|
||||
svc = line.split()[2]
|
||||
if "upstream" in line:
|
||||
counts[svc] = counts.get(svc, 0) + 1
|
||||
return max(counts, key=lambda k: counts[k]) if counts else ""
|
||||
@@ -0,0 +1,119 @@
|
||||
"""fault-localisation: name what broke, not what is complaining.
|
||||
|
||||
One incident, one injected fault, and a log the fault's victims dominate. The service that
|
||||
actually failed emits a single line and falls silent; the two that depend on it emit a
|
||||
dozen timeouts between them. Ranking by error volume answers the loudest service, which is
|
||||
never the right one — the probe in this package asserts that, so the day the generator
|
||||
drifts into making them coincide the environment fails rather than quietly rewarding the
|
||||
heuristic it exists to punish.
|
||||
|
||||
Three answers are required and they are not redundant. The service can be guessed one time
|
||||
in four. The fault class can be read off a line without knowing which line matters. Citing
|
||||
the evidence line is what distinguishes having found the cause from having recognised the
|
||||
vocabulary, and it is why the gate needs all three rather than a majority.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
|
||||
from pydantic import Field
|
||||
|
||||
import verifiers.v1 as vf
|
||||
|
||||
from fault_localisation.incident import FAULTS, SERVICES, build
|
||||
|
||||
CLASSES = sorted(f[0] for f in FAULTS)
|
||||
|
||||
SYSTEM = f"""You are on call. Read the log and name the ROOT CAUSE.
|
||||
|
||||
Return one JSON object in a ```json code block with exactly three keys:
|
||||
"service" the service that actually failed, one of: {", ".join(SERVICES)}
|
||||
"fault" the failure class, one of: {", ".join(CLASSES)}
|
||||
"evidence" the id (e.g. "L07") of the single line that names the cause
|
||||
|
||||
The service that failed is not the service producing the most errors. Its dependents
|
||||
produce the errors; it produces one line and then stops."""
|
||||
|
||||
_BLOCK = re.compile(r"```(?:json)?\s*\n(.*?)```", re.DOTALL)
|
||||
|
||||
|
||||
def parse_answer(reply: str) -> dict[str, str]:
|
||||
blocks = _BLOCK.findall(reply or "")
|
||||
raw = blocks[-1] if blocks else (reply or "")
|
||||
try:
|
||||
parsed = json.loads(raw.strip())
|
||||
except json.JSONDecodeError:
|
||||
return {}
|
||||
if not isinstance(parsed, dict):
|
||||
return {}
|
||||
return {k: str(v).strip() for k, v in parsed.items() if isinstance(v, (str, int))}
|
||||
|
||||
|
||||
class FaultData(vf.TaskData):
|
||||
seed: int
|
||||
|
||||
|
||||
class FaultTask(vf.Task[FaultData]):
|
||||
@vf.stop
|
||||
async def single_turn(self, trace: vf.Trace) -> bool:
|
||||
return trace.num_turns >= 1
|
||||
|
||||
@vf.metric
|
||||
async def scan(self, trace: vf.Trace) -> dict[str, float]:
|
||||
incident = build(self.data.seed)
|
||||
answer = parse_answer(trace.last_reply)
|
||||
service = float(answer.get("service", "").lower() == incident.root)
|
||||
fault = float(answer.get("fault", "").lower() == incident.fault)
|
||||
evidence = float(answer.get("evidence", "").upper() == incident.evidence)
|
||||
return {
|
||||
"service": service,
|
||||
"fault": fault,
|
||||
"evidence": evidence,
|
||||
"clean": float(service and fault and evidence),
|
||||
}
|
||||
|
||||
@vf.reward(weight=0.30)
|
||||
async def service(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("service", 0.0)
|
||||
|
||||
@vf.reward(weight=0.25)
|
||||
async def fault(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("fault", 0.0)
|
||||
|
||||
@vf.reward(weight=0.25)
|
||||
async def evidence(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("evidence", 0.0)
|
||||
|
||||
@vf.reward(weight=0.20)
|
||||
async def gate(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("clean", 0.0)
|
||||
|
||||
|
||||
class FaultConfig(vf.TasksetConfig):
|
||||
num_tasks: int = Field(64, ge=1)
|
||||
|
||||
|
||||
class FaultTaskset(vf.Taskset[FaultTask, FaultConfig]):
|
||||
SEED_BASE = 50_000
|
||||
|
||||
def load(self) -> list[FaultTask]:
|
||||
tasks = []
|
||||
for i in range(self.config.num_tasks):
|
||||
seed = self.SEED_BASE + i
|
||||
incident = build(seed)
|
||||
log = "\n".join(incident.lines)
|
||||
tasks.append(
|
||||
FaultTask(
|
||||
FaultData(
|
||||
idx=i,
|
||||
name=f"incident-{seed}",
|
||||
prompt=f"```\n{log}\n```\n\nName the root cause.",
|
||||
system_prompt=SYSTEM,
|
||||
seed=seed,
|
||||
),
|
||||
self.config.task,
|
||||
)
|
||||
)
|
||||
return tasks
|
||||
@@ -0,0 +1,13 @@
|
||||
[project]
|
||||
name = "fault-localisation"
|
||||
version = "0.1.0"
|
||||
description = "fault-localisation — name the service that broke, not the ones complaining about it."
|
||||
requires-python = ">=3.11"
|
||||
dependencies = ["verifiers"]
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["fault_localisation"]
|
||||
@@ -23,6 +23,9 @@ otherwise hang a rollout until the harness killed it.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
import regex
|
||||
@@ -134,3 +137,24 @@ def measure(records: list[Record], rules: list[Rule]) -> Outcome:
|
||||
total += len(innocent)
|
||||
lost += sum(1 for w in innocent if w not in redacted)
|
||||
return Outcome(residual, secrets, collateral, decoys, lost, total, invalid)
|
||||
|
||||
|
||||
_BLOCK = re.compile(r"```(?:json)?\s*\n(.*?)```", re.DOTALL)
|
||||
|
||||
|
||||
def parse_rules(reply: str) -> list[Rule]:
|
||||
"""The last JSON array in the reply. A reply with no parsable ruleset is an empty
|
||||
ruleset, not an error — it scores what doing nothing scores."""
|
||||
candidates = _BLOCK.findall(reply or "")
|
||||
raw = candidates[-1] if candidates else (reply or "")
|
||||
try:
|
||||
parsed = json.loads(raw.strip())
|
||||
except json.JSONDecodeError:
|
||||
return []
|
||||
if not isinstance(parsed, list):
|
||||
return []
|
||||
rules = []
|
||||
for item in parsed:
|
||||
if isinstance(item, dict) and isinstance(item.get("pattern"), str):
|
||||
rules.append(Rule(item["pattern"], str(item.get("replacement", ""))))
|
||||
return rules
|
||||
|
||||
@@ -21,7 +21,6 @@ and grades in milliseconds. Deterministic from the task seed.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from typing import ClassVar
|
||||
|
||||
@@ -30,7 +29,7 @@ from pydantic import Field
|
||||
import verifiers.v1 as vf
|
||||
|
||||
from redaction_pressure.corpus import build_slices
|
||||
from redaction_pressure.scan import MAX_RULES, Rule, measure
|
||||
from redaction_pressure.scan import MAX_RULES, Rule, measure, parse_rules
|
||||
|
||||
SYSTEM = f"""You are writing redaction rules for a support-ticket corpus.
|
||||
|
||||
@@ -48,27 +47,6 @@ Example shape:
|
||||
[{{"pattern": "\\\\bfoo-\\\\d+\\\\b", "replacement": "[REF]"}}]
|
||||
```"""
|
||||
|
||||
_BLOCK = re.compile(r"```(?:json)?\s*\n(.*?)```", re.DOTALL)
|
||||
|
||||
|
||||
def parse_rules(reply: str) -> list[Rule]:
|
||||
"""The last JSON array in the reply. A reply with no parsable ruleset is an empty
|
||||
ruleset, not an error — it scores what doing nothing scores."""
|
||||
candidates = _BLOCK.findall(reply or "")
|
||||
raw = candidates[-1] if candidates else (reply or "")
|
||||
try:
|
||||
parsed = json.loads(raw.strip())
|
||||
except json.JSONDecodeError:
|
||||
return []
|
||||
if not isinstance(parsed, list):
|
||||
return []
|
||||
rules = []
|
||||
for item in parsed:
|
||||
if isinstance(item, dict) and isinstance(item.get("pattern"), str):
|
||||
rules.append(Rule(item["pattern"], str(item.get("replacement", ""))))
|
||||
return rules
|
||||
|
||||
|
||||
class RedactionData(vf.TaskData):
|
||||
seed: int
|
||||
"""Rebuilds both slices exactly; the held-out records are never serialized here."""
|
||||
|
||||
Generated
-3492
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,13 @@
|
||||
[project]
|
||||
name = "schema-migration"
|
||||
version = "0.1.0"
|
||||
description = "schema-migration — split a text column into a number and a unit, graded on rows you never saw."
|
||||
requires-python = ">=3.11"
|
||||
dependencies = ["verifiers"]
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["schema_migration"]
|
||||
@@ -0,0 +1,3 @@
|
||||
from schema_migration.taskset import MigrationTaskset
|
||||
|
||||
__all__ = ["MigrationTaskset"]
|
||||
@@ -0,0 +1,69 @@
|
||||
"""The table before the migration, and the rows the migration is graded on.
|
||||
|
||||
One `readings` table carries a `value_text` column that should never have been text:
|
||||
"1400 ms", "412 rps", "-3.5 degC". The migration has to split it into a real number and a
|
||||
unit and drop the original column, without losing a row or rounding a value.
|
||||
|
||||
The visible rows are deliberately tidy. The held-out rows are not: they carry thousands
|
||||
separators, negatives, decimals, a unit with a digit in it, a value with no unit at all,
|
||||
and one with extra internal whitespace. A migration written by pattern-matching the five
|
||||
rows on screen parses about half of them, which is the entire point of grading on rows the
|
||||
author never read.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import random
|
||||
import sqlite3
|
||||
|
||||
TIDY = [
|
||||
("1400 ms", 1400.0, "ms"),
|
||||
("412 rps", 412.0, "rps"),
|
||||
("16 conns", 16.0, "conns"),
|
||||
("88 waiters", 88.0, "waiters"),
|
||||
("7 replicas", 7.0, "replicas"),
|
||||
]
|
||||
|
||||
# Every awkward shape a real column of this kind contains.
|
||||
AWKWARD = [
|
||||
("1,400 ms", 1400.0, "ms"), # thousands separator
|
||||
("-3.5 degC", -3.5, "degC"), # negative, decimal
|
||||
("0.25 s", 0.25, "s"), # leading zero decimal
|
||||
("64 KiB/s", 64.0, "KiB/s"), # unit containing a digit and a slash
|
||||
("120", 120.0, ""), # no unit at all
|
||||
("55 rps", 55.0, "rps"), # doubled internal space
|
||||
("2,048 MiB", 2048.0, "MiB"),
|
||||
("-12 ms", -12.0, "ms"),
|
||||
("3.0 x", 3.0, "x"),
|
||||
("1,000,000 rows", 1000000.0, "rows"),
|
||||
]
|
||||
|
||||
SOURCE_DDL = """
|
||||
CREATE TABLE readings (
|
||||
id INTEGER PRIMARY KEY,
|
||||
probe TEXT NOT NULL,
|
||||
value_text TEXT NOT NULL
|
||||
);
|
||||
"""
|
||||
|
||||
TARGET_COLUMNS = {"id", "probe", "value_num", "unit"}
|
||||
|
||||
|
||||
def rows(seed: int, count: int, awkward: bool) -> list[tuple[str, float, str]]:
|
||||
rng = random.Random(seed)
|
||||
pool = AWKWARD + TIDY if awkward else TIDY
|
||||
return [pool[rng.randrange(len(pool))] for _ in range(count)]
|
||||
|
||||
|
||||
def connect(seed: int, count: int, awkward: bool) -> tuple[sqlite3.Connection, list[tuple]]:
|
||||
"""A fresh in-memory database holding one slice. Nothing touches disk, so an agent's
|
||||
SQL cannot reach anything — the environment is the sandbox."""
|
||||
con = sqlite3.connect(":memory:")
|
||||
con.executescript(SOURCE_DDL)
|
||||
data = rows(seed, count, awkward)
|
||||
con.executemany(
|
||||
"INSERT INTO readings (id, probe, value_text) VALUES (?, ?, ?)",
|
||||
[(i + 1, f"probe_{i:03d}", text) for i, (text, _, _) in enumerate(data)],
|
||||
)
|
||||
con.commit()
|
||||
return con, data
|
||||
@@ -0,0 +1,100 @@
|
||||
"""Running the agent's migration, and measuring what survived it.
|
||||
|
||||
Four things are checked, and they are in tension by construction:
|
||||
|
||||
schema the target columns exist and `value_text` is gone. A migration that adds the
|
||||
new columns and leaves the old one is not a migration, it is a copy.
|
||||
fidelity per row, does (value_num, unit) recompose to the original value. This is the
|
||||
expensive one and the only one that punishes a parser fitted to the visible
|
||||
rows.
|
||||
rows the row count is unchanged. Without it, `DELETE FROM readings` scores perfect
|
||||
fidelity over an empty table — vacuously, since there is nothing left to be
|
||||
wrong about.
|
||||
gate all three, exactly. `forge verify`'s exit code: a migration is correct or it
|
||||
is not run in production.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlite3
|
||||
from dataclasses import dataclass
|
||||
|
||||
from schema_migration.fixture import TARGET_COLUMNS, connect
|
||||
|
||||
STATEMENT_LIMIT = 40
|
||||
# ATTACH would reach outside the in-memory database; the rest cannot, but there is no
|
||||
# reason for a schema migration to use any of them.
|
||||
FORBIDDEN = ("attach", "pragma", "vacuum")
|
||||
TOLERANCE = 1e-9
|
||||
|
||||
|
||||
@dataclass
|
||||
class Outcome:
|
||||
schema_ok: bool = False
|
||||
matched: int = 0
|
||||
graded: int = 0
|
||||
rows_before: int = 0
|
||||
rows_after: int = 0
|
||||
error: str = ""
|
||||
|
||||
@property
|
||||
def fidelity(self) -> float:
|
||||
return self.matched / self.graded if self.graded else 0.0
|
||||
|
||||
@property
|
||||
def rows_kept(self) -> float:
|
||||
if not self.rows_before:
|
||||
return 0.0
|
||||
# Inserting extra rows is as wrong as dropping them, so this is a distance, not a ratio.
|
||||
return max(0.0, 1.0 - abs(self.rows_after - self.rows_before) / self.rows_before)
|
||||
|
||||
@property
|
||||
def clean(self) -> bool:
|
||||
return self.schema_ok and self.graded > 0 and self.matched == self.graded and self.rows_after == self.rows_before
|
||||
|
||||
|
||||
def _columns(con: sqlite3.Connection) -> set[str]:
|
||||
return {r[1] for r in con.execute("PRAGMA table_info(readings)")}
|
||||
|
||||
|
||||
def measure(seed: int, count: int, migration: str) -> Outcome:
|
||||
con, data = connect(seed, count, awkward=True)
|
||||
outcome = Outcome(rows_before=len(data), graded=len(data))
|
||||
|
||||
lowered = (migration or "").lower()
|
||||
if any(word in lowered for word in FORBIDDEN):
|
||||
outcome.error = "forbidden statement"
|
||||
return outcome
|
||||
if lowered.count(";") > STATEMENT_LIMIT:
|
||||
outcome.error = "too many statements"
|
||||
return outcome
|
||||
if not lowered.strip():
|
||||
outcome.error = "empty migration"
|
||||
return outcome
|
||||
|
||||
try:
|
||||
con.executescript(migration)
|
||||
except sqlite3.Error as exc:
|
||||
# A migration that does not run scores what doing nothing scores. It is not an
|
||||
# error for the rollout — the reward is the signal.
|
||||
outcome.error = str(exc)[:120]
|
||||
return outcome
|
||||
|
||||
columns = _columns(con)
|
||||
outcome.schema_ok = TARGET_COLUMNS.issubset(columns) and "value_text" not in columns
|
||||
try:
|
||||
outcome.rows_after = con.execute("SELECT count(*) FROM readings").fetchone()[0]
|
||||
if outcome.schema_ok:
|
||||
got = {
|
||||
r[0]: (r[1], r[2])
|
||||
for r in con.execute("SELECT id, value_num, unit FROM readings")
|
||||
}
|
||||
for i, (_, num, unit) in enumerate(data):
|
||||
have = got.get(i + 1)
|
||||
if not have or have[0] is None:
|
||||
continue
|
||||
if abs(float(have[0]) - num) < TOLERANCE and (have[1] or "").strip() == unit:
|
||||
outcome.matched += 1
|
||||
except (sqlite3.Error, TypeError, ValueError) as exc:
|
||||
outcome.error = str(exc)[:120]
|
||||
return outcome
|
||||
@@ -0,0 +1,121 @@
|
||||
"""schema-migration: a migration is judged on the rows you did not look at.
|
||||
|
||||
`readings.value_text` holds numbers and units as one string. The migration has to become
|
||||
`value_num REAL` plus `unit TEXT`, drop the original column, and lose nothing. The agent
|
||||
sees five tidy rows; it is graded on forty it never sees, which carry thousands separators,
|
||||
negatives, decimals, a unit containing a digit and a slash, a value with no unit, and
|
||||
doubled internal whitespace.
|
||||
|
||||
The reward is the migration actually executing against a real database, so there is no
|
||||
judge and no rubric — SQLite either recomposes the value or it does not. Row count is
|
||||
scored separately from fidelity because otherwise `DELETE FROM readings` is a perfect
|
||||
migration: nothing left to be wrong about.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
from pydantic import Field
|
||||
|
||||
import verifiers.v1 as vf
|
||||
|
||||
from schema_migration.fixture import connect
|
||||
from schema_migration.run import measure
|
||||
|
||||
SYSTEM = """You are writing a SQLite migration.
|
||||
|
||||
The table is:
|
||||
CREATE TABLE readings (id INTEGER PRIMARY KEY, probe TEXT NOT NULL, value_text TEXT NOT NULL);
|
||||
|
||||
Migrate it so that `readings` has columns id, probe, value_num (REAL) and unit (TEXT), and
|
||||
no longer has value_text. `value_num` is the number in value_text; `unit` is what follows
|
||||
it, trimmed, or an empty string when there is no unit.
|
||||
|
||||
Return ONLY SQL in one ```sql code block. It is run with executescript(), so several
|
||||
statements are fine. Preserve every row and every id.
|
||||
|
||||
You are graded on rows you have not seen, from the same column. They are messier than the
|
||||
ones below."""
|
||||
|
||||
_BLOCK = re.compile(r"```(?:sql)?\s*\n(.*?)```", re.DOTALL)
|
||||
|
||||
|
||||
def parse_sql(reply: str) -> str:
|
||||
blocks = _BLOCK.findall(reply or "")
|
||||
return (blocks[-1] if blocks else (reply or "")).strip()
|
||||
|
||||
|
||||
class MigrationData(vf.TaskData):
|
||||
seed: int
|
||||
held_out: int
|
||||
|
||||
|
||||
class MigrationTask(vf.Task[MigrationData]):
|
||||
@vf.stop
|
||||
async def single_turn(self, trace: vf.Trace) -> bool:
|
||||
return trace.num_turns >= 1
|
||||
|
||||
@vf.metric
|
||||
async def scan(self, trace: vf.Trace) -> dict[str, float]:
|
||||
outcome = measure(self.data.seed, self.data.held_out, parse_sql(trace.last_reply))
|
||||
return {
|
||||
"schema": float(outcome.schema_ok),
|
||||
"fidelity": outcome.fidelity,
|
||||
"rows_kept": outcome.rows_kept,
|
||||
# Rows are a MULTIPLIER on fidelity, not a reward beside it. Scored separately
|
||||
# they paid 0.15 for adding two empty columns and touching nothing — a floor
|
||||
# above zero for near-inaction, which the house rules forbid. As a multiplier,
|
||||
# losing rows still zeroes an otherwise perfect migration and preserving them
|
||||
# on its own earns nothing.
|
||||
"integrity": outcome.fidelity * outcome.rows_kept,
|
||||
"clean": float(outcome.clean),
|
||||
"errored": float(bool(outcome.error)),
|
||||
}
|
||||
|
||||
@vf.reward(weight=0.30)
|
||||
async def schema(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("schema", 0.0)
|
||||
|
||||
@vf.reward(weight=0.45)
|
||||
async def integrity(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("integrity", 0.0)
|
||||
|
||||
@vf.reward(weight=0.25)
|
||||
async def gate(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("clean", 0.0)
|
||||
|
||||
|
||||
class MigrationConfig(vf.TasksetConfig):
|
||||
num_tasks: int = Field(48, ge=1)
|
||||
visible: int = Field(5, ge=1)
|
||||
held_out: int = Field(40, ge=1)
|
||||
|
||||
|
||||
class MigrationTaskset(vf.Taskset[MigrationTask, MigrationConfig]):
|
||||
SEED_BASE = 30_000
|
||||
|
||||
def load(self) -> list[MigrationTask]:
|
||||
tasks = []
|
||||
for i in range(self.config.num_tasks):
|
||||
seed = self.SEED_BASE + i
|
||||
# The visible slice is tidy on purpose; the graded one is not.
|
||||
con, data = connect(seed, self.config.visible, awkward=False)
|
||||
sample = "\n".join(
|
||||
f" {i + 1} | probe_{i:03d} | {text}" for i, (text, _, _) in enumerate(data)
|
||||
)
|
||||
con.close()
|
||||
tasks.append(
|
||||
MigrationTask(
|
||||
MigrationData(
|
||||
idx=i,
|
||||
name=f"readings-{seed}",
|
||||
prompt=f"Rows in the table:\n\n id | probe | value_text\n{sample}\n\nWrite the migration.",
|
||||
system_prompt=SYSTEM,
|
||||
seed=seed,
|
||||
held_out=self.config.held_out,
|
||||
),
|
||||
self.config.task,
|
||||
)
|
||||
)
|
||||
return tasks
|
||||
Reference in New Issue
Block a user