Three more environments, and probe.py to gate all four
canary-trap, fault-localisation and schema-migration, in the three domains the Environments Hub has nothing in: contamination, operations, data engineering. None is a port. Each is graded on a slice the model never read. canary-trap runs a probe set against a model that memorised the corpus AND one that learned the same facts from a reworded copy with every identifier regenerated. The obvious probe -- plant a GUID, ask for it back -- is sound and caps at 0.525 because it cannot see the paraphrase. It scored higher than that at first: facts with no distinct rewording had the paraphrase restating the corpus verbatim, so GUID probes caught it by accident. Every fact now carries a "core" token present in both wordings and in no public one, asserted at import, which is also what makes the ceiling reachable. fault-localisation puts the fault upstream and the noise downstream, the way incidents actually present. The broken service logs one line and goes quiet while its dependents log a dozen timeouts, so ranking by error volume answers the victim -- 0/64 incidents have the loudest service as the root, and probe.py asserts it rather than trusting the generator to stay that way. schema-migration executes the model's SQL against forty held-out rows carrying thousands separators, negatives, a unit with a slash in it and a value with no unit. Row preservation started as a reward beside fidelity and paid 0.15 for adding two empty columns and touching nothing; it is a multiplier now, so losing rows still zeroes a perfect migration and keeping them earns nothing on its own. probe.py is house rule 3 as an exit code: 0.000 for inaction and 1.000 for an oracle, or it returns 1. It loads the leaf modules through a namespace shim rather than the packages, so it runs without verifiers installed -- a self-check that needs the training stack is a self-check nobody runs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
@@ -1,30 +1,53 @@
|
||||
# Arena
|
||||
|
||||
RL environments from Lumbridge. Apache-2.0, built to the [`verifiers`](https://github.com/PrimeIntellect-ai/verifiers)
|
||||
v1 spec so they install and run on the Prime Intellect Environments Hub unchanged.
|
||||
v1 spec so the same wheel installs from the Prime Intellect Environments Hub and runs here
|
||||
unchanged.
|
||||
|
||||
An environment here is an eval you take the gradient of. That is not a rename — it changes
|
||||
what the artifact has to carry. A benchmark's score is its output, so a benchmark can be a
|
||||
number in a table. An environment's score is the input to the next weight update, so it has
|
||||
to be reproducible, gameable-proof, and traceable back to the code that produced it, or the
|
||||
model that trained on it inherits whatever was wrong with it silently.
|
||||
An environment is an eval you take the gradient of. That is not a rename — it changes what
|
||||
the artifact has to carry. A benchmark's score is its output, so a benchmark can be a number
|
||||
in a table. An environment's score is the input to the next weight update, so a defect in one
|
||||
is inherited by the model rather than printed beside it.
|
||||
|
||||
Three rules follow, and every environment in this repository is held to them:
|
||||
Most of what is on the Hub today is a benchmark that was ported into the spec. A port cannot
|
||||
generate a held-out slice, because its dataset is fixed — so it is contaminated the moment
|
||||
anyone trains on it. And an exact-match reward is single-sided by construction, so it teaches
|
||||
whatever maxes the string comparison. Nothing here is a port.
|
||||
|
||||
1. **Graded on what the model did not see.** Rules, plans and policies fitted to visible
|
||||
examples pass on visible examples. If a held-out slice cannot be generated, the
|
||||
## The house rules
|
||||
|
||||
Every environment in this repository is held to three, and ships the probe that proves it.
|
||||
|
||||
1. **Graded on what the model did not see.** If a held-out slice cannot be generated, the
|
||||
environment does not measure generalisation and does not ship.
|
||||
2. **No single-sided reward.** Every reward has a counterweight in the same episode. A
|
||||
reward you can max by doing the crude thing is a reward that teaches the crude thing.
|
||||
3. **A reachable ceiling and a zero floor, demonstrated.** Each environment ships the check
|
||||
that shows an oracle scores 1.0 and inaction scores 0.0. An unreachable component is
|
||||
dead weight in the gradient; a floor above zero pays for doing nothing.
|
||||
2. **No single-sided reward.** Every reward has a counterweight in the same episode. A reward
|
||||
you can max by doing the crude thing is a reward that teaches the crude thing.
|
||||
3. **A zero floor and a reachable ceiling, demonstrated.** Inaction scores 0.0 and an oracle
|
||||
scores 1.0, measured, in `probe.py`. An unreachable component is dead weight in the
|
||||
gradient; a floor above zero pays for doing nothing.
|
||||
|
||||
## Environments
|
||||
Rule 3 is not decoration. Every environment here was wrong the first time and the probe is
|
||||
what caught it — see the table below, and the git history.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| [`redaction-pressure`](environments/redaction_pressure) | Write redaction rules for a ticket corpus. Graded on the next records the generator would have produced. Every secret class is shape-matched to a decoy, so a rule keyed on shape earns on one and pays on the other. Reward is a scanner — no judge model, no runtime, milliseconds per rollout. |
|
||||
## The environments
|
||||
|
||||
| | What it measures | The trap |
|
||||
|---|---|---|
|
||||
| [`redaction-pressure`](environments/redaction_pressure) | Write redaction rules for a ticket corpus, run against the next records the generator would have produced. | Every secret class is shape-matched to a decoy: a live key beside a published test key, a card number beside a build id. A rule keyed on shape earns on one and pays on the other. |
|
||||
| [`canary-trap`](environments/canary_trap) | Write contamination probes that catch a model trained on your eval set. | Probes are run against a model that memorised the corpus **and** one that learned the same facts from a reworded copy. GUID-recall probes are sound and see only half of what is there. |
|
||||
| [`fault-localisation`](environments/fault_localisation) | Read an incident log and name the root cause, its class, and the line proving it. | The service that broke emits one line and goes quiet; its dependents emit a dozen timeouts. Ranking by error volume answers the victim, every time. |
|
||||
| [`schema-migration`](environments/schema_migration) | Split a text column into a number and a unit, executed against held-out rows. | The visible rows are tidy. The graded ones carry thousands separators, negatives, a unit containing a slash, and a value with no unit at all. |
|
||||
|
||||
## What the probes measure
|
||||
|
||||
Reward for the degenerate strategies and for an oracle, from `uv run python probe.py`:
|
||||
|
||||
| environment | inaction | crude maximiser | plausible attempt | oracle |
|
||||
|---|---|---|---|---|
|
||||
| `redaction-pressure` | 0.000 | 0.380 *(redact everything)* | 0.552 | **1.000** |
|
||||
| `canary-trap` | 0.000 | 0.000 *(public-knowledge probes)* | 0.525 *(GUID recall)* | **1.000** |
|
||||
| `fault-localisation` | 0.000 | 0.250 *(blame the loudest)* | 0.500 | **1.000** |
|
||||
| `schema-migration` | 0.000 | 0.000 *(add columns, touch nothing)* | 0.630 | **1.000** |
|
||||
|
||||
## Running one
|
||||
|
||||
@@ -32,10 +55,13 @@ Three rules follow, and every environment in this repository is held to them:
|
||||
uv run eval @ configs/redaction_pressure.toml --model <model-id>
|
||||
```
|
||||
|
||||
## Why these, and not benchmark reimplementations
|
||||
## Publishing
|
||||
|
||||
The Hub is filling with reimplementations, which are worth doing and are not what we have.
|
||||
What Lumbridge has is a set of tools that already do verifiable work — Bench's
|
||||
contamination gate, Forge's signed build chain — and those gates make good rewards
|
||||
precisely because they were built to be un-arguable about a machine. `redaction-pressure`
|
||||
is Forge's `redact` stage with the exit code turned into a gradient.
|
||||
The layout mirrors `verifiers`' own, so an environment goes to the Hub without a fork:
|
||||
|
||||
```bash
|
||||
prime env push redaction-pressure -v PUBLIC
|
||||
```
|
||||
|
||||
No model has been evaluated in any of these yet. That is a different sentence from having a
|
||||
leaderboard, and this file will say so until it is not true.
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
# canary-trap — see environments/canary_trap for what it measures.
|
||||
#
|
||||
# uv run eval @ configs/canary_trap.toml --model <model-id>
|
||||
num_tasks = 16
|
||||
num_rollouts = 2
|
||||
|
||||
[env.taskset]
|
||||
id = "canary-trap"
|
||||
|
||||
[env.agent.harness]
|
||||
id = "null" # pure-trace reward: one completion, no tools, no runtime
|
||||
@@ -0,0 +1,11 @@
|
||||
# fault-localisation — see environments/fault_localisation for what it measures.
|
||||
#
|
||||
# uv run eval @ configs/fault_localisation.toml --model <model-id>
|
||||
num_tasks = 16
|
||||
num_rollouts = 2
|
||||
|
||||
[env.taskset]
|
||||
id = "fault-localisation"
|
||||
|
||||
[env.agent.harness]
|
||||
id = "null" # pure-trace reward: one completion, no tools, no runtime
|
||||
@@ -0,0 +1,11 @@
|
||||
# schema-migration — see environments/schema_migration for what it measures.
|
||||
#
|
||||
# uv run eval @ configs/schema_migration.toml --model <model-id>
|
||||
num_tasks = 16
|
||||
num_rollouts = 2
|
||||
|
||||
[env.taskset]
|
||||
id = "schema-migration"
|
||||
|
||||
[env.agent.harness]
|
||||
id = "null" # pure-trace reward: one completion, no tools, no runtime
|
||||
@@ -0,0 +1,3 @@
|
||||
from canary_trap.taskset import CanaryTaskset
|
||||
|
||||
__all__ = ["CanaryTaskset"]
|
||||
@@ -0,0 +1,121 @@
|
||||
"""The three knowledge states a probe has to tell apart.
|
||||
|
||||
A contamination probe is only useful if it separates a model that has seen your eval set
|
||||
from one that has not. The trap is that the obvious probe — carry a GUID into the corpus
|
||||
and ask for it back — separates only the *verbatim* case. Training data gets reworded,
|
||||
deduplicated and paraphrased on the way into a model, and a model that learned every fact
|
||||
in your eval set from a reworded copy answers no GUID and is contaminated anyway.
|
||||
|
||||
So a task builds three knowledge states from one seed:
|
||||
|
||||
clean public facts only. Answering from here is not evidence of anything.
|
||||
verbatim public + the corpus exactly as written, GUIDs and all.
|
||||
paraphrase public + the same FACTS, reworded, with every GUID regenerated.
|
||||
|
||||
A probe set is scored on catching both contaminated states without firing on the clean
|
||||
one. Verbatim-only probes cap at half the detection score, and that ceiling is the whole
|
||||
lesson of the environment.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import random
|
||||
import string
|
||||
from dataclasses import dataclass
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Fact:
|
||||
"""One fact in three parts.
|
||||
|
||||
`canonical` is how the corpus states it. `alias` is how the paraphrase states it, and
|
||||
it shares no wording with `canonical` beyond `core`. `core` is the token a probe has to
|
||||
key on to see BOTH — and it is deliberately absent from public knowledge, so keying on
|
||||
it is also sound. Every fact therefore has an answer that scores 1.0, and none of them
|
||||
is the sentence the corpus actually contains.
|
||||
"""
|
||||
|
||||
subject: str
|
||||
attribute: str
|
||||
canonical: str
|
||||
alias: str
|
||||
core: str
|
||||
|
||||
|
||||
SUBJECTS = [
|
||||
Fact("the Lisbon relay", "throughput", "412 requests per second", "412 rps", "412"),
|
||||
Fact("the Osaka mirror", "operator", "Ines Baptiste", "I. Baptiste", "Baptiste"),
|
||||
Fact("the Tallinn cache", "eviction policy", "least-recently-used (LRU)", "LRU ordering", "LRU"),
|
||||
Fact("the Bogota gateway", "timeout", "1400 milliseconds", "1400 ms", "1400"),
|
||||
Fact("the Leeds shard", "replica count", "seven replicas", "seven copies", "seven"),
|
||||
Fact("the Dakar queue", "retry budget", "nine retries", "nine attempts", "nine"),
|
||||
]
|
||||
|
||||
# Facts a model could know without ever seeing our corpus. A probe whose answer lives here
|
||||
# fires on a clean model, which is what makes it a false positive rather than a probe.
|
||||
PUBLIC = [
|
||||
"TCP uses a three-way handshake.",
|
||||
"Postgres defaults to port 5432.",
|
||||
"HTTP 429 means too many requests.",
|
||||
"UTC does not observe daylight saving.",
|
||||
"SHA-256 emits 32 bytes.",
|
||||
"The capital of Portugal is Lisbon.",
|
||||
]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Knowledge:
|
||||
"""What one model can answer from. Membership is a substring test on `text`."""
|
||||
|
||||
name: str
|
||||
text: str
|
||||
|
||||
|
||||
def _check_facts() -> None:
|
||||
"""The environment's own floor-and-ceiling guarantee, asserted at import.
|
||||
|
||||
Every fact must have an answer that a probe can key on to see both contaminated states
|
||||
(`core` in canonical AND in alias) without accusing a clean model (`core` absent from
|
||||
public). Without this the ceiling is unreachable for some seeds and the reward quietly
|
||||
stops being solvable — which is exactly the failure the house rules exist to catch.
|
||||
"""
|
||||
public = " ".join(PUBLIC).lower()
|
||||
for fact in SUBJECTS:
|
||||
assert fact.core.lower() in fact.canonical.lower(), fact
|
||||
assert fact.core.lower() in fact.alias.lower(), fact
|
||||
assert fact.core.lower() not in public, f"{fact.core} collides with public knowledge"
|
||||
# And the paraphrase must not simply restate the corpus, or a verbatim probe would
|
||||
# catch it for free and the environment would teach nothing.
|
||||
assert fact.alias.lower() != fact.canonical.lower(), fact
|
||||
|
||||
|
||||
_check_facts()
|
||||
|
||||
|
||||
def _guid(rng: random.Random) -> str:
|
||||
hexd = "0123456789abcdef"
|
||||
return "-".join("".join(rng.choice(hexd) for _ in range(n)) for n in (8, 4, 4, 12))
|
||||
|
||||
|
||||
def build(seed: int, facts: int) -> tuple[str, dict[str, Knowledge]]:
|
||||
"""One corpus, and the three knowledge states built from it."""
|
||||
rng = random.Random(seed)
|
||||
picked = rng.sample(SUBJECTS, min(facts, len(SUBJECTS)))
|
||||
public_text = " ".join(PUBLIC)
|
||||
|
||||
verbatim_lines, paraphrase_lines = [], []
|
||||
for fact in picked:
|
||||
verbatim_lines.append(
|
||||
f"[{_guid(rng)}] Record for {fact.subject}: {fact.attribute} is {fact.canonical}."
|
||||
)
|
||||
# Same fact, different sentence, and a REGENERATED tag — the paraphrase remembers
|
||||
# what is true and has forgotten every string that carried it.
|
||||
paraphrase_lines.append(
|
||||
f"[{_guid(rng)}] Concerning {fact.subject}, its {fact.attribute} sits at {fact.alias}."
|
||||
)
|
||||
|
||||
corpus = "\n".join(verbatim_lines)
|
||||
return corpus, {
|
||||
"clean": Knowledge("clean", public_text),
|
||||
"verbatim": Knowledge("verbatim", public_text + "\n" + corpus),
|
||||
"paraphrase": Knowledge("paraphrase", public_text + "\n" + "\n".join(paraphrase_lines)),
|
||||
}
|
||||
@@ -0,0 +1,105 @@
|
||||
"""Scoring a probe set against the three knowledge states.
|
||||
|
||||
Two properties, and they pull against each other:
|
||||
|
||||
sound the probe does NOT answer on the clean model. An unsound probe accuses a model
|
||||
that has seen nothing, which is worse than no probe at all — Bench VOIDS a
|
||||
score on a contamination verdict, so a false positive destroys a real result.
|
||||
catching a sound probe answers on a contaminated model. Both contaminated states have
|
||||
to be caught, and the paraphrase is the one that costs: it kept every fact and
|
||||
not one of the strings, so a GUID-recall probe cannot see it.
|
||||
|
||||
Detection is scored per state rather than per probe, because catching the same state with
|
||||
six probes is not six times the evidence. Specificity is per probe, so shotgunning a
|
||||
hundred guesses to land one is paid for.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from canary_trap.corpus import Knowledge
|
||||
|
||||
MAX_PROBES = 12
|
||||
MIN_ANSWER_CHARS = 3
|
||||
CONTAMINATED = ("verbatim", "paraphrase")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Probe:
|
||||
question: str
|
||||
answer: str
|
||||
|
||||
|
||||
@dataclass
|
||||
class Outcome:
|
||||
sound: int = 0
|
||||
total: int = 0
|
||||
caught: set[str] = field(default_factory=set)
|
||||
rejected: list[str] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def specificity(self) -> float:
|
||||
"""Share of probes that do not accuse a clean model. Zero probes scores zero: an
|
||||
empty set has no false positives and also no evidence, and rewarding it would make
|
||||
returning nothing the safe play."""
|
||||
return self.sound / self.total if self.total else 0.0
|
||||
|
||||
@property
|
||||
def detection(self) -> float:
|
||||
return len(self.caught) / len(CONTAMINATED)
|
||||
|
||||
@property
|
||||
def clean_gate(self) -> bool:
|
||||
"""Every contaminated state caught, and not one probe firing on the clean model."""
|
||||
return self.detection == 1.0 and self.total > 0 and self.sound == self.total
|
||||
|
||||
|
||||
def _answers(knowledge: Knowledge, probe: Probe) -> bool:
|
||||
"""The model stands in for itself: it can produce an answer exactly when that answer is
|
||||
present in what it knows. Crude, and the crudeness is load-bearing — it makes the whole
|
||||
environment deterministic and free, and it still distinguishes recall of a string from
|
||||
recall of a fact, which is the only distinction being taught."""
|
||||
return probe.answer.lower() in knowledge.text.lower()
|
||||
|
||||
|
||||
def measure(states: dict[str, Knowledge], probes: list[Probe]) -> Outcome:
|
||||
outcome = Outcome()
|
||||
for probe in probes[:MAX_PROBES]:
|
||||
# A blank or near-blank answer is a substring of everything, which would read as a
|
||||
# probe that catches all three states. Rejected rather than scored as unsound, so
|
||||
# the reason shows up in the trace.
|
||||
if len(probe.answer.strip()) < MIN_ANSWER_CHARS:
|
||||
outcome.rejected.append(probe.answer[:24])
|
||||
outcome.total += 1
|
||||
continue
|
||||
outcome.total += 1
|
||||
if _answers(states["clean"], probe):
|
||||
continue # unsound: fires on a model that has seen nothing
|
||||
outcome.sound += 1
|
||||
for name in CONTAMINATED:
|
||||
if _answers(states[name], probe):
|
||||
outcome.caught.add(name)
|
||||
return outcome
|
||||
|
||||
|
||||
_BLOCK = re.compile(r"```(?:json)?\s*\n(.*?)```", re.DOTALL)
|
||||
|
||||
|
||||
def parse_probes(reply: str) -> list[Probe]:
|
||||
blocks = _BLOCK.findall(reply or "")
|
||||
raw = blocks[-1] if blocks else (reply or "")
|
||||
try:
|
||||
parsed = json.loads(raw.strip())
|
||||
except json.JSONDecodeError:
|
||||
return []
|
||||
if not isinstance(parsed, list):
|
||||
return []
|
||||
return [
|
||||
Probe(str(p.get("question", "")), p["answer"])
|
||||
for p in parsed
|
||||
if isinstance(p, dict) and isinstance(p.get("answer"), str)
|
||||
]
|
||||
@@ -0,0 +1,110 @@
|
||||
"""canary-trap: write contamination probes that survive a paraphrase.
|
||||
|
||||
The agent reads a corpus destined to become an eval set and returns probes that would
|
||||
reveal a model trained on it. Probes are then run against three stand-in models: one that
|
||||
has seen only public facts, one that memorised the corpus verbatim, and one that learned
|
||||
the same facts from a reworded copy with every identifier regenerated.
|
||||
|
||||
The obvious probe — ask for a GUID that was planted in the corpus — is sound and catches
|
||||
exactly one of the two contaminated models, so it ceilings at 0.5 detection. The probe that
|
||||
scores is one keyed on a FACT the paraphrase kept, whose answer does not also appear in
|
||||
public knowledge. Learning that difference is the environment.
|
||||
|
||||
This is Bench's contamination gate as a task. There, a canary hit does not annotate a score,
|
||||
it voids it — which is why a probe that fires on a clean model is the expensive failure here
|
||||
and specificity carries as much weight as detection.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pydantic import Field
|
||||
|
||||
import verifiers.v1 as vf
|
||||
|
||||
from canary_trap.corpus import build
|
||||
from canary_trap.scan import MAX_PROBES, Probe, measure, parse_probes
|
||||
|
||||
SYSTEM = f"""You are designing contamination probes for an evaluation corpus.
|
||||
|
||||
Return a JSON array of at most {MAX_PROBES} probes in one ```json code block. Each probe is
|
||||
an object with "question" (what you would ask a model) and "answer" (the exact string a
|
||||
model that has seen this corpus would produce).
|
||||
|
||||
Your probes are run against three models: one that has seen only public knowledge, one that
|
||||
memorised this corpus word for word, and one that learned the same facts from a reworded
|
||||
copy in which every bracketed identifier was regenerated.
|
||||
|
||||
A probe that the public-knowledge model can answer is worse than no probe — it accuses an
|
||||
innocent model. A probe only the word-for-word model can answer catches half of what is
|
||||
there.
|
||||
|
||||
Example shape:
|
||||
```json
|
||||
[{{"question": "What is the retry budget for the Dakar queue?", "answer": "three retries"}}]
|
||||
```"""
|
||||
|
||||
class CanaryData(vf.TaskData):
|
||||
seed: int
|
||||
facts: int
|
||||
|
||||
|
||||
class CanaryTask(vf.Task[CanaryData]):
|
||||
@vf.stop
|
||||
async def single_turn(self, trace: vf.Trace) -> bool:
|
||||
return trace.num_turns >= 1
|
||||
|
||||
@vf.metric
|
||||
async def scan(self, trace: vf.Trace) -> dict[str, float]:
|
||||
_, states = build(self.data.seed, self.data.facts)
|
||||
outcome = measure(states, parse_probes(trace.last_reply))
|
||||
return {
|
||||
"detection": outcome.detection,
|
||||
"specificity": outcome.specificity,
|
||||
"clean": float(outcome.clean_gate),
|
||||
"probes": float(outcome.total),
|
||||
"rejected": float(len(outcome.rejected)),
|
||||
# Reported separately because it is the finding the environment exists to
|
||||
# teach: a probe set can be perfectly sound and still blind to a paraphrase.
|
||||
"caught_paraphrase": float("paraphrase" in outcome.caught),
|
||||
}
|
||||
|
||||
@vf.reward(weight=0.35)
|
||||
async def detection(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("detection", 0.0)
|
||||
|
||||
@vf.reward(weight=0.35)
|
||||
async def specificity(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("specificity", 0.0)
|
||||
|
||||
@vf.reward(weight=0.30)
|
||||
async def gate(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("clean", 0.0)
|
||||
|
||||
|
||||
class CanaryConfig(vf.TasksetConfig):
|
||||
num_tasks: int = Field(48, ge=1)
|
||||
facts: int = Field(4, ge=1, le=6)
|
||||
|
||||
|
||||
class CanaryTaskset(vf.Taskset[CanaryTask, CanaryConfig]):
|
||||
SEED_BASE = 70_000
|
||||
|
||||
def load(self) -> list[CanaryTask]:
|
||||
tasks = []
|
||||
for i in range(self.config.num_tasks):
|
||||
seed = self.SEED_BASE + i
|
||||
corpus, _ = build(seed, self.config.facts)
|
||||
tasks.append(
|
||||
CanaryTask(
|
||||
CanaryData(
|
||||
idx=i,
|
||||
name=f"corpus-{seed}",
|
||||
prompt=f"The corpus:\n\n{corpus}\n\nWrite the probes.",
|
||||
system_prompt=SYSTEM,
|
||||
seed=seed,
|
||||
facts=self.config.facts,
|
||||
),
|
||||
self.config.task,
|
||||
)
|
||||
)
|
||||
return tasks
|
||||
@@ -0,0 +1,13 @@
|
||||
[project]
|
||||
name = "canary-trap"
|
||||
version = "0.1.0"
|
||||
description = "canary-trap — write contamination probes that survive a paraphrase, without accusing a clean model."
|
||||
requires-python = ">=3.11"
|
||||
dependencies = ["verifiers"]
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["canary_trap"]
|
||||
@@ -0,0 +1,3 @@
|
||||
from fault_localisation.taskset import FaultTaskset
|
||||
|
||||
__all__ = ["FaultTaskset"]
|
||||
@@ -0,0 +1,94 @@
|
||||
"""Generating one incident: a service graph, an injected fault, and the logs it produced.
|
||||
|
||||
The fault is placed upstream and the noise is emitted downstream, because that is how real
|
||||
incidents present and it is the one thing a log-reading agent has to get right. A service
|
||||
whose dependency has stalled logs timeouts, retries and 503s — loudly, once per request —
|
||||
while the service that actually broke logs a single line about its connection pool and then
|
||||
goes quiet. Ranking services by error volume therefore points at the victim every time.
|
||||
|
||||
Everything is derived from one seed, so a task is reproducible from its integer id and the
|
||||
answer key is generated alongside the logs rather than written by hand.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import random
|
||||
from dataclasses import dataclass
|
||||
|
||||
SERVICES = ["edge", "checkout", "ledger", "inventory", "pricing", "vault"]
|
||||
|
||||
# (fault class, the single line the ROOT service emits, what its callers emit)
|
||||
FAULTS = [
|
||||
("pool_exhausted", "connection pool exhausted (size=16, waiters=88)", "upstream timeout after 3000ms"),
|
||||
("disk_full", "write failed: no space left on device", "upstream returned 503"),
|
||||
("cert_expired", "TLS handshake failed: certificate expired", "upstream connection reset"),
|
||||
("deadlock", "transaction aborted: deadlock detected", "upstream timeout after 3000ms"),
|
||||
("oom_kill", "worker killed: cgroup memory limit exceeded", "upstream returned 502"),
|
||||
("clock_skew", "rejecting request: token nbf 41s in the future", "upstream returned 401"),
|
||||
]
|
||||
|
||||
|
||||
@dataclass
|
||||
class Incident:
|
||||
lines: list[str]
|
||||
"""Log lines, each prefixed with a stable `L07` style id the answer can cite."""
|
||||
root: str
|
||||
fault: str
|
||||
evidence: str
|
||||
"""The id of the one line that names the cause. Every other line is a symptom."""
|
||||
|
||||
|
||||
def build(seed: int) -> Incident:
|
||||
rng = random.Random(seed)
|
||||
root, *rest = rng.sample(SERVICES, 4)
|
||||
# The callers: they depend on root, so they are the ones that will look broken.
|
||||
callers = rest[:2]
|
||||
bystander = rest[2]
|
||||
fault, root_line, caller_line = rng.choice(FAULTS)
|
||||
|
||||
raw: list[tuple[int, str, str]] = []
|
||||
clock = rng.randint(0, 40)
|
||||
|
||||
# Ordinary traffic first, from everyone, so "the service with any errors" is not free.
|
||||
for _ in range(6):
|
||||
clock += rng.randint(1, 4)
|
||||
svc = rng.choice([root, *callers, bystander])
|
||||
raw.append((clock, svc, rng.choice([
|
||||
"handled request in 24ms", "cache hit", "healthcheck ok", "handled request in 61ms",
|
||||
])))
|
||||
|
||||
# The cause: one line, once, and then the root goes quiet.
|
||||
clock += rng.randint(1, 3)
|
||||
cause_at = clock
|
||||
raw.append((clock, root, root_line))
|
||||
|
||||
# The symptoms: loud, repeated, and in the wrong place.
|
||||
for _ in range(rng.randint(9, 14)):
|
||||
clock += rng.randint(1, 3)
|
||||
raw.append((clock, rng.choice(callers), caller_line))
|
||||
# A red herring the bystander emits on its own schedule, unrelated to the fault.
|
||||
clock += rng.randint(1, 2)
|
||||
raw.append((clock, bystander, "retrying scheduled job (attempt 2)"))
|
||||
|
||||
raw.sort(key=lambda r: r[0])
|
||||
lines, evidence = [], ""
|
||||
for i, (at, svc, text) in enumerate(raw):
|
||||
line_id = f"L{i:02d}"
|
||||
if at == cause_at and svc == root and text == root_line:
|
||||
evidence = line_id
|
||||
lines.append(f"{line_id} t+{at:03d}s {svc:<10} {text}")
|
||||
return Incident(lines=lines, root=root, fault=fault, evidence=evidence)
|
||||
|
||||
|
||||
def loudest(incident: Incident) -> str:
|
||||
"""The service with the most error lines — what ranking by volume would answer.
|
||||
|
||||
Exported because the probe uses it: if this ever starts coinciding with the root, the
|
||||
environment has stopped testing the thing it was built to test.
|
||||
"""
|
||||
counts: dict[str, int] = {}
|
||||
for line in incident.lines:
|
||||
svc = line.split()[2]
|
||||
if "upstream" in line:
|
||||
counts[svc] = counts.get(svc, 0) + 1
|
||||
return max(counts, key=lambda k: counts[k]) if counts else ""
|
||||
@@ -0,0 +1,119 @@
|
||||
"""fault-localisation: name what broke, not what is complaining.
|
||||
|
||||
One incident, one injected fault, and a log the fault's victims dominate. The service that
|
||||
actually failed emits a single line and falls silent; the two that depend on it emit a
|
||||
dozen timeouts between them. Ranking by error volume answers the loudest service, which is
|
||||
never the right one — the probe in this package asserts that, so the day the generator
|
||||
drifts into making them coincide the environment fails rather than quietly rewarding the
|
||||
heuristic it exists to punish.
|
||||
|
||||
Three answers are required and they are not redundant. The service can be guessed one time
|
||||
in four. The fault class can be read off a line without knowing which line matters. Citing
|
||||
the evidence line is what distinguishes having found the cause from having recognised the
|
||||
vocabulary, and it is why the gate needs all three rather than a majority.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
|
||||
from pydantic import Field
|
||||
|
||||
import verifiers.v1 as vf
|
||||
|
||||
from fault_localisation.incident import FAULTS, SERVICES, build
|
||||
|
||||
CLASSES = sorted(f[0] for f in FAULTS)
|
||||
|
||||
SYSTEM = f"""You are on call. Read the log and name the ROOT CAUSE.
|
||||
|
||||
Return one JSON object in a ```json code block with exactly three keys:
|
||||
"service" the service that actually failed, one of: {", ".join(SERVICES)}
|
||||
"fault" the failure class, one of: {", ".join(CLASSES)}
|
||||
"evidence" the id (e.g. "L07") of the single line that names the cause
|
||||
|
||||
The service that failed is not the service producing the most errors. Its dependents
|
||||
produce the errors; it produces one line and then stops."""
|
||||
|
||||
_BLOCK = re.compile(r"```(?:json)?\s*\n(.*?)```", re.DOTALL)
|
||||
|
||||
|
||||
def parse_answer(reply: str) -> dict[str, str]:
|
||||
blocks = _BLOCK.findall(reply or "")
|
||||
raw = blocks[-1] if blocks else (reply or "")
|
||||
try:
|
||||
parsed = json.loads(raw.strip())
|
||||
except json.JSONDecodeError:
|
||||
return {}
|
||||
if not isinstance(parsed, dict):
|
||||
return {}
|
||||
return {k: str(v).strip() for k, v in parsed.items() if isinstance(v, (str, int))}
|
||||
|
||||
|
||||
class FaultData(vf.TaskData):
|
||||
seed: int
|
||||
|
||||
|
||||
class FaultTask(vf.Task[FaultData]):
|
||||
@vf.stop
|
||||
async def single_turn(self, trace: vf.Trace) -> bool:
|
||||
return trace.num_turns >= 1
|
||||
|
||||
@vf.metric
|
||||
async def scan(self, trace: vf.Trace) -> dict[str, float]:
|
||||
incident = build(self.data.seed)
|
||||
answer = parse_answer(trace.last_reply)
|
||||
service = float(answer.get("service", "").lower() == incident.root)
|
||||
fault = float(answer.get("fault", "").lower() == incident.fault)
|
||||
evidence = float(answer.get("evidence", "").upper() == incident.evidence)
|
||||
return {
|
||||
"service": service,
|
||||
"fault": fault,
|
||||
"evidence": evidence,
|
||||
"clean": float(service and fault and evidence),
|
||||
}
|
||||
|
||||
@vf.reward(weight=0.30)
|
||||
async def service(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("service", 0.0)
|
||||
|
||||
@vf.reward(weight=0.25)
|
||||
async def fault(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("fault", 0.0)
|
||||
|
||||
@vf.reward(weight=0.25)
|
||||
async def evidence(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("evidence", 0.0)
|
||||
|
||||
@vf.reward(weight=0.20)
|
||||
async def gate(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("clean", 0.0)
|
||||
|
||||
|
||||
class FaultConfig(vf.TasksetConfig):
|
||||
num_tasks: int = Field(64, ge=1)
|
||||
|
||||
|
||||
class FaultTaskset(vf.Taskset[FaultTask, FaultConfig]):
|
||||
SEED_BASE = 50_000
|
||||
|
||||
def load(self) -> list[FaultTask]:
|
||||
tasks = []
|
||||
for i in range(self.config.num_tasks):
|
||||
seed = self.SEED_BASE + i
|
||||
incident = build(seed)
|
||||
log = "\n".join(incident.lines)
|
||||
tasks.append(
|
||||
FaultTask(
|
||||
FaultData(
|
||||
idx=i,
|
||||
name=f"incident-{seed}",
|
||||
prompt=f"```\n{log}\n```\n\nName the root cause.",
|
||||
system_prompt=SYSTEM,
|
||||
seed=seed,
|
||||
),
|
||||
self.config.task,
|
||||
)
|
||||
)
|
||||
return tasks
|
||||
@@ -0,0 +1,13 @@
|
||||
[project]
|
||||
name = "fault-localisation"
|
||||
version = "0.1.0"
|
||||
description = "fault-localisation — name the service that broke, not the ones complaining about it."
|
||||
requires-python = ">=3.11"
|
||||
dependencies = ["verifiers"]
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["fault_localisation"]
|
||||
@@ -23,6 +23,9 @@ otherwise hang a rollout until the harness killed it.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
import regex
|
||||
@@ -134,3 +137,24 @@ def measure(records: list[Record], rules: list[Rule]) -> Outcome:
|
||||
total += len(innocent)
|
||||
lost += sum(1 for w in innocent if w not in redacted)
|
||||
return Outcome(residual, secrets, collateral, decoys, lost, total, invalid)
|
||||
|
||||
|
||||
_BLOCK = re.compile(r"```(?:json)?\s*\n(.*?)```", re.DOTALL)
|
||||
|
||||
|
||||
def parse_rules(reply: str) -> list[Rule]:
|
||||
"""The last JSON array in the reply. A reply with no parsable ruleset is an empty
|
||||
ruleset, not an error — it scores what doing nothing scores."""
|
||||
candidates = _BLOCK.findall(reply or "")
|
||||
raw = candidates[-1] if candidates else (reply or "")
|
||||
try:
|
||||
parsed = json.loads(raw.strip())
|
||||
except json.JSONDecodeError:
|
||||
return []
|
||||
if not isinstance(parsed, list):
|
||||
return []
|
||||
rules = []
|
||||
for item in parsed:
|
||||
if isinstance(item, dict) and isinstance(item.get("pattern"), str):
|
||||
rules.append(Rule(item["pattern"], str(item.get("replacement", ""))))
|
||||
return rules
|
||||
|
||||
@@ -21,7 +21,6 @@ and grades in milliseconds. Deterministic from the task seed.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from typing import ClassVar
|
||||
|
||||
@@ -30,7 +29,7 @@ from pydantic import Field
|
||||
import verifiers.v1 as vf
|
||||
|
||||
from redaction_pressure.corpus import build_slices
|
||||
from redaction_pressure.scan import MAX_RULES, Rule, measure
|
||||
from redaction_pressure.scan import MAX_RULES, Rule, measure, parse_rules
|
||||
|
||||
SYSTEM = f"""You are writing redaction rules for a support-ticket corpus.
|
||||
|
||||
@@ -48,27 +47,6 @@ Example shape:
|
||||
[{{"pattern": "\\\\bfoo-\\\\d+\\\\b", "replacement": "[REF]"}}]
|
||||
```"""
|
||||
|
||||
_BLOCK = re.compile(r"```(?:json)?\s*\n(.*?)```", re.DOTALL)
|
||||
|
||||
|
||||
def parse_rules(reply: str) -> list[Rule]:
|
||||
"""The last JSON array in the reply. A reply with no parsable ruleset is an empty
|
||||
ruleset, not an error — it scores what doing nothing scores."""
|
||||
candidates = _BLOCK.findall(reply or "")
|
||||
raw = candidates[-1] if candidates else (reply or "")
|
||||
try:
|
||||
parsed = json.loads(raw.strip())
|
||||
except json.JSONDecodeError:
|
||||
return []
|
||||
if not isinstance(parsed, list):
|
||||
return []
|
||||
rules = []
|
||||
for item in parsed:
|
||||
if isinstance(item, dict) and isinstance(item.get("pattern"), str):
|
||||
rules.append(Rule(item["pattern"], str(item.get("replacement", ""))))
|
||||
return rules
|
||||
|
||||
|
||||
class RedactionData(vf.TaskData):
|
||||
seed: int
|
||||
"""Rebuilds both slices exactly; the held-out records are never serialized here."""
|
||||
|
||||
Generated
-3492
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,13 @@
|
||||
[project]
|
||||
name = "schema-migration"
|
||||
version = "0.1.0"
|
||||
description = "schema-migration — split a text column into a number and a unit, graded on rows you never saw."
|
||||
requires-python = ">=3.11"
|
||||
dependencies = ["verifiers"]
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["schema_migration"]
|
||||
@@ -0,0 +1,3 @@
|
||||
from schema_migration.taskset import MigrationTaskset
|
||||
|
||||
__all__ = ["MigrationTaskset"]
|
||||
@@ -0,0 +1,69 @@
|
||||
"""The table before the migration, and the rows the migration is graded on.
|
||||
|
||||
One `readings` table carries a `value_text` column that should never have been text:
|
||||
"1400 ms", "412 rps", "-3.5 degC". The migration has to split it into a real number and a
|
||||
unit and drop the original column, without losing a row or rounding a value.
|
||||
|
||||
The visible rows are deliberately tidy. The held-out rows are not: they carry thousands
|
||||
separators, negatives, decimals, a unit with a digit in it, a value with no unit at all,
|
||||
and one with extra internal whitespace. A migration written by pattern-matching the five
|
||||
rows on screen parses about half of them, which is the entire point of grading on rows the
|
||||
author never read.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import random
|
||||
import sqlite3
|
||||
|
||||
TIDY = [
|
||||
("1400 ms", 1400.0, "ms"),
|
||||
("412 rps", 412.0, "rps"),
|
||||
("16 conns", 16.0, "conns"),
|
||||
("88 waiters", 88.0, "waiters"),
|
||||
("7 replicas", 7.0, "replicas"),
|
||||
]
|
||||
|
||||
# Every awkward shape a real column of this kind contains.
|
||||
AWKWARD = [
|
||||
("1,400 ms", 1400.0, "ms"), # thousands separator
|
||||
("-3.5 degC", -3.5, "degC"), # negative, decimal
|
||||
("0.25 s", 0.25, "s"), # leading zero decimal
|
||||
("64 KiB/s", 64.0, "KiB/s"), # unit containing a digit and a slash
|
||||
("120", 120.0, ""), # no unit at all
|
||||
("55 rps", 55.0, "rps"), # doubled internal space
|
||||
("2,048 MiB", 2048.0, "MiB"),
|
||||
("-12 ms", -12.0, "ms"),
|
||||
("3.0 x", 3.0, "x"),
|
||||
("1,000,000 rows", 1000000.0, "rows"),
|
||||
]
|
||||
|
||||
SOURCE_DDL = """
|
||||
CREATE TABLE readings (
|
||||
id INTEGER PRIMARY KEY,
|
||||
probe TEXT NOT NULL,
|
||||
value_text TEXT NOT NULL
|
||||
);
|
||||
"""
|
||||
|
||||
TARGET_COLUMNS = {"id", "probe", "value_num", "unit"}
|
||||
|
||||
|
||||
def rows(seed: int, count: int, awkward: bool) -> list[tuple[str, float, str]]:
|
||||
rng = random.Random(seed)
|
||||
pool = AWKWARD + TIDY if awkward else TIDY
|
||||
return [pool[rng.randrange(len(pool))] for _ in range(count)]
|
||||
|
||||
|
||||
def connect(seed: int, count: int, awkward: bool) -> tuple[sqlite3.Connection, list[tuple]]:
|
||||
"""A fresh in-memory database holding one slice. Nothing touches disk, so an agent's
|
||||
SQL cannot reach anything — the environment is the sandbox."""
|
||||
con = sqlite3.connect(":memory:")
|
||||
con.executescript(SOURCE_DDL)
|
||||
data = rows(seed, count, awkward)
|
||||
con.executemany(
|
||||
"INSERT INTO readings (id, probe, value_text) VALUES (?, ?, ?)",
|
||||
[(i + 1, f"probe_{i:03d}", text) for i, (text, _, _) in enumerate(data)],
|
||||
)
|
||||
con.commit()
|
||||
return con, data
|
||||
@@ -0,0 +1,100 @@
|
||||
"""Running the agent's migration, and measuring what survived it.
|
||||
|
||||
Four things are checked, and they are in tension by construction:
|
||||
|
||||
schema the target columns exist and `value_text` is gone. A migration that adds the
|
||||
new columns and leaves the old one is not a migration, it is a copy.
|
||||
fidelity per row, does (value_num, unit) recompose to the original value. This is the
|
||||
expensive one and the only one that punishes a parser fitted to the visible
|
||||
rows.
|
||||
rows the row count is unchanged. Without it, `DELETE FROM readings` scores perfect
|
||||
fidelity over an empty table — vacuously, since there is nothing left to be
|
||||
wrong about.
|
||||
gate all three, exactly. `forge verify`'s exit code: a migration is correct or it
|
||||
is not run in production.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlite3
|
||||
from dataclasses import dataclass
|
||||
|
||||
from schema_migration.fixture import TARGET_COLUMNS, connect
|
||||
|
||||
STATEMENT_LIMIT = 40
|
||||
# ATTACH would reach outside the in-memory database; the rest cannot, but there is no
|
||||
# reason for a schema migration to use any of them.
|
||||
FORBIDDEN = ("attach", "pragma", "vacuum")
|
||||
TOLERANCE = 1e-9
|
||||
|
||||
|
||||
@dataclass
|
||||
class Outcome:
|
||||
schema_ok: bool = False
|
||||
matched: int = 0
|
||||
graded: int = 0
|
||||
rows_before: int = 0
|
||||
rows_after: int = 0
|
||||
error: str = ""
|
||||
|
||||
@property
|
||||
def fidelity(self) -> float:
|
||||
return self.matched / self.graded if self.graded else 0.0
|
||||
|
||||
@property
|
||||
def rows_kept(self) -> float:
|
||||
if not self.rows_before:
|
||||
return 0.0
|
||||
# Inserting extra rows is as wrong as dropping them, so this is a distance, not a ratio.
|
||||
return max(0.0, 1.0 - abs(self.rows_after - self.rows_before) / self.rows_before)
|
||||
|
||||
@property
|
||||
def clean(self) -> bool:
|
||||
return self.schema_ok and self.graded > 0 and self.matched == self.graded and self.rows_after == self.rows_before
|
||||
|
||||
|
||||
def _columns(con: sqlite3.Connection) -> set[str]:
|
||||
return {r[1] for r in con.execute("PRAGMA table_info(readings)")}
|
||||
|
||||
|
||||
def measure(seed: int, count: int, migration: str) -> Outcome:
|
||||
con, data = connect(seed, count, awkward=True)
|
||||
outcome = Outcome(rows_before=len(data), graded=len(data))
|
||||
|
||||
lowered = (migration or "").lower()
|
||||
if any(word in lowered for word in FORBIDDEN):
|
||||
outcome.error = "forbidden statement"
|
||||
return outcome
|
||||
if lowered.count(";") > STATEMENT_LIMIT:
|
||||
outcome.error = "too many statements"
|
||||
return outcome
|
||||
if not lowered.strip():
|
||||
outcome.error = "empty migration"
|
||||
return outcome
|
||||
|
||||
try:
|
||||
con.executescript(migration)
|
||||
except sqlite3.Error as exc:
|
||||
# A migration that does not run scores what doing nothing scores. It is not an
|
||||
# error for the rollout — the reward is the signal.
|
||||
outcome.error = str(exc)[:120]
|
||||
return outcome
|
||||
|
||||
columns = _columns(con)
|
||||
outcome.schema_ok = TARGET_COLUMNS.issubset(columns) and "value_text" not in columns
|
||||
try:
|
||||
outcome.rows_after = con.execute("SELECT count(*) FROM readings").fetchone()[0]
|
||||
if outcome.schema_ok:
|
||||
got = {
|
||||
r[0]: (r[1], r[2])
|
||||
for r in con.execute("SELECT id, value_num, unit FROM readings")
|
||||
}
|
||||
for i, (_, num, unit) in enumerate(data):
|
||||
have = got.get(i + 1)
|
||||
if not have or have[0] is None:
|
||||
continue
|
||||
if abs(float(have[0]) - num) < TOLERANCE and (have[1] or "").strip() == unit:
|
||||
outcome.matched += 1
|
||||
except (sqlite3.Error, TypeError, ValueError) as exc:
|
||||
outcome.error = str(exc)[:120]
|
||||
return outcome
|
||||
@@ -0,0 +1,121 @@
|
||||
"""schema-migration: a migration is judged on the rows you did not look at.
|
||||
|
||||
`readings.value_text` holds numbers and units as one string. The migration has to become
|
||||
`value_num REAL` plus `unit TEXT`, drop the original column, and lose nothing. The agent
|
||||
sees five tidy rows; it is graded on forty it never sees, which carry thousands separators,
|
||||
negatives, decimals, a unit containing a digit and a slash, a value with no unit, and
|
||||
doubled internal whitespace.
|
||||
|
||||
The reward is the migration actually executing against a real database, so there is no
|
||||
judge and no rubric — SQLite either recomposes the value or it does not. Row count is
|
||||
scored separately from fidelity because otherwise `DELETE FROM readings` is a perfect
|
||||
migration: nothing left to be wrong about.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
from pydantic import Field
|
||||
|
||||
import verifiers.v1 as vf
|
||||
|
||||
from schema_migration.fixture import connect
|
||||
from schema_migration.run import measure
|
||||
|
||||
SYSTEM = """You are writing a SQLite migration.
|
||||
|
||||
The table is:
|
||||
CREATE TABLE readings (id INTEGER PRIMARY KEY, probe TEXT NOT NULL, value_text TEXT NOT NULL);
|
||||
|
||||
Migrate it so that `readings` has columns id, probe, value_num (REAL) and unit (TEXT), and
|
||||
no longer has value_text. `value_num` is the number in value_text; `unit` is what follows
|
||||
it, trimmed, or an empty string when there is no unit.
|
||||
|
||||
Return ONLY SQL in one ```sql code block. It is run with executescript(), so several
|
||||
statements are fine. Preserve every row and every id.
|
||||
|
||||
You are graded on rows you have not seen, from the same column. They are messier than the
|
||||
ones below."""
|
||||
|
||||
_BLOCK = re.compile(r"```(?:sql)?\s*\n(.*?)```", re.DOTALL)
|
||||
|
||||
|
||||
def parse_sql(reply: str) -> str:
|
||||
blocks = _BLOCK.findall(reply or "")
|
||||
return (blocks[-1] if blocks else (reply or "")).strip()
|
||||
|
||||
|
||||
class MigrationData(vf.TaskData):
|
||||
seed: int
|
||||
held_out: int
|
||||
|
||||
|
||||
class MigrationTask(vf.Task[MigrationData]):
|
||||
@vf.stop
|
||||
async def single_turn(self, trace: vf.Trace) -> bool:
|
||||
return trace.num_turns >= 1
|
||||
|
||||
@vf.metric
|
||||
async def scan(self, trace: vf.Trace) -> dict[str, float]:
|
||||
outcome = measure(self.data.seed, self.data.held_out, parse_sql(trace.last_reply))
|
||||
return {
|
||||
"schema": float(outcome.schema_ok),
|
||||
"fidelity": outcome.fidelity,
|
||||
"rows_kept": outcome.rows_kept,
|
||||
# Rows are a MULTIPLIER on fidelity, not a reward beside it. Scored separately
|
||||
# they paid 0.15 for adding two empty columns and touching nothing — a floor
|
||||
# above zero for near-inaction, which the house rules forbid. As a multiplier,
|
||||
# losing rows still zeroes an otherwise perfect migration and preserving them
|
||||
# on its own earns nothing.
|
||||
"integrity": outcome.fidelity * outcome.rows_kept,
|
||||
"clean": float(outcome.clean),
|
||||
"errored": float(bool(outcome.error)),
|
||||
}
|
||||
|
||||
@vf.reward(weight=0.30)
|
||||
async def schema(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("schema", 0.0)
|
||||
|
||||
@vf.reward(weight=0.45)
|
||||
async def integrity(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("integrity", 0.0)
|
||||
|
||||
@vf.reward(weight=0.25)
|
||||
async def gate(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("clean", 0.0)
|
||||
|
||||
|
||||
class MigrationConfig(vf.TasksetConfig):
|
||||
num_tasks: int = Field(48, ge=1)
|
||||
visible: int = Field(5, ge=1)
|
||||
held_out: int = Field(40, ge=1)
|
||||
|
||||
|
||||
class MigrationTaskset(vf.Taskset[MigrationTask, MigrationConfig]):
|
||||
SEED_BASE = 30_000
|
||||
|
||||
def load(self) -> list[MigrationTask]:
|
||||
tasks = []
|
||||
for i in range(self.config.num_tasks):
|
||||
seed = self.SEED_BASE + i
|
||||
# The visible slice is tidy on purpose; the graded one is not.
|
||||
con, data = connect(seed, self.config.visible, awkward=False)
|
||||
sample = "\n".join(
|
||||
f" {i + 1} | probe_{i:03d} | {text}" for i, (text, _, _) in enumerate(data)
|
||||
)
|
||||
con.close()
|
||||
tasks.append(
|
||||
MigrationTask(
|
||||
MigrationData(
|
||||
idx=i,
|
||||
name=f"readings-{seed}",
|
||||
prompt=f"Rows in the table:\n\n id | probe | value_text\n{sample}\n\nWrite the migration.",
|
||||
system_prompt=SYSTEM,
|
||||
seed=seed,
|
||||
held_out=self.config.held_out,
|
||||
),
|
||||
self.config.task,
|
||||
)
|
||||
)
|
||||
return tasks
|
||||
@@ -0,0 +1,173 @@
|
||||
"""House rule 3, executable: inaction scores 0.0 and an oracle scores 1.0.
|
||||
|
||||
Run this before believing any reward in this repository. Every environment here was wrong
|
||||
the first time and this is what caught it:
|
||||
|
||||
redaction-pressure scored SURVIVORS — secrets caught, decoys kept, prose kept. An empty
|
||||
ruleset destroys nothing so it keeps everything, and it tied the best
|
||||
real attempt at 0.500. Counting removals instead put inaction at zero.
|
||||
canary-trap let the paraphrase restate the corpus verbatim for facts that had no
|
||||
distinct rewording, so GUID probes caught it by accident and the
|
||||
environment taught nothing. Every fact now has a `core` token that is
|
||||
in both wordings and in no public one, asserted at import.
|
||||
schema-migration paid 0.15 for adding two empty columns and touching nothing. Row
|
||||
preservation is a multiplier on fidelity now, not a reward beside it.
|
||||
|
||||
Exit code is the API: 1 if any environment's floor is above zero or its ceiling is
|
||||
unreachable.
|
||||
|
||||
uv run --with regex python probe.py
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
import types
|
||||
from pathlib import Path
|
||||
|
||||
ENVS = Path(__file__).parent / "environments"
|
||||
|
||||
# Each package's __init__ imports its taskset, which imports verifiers — a heavy dependency
|
||||
# this file does not need and should not require. A self-check that only runs once the
|
||||
# training stack is installed is a self-check nobody runs. So the package name is registered
|
||||
# as a namespace pointing at the source directory, and the leaf modules are imported under
|
||||
# it directly: intra-package imports still resolve, __init__ never executes.
|
||||
for _pkg in ("redaction_pressure", "canary_trap", "fault_localisation", "schema_migration"):
|
||||
_shim = types.ModuleType(_pkg)
|
||||
_shim.__path__ = [str(ENVS / _pkg / _pkg)]
|
||||
sys.modules[_pkg] = _shim
|
||||
|
||||
TASKS = 24
|
||||
|
||||
|
||||
def redaction() -> dict[str, float]:
|
||||
from redaction_pressure.corpus import build_slices
|
||||
from redaction_pressure.scan import measure, parse_rules
|
||||
|
||||
def run(rules: list[dict]) -> float:
|
||||
total = 0.0
|
||||
for i in range(TASKS):
|
||||
_, held = build_slices(90_000 + i, 4, 12)
|
||||
o = measure(held, parse_rules("```json\n" + json.dumps(rules) + "\n```"))
|
||||
total += 0.35 * o.recall + 0.35 * o.precision + 0.30 * float(o.clean)
|
||||
return total / TASKS
|
||||
|
||||
oracle = [
|
||||
{"pattern": r"sk-live-\w+", "replacement": "[KEY]"},
|
||||
{"pattern": r"[\w.]+@(?!lumbridgecorp\.com)[\w.]+\.com", "replacement": "[EMAIL]"},
|
||||
{"pattern": r"\+1-\d{3}-\d{3}-\d{4}", "replacement": "[PHONE]"},
|
||||
{"pattern": r"(?<!ORD-)\b\d{4}-\d{4}-\d{4}-\d{4}\b", "replacement": "[CARD]"},
|
||||
{"pattern": r"\b[\w-]+\.internal\.lumbridge\b", "replacement": "[HOST]"},
|
||||
{"pattern": r"\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b", "replacement": "[GUID]"},
|
||||
{"pattern": r"\b(Dana|Priya|Marcus|Ines|Tomas|Rui|Nadia|Owen) (Okafor|Lindqvist|Baptiste|Moreau|Sato|Ahmed|Vance|Duarte)\b", "replacement": "[PERSON]"},
|
||||
]
|
||||
return {
|
||||
"inaction": run([]),
|
||||
"crude": run([{"pattern": r"\S+", "replacement": "[X]"}]),
|
||||
"plausible": run(oracle[:4]),
|
||||
"oracle": run(oracle),
|
||||
}
|
||||
|
||||
|
||||
def canary() -> dict[str, float]:
|
||||
from canary_trap.corpus import SUBJECTS, build
|
||||
from canary_trap.scan import measure, parse_probes
|
||||
|
||||
core = {f.subject: f.core for f in SUBJECTS}
|
||||
pat = re.compile(r"Record for ([^:]+): [a-z ]+is ([^.]+)\.")
|
||||
|
||||
def run(fn) -> float:
|
||||
total = 0.0
|
||||
for i in range(TASKS):
|
||||
corpus, states = build(70_000 + i, 4)
|
||||
o = measure(states, parse_probes("```json\n" + json.dumps(fn(corpus)) + "\n```"))
|
||||
total += 0.35 * o.detection + 0.35 * o.specificity + 0.30 * float(o.clean_gate)
|
||||
return total / TASKS
|
||||
|
||||
return {
|
||||
"inaction": run(lambda c: []),
|
||||
"crude": run(lambda c: [{"question": "?", "answer": "three-way"}, {"question": "?", "answer": "5432"}]),
|
||||
"plausible": run(lambda c: [{"question": "id?", "answer": t} for t in re.findall(r"\[([0-9a-f-]+)\]", c)]),
|
||||
"oracle": run(lambda c: [{"question": f"{s}?", "answer": core[s]} for s, _ in pat.findall(c)]),
|
||||
}
|
||||
|
||||
|
||||
def fault() -> dict[str, float]:
|
||||
from fault_localisation.incident import build, loudest
|
||||
|
||||
def run(fn) -> float:
|
||||
total = 0.0
|
||||
for i in range(TASKS):
|
||||
inc = build(50_000 + i)
|
||||
a = fn(inc)
|
||||
s = float(a.get("service", "") == inc.root)
|
||||
f = float(a.get("fault", "") == inc.fault)
|
||||
e = float(a.get("evidence", "") == inc.evidence)
|
||||
total += 0.30 * s + 0.25 * f + 0.25 * e + 0.20 * float(s and f and e)
|
||||
return total / TASKS
|
||||
|
||||
# The environment's own premise: blaming the loudest service must never be right.
|
||||
assert all(loudest(build(50_000 + i)) != build(50_000 + i).root for i in range(TASKS)), \
|
||||
"the loudest service is the root cause — the environment is rewarding the heuristic it punishes"
|
||||
return {
|
||||
"inaction": run(lambda inc: {}),
|
||||
"crude": run(lambda inc: {"service": loudest(inc), "fault": inc.fault, "evidence": "L00"}),
|
||||
"plausible": run(lambda inc: {"service": loudest(inc), "fault": inc.fault, "evidence": inc.evidence}),
|
||||
"oracle": run(lambda inc: {"service": inc.root, "fault": inc.fault, "evidence": inc.evidence}),
|
||||
}
|
||||
|
||||
|
||||
def migration() -> dict[str, float]:
|
||||
from schema_migration.run import measure
|
||||
|
||||
def run(sql: str) -> float:
|
||||
total = 0.0
|
||||
for i in range(TASKS):
|
||||
o = measure(30_000 + i, 40, sql)
|
||||
total += 0.30 * float(o.schema_ok) + 0.45 * (o.fidelity * o.rows_kept) + 0.25 * float(o.clean)
|
||||
return total / TASKS
|
||||
|
||||
naive = ("ALTER TABLE readings ADD COLUMN value_num REAL;"
|
||||
"ALTER TABLE readings ADD COLUMN unit TEXT;"
|
||||
"UPDATE readings SET value_num=CAST(substr(value_text,1,instr(value_text,' ')-1) AS REAL),"
|
||||
" unit=substr(value_text,instr(value_text,' ')+1);"
|
||||
"ALTER TABLE readings DROP COLUMN value_text;")
|
||||
clean = "replace(value_text,',','')"
|
||||
oracle = (f"ALTER TABLE readings ADD COLUMN value_num REAL;"
|
||||
f"ALTER TABLE readings ADD COLUMN unit TEXT;"
|
||||
f"UPDATE readings SET"
|
||||
f" value_num=CAST(CASE WHEN instr({clean},' ')>0"
|
||||
f" THEN substr({clean},1,instr({clean},' ')-1) ELSE {clean} END AS REAL),"
|
||||
f" unit=CASE WHEN instr({clean},' ')>0"
|
||||
f" THEN trim(substr({clean},instr({clean},' ')+1)) ELSE '' END;"
|
||||
f"ALTER TABLE readings DROP COLUMN value_text;")
|
||||
return {
|
||||
"inaction": run(""),
|
||||
"crude": run("ALTER TABLE readings ADD COLUMN value_num REAL; ALTER TABLE readings ADD COLUMN unit TEXT;"),
|
||||
"plausible": run(naive),
|
||||
"oracle": run(oracle),
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
results = {
|
||||
"redaction-pressure": redaction(),
|
||||
"canary-trap": canary(),
|
||||
"fault-localisation": fault(),
|
||||
"schema-migration": migration(),
|
||||
}
|
||||
print(f"{'environment':22}{'inaction':>10}{'crude':>9}{'plausible':>11}{'oracle':>9} verdict")
|
||||
failed = False
|
||||
for name, r in results.items():
|
||||
ok = r["inaction"] <= 1e-9 and r["oracle"] >= 1.0 - 1e-9
|
||||
failed |= not ok
|
||||
print(f"{name:22}{r['inaction']:10.3f}{r['crude']:9.3f}{r['plausible']:11.3f}"
|
||||
f"{r['oracle']:9.3f} {'ok' if ok else 'FAILS RULE 3'}")
|
||||
return 1 if failed else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Reference in New Issue
Block a user