canary-trap, fault-localisation and schema-migration, in the three domains the
Environments Hub has nothing in: contamination, operations, data engineering.
None is a port. Each is graded on a slice the model never read.
canary-trap runs a probe set against a model that memorised the corpus AND one
that learned the same facts from a reworded copy with every identifier
regenerated. The obvious probe -- plant a GUID, ask for it back -- is sound and
caps at 0.525 because it cannot see the paraphrase. It scored higher than that
at first: facts with no distinct rewording had the paraphrase restating the
corpus verbatim, so GUID probes caught it by accident. Every fact now carries a
"core" token present in both wordings and in no public one, asserted at import,
which is also what makes the ceiling reachable.
fault-localisation puts the fault upstream and the noise downstream, the way
incidents actually present. The broken service logs one line and goes quiet
while its dependents log a dozen timeouts, so ranking by error volume answers
the victim -- 0/64 incidents have the loudest service as the root, and probe.py
asserts it rather than trusting the generator to stay that way.
schema-migration executes the model's SQL against forty held-out rows carrying
thousands separators, negatives, a unit with a slash in it and a value with no
unit. Row preservation started as a reward beside fidelity and paid 0.15 for
adding two empty columns and touching nothing; it is a multiplier now, so
losing rows still zeroes a perfect migration and keeping them earns nothing on
its own.
probe.py is house rule 3 as an exit code: 0.000 for inaction and 1.000 for an
oracle, or it returns 1. It loads the leaf modules through a namespace shim
rather than the packages, so it runs without verifiers installed -- a self-check
that needs the training stack is a self-check nobody runs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An environment is an eval you take the gradient of, so it carries obligations a
benchmark does not: a held-out slice, a counterweighted reward, and a demonstrated
0.0 floor and 1.0 ceiling. README states the three as house rules.
redaction-pressure is Forge's redact stage as a task. The first reward design scored
survivors — secrets caught, decoys kept, prose kept — and an empty ruleset tied the
best real attempt at 0.500, because destroying nothing keeps everything. Counting
removals instead puts inaction at 0.000, shredding at 0.380, and an oracle at 1.000.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
karti
