Alert Triage: environment #2, built end to end by the pipeline
The first environment shipped through .claude/workflows/new-environment.js: specification, three adversarial reviews (all 'fixable', none fatal), the Python environment, the TypeScript port, captured rollouts, and the demo page. Eleven agents, no errors. The proof that the platform scales is one line long. Alert Triage has a completely different shape from Word Five — JSON actions, priced lookups, an analyst screen instead of a grid — and the only change under src/components/demo/ is a comment edit, because the isolation lint refused the word "wordle" there. Zero shell code changed. 415 contract checks now pass against two demos, up from 206 against one. The environment is honest by construction. Every alert is synthetic, generated from the seed, and the banner saying so sits inside the board surface. Two of the eleven scenario templates are hidden-suspicious: generated by the same code as their benign twin with the signal overlaid only in lookup data, so the free screen is identically distributed and a screen-only policy STRUCTURALLY cannot tell them apart. The probe ladder measures it: `fast` catches 0.0 of hidden seeds. That is the counterweight made real rather than asserted. Twelve policies, thirteen ladder assertions, a genuine three-way trade: fast 0.846 wins hours (0.85), misses every hidden case targeted 0.894 wins the shipped total thorough 0.820 wins evidence (1.00), spends 2.9 hours None dominates. 92 Python tests, 35 TypeScript tests, 65 fixtures replaying at delta 0, and conformance gated on world + scorer + protocol so the browser shows the same alert for ?seed= that Python generated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019mt6sHQHEnEYrJZvoMCJSB
This commit is contained in:
karti-ai
@@ -0,0 +1,259 @@
|
||||
# alert-triage — the environment
|
||||
|
||||
One transaction-monitoring alert on one synthetic customer. The agent reads
|
||||
the analyst's screen, may pay analyst-hours for lookups, and ends the episode
|
||||
by closing the alert with a cited rationale or escalating it with a typology
|
||||
and the transactions that constitute it. The grader compares the disposition
|
||||
and the citations to what the generator planted and charges the hours. No
|
||||
model, human or prose is read anywhere in it.
|
||||
|
||||
`SPEC.md` is the specification as written and reviewed before this code
|
||||
existed. This file records **what was implemented**, including every change
|
||||
the reviewers demanded; where the two differ, this file is what ships.
|
||||
|
||||
```bash
|
||||
uv run python envs/probe.py --taskset alert-triage # the ladder
|
||||
uv run pytest envs/alert_triage/tests # ~30 s; hashes 4096 worlds
|
||||
uv run python -m alert_triage.conformance # the three digests
|
||||
```
|
||||
|
||||
## Layout
|
||||
|
||||
| file | what |
|
||||
|---|---|
|
||||
| `alert_triage/rng.py` | FNV-1a seeding, xorshift32, civil dates. No platform RNG, no platform date. |
|
||||
| `alert_triage/generator.py` | Templates, overlays, the canonical world, `reference_minutes`, digest (a). |
|
||||
| `alert_triage/engine.py` | Screen, lookups, dispositions, shown-id discipline, the minute meter. |
|
||||
| `alert_triage/protocol.py` | The pinned parse rules, the system prompt, text renderers. |
|
||||
| `alert_triage/reward.py` | The three components. The `# region: pig-demo/reward` block is what the page quotes. |
|
||||
| `alert_triage/policies.py` | `fast`, `targeted`, `thorough` and the seven cheats. Read only the engine's view. |
|
||||
| `alert_triage/ladder.py` | The probe rungs and their thirteen assertions. |
|
||||
| `alert_triage/conformance.py` | Digests (b) and (c); writes `CONFORMANCE.txt`. |
|
||||
| `alert_triage/taskset.py` | `play_episode`, `replay`, and the optional verifiers Env. |
|
||||
| `conformance/replies.json` | The reply corpus digest (c) runs. |
|
||||
| `names/` | The synthetic name and word lists, with `PROVENANCE.md`. |
|
||||
|
||||
## The world, and what the reviewers changed
|
||||
|
||||
**Templates.** Six benign builders (B1 seasonal cash, B2 processor
|
||||
settlement, B3 property sale, B4 cash restaurant, B5 family remittance, B6
|
||||
inheritance) and three visible-suspicious ones (S1 structuring, S3 money mule,
|
||||
S6 smurfing over time). The hidden tier is no longer two fixed twins. A
|
||||
hidden alert is any benign builder run to completion — screen frozen — with
|
||||
one of three **overlays** applied to lookup data only:
|
||||
|
||||
| overlay | where it lives | compatible with | planted set |
|
||||
|---|---|---|---|
|
||||
| `funnel` | the counterparty lookup: the top outbound counterparty on screen is shared with 5–14 other alerted accounts | every benign template | `CP-x` + its transactions; alternate: the transactions |
|
||||
| `serial_closer` | the prior-alerts lookup: every prior alert on the same rule, closed within a day on one identical boilerplate, by different analysts | screens showing ≥ 3 prior alerts and no prior report | the `A-` ids + the triggering transactions; alternate: the `A-` ids |
|
||||
| `doc_mismatch` | the documents lookup against the counterparty lookup: the document names a party; the funds came from someone else | B2, B3, B6 | the document + `CP-x` + its triggering credits; alternate: the document + `CP-x` |
|
||||
|
||||
The tier is drawn *before* the template and the screen, and the prior-alert
|
||||
count on the KYC block is drawn before any template code runs, so the screen
|
||||
distribution is the same for hidden and benign by construction, not by
|
||||
effort. `tests/test_triage_screen_leak.py` asserts the screen is
|
||||
byte-identical with every compatible overlay on or off for every seed, and
|
||||
that no screen feature separates the populations beyond four sigma.
|
||||
|
||||
This answers three reviewer findings at once: every rule id now fires on both
|
||||
labels (`R-WIRE-04` and `R-DORM-05` carry funnel and serial-closer twins);
|
||||
each hidden signal lands on several templates so there is no template→lookup
|
||||
mapping to memorise; and no screen-identifiable segment carries a suspicious
|
||||
prior above the overall 15 / (55 + 15) ≈ 21 %.
|
||||
|
||||
**Removed:** B4's standalone `PREVIOUSLY_REVIEWED` alternate. A prior closure
|
||||
never resolves an alert on its own. The reason stays in the enum as legal and
|
||||
never planted, so its use renders as a diagnostic.
|
||||
|
||||
**Documents** carry per-world ids (`doc.D-<4 hex>`), and the engine rejects
|
||||
any citation of an id it has not rendered. `fast` therefore pulls
|
||||
`documents` on the templates a document explains, and the reference on those
|
||||
seeds is 0.70 h, not 0.55 h.
|
||||
|
||||
**The held-out slice** is `fnv1a(str(seed)) % 8 == 7`, about 12.5 % of
|
||||
seeds. It is generated in a **fictional** currency, KRN, with a fictional
|
||||
20,000 KRN reporting threshold (amounts scale ×2), rule ids renamed
|
||||
`TM-A1..A5`, the KYC block before the alert and the summary newest-first. It
|
||||
is never probed and never used to tune anything; the capture stage runs it
|
||||
once, after the arms are frozen, for the side-by-side numbers on the page.
|
||||
No claim about any real jurisdiction's threshold is made anywhere.
|
||||
|
||||
## Canonical world
|
||||
|
||||
Sorted keys, no whitespace, `null` for absent. Every amount is an integer in
|
||||
minor units; every date is a `YYYY-MM-DD` string produced by
|
||||
`rng.iso_date` from a day ordinal. `reference_minutes` and
|
||||
`reference_policy` are **part of the world** and therefore of digest (a):
|
||||
the browser recomputes them by running the ported policies rather than
|
||||
trusting the recorded number, and a mismatch is a verification failure, not
|
||||
an "unverifiable".
|
||||
|
||||
Draw order is the contract. Reading `generator.py` top to bottom in the
|
||||
order the builder calls things is the port; the port must not reorder a
|
||||
single `rng` call.
|
||||
|
||||
## The action set and the pinned parse rules
|
||||
|
||||
One JSON object per reply. A ```` ```json ```` fence anywhere wins; otherwise
|
||||
the first string-aware balanced `{…}` span; an unparseable or non-object
|
||||
candidate is a rejection with no further scanning. Enums exact-case. Cites
|
||||
deduplicated in order, matched exact-case against the shown set. The
|
||||
twelve-month window is the twelve calendar months ending in the fire month
|
||||
inclusive. A repeated lookup is rejected. Only accepted lookups add their
|
||||
price; every turn, accepted or rejected, costs `TURN_MINUTES`.
|
||||
|
||||
`aborted` is exactly `MAX_TURNS = 8` turns spent with no accepted
|
||||
disposition and scores 0.000. A run that stops earlier with no disposition is
|
||||
`truncated` and every component is `null` — rendered "not scored", never
|
||||
zero. Correct disposition → `solved`, wrong → `failed`.
|
||||
|
||||
## Hours
|
||||
|
||||
Integers in minutes, shown as hours. **The generator's config, not
|
||||
measurements** — nobody timed an analyst:
|
||||
|
||||
| constant | minutes | hours |
|
||||
|---|---|---|
|
||||
| `SCREEN_MINUTES` | 30 | 0.50 |
|
||||
| `TURN_MINUTES` | 3 | 0.05 |
|
||||
| `LOOKUP_MINUTES` | 24 | 0.40 |
|
||||
| `DOC_MINUTES` | 6 | 0.10 |
|
||||
|
||||
`LOOKUP_MINUTES` was raised from the spec's 0.25 h to 0.40 h so that the
|
||||
hours spent investigating clean screens genuinely exceed the hours a
|
||||
screen-only analyst loses to misses — the trade-off the reviewers said was
|
||||
manufactured in the spec is real at these constants and the ladder measures
|
||||
it (rung 8).
|
||||
|
||||
## The reward
|
||||
|
||||
| key | weight | role | definition |
|
||||
|---|---|---|---|
|
||||
| `caught` | 0.55 | counterweight | 1.0 on a correct call; 0.0 on a miss or no disposition; **0.5 on a false escalation** ("filed on a clean customer"). |
|
||||
| `hours` | 0.25 | objective | `min(1, reference / spent)` if correct, else 0. `null` if no shipped policy got the seed right (CI asserts this never happens). Capped: faster than the reference earns 1.0, not more. |
|
||||
| `evidence` | 0.20 | objective | F1 of the cites against the best planted alternate; 0 if the call was wrong. |
|
||||
|
||||
The 0.5 credit is the reviewers' fix for segment-blind defensive filing: with
|
||||
it, a blind escalation only beats a well-cited close when P(benign) is below
|
||||
**0.52** (`reward.escalate_break_even()`, printed by the probe), and no
|
||||
benign-looking screen segment is that suspicious.
|
||||
|
||||
## The ladder, measured
|
||||
|
||||
`uv run python envs/probe.py --taskset alert-triage`, 894 main-slice seeds
|
||||
(seeds 0–1023 minus the held-out bucket). These are the numbers the page may
|
||||
quote; the spec's `~0.86 vs ~0.85` text may not.
|
||||
|
||||
```
|
||||
policy caught hours evidence TOTAL hours@0.45
|
||||
inaction 0.0000 0.0000 0.0000 0.0000 0.0000
|
||||
malformed 0.0000 0.0000 0.0000 0.0000 0.0000
|
||||
close_blind 0.5705 0.5705 0.0000 0.4564 0.4868
|
||||
close_cites_all 0.5705 0.5705 0.0815 0.4727 0.4988
|
||||
cite_unseen 0.5705 0.5473 0.0000 0.4506 0.4764
|
||||
escalate_blind 0.7148 0.4295 0.3006 0.5606 0.5257
|
||||
fast_defensive 0.7148 0.4295 0.3006 0.5606 0.5257
|
||||
rule_table 0.7550 0.5593 0.3680 0.6287 0.6102
|
||||
fast 0.8535 0.8535 0.8159 0.8460 0.8480
|
||||
targeted 1.0000 0.6039 0.9625 0.8935 0.8163
|
||||
thorough 1.0000 0.2784 1.0000 0.8196 0.6753
|
||||
thorough_wasteful 1.0000 0.2412 1.0000 0.8103 0.6585
|
||||
```
|
||||
|
||||
Mean hours: fast 0.60, targeted 1.63, thorough 2.92. Caught on hidden seeds:
|
||||
fast 0.0, targeted 1.0, thorough 1.0.
|
||||
|
||||
The three shipped analysts, and the claim the page can make:
|
||||
|
||||
- **`fast`** decides from the screen (plus `documents` where a document
|
||||
should explain the alert). Wins `hours` outright. Misses every hidden
|
||||
seed, and only those.
|
||||
- **`targeted`** is `fast` on a screen that shows a typology, and otherwise
|
||||
pulls exactly the three lookups that could carry a hidden signal. Catches
|
||||
everything. **Wins at shipped weights, 0.894 to 0.846.** It defines
|
||||
`reference_minutes` on hidden seeds.
|
||||
- **`thorough`** always runs the full procedure, including two months of
|
||||
itemised history, and cites every constituting transaction it saw:
|
||||
evidence 1.000 exactly, and the lowest hours.
|
||||
|
||||
Move `hours` to **0.45** (the other two weights keeping their ratio) and
|
||||
`fast` wins, 0.848 to 0.816. Nothing else on the page depends on the
|
||||
direction; rung 8 asserts both margins are at least 0.02.
|
||||
|
||||
Note the weakest honest reading: `thorough` — which gets every call and
|
||||
every citation right — scores *below* `fast` at shipped weights, because
|
||||
spending five times the hours on the 85 % of alerts the screen settles costs
|
||||
more than the misses. That is the environment's argument, not a defect.
|
||||
|
||||
### Rungs a reviewer named
|
||||
|
||||
| rung | the cheat | assertion |
|
||||
|---|---|---|
|
||||
| `fast_defensive` | file blind on every benign-looking screen | total < every good policy |
|
||||
| `cite_unseen` | close citing a document it never opened | rejected on every seed; total < fast |
|
||||
| `rule_table` | close blind on wire/dormancy rules, file blind elsewhere | total < every good policy |
|
||||
| `close_cites_all` | cite every id on the screen | evidence < ½ its caught share; < fast's evidence |
|
||||
| `thorough_wasteful` | pull a month that carries nothing | total < thorough, same caught and evidence |
|
||||
|
||||
## The reference analysts see only the view
|
||||
|
||||
`policies.py` functions take `Engine.view()` and nothing else — the
|
||||
screen, the lookups so far, the turn and minute counters. Tests assert the
|
||||
signature, that the view carries none of `label`, `typology`, `planted`,
|
||||
`overlay`, `tier`, `template`, `reference_*`, and that every policy emits
|
||||
identical replies against a world with those fields deleted.
|
||||
|
||||
## Conformance
|
||||
|
||||
Three digests in `CONFORMANCE.txt`, each a CI gate for the TypeScript port:
|
||||
|
||||
1. **world** — canonical world JSON for seeds 0–4095, reference included.
|
||||
The port must carry the generator, not only the scorer: the Play tab and
|
||||
`?seed=` need the same alert on both sides.
|
||||
2. **scorer** — `score_exact()` over a fixed grid of episodes. Exact
|
||||
fractions, so no float formatting is hashed.
|
||||
3. **protocol** — parse + engine over `conformance/replies.json` on fixed
|
||||
worlds: accepted actions, rejection reasons, and the minute meter after
|
||||
every turn. This is the digest wordle does not need.
|
||||
|
||||
## What this is easier than, for the page
|
||||
|
||||
1. Every alert has a knowable answer; real SAR decisions are judgements, and a
|
||||
filed report is not a proven crime. Nine builders and three overlays stand
|
||||
in for a typology space in the hundreds.
|
||||
2. The 55/30/15 mix is deliberately not the real base rate. The great
|
||||
majority of production alerts close as false positives — no figure is
|
||||
quoted here because none was measured by us. "Hours per thousand alerts"
|
||||
on the page is Σ `hours_spent` over recorded seeds inside this
|
||||
environment, with the constants above beside it.
|
||||
3. The hour constants are assumptions.
|
||||
4. **The lookups are idealised.** Each returns the decisive fact in one
|
||||
call: a counterparty's other-alerted-account count, a prior alert's
|
||||
analyst and rationale, a document's named party. In production those
|
||||
signals are spread across systems and often absent. This environment
|
||||
tests whether the agent asks; it does not test whether the answer exists.
|
||||
5. The narrative prose is unscored.
|
||||
6. The adversary is static: the launderer shaped the data to pass the screen
|
||||
and the held-out slice moves the threshold once. Nothing responds to the
|
||||
agent's policy. The demo must claim exactly that and no more, and the
|
||||
wordle limit that names this demo as its answer should be softened to
|
||||
"nobody designed the data to pass the screen".
|
||||
7. `caught` is 1.0 on every benign alert by construction; the counterweight
|
||||
is argued from the ladder over seeds, never from one run.
|
||||
|
||||
## For the next stages
|
||||
|
||||
- **Port**: `rng.py`, `generator.py`, `engine.py`, `protocol.py`,
|
||||
`reward.py`, `policies.py` (`fast`, `targeted`, `thorough` — verify
|
||||
recomputes `reference_minutes`). Gate on all three digests.
|
||||
- **Capture**: `taskset.play_episode(seed, respond)` is the loop;
|
||||
`taskset.replay(seed, replies)` re-scores a fixture from its reply strings.
|
||||
Scripted arms: `fast`, `targeted`, `thorough`, and at least one cheat.
|
||||
Capture the held-out bucket exactly once, after the arms are frozen.
|
||||
- **Demo**: the board must carry "Synthetic data — no real customer, account
|
||||
or transaction"; the hour constants as config; the narrative marked "not
|
||||
scored"; the label reveal only after a disposition. The vertical card's
|
||||
`reward` line must be amended to say "hours as a ratio against a reference
|
||||
analyst policy", not "hours per thousand alerts".
|
||||
- `SPEC.md` is stage 1's document and was not edited; §3, §4, §6, §7 and
|
||||
§8 are superseded by this file where they differ.
|
||||
Reference in New Issue
Block a user