# alert-triage — the environment One transaction-monitoring alert on one synthetic customer. The agent reads the analyst's screen, may pay analyst-hours for lookups, and ends the episode by closing the alert with a cited rationale or escalating it with a typology and the transactions that constitute it. The grader compares the disposition and the citations to what the generator planted and charges the hours. No model, human or prose is read anywhere in it. `SPEC.md` is the specification as written and reviewed before this code existed. This file records **what was implemented**, including every change the reviewers demanded; where the two differ, this file is what ships. ```bash uv run python envs/probe.py --taskset alert-triage # the ladder uv run pytest envs/alert_triage/tests # ~30 s; hashes 4096 worlds uv run python -m alert_triage.conformance # the three digests ``` ## Layout | file | what | |---|---| | `alert_triage/rng.py` | FNV-1a seeding, xorshift32, civil dates. No platform RNG, no platform date. | | `alert_triage/generator.py` | Templates, overlays, the canonical world, `reference_minutes`, digest (a). | | `alert_triage/engine.py` | Screen, lookups, dispositions, shown-id discipline, the minute meter. | | `alert_triage/protocol.py` | The pinned parse rules, the system prompt, text renderers. | | `alert_triage/reward.py` | The three components. The `# region: pig-demo/reward` block is what the page quotes. | | `alert_triage/policies.py` | `fast`, `targeted`, `thorough` and the seven cheats. Read only the engine's view. | | `alert_triage/ladder.py` | The probe rungs and their thirteen assertions. | | `alert_triage/conformance.py` | Digests (b) and (c); writes `CONFORMANCE.txt`. | | `alert_triage/taskset.py` | `play_episode`, `replay`, and the optional verifiers Env. | | `conformance/replies.json` | The reply corpus digest (c) runs. | | `names/` | The synthetic name and word lists, with `PROVENANCE.md`. | ## The world, and what the reviewers changed **Templates.** Six benign builders (B1 seasonal cash, B2 processor settlement, B3 property sale, B4 cash restaurant, B5 family remittance, B6 inheritance) and three visible-suspicious ones (S1 structuring, S3 money mule, S6 smurfing over time). The hidden tier is no longer two fixed twins. A hidden alert is any benign builder run to completion — screen frozen — with one of three **overlays** applied to lookup data only: | overlay | where it lives | compatible with | planted set | |---|---|---|---| | `funnel` | the counterparty lookup: the top outbound counterparty on screen is shared with 5–14 other alerted accounts | every benign template | `CP-x` + its transactions; alternate: the transactions | | `serial_closer` | the prior-alerts lookup: every prior alert on the same rule, closed within a day on one identical boilerplate, by different analysts | screens showing ≥ 3 prior alerts and no prior report | the `A-` ids + the triggering transactions; alternate: the `A-` ids | | `doc_mismatch` | the documents lookup against the counterparty lookup: the document names a party; the funds came from someone else | B2, B3, B6 | the document + `CP-x` + its triggering credits; alternate: the document + `CP-x` | The tier is drawn *before* the template and the screen, and the prior-alert count on the KYC block is drawn before any template code runs, so the screen distribution is the same for hidden and benign by construction, not by effort. `tests/test_triage_screen_leak.py` asserts the screen is byte-identical with every compatible overlay on or off for every seed, and that no screen feature separates the populations beyond four sigma. This answers three reviewer findings at once: every rule id now fires on both labels (`R-WIRE-04` and `R-DORM-05` carry funnel and serial-closer twins); each hidden signal lands on several templates so there is no template→lookup mapping to memorise; and no screen-identifiable segment carries a suspicious prior above the overall 15 / (55 + 15) ≈ 21 %. **Removed:** B4's standalone `PREVIOUSLY_REVIEWED` alternate. A prior closure never resolves an alert on its own. The reason stays in the enum as legal and never planted, so its use renders as a diagnostic. **Documents** carry per-world ids (`doc.D-<4 hex>`), and the engine rejects any citation of an id it has not rendered. `fast` therefore pulls `documents` on the templates a document explains, and the reference on those seeds is 0.70 h, not 0.55 h. **The held-out slice** is `fnv1a(str(seed)) % 8 == 7`, about 12.5 % of seeds. It is generated in a **fictional** currency, KRN, with a fictional 20,000 KRN reporting threshold (amounts scale ×2), rule ids renamed `TM-A1..A5`, the KYC block before the alert and the summary newest-first. It is never probed and never used to tune anything; the capture stage runs it once, after the arms are frozen, for the side-by-side numbers on the page. No claim about any real jurisdiction's threshold is made anywhere. ## Canonical world Sorted keys, no whitespace, `null` for absent. Every amount is an integer in minor units; every date is a `YYYY-MM-DD` string produced by `rng.iso_date` from a day ordinal. `reference_minutes` and `reference_policy` are **part of the world** and therefore of digest (a): the browser recomputes them by running the ported policies rather than trusting the recorded number, and a mismatch is a verification failure, not an "unverifiable". Draw order is the contract. Reading `generator.py` top to bottom in the order the builder calls things is the port; the port must not reorder a single `rng` call. ## The action set and the pinned parse rules One JSON object per reply. A ```` ```json ```` fence anywhere wins; otherwise the first string-aware balanced `{…}` span; an unparseable or non-object candidate is a rejection with no further scanning. Enums exact-case. Cites deduplicated in order, matched exact-case against the shown set. The twelve-month window is the twelve calendar months ending in the fire month inclusive. A repeated lookup is rejected. Only accepted lookups add their price; every turn, accepted or rejected, costs `TURN_MINUTES`. `aborted` is exactly `MAX_TURNS = 8` turns spent with no accepted disposition and scores 0.000. A run that stops earlier with no disposition is `truncated` and every component is `null` — rendered "not scored", never zero. Correct disposition → `solved`, wrong → `failed`. ## Hours Integers in minutes, shown as hours. **The generator's config, not measurements** — nobody timed an analyst: | constant | minutes | hours | |---|---|---| | `SCREEN_MINUTES` | 30 | 0.50 | | `TURN_MINUTES` | 3 | 0.05 | | `LOOKUP_MINUTES` | 24 | 0.40 | | `DOC_MINUTES` | 6 | 0.10 | `LOOKUP_MINUTES` was raised from the spec's 0.25 h to 0.40 h so that the hours spent investigating clean screens genuinely exceed the hours a screen-only analyst loses to misses — the trade-off the reviewers said was manufactured in the spec is real at these constants and the ladder measures it (rung 8). ## The reward | key | weight | role | definition | |---|---|---|---| | `caught` | 0.55 | counterweight | 1.0 on a correct call; 0.0 on a miss or no disposition; **0.5 on a false escalation** ("filed on a clean customer"). | | `hours` | 0.25 | objective | `min(1, reference / spent)` if correct, else 0. `null` if no shipped policy got the seed right (CI asserts this never happens). Capped: faster than the reference earns 1.0, not more. | | `evidence` | 0.20 | objective | F1 of the cites against the best planted alternate; 0 if the call was wrong. | The 0.5 credit is the reviewers' fix for segment-blind defensive filing: with it, a blind escalation only beats a well-cited close when P(benign) is below **0.52** (`reward.escalate_break_even()`, printed by the probe), and no benign-looking screen segment is that suspicious. ## The ladder, measured `uv run python envs/probe.py --taskset alert-triage`, 894 main-slice seeds (seeds 0–1023 minus the held-out bucket). These are the numbers the page may quote; the spec's `~0.86 vs ~0.85` text may not. ``` policy caught hours evidence TOTAL hours@0.45 inaction 0.0000 0.0000 0.0000 0.0000 0.0000 malformed 0.0000 0.0000 0.0000 0.0000 0.0000 close_blind 0.5705 0.5705 0.0000 0.4564 0.4868 close_cites_all 0.5705 0.5705 0.0815 0.4727 0.4988 cite_unseen 0.5705 0.5473 0.0000 0.4506 0.4764 escalate_blind 0.7148 0.4295 0.3006 0.5606 0.5257 fast_defensive 0.7148 0.4295 0.3006 0.5606 0.5257 rule_table 0.7550 0.5593 0.3680 0.6287 0.6102 fast 0.8535 0.8535 0.8159 0.8460 0.8480 targeted 1.0000 0.6039 0.9625 0.8935 0.8163 thorough 1.0000 0.2784 1.0000 0.8196 0.6753 thorough_wasteful 1.0000 0.2412 1.0000 0.8103 0.6585 ``` Mean hours: fast 0.60, targeted 1.63, thorough 2.92. Caught on hidden seeds: fast 0.0, targeted 1.0, thorough 1.0. The three shipped analysts, and the claim the page can make: - **`fast`** decides from the screen (plus `documents` where a document should explain the alert). Wins `hours` outright. Misses every hidden seed, and only those. - **`targeted`** is `fast` on a screen that shows a typology, and otherwise pulls exactly the three lookups that could carry a hidden signal. Catches everything. **Wins at shipped weights, 0.894 to 0.846.** It defines `reference_minutes` on hidden seeds. - **`thorough`** always runs the full procedure, including two months of itemised history, and cites every constituting transaction it saw: evidence 1.000 exactly, and the lowest hours. Move `hours` to **0.45** (the other two weights keeping their ratio) and `fast` wins, 0.848 to 0.816. Nothing else on the page depends on the direction; rung 8 asserts both margins are at least 0.02. Note the weakest honest reading: `thorough` — which gets every call and every citation right — scores *below* `fast` at shipped weights, because spending five times the hours on the 85 % of alerts the screen settles costs more than the misses. That is the environment's argument, not a defect. ### Rungs a reviewer named | rung | the cheat | assertion | |---|---|---| | `fast_defensive` | file blind on every benign-looking screen | total < every good policy | | `cite_unseen` | close citing a document it never opened | rejected on every seed; total < fast | | `rule_table` | close blind on wire/dormancy rules, file blind elsewhere | total < every good policy | | `close_cites_all` | cite every id on the screen | evidence < ½ its caught share; < fast's evidence | | `thorough_wasteful` | pull a month that carries nothing | total < thorough, same caught and evidence | ## The reference analysts see only the view `policies.py` functions take `Engine.view()` and nothing else — the screen, the lookups so far, the turn and minute counters. Tests assert the signature, that the view carries none of `label`, `typology`, `planted`, `overlay`, `tier`, `template`, `reference_*`, and that every policy emits identical replies against a world with those fields deleted. ## Conformance Three digests in `CONFORMANCE.txt`, each a CI gate for the TypeScript port: 1. **world** — canonical world JSON for seeds 0–4095, reference included. The port must carry the generator, not only the scorer: the Play tab and `?seed=` need the same alert on both sides. 2. **scorer** — `score_exact()` over a fixed grid of episodes. Exact fractions, so no float formatting is hashed. 3. **protocol** — parse + engine over `conformance/replies.json` on fixed worlds: accepted actions, rejection reasons, and the minute meter after every turn. This is the digest wordle does not need. ## What this is easier than, for the page 1. Every alert has a knowable answer; real SAR decisions are judgements, and a filed report is not a proven crime. Nine builders and three overlays stand in for a typology space in the hundreds. 2. The 55/30/15 mix is deliberately not the real base rate. The great majority of production alerts close as false positives — no figure is quoted here because none was measured by us. "Hours per thousand alerts" on the page is Σ `hours_spent` over recorded seeds inside this environment, with the constants above beside it. 3. The hour constants are assumptions. 4. **The lookups are idealised.** Each returns the decisive fact in one call: a counterparty's other-alerted-account count, a prior alert's analyst and rationale, a document's named party. In production those signals are spread across systems and often absent. This environment tests whether the agent asks; it does not test whether the answer exists. 5. The narrative prose is unscored. 6. The adversary is static: the launderer shaped the data to pass the screen and the held-out slice moves the threshold once. Nothing responds to the agent's policy. The demo must claim exactly that and no more, and the wordle limit that names this demo as its answer should be softened to "nobody designed the data to pass the screen". 7. `caught` is 1.0 on every benign alert by construction; the counterweight is argued from the ladder over seeds, never from one run. ## For the next stages - **Port**: `rng.py`, `generator.py`, `engine.py`, `protocol.py`, `reward.py`, `policies.py` (`fast`, `targeted`, `thorough` — verify recomputes `reference_minutes`). Gate on all three digests. - **Capture**: `taskset.play_episode(seed, respond)` is the loop; `taskset.replay(seed, replies)` re-scores a fixture from its reply strings. Scripted arms: `fast`, `targeted`, `thorough`, and at least one cheat. Capture the held-out bucket exactly once, after the arms are frozen. - **Demo**: the board must carry "Synthetic data — no real customer, account or transaction"; the hour constants as config; the narrative marked "not scored"; the label reveal only after a disposition. The vertical card's `reward` line must be amended to say "hours as a ratio against a reference analyst policy", not "hours per thousand alerts". - `SPEC.md` is stage 1's document and was not edited; §3, §4, §6, §7 and §8 are superseded by this file where they differ.