PIG-Demo/envs/alert_triage/SPEC.md

alert-triage — specification

Vertical: financial-crime-alert-triage (src/content/verticals.ts, rank 4). Package: envs/alert_triage/. Demo: src/demos/alert-triage/. Traces: public/traces/alert-triage/.

This document is written before any code and is meant to be attacked. The two things it has to survive: a sceptic calling the grader subjective, and a sceptic finding a policy that maxes the reward while doing something a BSA officer would fire an analyst for.

In one paragraph. One episode is one transaction-monitoring alert on one synthetic customer. The agent sees what the analyst's screen shows — the alert, the triggering transactions, the KYC file, a twelve-month summary — and may pay analyst-hours for lookups (itemised history, a counterparty, prior alerts, documents on file). It ends the episode by closing the alert with a cited rationale or escalating it for a suspicious-activity filing with a typology and the transactions that constitute it. The generator planted the answer: the alert is either a benign scenario with a specific explaining fact, or a laundering typology with a specific set of transactions. The grader compares the disposition and the citations to what was planted, and charges the hours. Nothing in it reads prose.

---

1. The episode

One alert. Deterministic in the seed. The generator draws a scenario template from the seed (FNV-1a over the decimal seed selects the template and seeds a hand-written xorshift32 for its parameters — never a built-in RNG, see AGENTS.md), and produces:

• the customer (C-####): individual or business, occupation/business category, stated income or revenue, expected monthly cash, country, account opened, KYC last refreshed, PEP flag, prior alert and prior SAR counts;
• twelve months of account history: a monthly summary table (cash in/out, wires in/out, ACH in/out, transaction count), and the itemised transactions behind it, each with a stable id T-###, a date, a channel, an amount, a branch or counterparty id CP-##;
• the alert (A-####): the rule that fired (R-STR-01 structuring, R-VEL-02 rapid movement, R-CASH-03 cash above KYC expectation, R-WIRE-04 wires to a high-risk jurisdiction, R-DORM-05 dormant-account reactivation), the fire date, and the triggering transactions itemised;
• the hidden truth, never shown: label ∈ {benign, suspicious}, the typology if suspicious, and the planted citation set (§4).

The free screen. Turn 0 shows the alert, its triggering transactions, the KYC block and the monthly summary. This costs a fixed SCREEN_HOURS — the analyst has to read it whatever they do next.

Turns. Up to MAX_TURNS = 8 replies. Each reply is one lookup or one disposition (§3). Each turn spent — accepted or rejected — costs TURN_HOURS; a lookup adds its own price.

What ends it. An accepted close or escalate, or the eighth turn without one. The latter is outcome: aborted and scores 0.000 on every component: an alert nobody disposed of is still in the queue, which is the worst outcome for throughput and, on a suspicious alert, is a miss.

Scenario templates. Six benign, five suspicious. Every template is written from public typology descriptions (FinCEN SAR narrative guidance, FATF typology reports); none is a real case.

id	label	rule that fires	what the screen shows	where the answer lives
B1 seasonal cash	benign	R-CASH-03	landscaping / tax-prep business, cash spikes in season	kyc.expected_activity says seasonal, kyc.expected_monthly_cash covers peak
B2 processor settlement	benign	R-VEL-02	business receives processor wires, pays contractors within 48h	doc.processor_agreement on file
B3 property sale	benign	R-VEL-02 / R-DORM-05	individual receives title-company wire, moves it to brokerage	doc.closing_statement
B4 cash restaurant	benign	R-STR-01	daily cash deposits, three happen to land in 8,000–9,999	kyc.expected_monthly_cash ≥ observed; 12-month cash profile flat
B5 family remittance	benign	R-WIRE-04	monthly wire to one named individual in a high-risk country	kyc.remittance_declared names the destination
B6 inheritance	benign	R-DORM-05	dormant account receives estate wire	doc.probate_letter
S1 structuring	suspicious, visible	R-STR-01	45k stated income; 8 cash deposits 9,200–9,900 in 12 days across 3 branches	the 8 deposits
S3 money mule	suspicious, visible	R-VEL-02	student/retiree; 4 inbound P2P from unrelated individuals, each out within 24h to an exchange	the in/out pairs
S6 smurfing over time	suspicious, visible	R-CASH-03	cash business; monthly cash tripled over 12 months and no single deposit ever exceeded 9,999	the deposits in the last 3 months
S2 funnel account	suspicious, hidden	R-STR-01 or R-CASH-03	generated by B4's template — screen is identically distributed to a benign restaurant	counterparty lookup: its 3 outbound-wire counterparties are shared with 6–14 other alerted accounts; the wires are the evidence
S5 serial closer	suspicious, hidden	R-VEL-02	generated by B2's template — processor agreement on file, contractors paid	prior_alerts lookup: 3 prior alerts in 9 months, each closed by a different analyst with the same boilerplate; documents shows the agreement's counterparty is not the payer

The hidden tier is the environment's whole argument. S2 and S5 are produced by the same generator code as B4 and B2 respectively, with the suspicious signal overlaid only in the data a lookup returns. The free screen for a hidden case is therefore not merely similar to a benign one; it is drawn from the same distribution. A policy that decides from the screen cannot tell them apart, by construction, not by our failing to try hard enough.

Mix by seed: 55% benign, 30% visible-suspicious, 15% hidden. This is deliberately not the real base rate — real programmes close the great majority of alerts as false positives, and at that rate a 24-seed probe would contain about one suspicious alert and the ladder would measure noise. The page must say this (§8, risk 1).

2. The data

Synthetic, entirely, and labelled synthetic on the page — in the surface, not a footnote. No real customer, account, transaction, counterparty or analyst exists behind any of it. Names come from a synthetic name list committed under envs/alert_triage/names/ with its own PROVENANCE.md; business names are composed from committed word lists ("Harbor Lane Bistro LLC"). Amounts, dates, branch numbers and counterparties are generated.

Licence: the generator and its lists are ours, Apache-2.0 with the repository. The public documents the templates are written from — FinCEN's SAR narrative guidance and FATF typology reports — are cited as references on the evidence tab, not shipped, and not represented as data.

Why synthetic is the only honest option here: real alert data is customer financial data under BSA confidentiality; a SAR's existence may not be disclosed. There is no licensable real dataset for this task, and any page implying one would be lying.

3. The action set

Every reply is exactly one JSON object — the first {…} the parser can balance, or the contents of the first ```json fence if present. The system prompt says so and shows the four shapes.

action	fields	cost	what comes back
{"action":"lookup","what":"history","month":"2026-01"}	a month inside the 12	LOOKUP_HOURS	that month itemised, with T- ids and CP- ids
{"action":"lookup","what":"counterparty","id":"CP-7"}	a CP- id that appeared on screen or in a fetched month	LOOKUP_HOURS	name, type, jurisdiction, first seen, number of other alerted accounts sharing this counterparty in 12 months
{"action":"lookup","what":"prior_alerts"}	—	LOOKUP_HOURS	prior alerts: rule, date, disposition, closing analyst, rationale snippet
{"action":"lookup","what":"documents"}	—	DOC_HOURS	documents on file with doc. ids and one-line descriptions
{"action":"close","reason":"CONSISTENT_WITH_PROFILE","cites":["kyc.expected_monthly_cash"],"note":"…"}	reason ∈ {CONSISTENT_WITH_PROFILE, DOCUMENTED_SOURCE_OF_FUNDS, PREVIOUSLY_REVIEWED, RULE_ARTEFACT}; cites a non-empty list of ids; note free text	ends episode	—
{"action":"escalate","typology":"STRUCTURING","cites":["T-101","T-102"],"narrative":"…"}	typology ∈ {STRUCTURING, FUNNEL_ACCOUNT, MONEY_MULE, RAPID_MOVEMENT, UNKNOWN}; cites non-empty; narrative free text	ends episode	—

Citable ids: kyc.<field> for any field shown in the KYC block, doc.<id>, T-<n>, CP-<n>, A-<n> (a prior alert). An id the agent has not been shown is accepted and simply will not match anything planted — the grader does not need to know what the agent saw, only what it cited.

Malformed replies. No parseable object, an unknown action, an enum value outside its set, an empty cites, a month outside the window, a CP- id that does not exist, a lookup already made: all are rejected. A rejection costs a turn and TURN_HOURS, is counted in rejected_replies, returns a one-line reason in the format the prompt promised, and never raises. The engine is pure and cannot crash on input. A repeated lookup is rejected rather than re-charged so the transcript stays a record of distinct things learned.

Hour constants (the generator's config, shown on the page as such): SCREEN_HOURS 0.50, TURN_HOURS 0.05, LOOKUP_HOURS 0.25, DOC_HOURS 0.10. These are assumptions, not measurements, and the page must not present them as industry figures (§8, risk 2). The implement stage may tune them to make the ladder's ordering hold, and must record the values it ships in reward.py where the page quotes them.

4. The grader

Everything below is a pure function of (world, transcript). No model, no human, no prose is read anywhere in it.

From the world (regenerated from the seed): label, typology, planted. From the transcript: the accepted actions in order, the count of rejected replies, and the final disposition.

hours_spent   = SCREEN_HOURS + TURN_HOURS × turns_spent + Σ lookup prices
correct       = (label == benign  and disposition == close)
             or (label == suspicious and disposition == escalate)

planted is the minimal set of ids that resolves the alert, fixed by the template at generation time:

• benign: the explaining fact(s) — one or two kyc./doc. ids (table in §1);
• suspicious: the transaction ids that constitute the typology (S1: the eight deposits; S2: the outbound wires to the shared counterparties; S3: the in/out pairs; and so on).

Where a template admits two reasonable rationales (B4 can be closed on the KYC expectation or on a prior closed alert with the same rationale), planted is a list of alternates, and the score takes the best-matching one. Still deterministic; less opinionated.

evidence = F1 between cites and the best alternate of planted. If correct is false, evidence is 0 — a false escalation is checked against an empty set, a missed one against facts that do not exist for that label. Cite-everything scores what its precision earns (one planted id among eight cited is F1 0.22); cite-nothing is rejected at parse.

reference_hours — the hours denominator — is the hours of the cheapest shipped policy that reached the correct disposition on this seed, computed by running the two shipped reference policies (§6) through the engine at generation time, exactly as wordle computes reference_depth by running the solver. On benign and visible seeds that is fast; on hidden seeds it is thorough. It is stored in the trace as a metric so the browser can verify without running the policies, and its absence makes a run unverifiable, never zero.

Conformance. Two digests, both gated in CI: (a) SHA-256 over the canonical JSON of the generated world for seeds 0–4095 — the browser must show the same alert for ?seed= as the Python generated; (b) SHA-256 over the scorer applied to a fixed cross-product of synthetic dispositions (every template × every disposition × a grid of citation sets and hour totals). The generator has to be ported, not just the scorer, because the Play tab needs it.

5. The reward components

Weights sum to 1.0. One counterweight.

key	label (exec-facing)	weight	role	definition
caught	Nothing slipped through	0.55	counterweight	1.0 if a disposition was reached and either the alert was benign or it was escalated. 0.0 if the alert was suspicious and closed, or no disposition was reached.
hours	Cleared it in the hours budgeted	0.25	objective	min(1, reference_hours / hours_spent) if correct, else 0.0. A fast wrong answer saves nothing.
evidence	Named the facts that decide it	0.20	objective	F1 of cites against planted (best alternate), 0.0 if the disposition was wrong. The prose is shown, never scored.

Why this is the asymmetry the vertical promised. A missed escalation zeroes caught (0.55), and because the call was wrong it also zeroes hours and evidence: the episode scores 0.000. The most any policy can gain by being fast is hours's 0.25. One miss costs more than every hour saved, and you can read that off the weights rather than off a policy memo. The reward editor is where a BSA officer argues with it.

The tension, named.

• The crude policy that maxes the objective: close on turn one, read nothing. It takes hours (1.0 on every benign seed — it spent less than any reference) and is stopped by caught, which it scores at the benign base rate and no higher.
• The crude policy that maxes the counterweight: escalate on turn one, everything. caught 1.0, stopped by hours (0 on every benign seed — wrong call) and evidence (0 on every benign seed).
• The good policy that pays the counterweight's price: fast — decides from the screen and the KYC file, gets every benign and every visible case right, cites correctly, spends SCREEN_HOURS and one turn. It defines the hours denominator on 85% of seeds and scores 1.0 there. It structurally cannot see a funnel counterparty or a serial-closer history, so it scores caught ≈ 0.85 and is not 1.0.
• The good policy that takes the counterweight outright: thorough — pulls documents, prior alerts, every counterparty on screen and the itemised peak months, then decides with everything. caught 1.0, evidence 1.0, hours ≈ 0.4 (it spends roughly four times the screen cost on the 85% of seeds where the screen was enough).

Neither dominates: thorough wins caught and evidence; fast wins hours. At the shipped weights thorough is intended to edge fast by a small margin (≈0.86 vs ≈0.85 on the rough arithmetic); moving hours to 0.35 flips it. That the flip is one slider away is the demonstration.

Metrics (weight zero — rendered, never summed): hours_spent, reference_hours, lookups, turns_used, rejected_replies, false_escalation (0/1), missed (0/1), typology_match (1/0, null on a close), label (rendered as the reveal after the episode ends).

typology and reason are required for a well-formed action and are otherwise unscored. typology_match is a diagnostic. Scoring it would double-count caught with an enum the agent can guess.

6. The probe ladder

Run over 24 seeds; the probe asserts the seed set contains at least one seed of each tier (benign, visible, hidden) or aborts before measuring anything.

policy	behaviour	expected
inaction	reasons in prose, never emits an object	0.000 on every component, exactly
malformed	emits an object every turn with an unknown action or an empty cites	0.000 exactly; rejected_replies = 8
close_blind	turn one: close, reason CONSISTENT_WITH_PROFILE, cites kyc.business_type	caught = benign share (≈0.55); hours = benign share; evidence small; total ≈ 0.45
close_cites_all	turn one: close citing every KYC field and every triggering transaction	same as above with evidence ≈ 0.55 × 0.2; must score strictly below fast
escalate_blind	turn one: escalate, typology from the rule's nominal mapping, cites all triggering transactions	caught 1.0; hours = suspicious share (≈0.45); evidence ≈ 0.3; total ≈ 0.7; must score strictly below both good policies
fast	decides from the screen + KYC; closes when the KYC or a screen-visible fact explains the alert, escalates when the screen shows a typology; cites the explaining facts / constituting transactions	hours 1.0 on non-hidden seeds, 0 on hidden; caught ≈ 0.85 and < 1.0; evidence ≈ 0.85
thorough	lookups: documents, prior_alerts, every CP- on screen, itemised history for the two peak months; then decides with everything	caught = 1.0 exactly; evidence = 1.0; hours ≈ 0.4 and < fast's
thorough_wasteful	pulls all seven lookups the turn budget allows — including months that carry nothing — then decides exactly as thorough	same caught and evidence as thorough; total strictly below thorough — proves the hour budget binds

Assertions, in the style of envs/probe.py:

1. inaction and malformed total exactly 0.000.
2. thorough scores exactly 1.0 on caught — the counterweight is attainable.
3. fast scores strictly below 1.0 on caught — the counterweight bites the policy that defines the objective's denominator.
4. fast strictly beats thorough on hours — paying the counterweight's price is a real price.
5. Neither fast nor thorough dominates the other on every component.
6. escalate_blind, close_blind and close_cites_all each total strictly below both good policies. A cheat the reviewers name gets a rung here.
7. thorough_wasteful totals strictly below thorough.
8. No weighted component is flat across the ladder.

Note hours is scored over turns spent, rejected replies included, via TURN_HOURS, so a policy that jams the parser and then decides does not read as cheaper than one that decided cleanly.

7. The held-out slice

Three cuts, all deterministic in the seed:

1. Seed bucket. fnv1a(str(seed)) % 8 == 7 is held out — never captured, never probed, never in a training sweep. About 12.5% of seeds.
2. Moved threshold. Within the held-out bucket, structuring and cash templates (B4, S1, S2, S6) are generated in a second currency with that jurisdiction's reporting threshold, so the band is not 8,000–9,999 USD, and the rule ids are renamed. An agent that learned "9,xxx and R-STR-01" rather than "just under the threshold, repeatedly, across branches" is caught here. This is the honest form of the wordle page's "nobody is pushing back" limit: the adversary has moved, not responded (§8, risk 4).
3. Re-ordered screen. The held-out screen renders the KYC block before the alert and the summary in reverse chronological order. Format memorisation is not triage.

The page reports the training-slice and held-out numbers side by side, and never one as the other.

8. Risks — stated, not hidden

1. The ground truth is what the generator planted, and that is easier than reality in three ways. Every alert has a knowable answer; real SAR decisions are human judgements and a filed SAR is not a proven crime. Eleven templates stand in for a typology space in the hundreds. And the 55/45 mix is far from the real false-positive rate, which is widely reported to be above ninety percent; we do not re-quote a figure, we state that our mix is chosen so every arm meets both labels in a small seed set. The page must say that "hours per thousand alerts" here is a ratio inside this environment and not a production number.
2. The hour constants are assumptions. 0.50h to read a screen, 0.25h per lookup, 0.05h per turn. They are shown as the generator's config. No industry figure is cited for them because we have not measured one.
3. The narrative prose is unscored. Only the disposition, the citations and (as a metric) the typology are graded. A fluent wrong narrative and a terse right one score the same, and the page says so beside the narrative. This is the legal-playbook-redline move: score the checkable core, display the judgement half, never put a judge behind a bar.
4. The adversary is static. The launderer shaped the data to pass the screen, and the held-out slice moves the threshold, but nothing responds to the agent's policy during training. The wordle page's limit "nobody is pushing back… no counterparty adapting" names this demo as its answer; the demo stage should claim only what §1 and §7 deliver — an adversary who designed the data to pass, and moved once — and the ship stage should consider softening that wordle limit's wording. (That file belongs to another stage; not edited here.)
5. The hidden tier is hidden only if the generator keeps it so. If the implement stage lets any tell leak into the free screen — a different name distribution, a rounder amount, a branch number — a policy tuned to the leak catches S2/S5 from the screen and the tension collapses. Mitigation is structural (S2 and S5 are B4 and B2's generator with a lookup-only overlay) and assertion 3 catches a leak the shipped fast policy could exploit, but not one only an RL agent would find. A future review should try to classify hidden-vs-benign from the screen alone and expect chance.
6. caught is 1.0 on every benign alert by construction. On a single benign seed, a close-everything run looks perfect. The page must argue the counterweight from the ladder over seeds, never from one run — the same caveat as a lucky first guess in wordle.
7. The planted set is an opinion about what "the" evidence is. Alternates soften it; they do not remove it. Where a reviewer finds a rationale the grader refuses that a BSA officer would accept, the fix is another alternate in the template, not a judge.
8. Capture may find the weak arm cannot emit JSON. Qwen3-8B without thinking may reject its way to 0.000 on most seeds. That is a finding to show, not a bug to hide, but a ladder with three arms at 0.000 teaches nothing — the capture stage must include fast and thorough as generated arms so the reward has something to rank.
9. Terminology is real, cases are not. BSA, SAR, FinCEN, FATF and the typology names are public vocabulary and are used correctly. No scenario is drawn from a real case, and no institution is named or implied.

9. What the next stages need from this

• envs/alert_triage/pyproject.toml (alert-triage, workspace member), package alert_triage/ with generator.py (templates, xorshift32, FNV-1a seed), engine.py (state, lookups, dispositions, never raises), protocol.py (JSON parse, renderers, system prompt), reward.py (with the single # region: pig-demo/reward pair), policies.py (fast, thorough), taskset.py, tests/, and names/PROVENANCE.md.
• Probe policies registered for taskset alert-triage; eval command uv run python envs/probe.py.
• The TypeScript port carries the generator, not only the scorer.
• The demo's surface renders a "Synthetic data — no real customer, account or transaction" label on the board itself, the hour constants as config, the narrative with "not scored" beside it, and the label reveal only after a disposition.
• Interactive mode: yes. A person can read the screen, click the four lookups (each showing its hour price), and close with a citation checklist or escalate with a typology and a transaction tick-list.
• Suggested caveat for the vertical entry at ship time: "The alerts are synthetic and every one has a planted answer. The narrative prose is shown and not scored; the disposition and the cited facts are. The adversary designed the data and moved the threshold once — it does not adapt to the agent."