diff --git a/envs/alert_triage/CONFORMANCE.txt b/envs/alert_triage/CONFORMANCE.txt new file mode 100644 index 0000000..747353d --- /dev/null +++ b/envs/alert_triage/CONFORMANCE.txt @@ -0,0 +1,3 @@ +505ae48b664aae23aea01208879d899701cc2f6babd8e76ccbe36e75f9ba1b34 world sha256 over canonical world JSON, seeds 0-4095, newline-joined, reference_minutes included +f370d5f6f28964a070e926e7469ffbdea595fcb9ce45b4dbc7e16f607ed4cc47 scorer sha256 over score_exact() of the fixed grid in conformance.py, newline-joined +63dafb1034f19973fd208345f5bec8c7abd9839d8e1c11c2bb1ed8d581f34205 protocol sha256 over protocol_trace() of conformance/replies.json, newline-joined diff --git a/envs/alert_triage/README.md b/envs/alert_triage/README.md new file mode 100644 index 0000000..5292cdd --- /dev/null +++ b/envs/alert_triage/README.md @@ -0,0 +1,259 @@ +# alert-triage — the environment + +One transaction-monitoring alert on one synthetic customer. The agent reads +the analyst's screen, may pay analyst-hours for lookups, and ends the episode +by closing the alert with a cited rationale or escalating it with a typology +and the transactions that constitute it. The grader compares the disposition +and the citations to what the generator planted and charges the hours. No +model, human or prose is read anywhere in it. + +`SPEC.md` is the specification as written and reviewed before this code +existed. This file records **what was implemented**, including every change +the reviewers demanded; where the two differ, this file is what ships. + +```bash +uv run python envs/probe.py --taskset alert-triage # the ladder +uv run pytest envs/alert_triage/tests # ~30 s; hashes 4096 worlds +uv run python -m alert_triage.conformance # the three digests +``` + +## Layout + +| file | what | +|---|---| +| `alert_triage/rng.py` | FNV-1a seeding, xorshift32, civil dates. No platform RNG, no platform date. | +| `alert_triage/generator.py` | Templates, overlays, the canonical world, `reference_minutes`, digest (a). | +| `alert_triage/engine.py` | Screen, lookups, dispositions, shown-id discipline, the minute meter. | +| `alert_triage/protocol.py` | The pinned parse rules, the system prompt, text renderers. | +| `alert_triage/reward.py` | The three components. The `# region: pig-demo/reward` block is what the page quotes. | +| `alert_triage/policies.py` | `fast`, `targeted`, `thorough` and the seven cheats. Read only the engine's view. | +| `alert_triage/ladder.py` | The probe rungs and their thirteen assertions. | +| `alert_triage/conformance.py` | Digests (b) and (c); writes `CONFORMANCE.txt`. | +| `alert_triage/taskset.py` | `play_episode`, `replay`, and the optional verifiers Env. | +| `conformance/replies.json` | The reply corpus digest (c) runs. | +| `names/` | The synthetic name and word lists, with `PROVENANCE.md`. | + +## The world, and what the reviewers changed + +**Templates.** Six benign builders (B1 seasonal cash, B2 processor +settlement, B3 property sale, B4 cash restaurant, B5 family remittance, B6 +inheritance) and three visible-suspicious ones (S1 structuring, S3 money mule, +S6 smurfing over time). The hidden tier is no longer two fixed twins. A +hidden alert is any benign builder run to completion — screen frozen — with +one of three **overlays** applied to lookup data only: + +| overlay | where it lives | compatible with | planted set | +|---|---|---|---| +| `funnel` | the counterparty lookup: the top outbound counterparty on screen is shared with 5–14 other alerted accounts | every benign template | `CP-x` + its transactions; alternate: the transactions | +| `serial_closer` | the prior-alerts lookup: every prior alert on the same rule, closed within a day on one identical boilerplate, by different analysts | screens showing ≥ 3 prior alerts and no prior report | the `A-` ids + the triggering transactions; alternate: the `A-` ids | +| `doc_mismatch` | the documents lookup against the counterparty lookup: the document names a party; the funds came from someone else | B2, B3, B6 | the document + `CP-x` + its triggering credits; alternate: the document + `CP-x` | + +The tier is drawn *before* the template and the screen, and the prior-alert +count on the KYC block is drawn before any template code runs, so the screen +distribution is the same for hidden and benign by construction, not by +effort. `tests/test_triage_screen_leak.py` asserts the screen is +byte-identical with every compatible overlay on or off for every seed, and +that no screen feature separates the populations beyond four sigma. + +This answers three reviewer findings at once: every rule id now fires on both +labels (`R-WIRE-04` and `R-DORM-05` carry funnel and serial-closer twins); +each hidden signal lands on several templates so there is no template→lookup +mapping to memorise; and no screen-identifiable segment carries a suspicious +prior above the overall 15 / (55 + 15) ≈ 21 %. + +**Removed:** B4's standalone `PREVIOUSLY_REVIEWED` alternate. A prior closure +never resolves an alert on its own. The reason stays in the enum as legal and +never planted, so its use renders as a diagnostic. + +**Documents** carry per-world ids (`doc.D-<4 hex>`), and the engine rejects +any citation of an id it has not rendered. `fast` therefore pulls +`documents` on the templates a document explains, and the reference on those +seeds is 0.70 h, not 0.55 h. + +**The held-out slice** is `fnv1a(str(seed)) % 8 == 7`, about 12.5 % of +seeds. It is generated in a **fictional** currency, KRN, with a fictional +20,000 KRN reporting threshold (amounts scale ×2), rule ids renamed +`TM-A1..A5`, the KYC block before the alert and the summary newest-first. It +is never probed and never used to tune anything; the capture stage runs it +once, after the arms are frozen, for the side-by-side numbers on the page. +No claim about any real jurisdiction's threshold is made anywhere. + +## Canonical world + +Sorted keys, no whitespace, `null` for absent. Every amount is an integer in +minor units; every date is a `YYYY-MM-DD` string produced by +`rng.iso_date` from a day ordinal. `reference_minutes` and +`reference_policy` are **part of the world** and therefore of digest (a): +the browser recomputes them by running the ported policies rather than +trusting the recorded number, and a mismatch is a verification failure, not +an "unverifiable". + +Draw order is the contract. Reading `generator.py` top to bottom in the +order the builder calls things is the port; the port must not reorder a +single `rng` call. + +## The action set and the pinned parse rules + +One JSON object per reply. A ```` ```json ```` fence anywhere wins; otherwise +the first string-aware balanced `{…}` span; an unparseable or non-object +candidate is a rejection with no further scanning. Enums exact-case. Cites +deduplicated in order, matched exact-case against the shown set. The +twelve-month window is the twelve calendar months ending in the fire month +inclusive. A repeated lookup is rejected. Only accepted lookups add their +price; every turn, accepted or rejected, costs `TURN_MINUTES`. + +`aborted` is exactly `MAX_TURNS = 8` turns spent with no accepted +disposition and scores 0.000. A run that stops earlier with no disposition is +`truncated` and every component is `null` — rendered "not scored", never +zero. Correct disposition → `solved`, wrong → `failed`. + +## Hours + +Integers in minutes, shown as hours. **The generator's config, not +measurements** — nobody timed an analyst: + +| constant | minutes | hours | +|---|---|---| +| `SCREEN_MINUTES` | 30 | 0.50 | +| `TURN_MINUTES` | 3 | 0.05 | +| `LOOKUP_MINUTES` | 24 | 0.40 | +| `DOC_MINUTES` | 6 | 0.10 | + +`LOOKUP_MINUTES` was raised from the spec's 0.25 h to 0.40 h so that the +hours spent investigating clean screens genuinely exceed the hours a +screen-only analyst loses to misses — the trade-off the reviewers said was +manufactured in the spec is real at these constants and the ladder measures +it (rung 8). + +## The reward + +| key | weight | role | definition | +|---|---|---|---| +| `caught` | 0.55 | counterweight | 1.0 on a correct call; 0.0 on a miss or no disposition; **0.5 on a false escalation** ("filed on a clean customer"). | +| `hours` | 0.25 | objective | `min(1, reference / spent)` if correct, else 0. `null` if no shipped policy got the seed right (CI asserts this never happens). Capped: faster than the reference earns 1.0, not more. | +| `evidence` | 0.20 | objective | F1 of the cites against the best planted alternate; 0 if the call was wrong. | + +The 0.5 credit is the reviewers' fix for segment-blind defensive filing: with +it, a blind escalation only beats a well-cited close when P(benign) is below +**0.52** (`reward.escalate_break_even()`, printed by the probe), and no +benign-looking screen segment is that suspicious. + +## The ladder, measured + +`uv run python envs/probe.py --taskset alert-triage`, 894 main-slice seeds +(seeds 0–1023 minus the held-out bucket). These are the numbers the page may +quote; the spec's `~0.86 vs ~0.85` text may not. + +``` +policy caught hours evidence TOTAL hours@0.45 +inaction 0.0000 0.0000 0.0000 0.0000 0.0000 +malformed 0.0000 0.0000 0.0000 0.0000 0.0000 +close_blind 0.5705 0.5705 0.0000 0.4564 0.4868 +close_cites_all 0.5705 0.5705 0.0815 0.4727 0.4988 +cite_unseen 0.5705 0.5473 0.0000 0.4506 0.4764 +escalate_blind 0.7148 0.4295 0.3006 0.5606 0.5257 +fast_defensive 0.7148 0.4295 0.3006 0.5606 0.5257 +rule_table 0.7550 0.5593 0.3680 0.6287 0.6102 +fast 0.8535 0.8535 0.8159 0.8460 0.8480 +targeted 1.0000 0.6039 0.9625 0.8935 0.8163 +thorough 1.0000 0.2784 1.0000 0.8196 0.6753 +thorough_wasteful 1.0000 0.2412 1.0000 0.8103 0.6585 +``` + +Mean hours: fast 0.60, targeted 1.63, thorough 2.92. Caught on hidden seeds: +fast 0.0, targeted 1.0, thorough 1.0. + +The three shipped analysts, and the claim the page can make: + +- **`fast`** decides from the screen (plus `documents` where a document + should explain the alert). Wins `hours` outright. Misses every hidden + seed, and only those. +- **`targeted`** is `fast` on a screen that shows a typology, and otherwise + pulls exactly the three lookups that could carry a hidden signal. Catches + everything. **Wins at shipped weights, 0.894 to 0.846.** It defines + `reference_minutes` on hidden seeds. +- **`thorough`** always runs the full procedure, including two months of + itemised history, and cites every constituting transaction it saw: + evidence 1.000 exactly, and the lowest hours. + +Move `hours` to **0.45** (the other two weights keeping their ratio) and +`fast` wins, 0.848 to 0.816. Nothing else on the page depends on the +direction; rung 8 asserts both margins are at least 0.02. + +Note the weakest honest reading: `thorough` — which gets every call and +every citation right — scores *below* `fast` at shipped weights, because +spending five times the hours on the 85 % of alerts the screen settles costs +more than the misses. That is the environment's argument, not a defect. + +### Rungs a reviewer named + +| rung | the cheat | assertion | +|---|---|---| +| `fast_defensive` | file blind on every benign-looking screen | total < every good policy | +| `cite_unseen` | close citing a document it never opened | rejected on every seed; total < fast | +| `rule_table` | close blind on wire/dormancy rules, file blind elsewhere | total < every good policy | +| `close_cites_all` | cite every id on the screen | evidence < ½ its caught share; < fast's evidence | +| `thorough_wasteful` | pull a month that carries nothing | total < thorough, same caught and evidence | + +## The reference analysts see only the view + +`policies.py` functions take `Engine.view()` and nothing else — the +screen, the lookups so far, the turn and minute counters. Tests assert the +signature, that the view carries none of `label`, `typology`, `planted`, +`overlay`, `tier`, `template`, `reference_*`, and that every policy emits +identical replies against a world with those fields deleted. + +## Conformance + +Three digests in `CONFORMANCE.txt`, each a CI gate for the TypeScript port: + +1. **world** — canonical world JSON for seeds 0–4095, reference included. + The port must carry the generator, not only the scorer: the Play tab and + `?seed=` need the same alert on both sides. +2. **scorer** — `score_exact()` over a fixed grid of episodes. Exact + fractions, so no float formatting is hashed. +3. **protocol** — parse + engine over `conformance/replies.json` on fixed + worlds: accepted actions, rejection reasons, and the minute meter after + every turn. This is the digest wordle does not need. + +## What this is easier than, for the page + +1. Every alert has a knowable answer; real SAR decisions are judgements, and a + filed report is not a proven crime. Nine builders and three overlays stand + in for a typology space in the hundreds. +2. The 55/30/15 mix is deliberately not the real base rate. The great + majority of production alerts close as false positives — no figure is + quoted here because none was measured by us. "Hours per thousand alerts" + on the page is Σ `hours_spent` over recorded seeds inside this + environment, with the constants above beside it. +3. The hour constants are assumptions. +4. **The lookups are idealised.** Each returns the decisive fact in one + call: a counterparty's other-alerted-account count, a prior alert's + analyst and rationale, a document's named party. In production those + signals are spread across systems and often absent. This environment + tests whether the agent asks; it does not test whether the answer exists. +5. The narrative prose is unscored. +6. The adversary is static: the launderer shaped the data to pass the screen + and the held-out slice moves the threshold once. Nothing responds to the + agent's policy. The demo must claim exactly that and no more, and the + wordle limit that names this demo as its answer should be softened to + "nobody designed the data to pass the screen". +7. `caught` is 1.0 on every benign alert by construction; the counterweight + is argued from the ladder over seeds, never from one run. + +## For the next stages + +- **Port**: `rng.py`, `generator.py`, `engine.py`, `protocol.py`, + `reward.py`, `policies.py` (`fast`, `targeted`, `thorough` — verify + recomputes `reference_minutes`). Gate on all three digests. +- **Capture**: `taskset.play_episode(seed, respond)` is the loop; + `taskset.replay(seed, replies)` re-scores a fixture from its reply strings. + Scripted arms: `fast`, `targeted`, `thorough`, and at least one cheat. + Capture the held-out bucket exactly once, after the arms are frozen. +- **Demo**: the board must carry "Synthetic data — no real customer, account + or transaction"; the hour constants as config; the narrative marked "not + scored"; the label reveal only after a disposition. The vertical card's + `reward` line must be amended to say "hours as a ratio against a reference + analyst policy", not "hours per thousand alerts". +- `SPEC.md` is stage 1's document and was not edited; §3, §4, §6, §7 and + §8 are superseded by this file where they differ. diff --git a/envs/alert_triage/alert_triage/__init__.py b/envs/alert_triage/alert_triage/__init__.py new file mode 100644 index 0000000..aa29f25 --- /dev/null +++ b/envs/alert_triage/alert_triage/__init__.py @@ -0,0 +1,11 @@ +"""alert-triage: one synthetic transaction-monitoring alert per seed. + +Public surface, in the order the next stages need it: + + generator.world_for_seed(seed) the world, canonical and portable + engine.Engine(world) lookups, dispositions, the hour meter + protocol.parse_reply(text) one JSON object -> action or rejection + reward.score(episode) the three components, or None + policies.POLICIES the scripted analysts + taskset.play_episode(seed, fn) the loop capture, probe and tests share +""" diff --git a/envs/alert_triage/alert_triage/conformance.py b/envs/alert_triage/alert_triage/conformance.py new file mode 100644 index 0000000..646cfa5 --- /dev/null +++ b/envs/alert_triage/alert_triage/conformance.py @@ -0,0 +1,124 @@ +"""The three cross-language digests. `CONFORMANCE.txt` commits them; CI gates them. + + (a) world SHA-256 over the canonical JSON world for seeds 0–4095, + reference minutes included. The browser must show the same + alert for ?seed= as Python generated. + (b) scorer SHA-256 over the exact-fraction scores of a fixed grid of + synthetic episodes: label × disposition × cites × planted × + minutes × reference. No floats are hashed. + (c) protocol SHA-256 over what parse + engine make of a committed corpus of + reply strings (`conformance/replies.json`) on fixed worlds: + accepted actions, rejection reasons and the minute meter after + every turn. This is the digest wordle does not need and this + environment cannot do without — JSON-in-prose with fences and a + brace scan is where two runtimes disagree. + +Run: uv run python -m alert_triage.conformance [--write] +""" + +from __future__ import annotations + +import hashlib +import json +import sys +from pathlib import Path + +from .engine import Engine +from .generator import canonical_json, world_digest, world_for_seed +from .reward import Episode, score_exact + +ROOT = Path(__file__).parent.parent +CORPUS = ROOT / "conformance" / "replies.json" +COMMITTED = ROOT / "CONFORMANCE.txt" + +GRID_LABELS = ("benign", "suspicious") +GRID_DISPOSITIONS = ("close", "escalate", "none", "truncated") +GRID_CITES = (["a"], ["a", "b"], ["a", "b", "c", "d"], ["x"], ["a", "x"], ["a", "b", "x", "y", "z"], ["a", "a", "b"], ["kyc.z"]) +GRID_PLANTED = ([["a"]], [["a", "b"]], [["a", "b"], ["a"]], [["c"], ["a", "b", "c", "d"]]) +GRID_MINUTES = (33, 42, 60, 96, 150, 204) +GRID_REFERENCE = (33, 42, 96, None) + + +def scorer_digest() -> str: + digest = hashlib.sha256() + for label in GRID_LABELS: + for disposition in GRID_DISPOSITIONS: + for cites in GRID_CITES: + for planted in GRID_PLANTED: + for minutes in GRID_MINUTES: + for reference in GRID_REFERENCE: + ep = Episode( + label=label, + planted=planted, + disposition=None if disposition in ("none", "truncated") else disposition, + cites=[] if disposition in ("none", "truncated") else cites, + minutes_spent=minutes, + reference_minutes=reference, + turns_spent=1, + rejected=0, + truncated=disposition == "truncated", + ) + digest.update(canonical_json(score_exact(ep)).encode()) + digest.update(b"\n") + return digest.hexdigest() + + +def protocol_trace(corpus: list[dict] | None = None) -> list[dict]: + corpus = corpus if corpus is not None else json.loads(CORPUS.read_text()) + out = [] + for case in corpus: + engine = Engine(world_for_seed(case["seed"])) + turns = [] + for reply in case["replies"]: + if engine.done: + break + step = engine.step(reply) + turns.append({"accepted": step.accepted, "action": step.action, "reason": step.reason, "minutes": step.minutes_spent, "turns": step.turns_spent}) + ep = engine.episode() + out.append({"seed": case["seed"], "turns": turns, "outcome": engine.outcome, "truncated": ep.truncated, "score": score_exact(ep)}) + return out + + +def protocol_digest() -> str: + digest = hashlib.sha256() + for entry in protocol_trace(): + digest.update(canonical_json(entry).encode()) + digest.update(b"\n") + return digest.hexdigest() + + +def all_digests() -> dict[str, str]: + return {"world": world_digest(), "scorer": scorer_digest(), "protocol": protocol_digest()} + + +def committed() -> dict[str, str]: + out = {} + for line in COMMITTED.read_text().splitlines(): + parts = line.split() + if len(parts) >= 2: + out[parts[1]] = parts[0] + return out + + +def main(argv: list[str]) -> int: + digests = all_digests() + lines = [ + f"{digests['world']} world sha256 over canonical world JSON, seeds 0-4095, newline-joined, reference_minutes included", + f"{digests['scorer']} scorer sha256 over score_exact() of the fixed grid in conformance.py, newline-joined", + f"{digests['protocol']} protocol sha256 over protocol_trace() of conformance/replies.json, newline-joined", + ] + if "--write" in argv: + COMMITTED.write_text("\n".join(lines) + "\n") + print(f"wrote {COMMITTED}") + for line in lines: + print(line) + if COMMITTED.exists(): + mismatched = [k for k, v in committed().items() if digests.get(k) != v] + if mismatched: + print(f"MISMATCH against CONFORMANCE.txt: {mismatched}") + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main(sys.argv[1:])) diff --git a/envs/alert_triage/alert_triage/engine.py b/envs/alert_triage/alert_triage/engine.py new file mode 100644 index 0000000..f6f4870 --- /dev/null +++ b/envs/alert_triage/alert_triage/engine.py @@ -0,0 +1,229 @@ +"""One episode: the free screen, up to eight replies, and the hour meter. + +Pure and deterministic in the world it is given. It never raises on input: a +bad reply is a rejection with a one-line reason, charged a turn. + +Two discipline rules the reviewers demanded, both enforced here rather than in +the grader: + + * **Shown ids.** A disposition may cite only ids the engine has actually + rendered — KYC fields and triggering transactions from the screen, plus + whatever lookups returned. "Closed per the probate letter on file" + without opening the file is a rejected reply, not a scored one. + * **Abort versus truncate.** `aborted` is exactly MAX_TURNS turns spent + with no accepted disposition, and scores 0.000. Anything that stops + earlier with no disposition is `truncated` and is not scored at all. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Any + +from .generator import MAX_TURNS, screen_of +from .protocol import ( + ACTIONS, + CLOSE_REASONS, + LOOKUPS, + TYPOLOGIES, + _MONTH, + parse_reply, +) +from .reward import DOC_MINUTES, LOOKUP_MINUTES, SCREEN_MINUTES, TURN_MINUTES, Episode + +LOOKUP_PRICE = {"history": LOOKUP_MINUTES, "counterparty": LOOKUP_MINUTES, "prior_alerts": LOOKUP_MINUTES, "documents": DOC_MINUTES} + + +@dataclass +class Step: + accepted: bool + action: dict[str, Any] | None # canonical form of an accepted action + reason: str | None # why a rejected reply was rejected + result: dict[str, Any] | None # what an accepted lookup returned + minutes_spent: int + turns_spent: int + done: bool + + +@dataclass +class Engine: + world: dict[str, Any] + turns: int = 0 + rejected: int = 0 + minutes: int = SCREEN_MINUTES + actions: list[dict[str, Any]] = field(default_factory=list) + lookups: list[dict[str, Any]] = field(default_factory=list) + disposition: dict[str, Any] | None = None + shown: set[str] = field(default_factory=set) + last_rejection: str | None = None + + def __post_init__(self) -> None: + screen = self.screen() + self.shown.update(f"kyc.{k}" for k in screen["customer"] if k not in ("id", "name")) + self.shown.add(screen["alert"]["id"]) + for t in screen["triggering_transactions"]: + self.shown.add(t["id"]) + if t["cp"]: + self.shown.add(t["cp"]) + for cp in screen["screen_counterparties"]: + self.shown.add(cp["id"]) + + # ------------------------------------------------------------ views -- + + def screen(self) -> dict[str, Any]: + screen = screen_of(self.world) + if screen["screen_order"] == "kyc_first": + screen["summary"] = list(reversed(screen["summary"])) + return screen + + def view(self) -> dict[str, Any]: + """Everything the model has been shown, and nothing it has not. + + The scripted policies take this and only this. There is no label, + typology, planted set, overlay or reference in it — `tests/` scans + the keys to make sure. + """ + return { + "screen": self.screen(), + "lookups": list(self.lookups), + "turns_spent": self.turns, + "turns_left": MAX_TURNS - self.turns, + "rejected": self.rejected, + "minutes_spent": self.minutes, + "last_rejection": self.last_rejection, + } + + @property + def done(self) -> bool: + return self.disposition is not None or self.turns >= MAX_TURNS + + @property + def aborted(self) -> bool: + return self.disposition is None and self.turns >= MAX_TURNS + + # ------------------------------------------------------------- step -- + + def step(self, reply: str | None) -> Step: + if self.done: + return Step(False, None, "the episode is over", None, self.minutes, self.turns, True) + obj, reason = parse_reply(reply) + action = None + if obj is not None: + action, reason = self._validate(obj) + self.turns += 1 + self.minutes += TURN_MINUTES + if action is None: + self.rejected += 1 + self.last_rejection = reason + return Step(False, None, reason, None, self.minutes, self.turns, self.done) + + self.last_rejection = None + self.actions.append(action) + result = None + if action["action"] == "lookup": + self.minutes += LOOKUP_PRICE[action["what"]] + result = self._lookup(action) + self.lookups.append({"what": action["what"], "key": action.get("month") or action.get("id"), "result": result}) + else: + self.disposition = action + return Step(True, action, None, result, self.minutes, self.turns, self.done) + + def _validate(self, obj: dict[str, Any]) -> tuple[dict[str, Any] | None, str | None]: + action = obj.get("action") + if action not in ACTIONS: + return None, f"unknown action {action!r}; use lookup, close or escalate" + if action == "lookup": + what = obj.get("what") + if what not in LOOKUPS: + return None, f"unknown lookup {what!r}; use history, counterparty, prior_alerts or documents" + if what == "history": + month = obj.get("month") + if not isinstance(month, str) or not _MONTH.match(month) or month not in self.world["months"]: + return None, f"month {month!r} is not in the twelve-month window" + if any(l["what"] == "history" and l["key"] == month for l in self.lookups): + return None, f"history for {month} has already been looked up" + return {"action": "lookup", "what": "history", "month": month}, None + if what == "counterparty": + cid = obj.get("id") + if not isinstance(cid, str) or cid not in self.shown or not cid.startswith("CP-"): + return None, f"counterparty {cid!r} has not appeared on your screen or in a fetched month" + if any(l["what"] == "counterparty" and l["key"] == cid for l in self.lookups): + return None, f"counterparty {cid} has already been looked up" + return {"action": "lookup", "what": "counterparty", "id": cid}, None + if any(l["what"] == what for l in self.lookups): + return None, f"{what} has already been looked up" + return {"action": "lookup", "what": what}, None + + cites = obj.get("cites") + if not isinstance(cites, list) or not cites: + return None, "cites must be a non-empty list of ids" + seen: list[str] = [] + for c in cites: + if not isinstance(c, str): + return None, "cites must be a non-empty list of ids" + if c not in seen: + seen.append(c) + for c in seen: + if c not in self.shown: + return None, f"cites an id you have not been shown: {c}" + if action == "close": + reason = obj.get("reason") + if reason not in CLOSE_REASONS: + return None, f"unknown close reason {reason!r}" + note = obj.get("note") + return {"action": "close", "reason": reason, "cites": seen, "note": note if isinstance(note, str) else ""}, None + typology = obj.get("typology") + if typology not in TYPOLOGIES: + return None, f"unknown typology {typology!r}" + narrative = obj.get("narrative") + return {"action": "escalate", "typology": typology, "cites": seen, "narrative": narrative if isinstance(narrative, str) else ""}, None + + def _lookup(self, action: dict[str, Any]) -> dict[str, Any]: + w = self.world + what = action["what"] + if what == "history": + rows = [t for t in w["transactions"] if t["month"] == action["month"]] + for t in rows: + self.shown.add(t["id"]) + if t["cp"]: + self.shown.add(t["cp"]) + return {"month": action["month"], "transactions": rows} + if what == "counterparty": + cp = next(c for c in w["counterparties"] if c["id"] == action["id"]) + self.shown.update(cp["transactions"]) + return dict(cp) + if what == "prior_alerts": + self.shown.update(p["id"] for p in w["prior_alerts"]) + return {"alerts": list(w["prior_alerts"])} + self.shown.update(d["id"] for d in w["documents"]) + return {"documents": list(w["documents"])} + + # ---------------------------------------------------------- outcome -- + + @property + def outcome(self) -> str | None: + """solved | failed | aborted, or None while running / if truncated.""" + if self.disposition is not None: + label = self.world["label"] + right = (label == "benign") == (self.disposition["action"] == "close") + return "solved" if right else "failed" + return "aborted" if self.aborted else None + + def episode(self, truncated: bool | None = None) -> Episode: + if truncated is None: + truncated = not self.done + d = self.disposition + return Episode( + label=self.world["label"], + planted=self.world["planted"], + disposition=None if d is None else d["action"], + cites=[] if d is None else list(d["cites"]), + minutes_spent=self.minutes, + reference_minutes=self.world.get("reference_minutes"), + turns_spent=self.turns, + rejected=self.rejected, + truncated=truncated, + typology=None if d is None else d.get("typology"), + true_typology=self.world.get("typology"), + lookups=len(self.lookups), + ) diff --git a/envs/alert_triage/alert_triage/generator.py b/envs/alert_triage/alert_triage/generator.py new file mode 100644 index 0000000..49026d4 --- /dev/null +++ b/envs/alert_triage/alert_triage/generator.py @@ -0,0 +1,881 @@ +"""The world for a seed: one alert on one synthetic customer. + +Everything here is synthetic. No customer, account, transaction, counterparty, +analyst or institution exists behind any of it — see `names/PROVENANCE.md`. + +Portability is the whole design. `src/demos/alert-triage/` carries a port of +this file, and CI hashes the canonical JSON of every world for seeds 0–4095 on +both sides. So: + + * every random draw goes through `XorShift32` seeded by FNV-1a over the + decimal seed, in the order written here — the order IS the contract; + * every amount is an integer in minor units (cents), never a float; + * every date is a day ordinal turned into YYYY-MM-DD by our own civil-date + function, never a platform date type; + * canonical JSON is sorted keys, no whitespace, `null` for absent. + +The hidden tier — a suspicious alert whose free screen is drawn from a benign +template — is produced by running the benign builder to completion, freezing +the screen, and only then overlaying the suspicious signal onto data that a +lookup returns. `screen_of(world)` is byte-identical with the overlay on or +off, and `tests/test_screen_leak.py` asserts it for every seed. +""" + +from __future__ import annotations + +import hashlib +import json +from pathlib import Path +from typing import Any + +from .rng import ( + XorShift32, + add_months, + days_from_civil, + days_in_month, + fnv1a32, + iso_date, + month_key, +) + +# ------------------------------------------------------------ constants -- + +MAX_TURNS = 8 +WINDOW_MONTHS = 12 +SCREEN_COUNTERPARTIES = 2 # top-N by volume shown on the free screen + +# Mix by seed. Deliberately not the real false-positive base rate — see +# README.md, "what this is easier than". +BENIGN_PCT = 55 +VISIBLE_PCT = 30 +# hidden = the remaining 15 + +FAMILIES = ("STR", "VEL", "CASH", "WIRE", "DORM") + +# Rule ids per slice. The held-out slice renames them so a policy that learned +# "9,xxx and R-STR-01" rather than the pattern is caught. +RULE_IDS = { + "main": {"STR": "R-STR-01", "VEL": "R-VEL-02", "CASH": "R-CASH-03", "WIRE": "R-WIRE-04", "DORM": "R-DORM-05"}, + "held_out": {"STR": "TM-A1", "VEL": "TM-A2", "CASH": "TM-A3", "WIRE": "TM-A4", "DORM": "TM-A5"}, +} + +RULE_TEXT = { + "STR": "Three or more cash deposits just under the reporting threshold within ten days", + "VEL": "Funds received and moved out again within 48 hours", + "CASH": "Monthly cash deposits materially above the trailing six-month average", + "WIRE": "Outbound wires to a jurisdiction on the institution's monitored list", + "DORM": "Large credit to an account with no material activity for nine months", +} + +# The typology the rule nominally points at — what a defensive filer would +# write. Diagnostic only; `typology` is never scored. +NOMINAL_TYPOLOGY = {"STR": "STRUCTURING", "VEL": "RAPID_MOVEMENT", "CASH": "STRUCTURING", "WIRE": "UNKNOWN", "DORM": "UNKNOWN"} + +CURRENCIES = { + # threshold in minor units; scale multiplies every generated amount + "main": {"currency": "USD", "threshold": 1_000_000, "scale": 1}, + "held_out": {"currency": "KRN", "threshold": 2_000_000, "scale": 2}, +} + +BENIGN_TEMPLATES = ("B1", "B2", "B3", "B4", "B5", "B6") +VISIBLE_TEMPLATES = ("S1", "S3", "S6") +OVERLAYS = ("funnel", "serial_closer", "doc_mismatch") +DOC_TEMPLATES = ("B2", "B3", "B6") # where a document is the explaining fact + +TEMPLATE_FAMILY = {"B1": "CASH", "B2": "VEL", "B3": "VEL", "B4": "STR", "B5": "WIRE", "B6": "DORM", "S1": "STR", "S3": "VEL", "S6": "CASH"} + +INBOUND = ("cash_in", "wire_in", "ach_in", "p2p_in") + +_NAMES = Path(__file__).parent.parent / "names" + + +def _load(name: str): + return json.loads((_NAMES / name).read_text()) + + +FIRST_NAMES: list[str] = _load("first_names.json") +SURNAMES: list[str] = _load("surnames.json") +WORDS: dict[str, list[str]] = _load("words.json") +JURISDICTIONS: dict[str, Any] = _load("jurisdictions.json") + +BOILERPLATE = "Reviewed. Activity consistent with customer profile. No further action." + +BENIGN_RATIONALE = { + "STR": "Closed: deposits are documented daily takings; amounts vary with trading days; within kyc.expected_monthly_cash.", + "VEL": "Closed: pass-through matches the settlement terms in the agreement on file; counterparties verified.", + "CASH": "Closed: seasonal peak consistent with kyc.expected_activity; volume within stated expectation.", + "WIRE": "Closed: destination and amount match the remittance declared at onboarding; within stated income.", + "DORM": "Closed: reactivation explained by a documented source of funds; document reviewed and retained.", +} + + +# ----------------------------------------------------------- utilities -- + + +def canonical_json(value: Any) -> str: + return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False) + + +def _round_to(cents: int, step: int) -> int: + return (cents // step) * step + + +class _Ledger: + """Transactions are collected unordered, then ids are assigned by date. + + A stable sort on the day ordinal, with insertion order breaking ties, is + what makes T-ids identical across the two implementations. + """ + + def __init__(self) -> None: + self.rows: list[dict[str, Any]] = [] + + def add(self, day: int, channel: str, amount: int, branch: str | None = None, cp: str | None = None) -> int: + self.rows.append({"day": day, "seq": len(self.rows), "channel": channel, "amount": amount, "branch": branch, "cp": cp}) + return len(self.rows) - 1 + + def finish(self) -> tuple[list[dict[str, Any]], dict[int, str]]: + ordered = sorted(self.rows, key=lambda r: (r["day"], r["seq"])) + id_of: dict[int, str] = {} + out = [] + for i, row in enumerate(ordered): + tid = f"T-{i + 1}" + id_of[row["seq"]] = tid + out.append({ + "id": tid, + "date": iso_date(row["day"]), + "month": iso_date(row["day"])[:7], + "channel": row["channel"], + "amount": row["amount"], + "branch": row["branch"], + "cp": row["cp"], + }) + return out, id_of + + +class _Ctx: + """Everything a template builder needs, plus the accumulating world.""" + + def __init__(self, seed: int, rng: XorShift32, slice_name: str) -> None: + self.seed = seed + self.rng = rng + self.slice = slice_name + cur = CURRENCIES[slice_name] + self.currency = cur["currency"] + self.threshold = cur["threshold"] + self.scale = cur["scale"] + self.rule_ids = RULE_IDS[slice_name] + self.home = JURISDICTIONS["home"][self.currency] + self.ledger = _Ledger() + self.cps: list[dict[str, Any]] = [] + self.customer: dict[str, Any] = {} + self.months: list[tuple[int, int]] = [] + self.fire_day = 0 + self.fire_y = 0 + self.fire_m = 0 + self.documents: list[dict[str, Any]] = [] + self.doc_party_cp: str | None = None # the CP a party-bearing document names + self.branches = rng.sample(WORDS["branches"], 2) + + # money: `usd(n)` is n whole units of the slice currency, in minor units. + def usd(self, n: int) -> int: + return n * 100 * self.scale + + def money(self, lo: int, hi: int, step: int = 10) -> int: + """A random whole-unit amount in [lo, hi], rounded down to `step` units.""" + return _round_to(self.usd(self.rng.between(lo, hi)), self.usd(step)) + + def set_fire(self, month_choices: list[int], day_lo: int = 12, day_hi: int = 27) -> None: + self.fire_y = 2026 + self.fire_m = self.rng.pick(month_choices) + self.fire_day = days_from_civil(self.fire_y, self.fire_m, self.rng.between(day_lo, day_hi)) + self.months = [add_months(self.fire_y, self.fire_m, -(WINDOW_MONTHS - 1 - i)) for i in range(WINDOW_MONTHS)] + + def fire_dom(self) -> int: + return self.fire_day - days_from_civil(self.fire_y, self.fire_m, 1) + 1 + + def day_in(self, i: int, lo: int = 1, hi: int | None = None) -> int: + """A day ordinal inside window month i. Nothing is ever dated on or after the fire date.""" + y, m = self.months[i] + last = days_in_month(y, m) if hi is None else hi + if i == WINDOW_MONTHS - 1: + last = min(last, self.fire_dom() - 1) + lo = min(lo, last) + return days_from_civil(y, m, self.rng.between(lo, last)) + + def days_in(self, i: int, k: int, lo: int = 1, hi: int | None = None) -> list[int]: + """k distinct days in month i, ascending.""" + y, m = self.months[i] + last = days_in_month(y, m) if hi is None else hi + if i == WINDOW_MONTHS - 1: + last = min(last, self.fire_dom() - 1) + lo = min(lo, last) + first = days_from_civil(y, m, 1) + picks = self.rng.sample(list(range(lo, last + 1)), k) + return sorted(first + d - 1 for d in picks) + + def person(self) -> str: + return f"{self.rng.pick(FIRST_NAMES)} {self.rng.pick(SURNAMES)}" + + def business(self, noun: str | None = None) -> str: + adj = self.rng.pick(WORDS["adjectives"]) + noun = noun or self.rng.pick(WORDS["nouns"]) + return f"{adj} {noun} {self.rng.pick(WORDS['suffixes'])}" + + def cp(self, name: str, kind: str, jurisdiction: str | None = None) -> str: + cid = f"CP-{len(self.cps) + 1}" + self.cps.append({"id": cid, "name": name, "type": kind, "jurisdiction": jurisdiction or self.home}) + return cid + + def doc(self, kind: str, description: str, party: str | None = None) -> str: + while True: + did = f"doc.D-{self.rng.hex4()}" + if all(d["id"] != did for d in self.documents): + break + self.documents.append({"id": did, "kind": kind, "description": description, "party": party}) + return did + + def add(self, day: int, channel: str, amount: int, branch: str | None = None, cp: str | None = None) -> int: + return self.ledger.add(day, channel, amount, branch, cp) + + def branch(self) -> str: + return self.rng.pick(self.branches) + + def max_month_cash(self) -> int: + """The largest month of cash actually deposited so far — what the KYC expectation must cover.""" + totals: dict[int, int] = {} + for row in self.ledger.rows: + if row["channel"] == "cash_in": + key = iso_date(row["day"])[:7] + idx = int(key[:4]) * 12 + int(key[5:]) + totals[idx] = totals.get(idx, 0) + row["amount"] + return max(totals.values()) if totals else 0 + + +def _kyc_common(ctx: _Ctx, customer_type: str) -> None: + """The KYC draws every template makes, in the same order. + + Prior-alert and prior-SAR counts are drawn HERE, before any template or + label-specific code runs, from one distribution. The serial-closer overlay + rewrites what the prior alerts say; it can never change how many there are, + which is the number the screen shows. + """ + rng = ctx.rng + ctx.customer = { + "id": f"C-{rng.between(1000, 9999)}", + "name": "", + "customer_type": customer_type, + "country": ctx.home, + "pep": rng.chance(3), + "prior_alerts": 0, + "prior_sars": 0, + "remittance_declared": "none declared", + } + roll = rng.below(100) + ctx.customer["prior_alerts"] = 0 if roll < 35 else 1 if roll < 55 else 2 if roll < 70 else 3 if roll < 88 else 4 + ctx.customer["prior_sars"] = 1 if ctx.customer["prior_alerts"] >= 1 and rng.chance(8) else 0 + opened_years = rng.between(2, 15) + ctx.customer["account_opened"] = iso_date(ctx.fire_day - opened_years * 365 - rng.between(0, 300)) + ctx.customer["kyc_refreshed"] = iso_date(ctx.fire_day - rng.between(40, 500)) + + +def _salary(ctx: _Ctx, monthly: int, employer: str) -> None: + for i in range(WINDOW_MONTHS): + amount = monthly + ctx.usd(ctx.rng.between(-60, 60)) + ctx.add(ctx.day_in(i, 1, 5), "ach_in", amount, cp=employer) + + +def _small_cash_out(ctx: _Ctx, lo: int, hi: int, per_month_lo: int, per_month_hi: int) -> None: + for i in range(WINDOW_MONTHS): + n = ctx.rng.between(per_month_lo, per_month_hi) + for day in ctx.days_in(i, n): + ctx.add(day, "cash_out", ctx.money(lo, hi), branch=ctx.branch()) + + +def _cash_deposits(ctx: _Ctx, i: int, total: int, n: int, lo_cap: int, hi_cap: int, days: list[int] | None = None) -> list[int]: + """n cash deposits in month i summing roughly to `total`, each clamped.""" + days = days if days is not None else ctx.days_in(i, n) + out = [] + for day in days: + amount = total // n * ctx.rng.between(75, 125) // 100 + amount = max(lo_cap, min(hi_cap, _round_to(amount, ctx.usd(10)))) + out.append(ctx.add(day, "cash_in", amount, branch=ctx.branch())) + return out + + +# --------------------------------------------------------------- benign -- + + +def _build_B1(ctx: _Ctx) -> dict[str, Any]: + """Seasonal cash business. CASH fires on the first peak month; the KYC file said it would.""" + rng = ctx.rng + kind = rng.pick([ + ("landscaping services", "Landscaping", [4, 5, 6, 7, 8, 9], [4, 5], "April to September"), + ("tax preparation", "Tax Services", [2, 3, 4], [3], "February to April"), + ("pool maintenance", "Pool Care", [5, 6, 7, 8], [5, 6], "May to August"), + ]) + business_type, noun, peak, fire_choices, peak_text = kind + ctx.set_fire(fire_choices) + _kyc_common(ctx, "business") + ctx.customer["name"] = ctx.business(noun) + ctx.customer["business_type"] = business_type + off = ctx.money(3000, 7000, 100) + mult = rng.between(30, 40) + supplier = ctx.cp(ctx.business("Supply"), "business") + equipment = ctx.cp(ctx.business("Equipment"), "business") + for i, (y, m) in enumerate(ctx.months): + in_peak = m in peak + total = off * mult // 10 if in_peak else off + total = total * rng.between(85, 115) // 100 + n = rng.between(8, 12) if in_peak else rng.between(3, 5) + _cash_deposits(ctx, i, total, n, ctx.usd(200), ctx.threshold * 8 // 10 - ctx.usd(100)) + ctx.add(ctx.day_in(i, 20, 28), "ach_out", total * rng.between(20, 35) // 100, cp=supplier) + if m % 3 == 0: + ctx.add(ctx.day_in(i, 6, 12), "wire_out", ctx.money(2500, 9000, 100), cp=equipment) + expected = _round_to(ctx.max_month_cash() * rng.between(110, 130) // 100, ctx.usd(1000)) + ctx.usd(1000) + ctx.customer["expected_monthly_cash"] = expected + ctx.customer["expected_activity"] = f"Seasonal business; cash receipts peak {peak_text}." + ctx.customer["stated_income"] = _round_to(expected * 7, ctx.usd(1000)) + ctx.doc("registration", f"Business registration — {ctx.customer['name']}") + ctx.doc("licence", f"Trade licence, {business_type}, renewed annually") + return { + "family": "CASH", + "triggering": "fire_month_cash", + "planted": [["kyc.expected_activity", "kyc.expected_monthly_cash"], ["kyc.expected_monthly_cash"]], + } + + +def _build_B4(ctx: _Ctx) -> dict[str, Any]: + """Cash restaurant. Three of the month's deposits land in the band; the file expected more cash than this.""" + rng = ctx.rng + business_type, noun = rng.pick([("restaurant", "Bistro"), ("diner", "Diner"), ("pizzeria", "Kitchen"), ("cafe", "Grill")]) + ctx.set_fire([3, 4, 5, 6], 14, 27) + _kyc_common(ctx, "business") + ctx.customer["name"] = ctx.business(noun) + ctx.customer["business_type"] = business_type + base = ctx.money(40000, 75000, 1000) + produce = ctx.cp(ctx.business("Produce"), "business") + foods = ctx.cp(ctx.business("Foods"), "business") + band_lo = ctx.threshold * 8 // 10 + for i in range(WINDOW_MONTHS): + total = base * rng.between(90, 110) // 100 + n = rng.between(11, 14) + if i == WINDOW_MONTHS - 1: + # Three window deposits in the band, on distinct days; the rest ordinary. + fire_dom = ctx.fire_dom() + window_days = ctx.days_in(i, 3, max(1, fire_dom - 10), fire_dom - 1) + for day in window_days: + ctx.add(day, "cash_in", ctx.money(band_lo // ctx.usd(1) + 100, ctx.threshold // ctx.usd(1) - 100), branch=ctx.branch()) + n -= 3 + _cash_deposits(ctx, i, total, n, ctx.usd(1500), band_lo - ctx.usd(100)) + for day in ctx.days_in(i, 4, 1, 28): + ctx.add(day, "ach_out", total * rng.between(4, 7) // 100, cp=produce) + ctx.add(ctx.day_in(i, 1, 10), "ach_out", total * rng.between(10, 16) // 100, cp=foods) + expected = _round_to(ctx.max_month_cash() * rng.between(110, 130) // 100, ctx.usd(1000)) + ctx.usd(1000) + ctx.customer["expected_monthly_cash"] = expected + ctx.customer["expected_activity"] = "Cash-intensive food service; daily takings deposited most trading days." + ctx.customer["stated_income"] = _round_to(base * 12, ctx.usd(1000)) + ctx.doc("registration", f"Business registration — {ctx.customer['name']}") + ctx.doc("licence", "Food service licence, renewed annually") + return { + "family": "STR", + "triggering": "band_window", + "planted": [["kyc.expected_monthly_cash"], ["kyc.expected_monthly_cash", "kyc.business_type"]], + } + + +def _build_B2(ctx: _Ctx) -> dict[str, Any]: + """Processor settlement. Wires in from a processor, contractors paid within two days; the agreement is on file.""" + rng = ctx.rng + business_type, noun = rng.pick([("staffing agency", "Staffing"), ("freight brokerage", "Freight"), ("event production", "Events"), ("general contractor", "Contracting")]) + ctx.set_fire([3, 4, 5, 6], 14, 27) + _kyc_common(ctx, "business") + ctx.customer["name"] = ctx.business(noun) + ctx.customer["business_type"] = business_type + processor_name = f"{rng.pick(WORDS['adjectives'])} Payments LLC" + processor = ctx.cp(processor_name, "processor") + ctx.doc_party_cp = processor + contractors = [ctx.cp(ctx.business(), "business") for _ in range(2)] + revenue = 0 + window: list[int] = [] + for i in range(WINDOW_MONTHS): + k = rng.between(3, 4) + days = ctx.days_in(i, k, 1, 24) + if i == WINDOW_MONTHS - 1: + # Guarantee one settlement inside the 14-day window with its payouts before the fire date. + fire_dom = ctx.fire_dom() + days = ctx.days_in(i, k - 1, 1, max(1, fire_dom - 8)) + [ctx.fire_day - rng.between(3, 6)] + for day in days: + amount = ctx.money(15000, 60000, 100) + revenue += amount + t_in = ctx.add(day, "wire_in", amount, cp=processor) + share = rng.between(70, 90) + first = amount * share // 100 * rng.between(50, 70) // 100 + second = amount * share // 100 - first + t_a = ctx.add(day + 1, "wire_out", _round_to(first, ctx.usd(10)), cp=contractors[0]) + t_b = ctx.add(day + rng.between(1, 2), "ach_out", _round_to(second, ctx.usd(10)), cp=contractors[1]) + if i == WINDOW_MONTHS - 1 and day >= ctx.fire_day - 14: + window += [t_in, t_a, t_b] + ctx.customer["expected_monthly_cash"] = ctx.usd(0) + ctx.customer["expected_activity"] = "Card settlements from a payment processor; subcontractor payouts within days of settlement." + ctx.customer["stated_income"] = _round_to(revenue, ctx.usd(10000)) + ctx.doc("registration", f"Business registration — {ctx.customer['name']}") + ctx.doc("processing_agreement", f"Merchant processing agreement between {ctx.customer['name']} and {processor_name}; settlement to this account on T+1.", party=processor_name) + return {"family": "VEL", "triggering": window, "planted": [["DOC"], ["DOC", "kyc.business_type"]]} + + +def _build_B3(ctx: _Ctx) -> dict[str, Any]: + """Property sale. One title-company wire in, moved to a brokerage; closing statement on file.""" + rng = ctx.rng + ctx.set_fire([3, 4, 5, 6], 14, 27) + _kyc_common(ctx, "individual") + ctx.customer["name"] = ctx.person() + ctx.customer["occupation"] = rng.pick(["teacher", "software engineer", "pharmacist", "electrician", "accountant"]) + salary = ctx.money(4000, 9000, 100) + employer = ctx.cp(ctx.business(), "business") + _salary(ctx, salary, employer) + _small_cash_out(ctx, 100, 400, 2, 4) + title_name = f"{rng.pick(SURNAMES)} Title & Escrow" + title = ctx.cp(title_name, "title_company") + ctx.doc_party_cp = title + brokerage = ctx.cp(f"{rng.pick(WORDS['adjectives'])} Securities", "brokerage") + proceeds = ctx.money(180000, 650000, 1000) + day_in = ctx.fire_day - rng.between(3, 6) + t_in = ctx.add(day_in, "wire_in", proceeds, cp=title) + t_out = ctx.add(day_in + rng.between(1, 2), "wire_out", _round_to(proceeds * rng.between(85, 100) // 100, ctx.usd(100)), cp=brokerage) + ctx.customer["expected_monthly_cash"] = ctx.usd(0) + ctx.customer["expected_activity"] = "Salaried individual; payroll deposits and routine spending." + ctx.customer["stated_income"] = _round_to(salary * 12, ctx.usd(1000)) + ctx.doc("identity", "Government-issued identity document, verified at onboarding") + address = f"{rng.between(12, 980)} {rng.pick(WORDS['streets'])}" + ctx.doc("closing_statement", f"Closing statement — sale of {address}; net proceeds to seller disbursed by {title_name}.", party=title_name) + return {"family": "VEL", "triggering": [t_in, t_out], "planted": [["DOC"], ["DOC", "kyc.occupation"]]} + + +def _build_B5(ctx: _Ctx) -> dict[str, Any]: + """Family remittance. A monthly wire to one relative in a monitored jurisdiction, declared at onboarding.""" + rng = ctx.rng + ctx.set_fire([3, 4, 5, 6]) + _kyc_common(ctx, "individual") + surname = rng.pick(SURNAMES) + ctx.customer["name"] = f"{rng.pick(FIRST_NAMES)} {surname}" + ctx.customer["occupation"] = rng.pick(["registered nurse", "civil engineer", "teacher", "pharmacist"]) + salary = ctx.money(4500, 8500, 100) + employer = ctx.cp(ctx.business(), "business") + _salary(ctx, salary, employer) + _small_cash_out(ctx, 100, 400, 2, 4) + jurisdiction = rng.pick(JURISDICTIONS["monitored"]) + relation = rng.pick(["mother", "father", "brother", "sister"]) + relative = ctx.cp(f"{rng.pick(FIRST_NAMES)} {surname}", "individual", jurisdiction) + monthly = ctx.money(800, 2500, 50) + wires = [] + for i in range(WINDOW_MONTHS): + wires.append(ctx.add(ctx.day_in(i, 1, 7), "wire_out", monthly + ctx.usd(rng.between(-50, 50)), cp=relative)) + ctx.customer["expected_monthly_cash"] = ctx.usd(0) + ctx.customer["expected_activity"] = "Salaried individual; payroll deposits and routine spending." + ctx.customer["remittance_declared"] = f"Monthly support to {relation} in {jurisdiction}, about {monthly // 100 // ctx.scale * ctx.scale:,} {ctx.currency} per month." + ctx.customer["stated_income"] = _round_to(salary * 12, ctx.usd(1000)) + ctx.doc("identity", "Government-issued identity document, verified at onboarding") + ctx.doc("address", "Proof of address, utility statement") + return {"family": "WIRE", "triggering": wires[-3:], "planted": [["kyc.remittance_declared"], ["kyc.remittance_declared", "kyc.stated_income"]]} + + +def _build_B6(ctx: _Ctx) -> dict[str, Any]: + """Inheritance. A dormant account receives an estate distribution; the executor's letter is on file.""" + rng = ctx.rng + ctx.set_fire([3, 4, 5, 6], 14, 27) + _kyc_common(ctx, "individual") + ctx.customer["name"] = ctx.person() + ctx.customer["occupation"] = "retired" + ctx.customer["account_opened"] = iso_date(ctx.fire_day - rng.between(8, 20) * 365 - rng.between(0, 300)) + for i in range(WINDOW_MONTHS - 1): + if rng.chance(35): + ctx.add(ctx.day_in(i), "cash_out", ctx.money(40, 200), branch=ctx.branch()) + firm = f"{rng.pick(SURNAMES)} & {rng.pick(SURNAMES)} LLP, client trust account" + trust = ctx.cp(firm, "law_firm_trust") + ctx.doc_party_cp = trust + brokerage = ctx.cp(f"{rng.pick(WORDS['adjectives'])} Securities", "brokerage") + amount = ctx.money(60000, 400000, 1000) + day_in = ctx.fire_day - rng.between(4, 7) + t_in = ctx.add(day_in, "wire_in", amount, cp=trust) + t_out = ctx.add(day_in + rng.between(1, 3), "wire_out", _round_to(amount * rng.between(50, 90) // 100, ctx.usd(100)), cp=brokerage) + ctx.customer["expected_monthly_cash"] = ctx.usd(0) + ctx.customer["expected_activity"] = "Retired; low activity expected." + ctx.customer["stated_income"] = ctx.money(20000, 40000, 1000) + ctx.doc("identity", "Government-issued identity document, verified at onboarding") + executor, decedent = ctx.person(), ctx.person() + ctx.doc("executor_letter", f"Letter from {executor}, executor, via {firm}: distribution from the estate of {decedent} to the named beneficiary.", party=firm) + return {"family": "DORM", "triggering": [t_in, t_out], "planted": [["DOC"]]} + + +# -------------------------------------------------------------- visible -- + + +def _build_S1(ctx: _Ctx) -> dict[str, Any]: + """Structuring. Eight deposits just under the threshold in twelve days across three branches, on a modest income.""" + rng = ctx.rng + ctx.set_fire([3, 4, 5, 6], 14, 27) + _kyc_common(ctx, "individual") + ctx.customer["name"] = ctx.person() + ctx.customer["occupation"] = rng.pick(["warehouse associate", "delivery driver", "retail supervisor", "line cook"]) + ctx.branches = rng.sample(WORDS["branches"], 3) + salary = ctx.money(2600, 3600, 100) + employer = ctx.cp(ctx.business(), "business") + _salary(ctx, salary, employer) + _small_cash_out(ctx, 60, 300, 2, 4) + fire_dom = ctx.fire_dom() + days = ctx.days_in(WINDOW_MONTHS - 1, 8, max(1, fire_dom - 12), fire_dom - 2) + lo = ctx.threshold // ctx.usd(1) * 92 // 100 + hi = ctx.threshold // ctx.usd(1) * 99 // 100 + deposits = [] + total = 0 + for j, day in enumerate(days): + amount = ctx.money(lo, hi) + total += amount + deposits.append(ctx.add(day, "cash_in", amount, branch=ctx.branches[j % 3])) + holdings = ctx.cp(ctx.business("Holdings"), "business") + ctx.add(ctx.fire_day - 1, "wire_out", _round_to(total * rng.between(85, 95) // 100, ctx.usd(100)), cp=holdings) + ctx.customer["expected_monthly_cash"] = ctx.money(0, 500, 100) + ctx.customer["expected_activity"] = "Salaried individual; payroll deposits and routine spending." + ctx.customer["stated_income"] = _round_to(salary * 12 + ctx.usd(rng.between(0, 6000)), ctx.usd(1000)) + ctx.doc("identity", "Government-issued identity document, verified at onboarding") + return {"family": "STR", "triggering": "band_window", "planted": [deposits], "typology": "STRUCTURING"} + + +def _build_S3(ctx: _Ctx) -> dict[str, Any]: + """Money mule. Four P2P credits from unrelated individuals, each wired to an exchange within a day.""" + rng = ctx.rng + ctx.set_fire([3, 4, 5, 6], 16, 27) + _kyc_common(ctx, "individual") + ctx.customer["name"] = ctx.person() + ctx.customer["occupation"] = rng.pick(["student", "retired", "part-time cashier"]) + stipend = ctx.money(900, 1800, 50) + source = ctx.cp(ctx.business(), "business") + _salary(ctx, stipend, source) + _small_cash_out(ctx, 40, 200, 2, 4) + exchange = ctx.cp(f"{rng.pick(WORDS['adjectives'])} Digital Assets", "exchange") + fire_dom = ctx.fire_dom() + days = ctx.days_in(WINDOW_MONTHS - 1, 4, max(1, fire_dom - 14), fire_dom - 2) + pairs = [] + for day in days: + sender = ctx.cp(ctx.person(), "individual") + amount = ctx.money(2000, 4900, 10) + t_in = ctx.add(day, "p2p_in", amount, cp=sender) + t_out = ctx.add(day + rng.between(0, 1), "wire_out", _round_to(amount * rng.between(90, 97) // 100, ctx.usd(10)), cp=exchange) + pairs += [t_in, t_out] + ctx.customer["expected_monthly_cash"] = ctx.money(0, 300, 100) + ctx.customer["expected_activity"] = "Low-income individual; small regular credits and routine spending." + ctx.customer["stated_income"] = _round_to(stipend * 12, ctx.usd(1000)) + ctx.doc("identity", "Government-issued identity document, verified at onboarding") + return {"family": "VEL", "triggering": pairs, "planted": [pairs], "typology": "MONEY_MULE"} + + +def _build_S6(ctx: _Ctx) -> dict[str, Any]: + """Smurfing over time. Monthly cash triples across the year and no single deposit ever reaches the threshold.""" + rng = ctx.rng + business_type, noun = rng.pick([("vending route", "Vending"), ("car wash", "Car Wash"), ("laundromat", "Laundry"), ("convenience store", "Market")]) + ctx.set_fire([3, 4, 5, 6], 14, 27) + _kyc_common(ctx, "business") + ctx.customer["name"] = ctx.business(noun) + ctx.customer["business_type"] = business_type + c0 = ctx.money(9000, 14000, 500) + supplier = ctx.cp(ctx.business("Supply"), "business") + cap = ctx.threshold - ctx.usd(100) + for i in range(WINDOW_MONTHS): + total = c0 * (11 + 2 * i) // 11 * rng.between(97, 103) // 100 + n = rng.between(6, 9) + _cash_deposits(ctx, i, total, n, ctx.usd(2000), cap) + ctx.add(ctx.day_in(i, 20, 28), "ach_out", total * rng.between(15, 25) // 100, cp=supplier) + ctx.customer["expected_monthly_cash"] = _round_to(c0 * 12 // 10, ctx.usd(500)) + ctx.customer["expected_activity"] = "Cash-intensive retail; steady volume expected." + ctx.customer["stated_income"] = _round_to(c0 * 14, ctx.usd(1000)) + ctx.doc("registration", f"Business registration — {ctx.customer['name']}") + return {"family": "CASH", "triggering": "fire_month_cash", "planted": "last_two_months", "typology": "STRUCTURING"} + + +BUILDERS = {"B1": _build_B1, "B2": _build_B2, "B3": _build_B3, "B4": _build_B4, "B5": _build_B5, "B6": _build_B6, "S1": _build_S1, "S3": _build_S3, "S6": _build_S6} + + +# ------------------------------------------------------------- assembly -- + + +def _prior_alerts(ctx: _Ctx, family: str) -> list[dict[str, Any]]: + """Benign prior alerts: mixed rules, specific rationales, a few days each.""" + rng = ctx.rng + out: list[dict[str, Any]] = [] + used: set[str] = set() + for j in range(ctx.customer["prior_alerts"]): + while True: + aid = f"A-{rng.between(1000, 4999)}" + if aid not in used: + used.add(aid) + break + fam = rng.pick(list(FAMILIES)) + fired = ctx.fire_day - rng.between(30, 720) + escalated = j == 0 and ctx.customer["prior_sars"] >= 1 + out.append({ + "id": aid, + "rule": ctx.rule_ids[fam], + "fired": iso_date(fired), + "disposition": "escalated — report filed" if escalated else "closed", + "analyst": ctx.person(), + "rationale": "Escalated: pattern not explained by the file at the time." if escalated else BENIGN_RATIONALE[fam], + "elapsed_days": rng.between(2, 9), + }) + out.sort(key=lambda a: a["fired"]) + return out + + +def _assemble(ctx: _Ctx, template: str, spec: dict[str, Any]) -> dict[str, Any]: + transactions, id_of = ctx.ledger.finish() + family = spec["family"] + + # Triggering set, from the rule where the rule is mechanical. + if spec["triggering"] == "band_window": + lo, hi = ctx.threshold * 8 // 10, ctx.threshold - 1 + triggering = [t["id"] for t in transactions + if t["channel"] == "cash_in" and lo <= t["amount"] <= hi + and ctx.fire_day - 10 <= days_from_civil(*map(int, t["date"].split("-"))) <= ctx.fire_day - 1] + elif spec["triggering"] == "fire_month_cash": + fire_month = month_key(ctx.fire_y, ctx.fire_m) + triggering = [t["id"] for t in transactions if t["channel"] == "cash_in" and t["month"] == fire_month] + else: + triggering = sorted((id_of[s] for s in spec["triggering"]), key=lambda x: int(x[2:])) + + # Counterparties: transactions and first-seen from the ledger, direction from the channel. + by_cp: dict[str, list[dict[str, Any]]] = {c["id"]: [] for c in ctx.cps} + for t in transactions: + if t["cp"]: + by_cp[t["cp"]].append(t) + counterparties = [] + for c in ctx.cps: + rows = by_cp[c["id"]] + counterparties.append({ + **c, + "first_seen": rows[0]["date"] if rows else iso_date(ctx.fire_day), + "other_alerted_accounts": 1 if ctx.rng.chance(15) else 0, + "transactions": [t["id"] for t in rows], + }) + ranked = sorted(counterparties, key=lambda c: (-sum(t["amount"] for t in by_cp[c["id"]]), int(c["id"][3:]))) + screen_cps = [] + for c in ranked[:SCREEN_COUNTERPARTIES]: + rows = by_cp[c["id"]] + screen_cps.append({ + "id": c["id"], + "direction": "in" if rows and rows[0]["channel"] in INBOUND else "out", + "count": len(rows), + "total": sum(t["amount"] for t in rows), + }) + + summary = [] + for (y, m) in ctx.months: + key = month_key(y, m) + row = {"month": key, "cash_in": 0, "cash_out": 0, "wires_in": 0, "wires_out": 0, "ach_in": 0, "ach_out": 0, "count": 0} + for t in transactions: + if t["month"] != key: + continue + row["count"] += 1 + ch = t["channel"] + col = {"cash_in": "cash_in", "cash_out": "cash_out", "wire_in": "wires_in", "wire_out": "wires_out", + "ach_in": "ach_in", "ach_out": "ach_out", "p2p_in": "wires_in", "p2p_out": "wires_out"}[ch] + row[col] += t["amount"] + summary.append(row) + + # Planted citation sets. "DOC" is the template's explaining document. + explaining = next((d["id"] for d in ctx.documents if d["party"]), None) + if spec["planted"] == "last_two_months": + last_two = {month_key(*ctx.months[-1]), month_key(*ctx.months[-2])} + planted = [[t["id"] for t in transactions if t["channel"] == "cash_in" and t["month"] in last_two]] + else: + planted = [] + for alt in spec["planted"]: + planted.append([ + (explaining if x == "DOC" else id_of[x] if isinstance(x, int) else x) for x in alt + ]) + + world = { + "seed": ctx.seed, + "held_out": ctx.slice == "held_out", + "currency": ctx.currency, + "threshold": ctx.threshold, + "screen_order": "kyc_first" if ctx.slice == "held_out" else "alert_first", + "customer": ctx.customer, + "months": [month_key(y, m) for (y, m) in ctx.months], + "transactions": transactions, + "counterparties": counterparties, + "screen_counterparties": screen_cps, + "alert": { + "id": f"A-{ctx.rng.between(5000, 9999)}", + "rule": ctx.rule_ids[family], + "family": family, + "rule_text": RULE_TEXT[family], + "fired": iso_date(ctx.fire_day), + "triggering": triggering, + }, + "prior_alerts": _prior_alerts(ctx, family), + "documents": ctx.documents, + "summary": summary, + "template": template, + "overlay": None, + "label": "suspicious" if template.startswith("S") else "benign", + "typology": spec.get("typology"), + "planted": planted, + "doc_party_cp": ctx.doc_party_cp, + } + return world + + +# -------------------------------------------------------------- overlay -- + + +def compatible_overlays(world: dict[str, Any]) -> list[str]: + """Which hidden signals this benign world can carry. Depends on the screen, never on the label.""" + kinds = ["funnel"] + if world["customer"]["prior_alerts"] >= 3 and world["customer"]["prior_sars"] == 0: + kinds.append("serial_closer") + if world["template"] in DOC_TEMPLATES: + kinds.append("doc_mismatch") + return kinds + + +def _overlay_urn(kinds: list[str]) -> list[str]: + """funnel is always available, so the other two are drawn twice as often where they can be.""" + urn: list[str] = [] + for kind in kinds: + urn += [kind] * (1 if kind == "funnel" else 2) + return urn + + +def _outbound_target(world: dict[str, Any]) -> str: + """The highest-volume outbound counterparty on the screen. Every benign template has one.""" + for c in world["screen_counterparties"]: + if c["direction"] == "out": + return c["id"] + raise AssertionError(f"template {world['template']} has no outbound counterparty on screen") + + +def apply_overlay(world: dict[str, Any], kind: str, rng: XorShift32) -> None: + """Overlay a suspicious signal onto lookup data only. Mutates `world`. + + The screen — alert, triggering set, KYC block, summary, screen + counterparties — is not touched, and `tests/test_screen_leak.py` proves it. + """ + family = world["alert"]["family"] + triggering = world["alert"]["triggering"] + world["overlay"] = kind + world["label"] = "suspicious" + + if kind == "funnel": + target = _outbound_target(world) + cp = next(c for c in world["counterparties"] if c["id"] == target) + cp["other_alerted_accounts"] = rng.between(5, 14) + world["typology"] = "FUNNEL_ACCOUNT" + world["planted"] = [[target] + cp["transactions"], list(cp["transactions"])] + + elif kind == "serial_closer": + priors = world["prior_alerts"] + analysts = [] + while len(analysts) < len(priors): + name = f"{rng.pick(FIRST_NAMES)} {rng.pick(SURNAMES)}" + if name not in analysts: + analysts.append(name) + fire_day = days_from_civil(*map(int, world["alert"]["fired"].split("-"))) + days = sorted(rng.sample(list(range(fire_day - 330, fire_day - 30)), len(priors))) + for j, prior in enumerate(priors): + prior["rule"] = world["alert"]["rule"] + prior["fired"] = iso_date(days[j]) + prior["disposition"] = "closed" + prior["analyst"] = analysts[j] + prior["rationale"] = BOILERPLATE + prior["elapsed_days"] = rng.between(0, 1) + world["typology"] = NOMINAL_TYPOLOGY[family] + ids = [p["id"] for p in priors] + world["planted"] = [ids + triggering, ids] + + elif kind == "doc_mismatch": + target = world["doc_party_cp"] + cp = next(c for c in world["counterparties"] if c["id"] == target) + cp["name"] = f"{rng.pick(WORDS['adjectives'])} {rng.pick(['Holdings', 'Ventures', 'Trading'])} {rng.pick(WORDS['suffixes'])}" + doc = next(d["id"] for d in world["documents"] if d["party"]) + inbound = [t["id"] for t in world["transactions"] if t["id"] in triggering and t["cp"] == target] + world["typology"] = NOMINAL_TYPOLOGY[family] + world["planted"] = [[doc, target] + inbound, [doc, target]] + else: + raise ValueError(kind) + + +# -------------------------------------------------------------- public -- + + +def is_held_out(seed: int) -> bool: + return fnv1a32(str(seed)) % 8 == 7 + + +def generate(seed: int, overlay: str | None = "auto") -> dict[str, Any]: + """The world before reference hours. `overlay` is for tests: + "auto" follows the seed; None forces the benign twin; a kind forces it on.""" + h = fnv1a32(str(seed)) + slice_name = "held_out" if h % 8 == 7 else "main" + rng = XorShift32(h) + ctx = _Ctx(seed, rng, slice_name) + + roll = rng.below(100) + tier = "benign" if roll < BENIGN_PCT else "visible" if roll < BENIGN_PCT + VISIBLE_PCT else "hidden" + if tier == "visible": + template = VISIBLE_TEMPLATES[rng.below(len(VISIBLE_TEMPLATES))] + else: + template = BENIGN_TEMPLATES[rng.below(len(BENIGN_TEMPLATES))] + + spec = BUILDERS[template](ctx) + world = _assemble(ctx, template, spec) + world["tier"] = tier + + if tier != "visible": + kinds = compatible_overlays(world) + if overlay == "auto": + if tier == "hidden": + apply_overlay(world, rng.pick(_overlay_urn(kinds)), rng) + elif overlay is not None: + apply_overlay(world, overlay, rng) + del world["doc_party_cp"] + return world + + +def world_for_seed(seed: int) -> dict[str, Any]: + """The world with `reference_minutes` attached — part of the digest. + + The reference is the cheapest shipped policy that reached the correct + disposition on this seed, found by running them through the engine. The + policies read only the engine's view, never this dict, so attaching their + result here is not circular; it is the same move as wordle's + `reference_depth`, made part of the world so the browser recomputes it + instead of trusting the recorded number. + """ + from .policies import reference_for # local import: policies -> engine -> generator + + world = generate(seed) + minutes, policy = reference_for(world) + world["reference_minutes"] = minutes + world["reference_policy"] = policy + return world + + +SCREEN_KEYS = ("seed", "held_out", "currency", "threshold", "screen_order", "customer", "months", + "summary", "screen_counterparties", "alert") + + +def screen_of(world: dict[str, Any]) -> dict[str, Any]: + """Exactly what turn 0 shows, including the triggering transactions in full.""" + triggering = set(world["alert"]["triggering"]) + screen = {k: world[k] for k in SCREEN_KEYS} + screen["triggering_transactions"] = [t for t in world["transactions"] if t["id"] in triggering] + return screen + + +def world_digest(seeds=range(4096)) -> str: + """Digest (a): SHA-256 over the canonical world for each seed, newline-joined.""" + digest = hashlib.sha256() + for seed in seeds: + digest.update(canonical_json(world_for_seed(seed)).encode()) + digest.update(b"\n") + return digest.hexdigest() diff --git a/envs/alert_triage/alert_triage/ladder.py b/envs/alert_triage/alert_triage/ladder.py new file mode 100644 index 0000000..4e34e1d --- /dev/null +++ b/envs/alert_triage/alert_triage/ladder.py @@ -0,0 +1,165 @@ +"""The probe ladder for alert-triage. Invoked by `envs/probe.py`. + +Every rung is a policy from `policies.py`, played over the main-slice seeds +(held-out seeds are never probed). The assertions are the reward's claims, +checked per weighted component rather than as one blended float. Any cheat a +reviewer named is a rung here with an assertion that it loses — a cheat that +is measured is a cheat that cannot come back unnoticed. +""" + +from __future__ import annotations + +from .generator import is_held_out, world_for_seed +from .policies import POLICIES, run_policy +from .reward import ( + CAUGHT_WEIGHT, + EVIDENCE_WEIGHT, + ROLES, + WEIGHTS, + escalate_break_even, + metrics, + score, + total, +) + +# Main-slice seeds only: fnv1a(seed) % 8 == 7 is never probed. 1024 seeds +# because the margins asserted in rung 8 are population claims, and a +# 200-seed sample can draw 11% hidden instead of 15% and turn one into noise. +SEEDS = [s for s in range(1024) if not is_held_out(s)] +COMPONENTS = ("caught", "hours", "evidence") +GOOD = ("fast", "targeted", "thorough") +CHEATS = ("close_blind", "close_cites_all", "cite_unseen", "escalate_blind", "fast_defensive", "rule_table") + +# The one-slider counterfactual the page names: move `hours` here and the +# screen-only analyst wins. The other two weights keep their shipped ratio. +COUNTERFACTUAL_HOURS = 0.45 +MARGIN = 0.02 + + +def counterfactual_weights(hours_weight: float) -> dict[str, float]: + rest = 1.0 - hours_weight + share = CAUGHT_WEIGHT + EVIDENCE_WEIGHT + return {"caught": rest * CAUGHT_WEIGHT / share, "hours": hours_weight, "evidence": rest * EVIDENCE_WEIGHT / share} + + +def run(seeds: list[int] | None = None) -> int: + seeds = list(seeds or SEEDS) + worlds = [world_for_seed(s) for s in seeds] + tiers = {w["tier"] for w in worlds} + print(f"Probing alert-triage over {len(seeds)} main-slice seeds " + f"({sum(w['tier'] == 'benign' for w in worlds)} benign, " + f"{sum(w['tier'] == 'visible' for w in worlds)} visible, " + f"{sum(w['tier'] == 'hidden' for w in worlds)} hidden)\n") + + results: dict[str, dict[str, float]] = {} + cf: dict[str, float] = {} + per_seed: dict[str, list] = {} + for name, policy in POLICIES.items(): + episodes = [run_policy(policy, w).episode() for w in worlds] + scores = [score(e) for e in episodes] + assert all(v is not None for s in scores for v in s.values()), f"{name}: a component was None on a probe seed" + results[name] = {c: sum(s[c] for s in scores) / len(scores) for c in COMPONENTS} # type: ignore[misc] + results[name]["total"] = sum(total(e) for e in episodes) / len(episodes) # type: ignore[misc] + cf[name] = sum(total(e, counterfactual_weights(COUNTERFACTUAL_HOURS)) for e in episodes) / len(episodes) # type: ignore[misc] + per_seed[name] = episodes + + width = max(len(n) for n in POLICIES) + header = f"{'policy':<{width}} " + " ".join(f"{c:>10}" for c in COMPONENTS) + f" {'TOTAL':>8} {'hours@' + str(COUNTERFACTUAL_HOURS):>10}" + print(header) + print("-" * len(header)) + for name in POLICIES: + row = results[name] + cells = " ".join(f"{row[c]:>10.4f}" for c in COMPONENTS) + print(f"{name:<{width}} {cells} {row['total']:>8.4f} {cf[name]:>10.4f}") + + print(f"\nweights: {WEIGHTS}") + print(f"roles: {ROLES}") + print(f"blind-escalation break-even: escalate whenever P(benign) < {escalate_break_even():.2f}") + mean_hours = {n: round(sum(metrics(e)['hours_spent'] for e in per_seed[n]) / len(seeds), 2) for n in GOOD} + print(f"hours spent (mean): {mean_hours}") + hidden = [i for i, w in enumerate(worlds) if w["tier"] == "hidden"] + print("caught on hidden seeds:", {n: round(sum(score(per_seed[n][i])['caught'] for i in hidden) / max(1, len(hidden)), 3) for n in GOOD}) # type: ignore[arg-type] + + failures: list[str] = [] + r = results + + # 0. The seed set must contain every tier, or the assertions below measure noise. + if tiers != {"benign", "visible", "hidden"}: + failures.append(f"probe seeds cover tiers {sorted(tiers)}; need all three") + if any(w["reference_minutes"] is None for w in worlds): + failures.append("a probe seed has no reference_minutes — no shipped policy got it right") + + # 1. Doing nothing, and doing it malformed, must measure exactly nothing. + for name in ("inaction", "malformed"): + if abs(r[name]["total"]) > 1e-12: + failures.append(f"{name} scored {r[name]['total']:.6f}, must be 0.000") + if any(e.rejected != 8 or e.disposition is not None for e in per_seed[name]): + failures.append(f"{name} must be rejected on all eight turns and abort") + + # 2. The counterweight is attainable: the investigating references catch everything. + for name in ("targeted", "thorough"): + if abs(r[name]["caught"] - 1.0) > 1e-12: + failures.append(f"{name} caught {r[name]['caught']:.4f}, must be exactly 1.0") + + # 3. And it bites the policy that defines the hours denominator on most seeds. + if r["fast"]["caught"] >= 1.0 - 1e-12: + failures.append("fast caught 1.0 — the hidden tier leaked into the screen") + + # 4. Paying the counterweight's price is a real price. + if not (r["fast"]["hours"] > r["targeted"]["hours"] > r["thorough"]["hours"]): + failures.append("hours must order fast > targeted > thorough") + + # 5. No good policy dominates another on every component. + for a, b in (("fast", "targeted"), ("fast", "thorough"), ("targeted", "thorough")): + if all(r[a][c] >= r[b][c] - 1e-12 for c in COMPONENTS): + failures.append(f"{a} dominates {b} on every component — no real trade-off") + if all(r[b][c] >= r[a][c] - 1e-12 for c in COMPONENTS): + failures.append(f"{b} dominates {a} on every component — no real trade-off") + + # 6. Every named cheat loses to every good policy. + for cheat in CHEATS: + for good in GOOD: + if r[cheat]["total"] >= r[good]["total"] - 1e-12: + failures.append(f"{cheat} ({r[cheat]['total']:.4f}) >= {good} ({r[good]['total']:.4f})") + + # 7. The hour meter binds: a wasted month costs, and nothing else changes. + if r["thorough_wasteful"]["total"] >= r["thorough"]["total"] - 1e-12: + failures.append("thorough_wasteful >= thorough — the hour budget does not bind") + for c in ("caught", "evidence"): + if abs(r["thorough_wasteful"][c] - r["thorough"][c]) > 1e-12: + failures.append(f"thorough_wasteful differs from thorough on {c}") + + # 8. The headline claim, measured: targeted wins at shipped weights by a + # stated margin, and the one-slider counterfactual hands it to fast. + if r["targeted"]["total"] - r["fast"]["total"] < MARGIN: + failures.append(f"targeted beats fast by only {r['targeted']['total'] - r['fast']['total']:.4f} at shipped weights (need {MARGIN})") + if cf["fast"] - cf["targeted"] < MARGIN: + failures.append(f"fast beats targeted by only {cf['fast'] - cf['targeted']:.4f} at hours={COUNTERFACTUAL_HOURS} (need {MARGIN})") + + # 9. Citing an unopened document is rejected, every time. + if any(e.rejected < 1 for e in per_seed["cite_unseen"]): + failures.append("cite_unseen was not rejected on some seed") + + # 10. F1's precision term works: cite-everything earns less than half of its caught share in evidence. + if r["close_cites_all"]["evidence"] >= 0.5 * r["close_cites_all"]["caught"]: + failures.append("close_cites_all earns too much evidence — precision is not biting") + if r["close_cites_all"]["evidence"] >= r["fast"]["evidence"]: + failures.append("close_cites_all evidence >= fast evidence") + + # 11. History buys evidence on the visible typologies, and nothing else. + if r["thorough"]["evidence"] <= r["fast"]["evidence"]: + failures.append("thorough evidence <= fast evidence — itemised history buys nothing") + + # 12. No weighted component may be flat across the ladder. + for component in COMPONENTS: + values = [r[n][component] for n in POLICIES] + if max(values) - min(values) < 1e-9: + failures.append(f"component '{component}' is flat across the ladder — it measures nothing") + + print() + if failures: + for f in failures: + print(f"FAIL: {f}") + return 1 + print("All 13 ladder assertions hold.") + return 0 diff --git a/envs/alert_triage/alert_triage/policies.py b/envs/alert_triage/alert_triage/policies.py new file mode 100644 index 0000000..ab2850b --- /dev/null +++ b/envs/alert_triage/alert_triage/policies.py @@ -0,0 +1,346 @@ +"""The scripted analysts: three shipped references and the probe ladder's cheats. + +Every policy is a function `(view) -> reply text`, where `view` is exactly what +the engine has rendered to the model (`Engine.view()`), and nothing else. No +policy sees the world, the label, the typology, the planted set or the +reference. `tests/test_policies.py` enforces the signature and scans the view +for hidden keys. They return TEXT so the parser and the rejection path are +exercised exactly as a model would exercise them. + +The decision rules below are heuristics, written down so a reviewer can see +that the reference "analysts" are players, not oracles: + + fast decides from the screen; pulls `documents` only where the screen + says a document should explain the alert (a processor, a title + company, an estate). One or two turns. Cannot see a hidden signal. + targeted `fast` on a screen that shows a typology; otherwise the three + lookups that could carry a hidden signal — documents, prior + alerts, each counterparty on screen — then decides. No history. + thorough always the full procedure: documents, prior alerts, each + counterparty on screen, itemised history for the two most recent + months, then decides; cites every constituting transaction it saw. + +Hidden-signal rules, shared by targeted and thorough: + * a counterparty shared with ≥ 3 other alerted accounts → FUNNEL_ACCOUNT, + cite the counterparty and its transactions; + * ≥ 3 prior alerts, all closed within a day on one identical rationale + → the rule's typology, + cite the prior alerts and the triggering transactions; + * a document naming a party whose counterparty lookup carries a different + name → the rule's typology, + cite the document, the counterparty and its triggering credits. +""" + +from __future__ import annotations + +import json +from typing import Any, Callable + +from .generator import INBOUND, NOMINAL_TYPOLOGY, RULE_IDS + +Policy = Callable[[dict[str, Any]], str] + +_FAMILY_OF = {rid: fam for table in RULE_IDS.values() for fam, rid in table.items()} +EXPLAINING_DOCS = ("processing_agreement", "closing_statement", "executor_letter") + + +def _emit(obj: dict[str, Any]) -> str: + return json.dumps(obj) + + +def _family(view: dict[str, Any]) -> str: + return _FAMILY_OF.get(view["screen"]["alert"]["rule"], "STR") + + +def _looked(view: dict[str, Any], what: str, key: str | None = None) -> dict[str, Any] | None: + for l in view["lookups"]: + if l["what"] == what and (key is None or l["key"] == key): + return l["result"] + return None + + +def _band(view: dict[str, Any], t: dict[str, Any]) -> bool: + threshold = view["screen"]["threshold"] + return t["channel"] == "cash_in" and threshold * 8 // 10 <= t["amount"] < threshold + + +def _close(reason: str, cites: list[str], note: str) -> dict[str, Any]: + return {"action": "close", "reason": reason, "cites": cites, "note": note} + + +def _escalate(typology: str, cites: list[str], narrative: str) -> dict[str, Any]: + return {"action": "escalate", "typology": typology, "cites": cites, "narrative": narrative} + + +def _dedupe(ids: list[str]) -> list[str]: + out: list[str] = [] + for i in ids: + if i not in out: + out.append(i) + return out + + +# ------------------------------------------------------------ decisions -- + + +def screen_decision(view: dict[str, Any]) -> dict[str, Any] | None: + """What the screen alone supports. None means "a document should explain this".""" + screen = view["screen"] + fam = _family(view) + trig = screen["triggering_transactions"] + trig_ids = [t["id"] for t in trig] + kyc = screen["customer"] + max_cash = max(r["cash_in"] for r in screen["summary"]) + + if fam == "STR": + band = [t for t in trig if _band(view, t)] + branches = {t["branch"] for t in band} + if len(band) >= 5 or (len(band) >= 4 and len(branches) >= 3): + return _escalate("STRUCTURING", [t["id"] for t in band], "Repeated cash deposits just under the threshold across branches.") + if kyc["expected_monthly_cash"] >= max_cash: + return _close("CONSISTENT_WITH_PROFILE", ["kyc.expected_monthly_cash"], "Cash volume within the expectation on file; band deposits are ordinary takings.") + return _escalate("STRUCTURING", [t["id"] for t in band] or trig_ids, "Band deposits with no expectation on file to cover them.") + + if fam == "CASH": + if kyc["expected_monthly_cash"] >= max_cash: + return _close("CONSISTENT_WITH_PROFILE", ["kyc.expected_activity", "kyc.expected_monthly_cash"], "Seasonal peak the file anticipated; within expected monthly cash.") + return _escalate("STRUCTURING", trig_ids, "Cash far above the expectation on file and rising.") + + if fam == "VEL": + senders = {t["cp"] for t in trig if t["channel"] == "p2p_in"} + if len(senders) >= 3: + return _escalate("MONEY_MULE", trig_ids, "Credits from unrelated individuals moved to an exchange within a day.") + return None + + if fam == "WIRE": + if kyc.get("remittance_declared", "none declared") != "none declared": + return _close("CONSISTENT_WITH_PROFILE", ["kyc.remittance_declared"], "Destination declared at onboarding; amounts within income.") + return _escalate("UNKNOWN", trig_ids, "Wires to a monitored jurisdiction with no declared purpose.") + + return None # DORM: needs the document + + +def document_decision(view: dict[str, Any]) -> dict[str, Any]: + """After `documents`: close on the explaining document, or escalate if there is none.""" + docs = _looked(view, "documents") + trig_ids = [t["id"] for t in view["screen"]["triggering_transactions"]] + if docs: + for d in docs["documents"]: + if d["kind"] in EXPLAINING_DOCS: + return _close("DOCUMENTED_SOURCE_OF_FUNDS", [d["id"]], f"Source of funds documented: {d['kind']}.") + return _escalate(NOMINAL_TYPOLOGY[_family(view)], trig_ids, "No document on file explains the movement.") + + +def lookup_decision(view: dict[str, Any]) -> dict[str, Any] | None: + """A hidden signal in the lookups, or None.""" + screen = view["screen"] + fam = _family(view) + trig = screen["triggering_transactions"] + trig_ids = [t["id"] for t in trig] + + for l in view["lookups"]: + if l["what"] == "counterparty" and l["result"]["other_alerted_accounts"] >= 3: + r = l["result"] + return _escalate("FUNNEL_ACCOUNT", [r["id"]] + list(r["transactions"]), + f"{r['id']} is shared with {r['other_alerted_accounts']} other alerted accounts; funds converge on it.") + + priors = _looked(view, "prior_alerts") + if priors and len(priors["alerts"]) >= 3: + alerts = priors["alerts"] + rationales = {p["rationale"] for p in alerts} + if all(p["disposition"] == "closed" and p["elapsed_days"] <= 1 for p in alerts) and len(rationales) == 1: + return _escalate(NOMINAL_TYPOLOGY[fam], [p["id"] for p in alerts] + trig_ids, + "Repeat alerts closed within a day on one boilerplate rationale; nobody has looked.") + + docs = _looked(view, "documents") + if docs: + for d in docs["documents"]: + if not d["party"]: + continue + for cp_id in _dedupe([t["cp"] for t in trig if t["channel"] in INBOUND and t["cp"]]): + cp = _looked(view, "counterparty", cp_id) + if cp and cp["name"] != d["party"]: + credits = [t["id"] for t in trig if t["cp"] == cp_id and t["channel"] in INBOUND] + return _escalate(NOMINAL_TYPOLOGY[fam], [d["id"], cp_id] + credits, + f"The document names {d['party']}; the funds came from {cp['name']}.") + return None + + +def enriched_decision(view: dict[str, Any]) -> dict[str, Any]: + """thorough's final call: the screen rules, with citations widened to everything itemised.""" + d = screen_decision(view) + fam = _family(view) + screen = view["screen"] + months = screen["months"] + seen: list[dict[str, Any]] = list(screen["triggering_transactions"]) + for l in view["lookups"]: + if l["what"] == "history": + seen += l["result"]["transactions"] + if d is not None and d["action"] == "escalate": + if fam == "STR": + d["cites"] = _dedupe([t["id"] for t in seen if _band(view, t) and t["month"] == months[-1]] + d["cites"]) + elif fam == "CASH": + last_two = set(months[-2:]) + d["cites"] = _dedupe([t["id"] for t in seen if t["channel"] == "cash_in" and t["month"] in last_two] + d["cites"]) + return d + if d is None: + return document_decision(view) + return d + + +def _plan_next(view: dict[str, Any], plan: list[tuple[str, str | None]]) -> str | None: + for what, key in plan: + if _looked(view, what, key) is None: + if what == "history": + return _emit({"action": "lookup", "what": "history", "month": key}) + if what == "counterparty": + return _emit({"action": "lookup", "what": "counterparty", "id": key}) + return _emit({"action": "lookup", "what": what}) + return None + + +# ------------------------------------------------------------- shipped -- + + +def fast(view: dict[str, Any]) -> str: + d = screen_decision(view) + if d is not None: + return _emit(d) + if _looked(view, "documents") is None: + return _emit({"action": "lookup", "what": "documents"}) + return _emit(document_decision(view)) + + +def _investigation_plan(view: dict[str, Any]) -> list[tuple[str, str | None]]: + plan: list[tuple[str, str | None]] = [("documents", None), ("prior_alerts", None)] + plan += [("counterparty", cp["id"]) for cp in view["screen"]["screen_counterparties"]] + return plan + + +def targeted(view: dict[str, Any]) -> str: + d = screen_decision(view) + if d is not None and d["action"] == "escalate": + return _emit(d) + nxt = _plan_next(view, _investigation_plan(view)) + if nxt: + return nxt + found = lookup_decision(view) + if found: + return _emit(found) + return _emit(d if d is not None else document_decision(view)) + + +def thorough(view: dict[str, Any]) -> str: + months = view["screen"]["months"] + plan = _investigation_plan(view) + [("history", months[-1]), ("history", months[-2])] + nxt = _plan_next(view, plan) + if nxt: + return nxt + found = lookup_decision(view) + return _emit(found if found else enriched_decision(view)) + + +def thorough_wasteful(view: dict[str, Any]) -> str: + """thorough plus a month that carries nothing. Proves the hour meter binds.""" + months = view["screen"]["months"] + plan = _investigation_plan(view) + [("history", months[-1]), ("history", months[-2]), ("history", months[-3])] + nxt = _plan_next(view, plan) + if nxt: + return nxt + found = lookup_decision(view) + return _emit(found if found else enriched_decision(view)) + + +# -------------------------------------------------------------- cheats -- + + +def inaction(view: dict[str, Any]) -> str: + return "Let me reason about this alert carefully before committing to a disposition." + + +def malformed(view: dict[str, Any]) -> str: + return _emit({"action": "file_sar", "cites": []}) + + +def close_blind(view: dict[str, Any]) -> str: + return _emit(_close("CONSISTENT_WITH_PROFILE", ["kyc.customer_type"], "Looks fine.")) + + +def close_cites_all(view: dict[str, Any]) -> str: + screen = view["screen"] + cites = [f"kyc.{k}" for k in screen["customer"] if k not in ("id", "name")] + cites += [t["id"] for t in screen["triggering_transactions"]] + return _emit(_close("CONSISTENT_WITH_PROFILE", cites, "See everything.")) + + +def escalate_blind(view: dict[str, Any]) -> str: + trig_ids = [t["id"] for t in view["screen"]["triggering_transactions"]] + return _emit(_escalate(NOMINAL_TYPOLOGY[_family(view)], trig_ids, "Filing defensively.")) + + +def fast_defensive(view: dict[str, Any]) -> str: + """fast where the screen shows a typology; a blind filing everywhere else.""" + d = screen_decision(view) + if d is not None and d["action"] == "escalate": + return _emit(d) + return escalate_blind(view) + + +def rule_table(view: dict[str, Any]) -> str: + """Closes every wire-corridor and dormancy alert blind; files on everything else.""" + fam = _family(view) + if fam == "WIRE": + return _emit(_close("CONSISTENT_WITH_PROFILE", ["kyc.remittance_declared"], "Wires are always remittances.")) + if fam == "DORM": + return _emit(_close("CONSISTENT_WITH_PROFILE", ["kyc.account_opened"], "Old accounts wake up.")) + return escalate_blind(view) + + +def cite_unseen(view: dict[str, Any]) -> str: + """Cites a document it never opened; rejected, then falls back to closing blind.""" + if view["rejected"] == 0: + return _emit(_close("DOCUMENTED_SOURCE_OF_FUNDS", ["doc.probate_letter"], "Per the letter on file.")) + return close_blind(view) + + +POLICIES: dict[str, Policy] = { + "inaction": inaction, + "malformed": malformed, + "close_blind": close_blind, + "close_cites_all": close_cites_all, + "cite_unseen": cite_unseen, + "escalate_blind": escalate_blind, + "fast_defensive": fast_defensive, + "rule_table": rule_table, + "fast": fast, + "targeted": targeted, + "thorough": thorough, + "thorough_wasteful": thorough_wasteful, +} + +SHIPPED = ("fast", "targeted", "thorough") + + +def run_policy(policy: Policy, world: dict[str, Any]): + """Play a world to the end under one policy. Returns the finished Engine.""" + from .engine import Engine + + engine = Engine(world) + while not engine.done: + engine.step(policy(engine.view())) + return engine + + +def reference_for(world: dict[str, Any]) -> tuple[int | None, str | None]: + """The cheapest shipped policy that reaches the correct disposition, in minutes. + + (None, None) if none does — which `tests/test_reference.py` asserts never + happens for seeds 0–4095, so a null here is a CI failure, not a silent + "not scored" on the page. + """ + best: tuple[int, str] | None = None + for name in SHIPPED: + engine = run_policy(POLICIES[name], world) + if engine.outcome == "solved" and (best is None or engine.minutes < best[0]): + best = (engine.minutes, name) + return best if best else (None, None) diff --git a/envs/alert_triage/alert_triage/protocol.py b/envs/alert_triage/alert_triage/protocol.py new file mode 100644 index 0000000..2749ee7 --- /dev/null +++ b/envs/alert_triage/alert_triage/protocol.py @@ -0,0 +1,209 @@ +"""Turning model text into one action, and engine results into text. + +The one rule, as in wordle_five: **never raise**. A malformed reply is a +thing the model did, not an error in the harness. It becomes a rejection that +costs a turn and shows up in `rejected_replies`. + +The parse rules are pinned here because this is where two runtimes disagree, +and CI hashes a committed corpus of reply strings through this parser and the +engine on both sides (digest (c) in CONFORMANCE.txt): + + 1. If a ```json fence appears anywhere, its contents are the candidate. + A bare `{` before the fence does not win. + 2. Otherwise the candidate is the first string-aware balanced `{…}` span: + braces inside JSON strings do not count, backslash escapes are honoured. + 3. If the candidate is not valid JSON, or is JSON but not an object, the + reply is rejected. There is no further scanning for a second candidate. + 4. Enum values are matched exact-case. Cites are deduplicated in order and + matched exact-case against what the engine has shown. +""" + +from __future__ import annotations + +import json +import re +from typing import Any + +from .generator import MAX_TURNS + +ACTIONS = ("lookup", "close", "escalate") +LOOKUPS = ("history", "counterparty", "prior_alerts", "documents") +CLOSE_REASONS = ("CONSISTENT_WITH_PROFILE", "DOCUMENTED_SOURCE_OF_FUNDS", "PREVIOUSLY_REVIEWED", "RULE_ARTEFACT") +TYPOLOGIES = ("STRUCTURING", "FUNNEL_ACCOUNT", "MONEY_MULE", "RAPID_MOVEMENT", "UNKNOWN") + +_FENCE = re.compile(r"```json\s*(.*?)```", re.S) +_MONTH = re.compile(r"^\d{4}-\d{2}$") + + +def _reject_constant(name: str): + # Python's json accepts NaN/Infinity by default; JSON.parse does not. + raise ValueError(name) + + +def extract_candidate(text: str) -> str | None: + match = _FENCE.search(text) + if match: + return match.group(1) + start = text.find("{") + if start < 0: + return None + depth = 0 + in_string = False + escaped = False + for i in range(start, len(text)): + ch = text[i] + if in_string: + if escaped: + escaped = False + elif ch == "\\": + escaped = True + elif ch == '"': + in_string = False + continue + if ch == '"': + in_string = True + elif ch == "{": + depth += 1 + elif ch == "}": + depth -= 1 + if depth == 0: + return text[start : i + 1] + return None + + +def parse_reply(text: str | None) -> tuple[dict[str, Any] | None, str | None]: + """(object, None) or (None, reason). Validation of fields is the engine's job.""" + if not text: + return None, "no JSON object found in the reply" + candidate = extract_candidate(text) + if candidate is None: + return None, "no JSON object found in the reply" + try: + value = json.loads(candidate, parse_constant=_reject_constant) + except (ValueError, RecursionError): + return None, "the JSON object could not be parsed" + if not isinstance(value, dict): + return None, "the reply parsed but is not a JSON object" + return value, None + + +# -------------------------------------------------------------- render -- + +SYSTEM_PROMPT = """You are a transaction-monitoring analyst at a bank. One alert has been assigned to you. + +All data is synthetic. No real customer, account, transaction or counterparty exists. + +Your screen shows the alert, the transactions that triggered it, the customer's KYC file and a twelve-month summary. Reading it has already cost {screen} analyst-hours. You have {turns} replies. Each reply must be exactly ONE JSON object (in a ```json fence, or as the first {{...}} in the reply), and every reply costs {turn} hours whether or not it is accepted. + +Lookups (each may be made once): + {{"action":"lookup","what":"history","month":"YYYY-MM"}} itemised transactions for a month in the window +{lookup} h + {{"action":"lookup","what":"counterparty","id":"CP-n"}} name, type, jurisdiction, first seen, how many OTHER alerted accounts share it, its transactions +{lookup} h + {{"action":"lookup","what":"prior_alerts"}} prior alerts on this customer with disposition, analyst and rationale +{lookup} h + {{"action":"lookup","what":"documents"}} documents on file +{doc} h + +Dispositions (end the episode): + {{"action":"close","reason":R,"cites":[ids],"note":"..."}} + R is one of CONSISTENT_WITH_PROFILE, DOCUMENTED_SOURCE_OF_FUNDS, PREVIOUSLY_REVIEWED, RULE_ARTEFACT + {{"action":"escalate","typology":T,"cites":[ids],"narrative":"..."}} + T is one of STRUCTURING, FUNNEL_ACCOUNT, MONEY_MULE, RAPID_MOVEMENT, UNKNOWN + +`cites` must be a non-empty list of ids you have actually been shown: kyc., T-, CP-, A- (a prior alert), doc.. Citing an id you have not been shown is rejected. The note or narrative is displayed and never scored. + +A close on an alert that should have been escalated scores zero. An escalation on a clean customer scores half on the counterweight and zero on the rest. Cite the facts that decide the case, not everything you saw. If you reach your last reply without a disposition the alert stays in the queue and the episode scores zero.""" + + +def system_prompt() -> str: + from .reward import DOC_MINUTES, LOOKUP_MINUTES, SCREEN_MINUTES, TURN_MINUTES + + return SYSTEM_PROMPT.format( + screen=f"{SCREEN_MINUTES / 60:.2f}", + turn=f"{TURN_MINUTES / 60:.2f}", + lookup=f"{LOOKUP_MINUTES / 60:.2f}", + doc=f"{DOC_MINUTES / 60:.2f}", + turns=MAX_TURNS, + ) + + +def money(cents: int, currency: str) -> str: + sign = "-" if cents < 0 else "" + cents = abs(cents) + return f"{sign}{cents // 100:,}.{cents % 100:02d} {currency}" + + +def _txn_line(t: dict[str, Any], currency: str) -> str: + where = f"branch {t['branch']}" if t["branch"] else f"{t['cp']}" + return f" {t['id']:<6} {t['date']} {t['channel']:<9} {money(t['amount'], currency):>18} {where}" + + +def render_screen(screen: dict[str, Any]) -> str: + cur = screen["currency"] + c = screen["customer"] + a = screen["alert"] + alert_block = [ + f"ALERT {a['id']} rule {a['rule']} — {a['rule_text']}", + f" fired {a['fired']}", + " triggering transactions:", + *[_txn_line(t, cur) for t in screen["triggering_transactions"]], + ] + kyc_lines = [f"KYC FILE {c['id']} {c['name']}"] + for key in sorted(c): + if key in ("id", "name"): + continue + value = c[key] + if key in ("stated_income", "expected_monthly_cash"): + value = money(value, cur) + kyc_lines.append(f" kyc.{key} = {value}") + rows = screen["summary"] + summary = ["TWELVE-MONTH SUMMARY (" + cur + ")", + f" {'month':<8}{'cash in':>16}{'cash out':>16}{'wires in':>16}{'wires out':>16}{'ach in':>16}{'ach out':>16}{'txns':>6}"] + for r in rows: + summary.append( + f" {r['month']:<8}{r['cash_in'] // 100:>16,}{r['cash_out'] // 100:>16,}{r['wires_in'] // 100:>16,}" + f"{r['wires_out'] // 100:>16,}{r['ach_in'] // 100:>16,}{r['ach_out'] // 100:>16,}{r['count']:>6}" + ) + cps = ["COUNTERPARTIES on screen (top by twelve-month volume):"] + for cp in screen["screen_counterparties"]: + cps.append(f" {cp['id']:<6} {cp['direction']:<4} {cp['count']:>3} txns {money(cp['total'], cur)}") + blocks = [kyc_lines, alert_block] if screen["screen_order"] == "kyc_first" else [alert_block, kyc_lines] + blocks += [summary, cps, [f"Reporting threshold in this jurisdiction: {money(screen['threshold'], cur)}", + "SYNTHETIC DATA — no real customer, account or transaction."]] + return "\n".join("\n".join(b) for b in blocks) + + +def render_lookup(what: str, result: dict[str, Any], currency: str) -> str: + if what == "history": + lines = [f"HISTORY {result['month']}: {len(result['transactions'])} transactions"] + lines += [_txn_line(t, currency) for t in result["transactions"]] + return "\n".join(lines) + if what == "counterparty": + r = result + return "\n".join([ + f"COUNTERPARTY {r['id']} {r['name']}", + f" type {r['type']}; jurisdiction {r['jurisdiction']}; first seen {r['first_seen']}", + f" other alerted accounts sharing this counterparty in twelve months: {r['other_alerted_accounts']}", + f" transactions: {', '.join(r['transactions'])}", + ]) + if what == "prior_alerts": + if not result["alerts"]: + return "PRIOR ALERTS: none on file." + lines = ["PRIOR ALERTS:"] + for p in result["alerts"]: + lines.append(f" {p['id']} {p['rule']} fired {p['fired']} {p['disposition']} after {p['elapsed_days']} day(s) by {p['analyst']}") + lines.append(f" {p['rationale']}") + return "\n".join(lines) + if what == "documents": + lines = ["DOCUMENTS ON FILE:"] + for d in result["documents"]: + lines.append(f" {d['id']} [{d['kind']}] {d['description']}") + return "\n".join(lines) + return json.dumps(result) + + +def render_rejection(reason: str, turns_left: int) -> str: + plural = "y" if turns_left == 1 else "ies" + return f"That reply was not accepted: {reason}. It still cost a turn. You have {turns_left} repl{plural} left." + + +def render_cost(minutes_spent: int, turns_left: int) -> str: + plural = "y" if turns_left == 1 else "ies" + return f"Hours spent so far: {minutes_spent / 60:.2f}. You have {turns_left} repl{plural} left." diff --git a/envs/alert_triage/alert_triage/reward.py b/envs/alert_triage/alert_triage/reward.py new file mode 100644 index 0000000..b913349 --- /dev/null +++ b/envs/alert_triage/alert_triage/reward.py @@ -0,0 +1,210 @@ +"""The reward. Quoted verbatim on the demo page, so written to be read by +someone who does not write Python. + +Three components, weights summing to 1.0. Two pull toward clearing the alert +quickly with the right citations. The third — the heaviest — pulls the other +way: it pays for not letting a suspicious customer through, and a policy that +never investigates cannot collect it. + +Hours are integers in MINUTES so that the world, the grader and the browser +agree to the last digit; the page shows them as hours. + +# region: pig-demo/reward +""" + +from __future__ import annotations + +from dataclasses import dataclass + +# The generator's config, NOT industry measurements. Nobody timed an analyst. +SCREEN_MINUTES = 30 # reading the free screen; charged to everyone +TURN_MINUTES = 3 # every reply, accepted or rejected +LOOKUP_MINUTES = 24 # history, counterparty, prior alerts +DOC_MINUTES = 6 # documents on file + +CAUGHT_WEIGHT = 0.55 +HOURS_WEIGHT = 0.25 +EVIDENCE_WEIGHT = 0.20 + +# What `caught` pays for filing on a clean customer. Not zero: a defensive +# escalation is a wasted filing, not a missed crime. Not one: otherwise a +# policy that escalates everything takes the counterweight for free. +FALSE_ESCALATION_CREDIT = 0.5 + + +@dataclass +class Episode: + """Everything the reward needs from one played alert.""" + + label: str # "benign" | "suspicious" — what the generator planted + planted: list[list[str]] # alternates; the best one is scored + disposition: str | None # "close" | "escalate" | None + cites: list[str] + minutes_spent: int + reference_minutes: int | None # cheapest shipped policy that got this seed right + turns_spent: int + rejected: int + truncated: bool = False # ended early with no disposition: not scored + typology: str | None = None # what the agent wrote; diagnostic only + true_typology: str | None = None # what the generator planted; diagnostic only + lookups: int = 0 + + +def correct(ep: Episode) -> bool: + """A close on a benign alert, or an escalation on a suspicious one.""" + return (ep.label == "benign" and ep.disposition == "close") or ( + ep.label == "suspicious" and ep.disposition == "escalate" + ) + + +def caught(ep: Episode) -> float: + """Nothing slipped through. The counterweight, and over half the reward. + + 1.0 on any correct call. 0.0 if a suspicious alert was closed, or nothing + was decided. A false escalation earns FALSE_ESCALATION_CREDIT: the + customer was clean, so nothing slipped, but a report was filed on them. + + Read the asymmetry off the weights: a miss zeroes this AND the two + objectives below (the call was wrong), so the episode scores 0.000. The + most any policy can gain by being fast is `hours`' 0.25. One miss costs + more than every hour saved. + """ + if ep.disposition is None: + return 0.0 + if correct(ep): + return 1.0 + if ep.label == "benign": + return FALSE_ESCALATION_CREDIT + return 0.0 + + +def hours(ep: Episode) -> float | None: + """Cleared it in the hours budgeted, as a ratio against a reference. + + The reference is the cheapest SHIPPED policy that reached the correct + disposition on this same alert — `fast` where the screen was enough, + `targeted` where a lookup was needed. Never a true optimum: grade against + an optimum and the shipped policy fails its own probe on some seeds. + + Capped at 1.0. A correct call faster than the reference is not paid + extra; the ceiling is our heuristic, not an optimum. A fast wrong answer + saves nothing. If no shipped policy got this seed right there is no + reference and this is None — not scored — which CI asserts never happens. + """ + if not correct(ep): + return 0.0 + if ep.reference_minutes is None: + return None + return min(1.0, ep.reference_minutes / max(1, ep.minutes_spent)) + + +def f1_exact(cites: list[str], planted: list[str]) -> tuple[int, int]: + """F1 as an exact fraction (numerator, denominator), so both runtimes agree bit for bit.""" + common = len(set(cites) & set(planted)) + return 2 * common, len(set(cites)) + len(set(planted)) + + +def evidence(ep: Episode) -> float: + """Named the facts that decide it: F1 between the cites and the planted set. + + The planted set is what the generator buried — the explaining KYC or + document fact on a benign alert, the transactions that constitute the + typology on a suspicious one. Where a template admits two reasonable + rationales the best alternate is scored. + + F1's precision term is what stops cite-everything: one planted id among + eight cited scores 0.22. Its recall term is what stops cite-one-thing on + a pattern of eight deposits. Zero if the disposition was wrong; a false + escalation is checked against an empty set. The narrative prose is never + read here. + """ + if not correct(ep) or not ep.cites: + return 0.0 + best_num, best_den = 0, 1 + for alt in ep.planted: + num, den = f1_exact(ep.cites, alt) + if num * best_den > best_num * den: + best_num, best_den = num, den + return best_num / best_den + + +def score(ep: Episode) -> dict[str, float | None]: + """The components, unweighted. None means "not scored", never zero.""" + if ep.truncated: + return {"caught": None, "hours": None, "evidence": None} + return {"caught": caught(ep), "hours": hours(ep), "evidence": evidence(ep)} + + +def total(ep: Episode, weights: dict[str, float] | None = None) -> float | None: + parts = score(ep) + if any(v is None for v in parts.values()): + return None + w = weights or WEIGHTS + return parts["caught"] * w["caught"] + parts["hours"] * w["hours"] + parts["evidence"] * w["evidence"] # type: ignore[operator] + + +WEIGHTS = {"caught": CAUGHT_WEIGHT, "hours": HOURS_WEIGHT, "evidence": EVIDENCE_WEIGHT} +ROLES = {"caught": "counterweight", "hours": "objective", "evidence": "objective"} + +# endregion: pig-demo/reward + + +def escalate_break_even(evidence_if_close: float = 1.0) -> float: + """P(benign) below which a blind escalation out-scores a well-cited close. + + Escalating blind pays CAUGHT_WEIGHT × (credit × p_b + (1 − p_b)) plus, on + the suspicious share, hours (paid 1.0: the blind filer is fastest) and + nothing for evidence. Closing pays p_b × (CAUGHT + HOURS + EVIDENCE × e). + Printed on the page so a BSA officer can read the threshold the weights + encode rather than infer it. + """ + close_per_benign = CAUGHT_WEIGHT + HOURS_WEIGHT + EVIDENCE_WEIGHT * evidence_if_close + esc_per_benign = CAUGHT_WEIGHT * FALSE_ESCALATION_CREDIT + esc_per_suspicious = CAUGHT_WEIGHT + HOURS_WEIGHT + # p_b · close = p_b · esc_b + (1 − p_b) · esc_s → p_b = esc_s / (close − esc_b + esc_s) + return esc_per_suspicious / (close_per_benign - esc_per_benign + esc_per_suspicious) + + +def score_exact(ep: Episode) -> dict[str, list[int] | None]: + """The components as exact fractions, for the cross-language digest.""" + if ep.truncated: + return {"caught": None, "hours": None, "evidence": None} + if ep.disposition is None: + c: list[int] = [0, 1] + elif correct(ep): + c = [1, 1] + elif ep.label == "benign": + c = [1, 2] + else: + c = [0, 1] + if not correct(ep): + h: list[int] | None = [0, 1] + elif ep.reference_minutes is None: + h = None + else: + spent = max(1, ep.minutes_spent) + h = [1, 1] if ep.reference_minutes >= spent else [ep.reference_minutes, spent] + if not correct(ep) or not ep.cites: + e = [0, 1] + else: + best = (0, 1) + for alt in ep.planted: + num, den = f1_exact(ep.cites, alt) + if num * best[1] > best[0] * den: + best = (num, den) + e = [best[0], best[1]] + return {"caught": c, "hours": h, "evidence": e} + + +def metrics(ep: Episode) -> dict[str, float | None]: + """Diagnostics. Weight zero — reported, never summed into the reward.""" + return { + "hours_spent": ep.minutes_spent / 60, + "reference_hours": None if ep.reference_minutes is None else ep.reference_minutes / 60, + "lookups": float(ep.lookups), + "turns_used": float(ep.turns_spent), + "rejected_replies": float(ep.rejected), + "false_escalation": 1.0 if ep.label == "benign" and ep.disposition == "escalate" else 0.0, + "missed": 1.0 if ep.label == "suspicious" and ep.disposition == "close" else 0.0, + "typology_match": None if ep.disposition != "escalate" else (1.0 if ep.typology == ep.true_typology else 0.0), + } diff --git a/envs/alert_triage/alert_triage/rng.py b/envs/alert_triage/alert_triage/rng.py new file mode 100644 index 0000000..96cfa7a --- /dev/null +++ b/envs/alert_triage/alert_triage/rng.py @@ -0,0 +1,120 @@ +"""Portable randomness and portable dates. + +Nothing in the generator may touch a language built-in for either. A built-in +RNG picks a different alert for the same seed in the browser, and a built-in +date type is timezone-sensitive in JavaScript — CI in UTC passes and a +visitor in another timezone sees a different fire date for the same `?seed=`. +Both are hand-written here and ported line for line to TypeScript. +""" + +from __future__ import annotations + +MASK = 0xFFFFFFFF + + +def fnv1a32(text: str) -> int: + """FNV-1a, 32-bit, over the UTF-8 bytes. Identical to wordle_five's.""" + h = 0x811C9DC5 + for byte in text.encode(): + h ^= byte + h = (h * 0x01000193) & MASK + return h + + +class XorShift32: + """Marsaglia xorshift32. Seven lines, and the browser has the same seven. + + `below(n)` is a plain modulo. It is biased for large `n`, which does not + matter here (every `n` is tiny) and keeps the port a one-liner — + `Math.imul`-free, since the state is only ever shifted and xored. + """ + + def __init__(self, seed: int) -> None: + self.state = seed & MASK or 0x9E3779B9 + + def next(self) -> int: + x = self.state + x ^= (x << 13) & MASK + x ^= x >> 17 + x ^= (x << 5) & MASK + self.state = x + return x + + def below(self, n: int) -> int: + return self.next() % n + + def between(self, lo: int, hi: int) -> int: + """Inclusive on both ends.""" + return lo + self.below(hi - lo + 1) + + def chance(self, percent: int) -> bool: + return self.below(100) < percent + + def pick(self, items): + return items[self.below(len(items))] + + def sample(self, items, k: int) -> list: + """k distinct items, Fisher-Yates over a copy, first k taken.""" + pool = list(items) + out = [] + for i in range(min(k, len(pool))): + j = i + self.below(len(pool) - i) + pool[i], pool[j] = pool[j], pool[i] + out.append(pool[i]) + return out + + def hex4(self) -> str: + return format(self.below(65536), "04x") + + +# ------------------------------------------------------------------ dates -- +# Howard Hinnant's civil-from-days / days-from-civil. Integer arithmetic only. +# Day 0 is 1970-01-01. + + +def days_from_civil(y: int, m: int, d: int) -> int: + y -= m <= 2 + era = (y if y >= 0 else y - 399) // 400 + yoe = y - era * 400 + mp = (m + 9) % 12 + doy = (153 * mp + 2) // 5 + d - 1 + doe = yoe * 365 + yoe // 4 - yoe // 100 + doy + return era * 146097 + doe - 719468 + + +def civil_from_days(z: int) -> tuple[int, int, int]: + z += 719468 + era = (z if z >= 0 else z - 146096) // 146097 + doe = z - era * 146097 + yoe = (doe - doe // 1460 + doe // 36524 - doe // 146096) // 365 + y = yoe + era * 400 + doy = doe - (365 * yoe + yoe // 4 - yoe // 100) + mp = (5 * doy + 2) // 153 + d = doy - (153 * mp + 2) // 5 + 1 + m = mp + 3 if mp < 10 else mp - 9 + return (y + (m <= 2), m, d) + + +def days_in_month(y: int, m: int) -> int: + if m == 12: + return days_from_civil(y + 1, 1, 1) - days_from_civil(y, 12, 1) + return days_from_civil(y, m + 1, 1) - days_from_civil(y, m, 1) + + +def iso_date(day: int) -> str: + y, m, d = civil_from_days(day) + return f"{y:04d}-{m:02d}-{d:02d}" + + +def month_key(y: int, m: int) -> str: + return f"{y:04d}-{m:02d}" + + +def month_of_day(day: int) -> str: + y, m, _ = civil_from_days(day) + return month_key(y, m) + + +def add_months(y: int, m: int, n: int) -> tuple[int, int]: + index = y * 12 + (m - 1) + n + return (index // 12, index % 12 + 1) diff --git a/envs/alert_triage/alert_triage/taskset.py b/envs/alert_triage/alert_triage/taskset.py new file mode 100644 index 0000000..0da1ce5 --- /dev/null +++ b/envs/alert_triage/alert_triage/taskset.py @@ -0,0 +1,116 @@ +"""The episode loop everything shares, and the optional verifiers taskset. + +`play_episode` is the whole game: capture, the probe, the tests and the +fixture verifier all drive it, so the thing evaluated and the thing tested +cannot drift apart. The verifiers Env is a thin adapter over the same engine. +""" + +from __future__ import annotations + +from typing import Any, Callable + +from .engine import Engine +from .generator import MAX_TURNS, world_for_seed +from .protocol import render_cost, render_lookup, render_rejection, render_screen, system_prompt +from .reward import ROLES, WEIGHTS, metrics, score + +TASKSET_ID = "alert-triage" + +Responder = Callable[[str, list[dict[str, Any]], dict[str, Any]], str | None] + + +def opening_message(engine: Engine) -> str: + return render_screen(engine.screen()) + "\n\n" + render_cost(engine.minutes, MAX_TURNS - engine.turns) + + +def observation_for(engine: Engine, step) -> str: + """The text the model sees after a reply.""" + left = MAX_TURNS - engine.turns + if not step.accepted: + return render_rejection(step.reason or "rejected", left) + if step.action["action"] == "lookup": + return render_lookup(step.action["what"], step.result, engine.world["currency"]) + "\n\n" + render_cost(engine.minutes, left) + return "Disposition recorded. The episode is over." + + +def play_episode(seed: int, respond: Responder, world: dict[str, Any] | None = None) -> dict[str, Any]: + """Drive one alert against a callable `(prompt, transcript, view) -> reply text`. + + `view` is the structured observation (what scripted policies read); + `transcript` is the text history (what a model reads). A `None` reply is + recorded as a rejected turn — a dropped model call is a thing that + happened, not a turn to discard. + """ + world = world or world_for_seed(seed) + engine = Engine(world) + prompt = system_prompt() + transcript: list[dict[str, Any]] = [{"reply": None, "action": None, "observation": opening_message(engine)}] + + while not engine.done: + reply = respond(prompt, transcript, engine.view()) + step = engine.step(reply) + transcript.append({"reply": reply, "action": step.action, "rejection": step.reason, "observation": observation_for(engine, step)}) + + episode = engine.episode() + return { + "seed": seed, + "outcome": engine.outcome, + "truncated": episode.truncated, + "transcript": transcript, + "rewards": score(episode), + "weights": WEIGHTS, + "roles": ROLES, + "metrics": metrics(episode), + "info": { + "label": world["label"], + "template": world["template"], + "overlay": world["overlay"], + "typology": world["typology"], + "planted": world["planted"], + "reference_minutes": world["reference_minutes"], + "reference_policy": world["reference_policy"], + "held_out": world["held_out"], + }, + } + + +def replay(seed: int, replies: list[str | None], world: dict[str, Any] | None = None) -> dict[str, Any]: + """Re-score a recorded run from its reply strings alone. What `verify` does.""" + replies = list(replies) + engine = Engine(world or world_for_seed(seed)) + for reply in replies: + if engine.done: + break + engine.step(reply) + episode = engine.episode() + return {"rewards": score(episode), "metrics": metrics(episode), "outcome": engine.outcome, "truncated": episode.truncated} + + +def build_taskset(): # pragma: no cover - requires the optional extra + """The verifiers v1 Taskset. Imported lazily so the module stays optional.""" + import verifiers as vf + + class AlertTriageEnv(vf.Env): + """One agent seat, `analyst`. The engine plays the user role: every + observation is computed, never generated, which is why the reward is + verifiable at all.""" + + async def run(self, task, agent, trace): + world = world_for_seed(task.data.info["seed"]) + engine = Engine(world) + async with agent.interaction(system=system_prompt()) as session: + message = opening_message(engine) + while not engine.done: + reply = await session.turn(message) + step = engine.step(reply) + message = observation_for(engine, step) + episode = engine.episode() + for key, value in score(episode).items(): + trace.record_reward(key, 0.0 if value is None else value, WEIGHTS[key]) + for key, value in metrics(episode).items(): + if value is not None: + trace.record_reward(key, value, 0.0) + trace.info["label"] = world["label"] + trace.info["outcome"] = engine.outcome + + return AlertTriageEnv diff --git a/envs/alert_triage/conformance/replies.json b/envs/alert_triage/conformance/replies.json new file mode 100644 index 0000000..dc206e6 --- /dev/null +++ b/envs/alert_triage/conformance/replies.json @@ -0,0 +1,88 @@ +[ + { + "seed": 0, + "note": "well-formed lookups, a fenced disposition after a bare brace, brace inside a string", + "replies": [ + "{\"action\":\"lookup\",\"what\":\"documents\"}", + "I will check priors first. {\"action\":\"lookup\",\"what\":\"prior_alerts\"} and then decide.", + "{\"action\":\"lookup\",\"what\":\"history\",\"month\":\"BADMONTH\"}", + "Let me look at CP-1 {not json} ```json\n{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-1\"}\n```", + "{\"action\":\"close\",\"reason\":\"CONSISTENT_WITH_PROFILE\",\"cites\":[\"kyc.customer_type\",\"kyc.customer_type\",\"kyc.country\"],\"note\":\"braces {inside} a string \\\" and an escaped quote\"}" + ] + }, + { + "seed": 1, + "note": "invalid first span then valid: rejected, no further scanning; lowercase enums; non-string cites", + "replies": [ + "{\"action\":\"lookup\",\"what\":\"documents\",} {\"action\":\"lookup\",\"what\":\"documents\"}", + "{\"action\":\"close\",\"reason\":\"consistent_with_profile\",\"cites\":[\"kyc.country\"]}", + "{\"action\":\"Lookup\",\"what\":\"documents\"}", + "{\"action\":\"close\",\"reason\":\"CONSISTENT_WITH_PROFILE\",\"cites\":[1,2]}", + "{\"action\":\"close\",\"reason\":\"CONSISTENT_WITH_PROFILE\",\"cites\":\"kyc.country\"}", + "{\"action\":\"escalate\",\"typology\":\"structuring\",\"cites\":[\"kyc.country\"]}", + "{\"action\":\"escalate\",\"typology\":\"STRUCTURING\",\"cites\":[],\"narrative\":\"x\"}", + "{\"action\":\"escalate\",\"typology\":\"STRUCTURING\",\"cites\":[\"kyc.country\"],\"narrative\":42}" + ] + }, + { + "seed": 2, + "note": "repeat lookups, nonexistent and unseen counterparties, out-of-window months, an unseen cite", + "replies": [ + "{\"action\":\"lookup\",\"what\":\"prior_alerts\"}", + "{\"action\":\"lookup\",\"what\":\"prior_alerts\"}", + "{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-99\"}", + "{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"cp-1\"}", + "{\"action\":\"lookup\",\"what\":\"history\",\"month\":\"2024-01\"}", + "{\"action\":\"lookup\",\"what\":\"history\",\"month\":\"2026-1\"}", + "{\"action\":\"close\",\"reason\":\"DOCUMENTED_SOURCE_OF_FUNDS\",\"cites\":[\"doc.probate_letter\"]}", + "{\"action\":\"close\",\"reason\":\"PREVIOUSLY_REVIEWED\",\"cites\":[\"kyc.prior_alerts\"]}" + ] + }, + { + "seed": 3, + "note": "prose only, empty, null, NaN, a fenced array, nested objects, unbalanced braces; runs out the budget -> aborted", + "replies": [ + "I would like to think about this alert a little longer before acting.", + "", + null, + "{\"action\":\"lookup\",\"what\":\"documents\",\"weight\":NaN}", + "```json\n[{\"action\":\"lookup\",\"what\":\"documents\"}]\n```", + "{\"outer\":{\"action\":\"lookup\",\"what\":\"documents\"}}", + "{\"action\":\"lookup\",\"what\":\"documents\"", + "{\"what\":\"documents\"}" + ] + }, + { + "seed": 4, + "note": "a full investigation that itemises the two most recent months and escalates on the last turn", + "replies": [ + "{\"action\":\"lookup\",\"what\":\"documents\"}", + "{\"action\":\"lookup\",\"what\":\"prior_alerts\"}", + "{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-2\"}", + "{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-3\"}", + "{\"action\":\"lookup\",\"what\":\"history\",\"month\":\"2026-03\"}", + "{\"action\":\"lookup\",\"what\":\"history\",\"month\":\"2026-04\"}", + "{\"action\":\"lookup\",\"what\":\"history\",\"month\":\"2026-05\"}", + "{\"action\":\"escalate\",\"typology\":\"UNKNOWN\",\"cites\":[\"kyc.country\",\"kyc.pep\"],\"narrative\":\"eighth turn\"}" + ] + }, + { + "seed": 5, + "note": "a disposition citing every shown KYC field; extra keys ignored; the episode ends and later replies are not counted", + "replies": [ + "```json\n{\"action\": \"close\", \"reason\": \"RULE_ARTEFACT\", \"cites\": [\"kyc.customer_type\", \"kyc.country\", \"kyc.pep\", \"kyc.prior_alerts\", \"kyc.prior_sars\", \"kyc.stated_income\", \"kyc.expected_monthly_cash\", \"kyc.expected_activity\", \"kyc.account_opened\", \"kyc.kyc_refreshed\", \"kyc.remittance_declared\"], \"note\": \"\", \"confidence\": 0.9}\n```", + "{\"action\":\"lookup\",\"what\":\"documents\"}" + ] + }, + { + "seed": 6, + "note": "stops after three turns with no disposition: truncated, not aborted; an unknown lookup kind", + "replies": [ + "{\"action\":\"lookup\",\"what\":\"documents\"}", + "thinking", + "{\"action\":\"lookup\",\"what\":\"prior_alerts\"}", + "{\"action\":\"lookup\",\"what\":\"emails\"}", + "[{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-1\"}] as a bare array: the inner object is the first balanced span" + ] + } +] \ No newline at end of file diff --git a/envs/alert_triage/names/PROVENANCE.md b/envs/alert_triage/names/PROVENANCE.md new file mode 100644 index 0000000..1268c8e --- /dev/null +++ b/envs/alert_triage/names/PROVENANCE.md @@ -0,0 +1,31 @@ +# Where these names came from + +Every list in this directory was written by hand for this environment. None +is derived from a customer file, a sanctions list, a leaked dataset, a public +records scrape or any other collection of real people or real businesses. + +- **`first_names.json`, `surnames.json`** — 40 given names and 40 surnames, + chosen to be common across many languages so that no generated customer + reads as belonging to one region. A generated name such as "Priya Whitlock" + or "Dmitri Okafor" is a random pairing; any resemblance to a real person is + the ordinary consequence of common names being common. Analysts who "closed" + prior alerts are drawn from the same two lists. +- **`words.json`** — business names are composed as `adjective + noun + + suffix` ("Harbor Lane Bistro LLC" is the shape). Branch codes and street + names are invented. +- **`jurisdictions.json`** — every counterparty jurisdiction is **fictional**. + The "monitored" list is the synthetic institution's own high-risk corridor + list; it names no real country and makes no claim about any real + jurisdiction's risk rating. The customer's home country is the United States + in the main seed slice and the fictional "Karnland" (currency KRN) in the + held-out slice, where the reporting threshold is a fictional 20,000 KRN + rather than the public 10,000 USD currency-transaction-report threshold. + +Licence: these lists are part of the repository and carry its Apache-2.0 +licence. + +The scenario *templates* in `../alert_triage/generator.py` are written from +public typology descriptions — FinCEN's SAR narrative guidance and the FATF +typology reports — which are cited on the demo's evidence tab as references. +Nothing from those documents is shipped as data, and no template is a real +case. diff --git a/envs/alert_triage/names/first_names.json b/envs/alert_triage/names/first_names.json new file mode 100644 index 0000000..5133ac7 --- /dev/null +++ b/envs/alert_triage/names/first_names.json @@ -0,0 +1,4 @@ +["Aaron","Alicia","Amara","Andre","Beatriz","Bilal","Camille","Carlos","Chidi","Dana", + "Dmitri","Elena","Emeka","Farah","Felix","Gabriela","Hana","Hiro","Ines","Ivan", + "Jamal","Jonas","Keiko","Lars","Leila","Luca","Malik","Marta","Mateo","Mira", + "Naomi","Nikolai","Olu","Priya","Rafael","Rosa","Samir","Sofia","Tomas","Yara"] diff --git a/envs/alert_triage/names/jurisdictions.json b/envs/alert_triage/names/jurisdictions.json new file mode 100644 index 0000000..b117889 --- /dev/null +++ b/envs/alert_triage/names/jurisdictions.json @@ -0,0 +1,5 @@ +{ + "home": {"USD": "United States", "KRN": "Karnland"}, + "monitored": ["Marovia","Kestrel Islands","Nordwall","Sarenne"], + "other": ["Ostmark","Vellan","Tirsa","Halmond","Brixa","Corvale"] +} diff --git a/envs/alert_triage/names/surnames.json b/envs/alert_triage/names/surnames.json new file mode 100644 index 0000000..9f0b338 --- /dev/null +++ b/envs/alert_triage/names/surnames.json @@ -0,0 +1,4 @@ +["Abara","Bergstrom","Calloway","Delacroix","Eskildsen","Farrow","Galloway","Haddad","Iwata","Jankowski", + "Kavanagh","Lindqvist","Marchetti","Nakamura","Oduya","Petrov","Quintero","Rasmussen","Sandoval","Tanaka", + "Ubaldi","Vasquez","Whitlock","Xiang","Yilmaz","Zapata","Okafor","Brennan","Castellano","Dubois", + "Ferreira","Grimaldi","Halvorsen","Ibarra","Jensen","Kowalczyk","Laurent","Moreau","Novak","Pereira"] diff --git a/envs/alert_triage/names/words.json b/envs/alert_triage/names/words.json new file mode 100644 index 0000000..02c39ac --- /dev/null +++ b/envs/alert_triage/names/words.json @@ -0,0 +1,11 @@ +{ + "adjectives": ["Harbor","Northgate","Summit","Blue Ridge","Cedar","Ironwood","Lakeside","Meridian","Copperfield","Silverline", + "Westbrook","Granite","Oakhill","Redfern","Stonebridge","Brightwater","Highland","Riverbend","Fairmont","Ashgrove", + "Kingsway","Maple Lane","Pinecrest","Eastfield"], + "nouns": ["Bistro","Grill","Kitchen","Diner","Landscaping","Tax Services","Pool Care","Staffing","Freight","Events", + "Contracting","Foods","Produce","Supply","Equipment","Payroll","Logistics","Holdings","Trading","Ventures", + "Vending","Car Wash","Laundry","Market"], + "suffixes": ["LLC","Inc.","Co.","Group"], + "branches": ["BR-014","BR-027","BR-031","BR-052","BR-068","BR-073","BR-089","BR-102"], + "streets": ["Alder Street","Birch Avenue","Cypress Road","Dogwood Lane","Elm Court","Fir Street","Hazel Way","Juniper Drive"] +} diff --git a/envs/alert_triage/pyproject.toml b/envs/alert_triage/pyproject.toml new file mode 100644 index 0000000..c215840 --- /dev/null +++ b/envs/alert_triage/pyproject.toml @@ -0,0 +1,21 @@ +[project] +name = "alert-triage" +version = "0.1.0" +description = "One synthetic transaction-monitoring alert, closed or escalated with citations, scored by a pure grader" +requires-python = ">=3.11,<3.14" +dependencies = [] + +[project.optional-dependencies] +# verifiers is optional for the same reason as wordle_five: the generator, the +# engine, the reward and the probe are gated in CI without an RL stack. +verifiers = ["verifiers>=0.3.2.dev12"] + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[tool.hatch.build.targets.wheel] +packages = ["alert_triage"] + +[tool.hatch.build] +include = ["alert_triage/**", "names/**", "conformance/**", "pyproject.toml", "README.md"] diff --git a/envs/alert_triage/tests/test_triage_conformance.py b/envs/alert_triage/tests/test_triage_conformance.py new file mode 100644 index 0000000..39cb943 --- /dev/null +++ b/envs/alert_triage/tests/test_triage_conformance.py @@ -0,0 +1,32 @@ +"""The cross-language gates. If any of these changes, the TypeScript port must be re-verified.""" + +from __future__ import annotations + +from alert_triage.conformance import committed, protocol_digest, protocol_trace, scorer_digest +from alert_triage.generator import world_digest + + +def test_scorer_digest_matches_the_committed_value() -> None: + assert scorer_digest() == committed()["scorer"] + + +def test_protocol_digest_matches_the_committed_value() -> None: + assert protocol_digest() == committed()["protocol"] + + +def test_protocol_corpus_exercises_every_rejection_path() -> None: + trace = protocol_trace() + reasons = {t["reason"] for case in trace for t in case["turns"] if not t["accepted"]} + fragments = ["no JSON object", "could not be parsed", "not a JSON object", "unknown action", "unknown lookup", + "not in the twelve-month window", "already been looked up", "has not appeared", "cites must be", + "not been shown", "unknown close reason", "unknown typology"] + for fragment in fragments: + assert any(fragment in r for r in reasons), fragment + outcomes = {case["outcome"] for case in trace} + assert "aborted" in outcomes + assert any(case["truncated"] for case in trace) + assert any(len(case["turns"]) == 8 and case["outcome"] in ("solved", "failed") for case in trace) + + +def test_world_digest_matches_the_committed_value() -> None: + assert world_digest() == committed()["world"] diff --git a/envs/alert_triage/tests/test_triage_engine.py b/envs/alert_triage/tests/test_triage_engine.py new file mode 100644 index 0000000..7f850c5 --- /dev/null +++ b/envs/alert_triage/tests/test_triage_engine.py @@ -0,0 +1,124 @@ +"""Rejections, charges, shown ids, and the abort/truncate line.""" + +from __future__ import annotations + +import json + +from alert_triage.engine import Engine +from alert_triage.generator import MAX_TURNS, world_for_seed +from alert_triage.reward import DOC_MINUTES, LOOKUP_MINUTES, SCREEN_MINUTES, TURN_MINUTES, score + +JUNK = ["", " ", None, "no json here", "{", "}", "{}", "[]", "null", "{\"action\":null}", "{\"action\":\"lookup\"}", + "{\"action\":\"lookup\",\"what\":\"history\"}", "{\"action\":\"lookup\",\"what\":\"history\",\"month\":7}", + "{\"action\":\"lookup\",\"what\":\"counterparty\"}", "{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":[]}", + "{\"action\":\"close\"}", "{\"action\":\"close\",\"reason\":\"RULE_ARTEFACT\"}", + "{\"action\":\"close\",\"reason\":\"RULE_ARTEFACT\",\"cites\":{}}", "{\"action\":\"escalate\",\"cites\":[\"kyc.country\"]}", + "{\"action\":\"lookup\",\"what\":\"documents\",\"x\":NaN}", "\x00\x01", "{" * 50] + + +def _e(seed: int = 0) -> Engine: + return Engine(world_for_seed(seed)) + + +def test_screen_costs_and_first_turn() -> None: + e = _e() + assert e.minutes == SCREEN_MINUTES and e.turns == 0 and not e.done + + +def test_junk_never_raises_and_costs_one_turn_each() -> None: + for junk in JUNK: + e = _e() + step = e.step(junk) + assert not step.accepted and step.reason and e.turns == 1 and e.rejected == 1 + assert e.minutes == SCREEN_MINUTES + TURN_MINUTES + + +def test_trailing_garbage_after_a_balanced_object_is_fine() -> None: + e = _e() + assert e.step('{"action":"lookup","what":"documents"}' + "}" * 50 + " and more prose").accepted + + +def test_lookup_charges_its_price_only_when_accepted() -> None: + e = _e() + e.step('{"action":"lookup","what":"documents"}') + assert e.minutes == SCREEN_MINUTES + TURN_MINUTES + DOC_MINUTES + e.step('{"action":"lookup","what":"documents"}') # repeat: rejected + assert e.minutes == SCREEN_MINUTES + 2 * TURN_MINUTES + DOC_MINUTES and e.rejected == 1 + e.step('{"action":"lookup","what":"prior_alerts"}') + assert e.minutes == SCREEN_MINUTES + 3 * TURN_MINUTES + DOC_MINUTES + LOOKUP_MINUTES + + +def test_history_window_is_the_twelve_months_ending_in_the_fire_month() -> None: + e = _e() + months = e.world["months"] + assert len(months) == 12 and months[-1] == e.world["alert"]["fired"][:7] + assert e.step(json.dumps({"action": "lookup", "what": "history", "month": months[0]})).accepted + before = months[0][:4] + "-" + f"{int(months[0][5:]) - 1:02d}" if months[0][5:] != "01" else f"{int(months[0][:4]) - 1}-12" + assert not e.step(json.dumps({"action": "lookup", "what": "history", "month": before})).accepted + + +def test_counterparty_must_have_been_shown() -> None: + e = _e() + on_screen = e.world["screen_counterparties"][0]["id"] + assert e.step(json.dumps({"action": "lookup", "what": "counterparty", "id": on_screen})).accepted + assert not e.step('{"action":"lookup","what":"counterparty","id":"CP-999"}').accepted + assert not e.step(json.dumps({"action": "lookup", "what": "counterparty", "id": on_screen})).accepted + + +def test_cites_must_be_shown_and_are_deduplicated() -> None: + e = _e() + step = e.step('{"action":"close","reason":"RULE_ARTEFACT","cites":["kyc.country","doc.D-0000"]}') + assert not step.accepted and "not been shown" in step.reason + step = e.step('{"action":"close","reason":"RULE_ARTEFACT","cites":["kyc.country","kyc.country","kyc.pep"]}') + assert step.accepted and step.action["cites"] == ["kyc.country", "kyc.pep"] and e.done + + +def test_documents_become_citable_after_the_lookup() -> None: + e = _e() + doc = e.world["documents"][0]["id"] + assert not e.step(json.dumps({"action": "close", "reason": "DOCUMENTED_SOURCE_OF_FUNDS", "cites": [doc]})).accepted + assert e.step('{"action":"lookup","what":"documents"}').accepted + assert e.step(json.dumps({"action": "close", "reason": "DOCUMENTED_SOURCE_OF_FUNDS", "cites": [doc]})).accepted + + +def test_enums_are_exact_case() -> None: + e = _e() + assert not e.step('{"action":"close","reason":"rule_artefact","cites":["kyc.country"]}').accepted + assert not e.step('{"action":"ESCALATE","typology":"UNKNOWN","cites":["kyc.country"]}').accepted + assert e.step('{"action":"escalate","typology":"UNKNOWN","cites":["kyc.country"]}').accepted + + +def test_eight_rejections_abort_and_score_zero() -> None: + e = _e() + for _ in range(MAX_TURNS): + e.step("nothing") + assert e.done and e.aborted and e.outcome == "aborted" + assert e.step("{}").reason == "the episode is over" and e.turns == MAX_TURNS + assert score(e.episode()) == {"caught": 0.0, "hours": 0.0, "evidence": 0.0} + + +def test_a_disposition_on_the_eighth_turn_is_accepted() -> None: + e = _e() + for _ in range(MAX_TURNS - 1): + e.step("nothing") + assert e.step('{"action":"close","reason":"RULE_ARTEFACT","cites":["kyc.country"]}').accepted + assert e.outcome in ("solved", "failed") + + +def test_stopping_early_is_truncated_not_aborted() -> None: + e = _e() + e.step("nothing") + e.step('{"action":"lookup","what":"documents"}') + assert not e.done and e.outcome is None + ep = e.episode() + assert ep.truncated + assert score(ep) == {"caught": None, "hours": None, "evidence": None} + + +def test_view_never_shows_hidden_fields() -> None: + e = _e(5) + e.step('{"action":"lookup","what":"documents"}') + e.step('{"action":"lookup","what":"prior_alerts"}') + text = json.dumps(e.view()) + for key in ("label", "planted", "typology", "overlay", "tier", "template", "reference_minutes", "reference_policy"): + assert f'"{key}"' not in text, key diff --git a/envs/alert_triage/tests/test_triage_generator.py b/envs/alert_triage/tests/test_triage_generator.py new file mode 100644 index 0000000..188df39 --- /dev/null +++ b/envs/alert_triage/tests/test_triage_generator.py @@ -0,0 +1,164 @@ +"""What the generator promises about every world, checked over every digested seed.""" + +from __future__ import annotations + +import json +from collections import Counter + +import pytest + +from alert_triage.generator import ( + BENIGN_PCT, + RULE_IDS, + VISIBLE_PCT, + canonical_json, + generate, + is_held_out, + screen_of, +) +from alert_triage.rng import days_from_civil + +ALL = range(4096) + + +@pytest.fixture(scope="module") +def worlds(): + return [generate(s) for s in ALL] + + +def _day(iso: str) -> int: + y, m, d = (int(x) for x in iso.split("-")) + return days_from_civil(y, m, d) + + +def test_deterministic() -> None: + assert canonical_json(generate(17)) == canonical_json(generate(17)) + assert canonical_json(generate(17)) != canonical_json(generate(18)) + + +def test_canonical_json_has_no_floats(worlds) -> None: + def walk(v): + if isinstance(v, float): + raise AssertionError("float in world") + if isinstance(v, dict): + for x in v.values(): + walk(x) + if isinstance(v, list): + for x in v: + walk(x) + for w in worlds[:512]: + walk(w) + assert json.loads(canonical_json(w)) == w + + +def test_mix_by_seed_is_close_to_declared(worlds) -> None: + tiers = Counter(w["tier"] for w in worlds) + n = len(worlds) + assert abs(tiers["benign"] / n - BENIGN_PCT / 100) < 0.03 + assert abs(tiers["visible"] / n - VISIBLE_PCT / 100) < 0.03 + assert abs(tiers["hidden"] / n - (100 - BENIGN_PCT - VISIBLE_PCT) / 100) < 0.03 + + +def test_every_rule_carries_both_labels(worlds) -> None: + """CHEAT 3: no rule id may be a label. Each fires on benign AND suspicious seeds.""" + seen = Counter((w["alert"]["rule"], w["label"]) for w in worlds) + for table in RULE_IDS.values(): + for rid in table.values(): + assert seen[(rid, "benign")] > 0, rid + assert seen[(rid, "suspicious")] > 0, rid + + +def test_every_hidden_signal_lands_on_several_templates(worlds) -> None: + """CHEAT 4: no fixed template -> lookup mapping to memorise.""" + pairs = Counter((w["overlay"], w["template"]) for w in worlds if w["overlay"]) + by_kind: dict[str, set[str]] = {} + for (kind, template), _ in pairs.items(): + by_kind.setdefault(kind, set()).add(template) + assert len(by_kind["funnel"]) == 6 + assert len(by_kind["serial_closer"]) >= 4 + assert len(by_kind["doc_mismatch"]) == 3 + + +def test_overlay_only_on_benign_screens(worlds) -> None: + for w in worlds: + if w["overlay"]: + assert w["template"].startswith("B") and w["tier"] == "hidden" and w["label"] == "suspicious" + if w["template"].startswith("S"): + assert w["label"] == "suspicious" and w["overlay"] is None + + +def test_nothing_is_dated_on_or_after_the_fire_date(worlds) -> None: + for w in worlds: + fired = _day(w["alert"]["fired"]) + assert all(_day(t["date"]) < fired for t in w["transactions"]), w["seed"] + + +def test_transaction_ids_are_chronological_and_dense(worlds) -> None: + for w in worlds[:512]: + ids = [t["id"] for t in w["transactions"]] + assert ids == [f"T-{i + 1}" for i in range(len(ids))] + days = [_day(t["date"]) for t in w["transactions"]] + assert days == sorted(days) + + +def test_triggering_set_is_non_empty_and_shown(worlds) -> None: + for w in worlds: + ids = {t["id"] for t in w["transactions"]} + assert w["alert"]["triggering"], w["seed"] + assert set(w["alert"]["triggering"]) <= ids + + +def test_planted_is_citable_after_the_right_lookups(worlds) -> None: + """Every planted id exists in the world: a kyc field, a T-id, a CP-id, a doc or a prior alert.""" + for w in worlds: + universe = {f"kyc.{k}" for k in w["customer"]} | {t["id"] for t in w["transactions"]} + universe |= {c["id"] for c in w["counterparties"]} | {d["id"] for d in w["documents"]} | {p["id"] for p in w["prior_alerts"]} + assert w["planted"], w["seed"] + for alt in w["planted"]: + assert alt and set(alt) <= universe, (w["seed"], alt) + + +def test_benign_screens_always_carry_an_outbound_counterparty(worlds) -> None: + for w in worlds: + if w["template"].startswith("B"): + assert any(c["direction"] == "out" for c in w["screen_counterparties"]), w["seed"] + assert len(w["screen_counterparties"]) == 2 + assert 1 <= len(w["screen_counterparties"]) <= 2 + + +def test_prior_alert_count_on_screen_matches_the_lookup(worlds) -> None: + for w in worlds: + assert len(w["prior_alerts"]) == w["customer"]["prior_alerts"] + + +def test_held_out_slice_moves_the_threshold_and_the_screen(worlds) -> None: + held = [w for w in worlds if w["held_out"]] + main = [w for w in worlds if not w["held_out"]] + assert 0.10 < len(held) / len(worlds) < 0.15 + for w in held: + assert is_held_out(w["seed"]) + assert w["currency"] == "KRN" and w["threshold"] == 2_000_000 and w["screen_order"] == "kyc_first" + assert w["alert"]["rule"] in RULE_IDS["held_out"].values() + assert w["customer"]["country"] == "Karnland" + for w in main: + assert w["currency"] == "USD" and w["threshold"] == 1_000_000 and w["screen_order"] == "alert_first" + assert w["alert"]["rule"] in RULE_IDS["main"].values() + + +def test_structuring_band_is_threshold_relative(worlds) -> None: + for w in worlds: + if w["template"] == "S1": + lo, hi = w["threshold"] * 92 // 100, w["threshold"] - 1 + deposits = [t for t in w["transactions"] if t["id"] in w["planted"][0]] + assert len(deposits) == 8 + assert all(lo <= t["amount"] <= hi for t in deposits) + assert len({t["branch"] for t in deposits}) == 3 + + +def test_screen_carries_no_hidden_field(worlds) -> None: + hidden = {"label", "typology", "planted", "overlay", "tier", "template", "reference_minutes", "reference_policy", + "prior_alerts_list", "documents", "counterparties", "transactions", "other_alerted_accounts", "party", "rationale"} + for w in worlds[:256]: + text = canonical_json(screen_of(w)) + for key in hidden: + assert f'"{key}"' not in text, key diff --git a/envs/alert_triage/tests/test_triage_policies.py b/envs/alert_triage/tests/test_triage_policies.py new file mode 100644 index 0000000..6b734a4 --- /dev/null +++ b/envs/alert_triage/tests/test_triage_policies.py @@ -0,0 +1,84 @@ +"""The reference analysts are players, not oracles, and the reference exists everywhere.""" + +from __future__ import annotations + +import copy +import inspect + +from alert_triage.engine import Engine +from alert_triage.generator import MAX_TURNS, canonical_json, generate, world_for_seed +from alert_triage.policies import POLICIES, SHIPPED, reference_for, run_policy + +ALL = range(4096) + + +def test_every_policy_takes_only_the_view() -> None: + for name, policy in POLICIES.items(): + params = list(inspect.signature(policy).parameters) + assert params == ["view"], name + + +def test_policies_never_mutate_the_world() -> None: + for seed in range(32): + world = world_for_seed(seed) + before = canonical_json(world) + for policy in POLICIES.values(): + run_policy(policy, world) + assert canonical_json(world) == before + + +def test_shipped_policies_are_never_rejected() -> None: + for seed in range(256): + world = world_for_seed(seed) + for name in SHIPPED + ("thorough_wasteful",): + engine = run_policy(POLICIES[name], world) + assert engine.rejected == 0, (seed, name) + assert engine.disposition is not None, (seed, name) + + +def test_turn_budgets_are_as_documented() -> None: + for seed in range(128): + world = world_for_seed(seed) + assert run_policy(POLICIES["fast"], world).turns <= 2 + assert run_policy(POLICIES["targeted"], world).turns <= 5 + thorough = run_policy(POLICIES["thorough"], world).turns + assert thorough <= MAX_TURNS - 1 + assert run_policy(POLICIES["thorough_wasteful"], world).turns == thorough + 1 + + +def test_reference_exists_and_the_investigators_catch_everything_on_every_seed() -> None: + """Lens 0: `reference_minutes` is never null. Lens 1/2: caught 1.0 over 4096, not 24.""" + for seed in ALL: + world = generate(seed) + minutes, policy = reference_for(world) + assert minutes is not None and policy is not None, seed + for name in ("targeted", "thorough"): + assert run_policy(POLICIES[name], world).outcome == "solved", (seed, name) + + +def test_fast_misses_exactly_the_hidden_tier() -> None: + for seed in range(512): + world = world_for_seed(seed) + outcome = run_policy(POLICIES["fast"], world).outcome + assert (outcome == "solved") == (world["tier"] != "hidden"), seed + + +def test_reference_is_fast_where_the_screen_suffices() -> None: + for seed in range(512): + world = world_for_seed(seed) + assert world["reference_policy"] == ("targeted" if world["tier"] == "hidden" else "fast"), seed + + +def test_policies_cannot_see_the_answer() -> None: + """Run every policy against a view whose world has the hidden fields deleted: identical replies.""" + world = world_for_seed(9) + stripped = copy.deepcopy(world) + for key in ("label", "typology", "planted", "overlay", "tier", "template", "reference_minutes", "reference_policy"): + stripped[key] = None if key != "planted" else [] + for name, policy in POLICIES.items(): + a, b = Engine(world), Engine(stripped) + while not a.done: + ra, rb = policy(a.view()), policy(b.view()) + assert ra == rb, name + a.step(ra) + b.step(rb) diff --git a/envs/alert_triage/tests/test_triage_protocol.py b/envs/alert_triage/tests/test_triage_protocol.py new file mode 100644 index 0000000..fc46b4f --- /dev/null +++ b/envs/alert_triage/tests/test_triage_protocol.py @@ -0,0 +1,48 @@ +"""The pinned parse rules — the ones two runtimes disagree on.""" + +from __future__ import annotations + +from alert_triage.protocol import extract_candidate, parse_reply + +FENCE = "```" + + +def test_fence_wins_over_an_earlier_bare_brace() -> None: + text = 'first {not it} then ' + FENCE + 'json\n{"action":"lookup","what":"documents"}\n' + FENCE + assert extract_candidate(text) == '{"action":"lookup","what":"documents"}\n' + obj, reason = parse_reply(text) + assert obj == {"action": "lookup", "what": "documents"} and reason is None + + +def test_first_balanced_span_is_string_aware() -> None: + text = 'ok {"note":"a } inside \\" quotes","action":"close"} trailing {"x":1}' + obj, _ = parse_reply(text) + assert obj == {"note": 'a } inside " quotes', "action": "close"} + + +def test_invalid_first_span_is_rejected_without_further_scanning() -> None: + obj, reason = parse_reply('{"a":1,} {"action":"lookup","what":"documents"}') + assert obj is None and reason + + +def test_non_object_json_is_rejected() -> None: + assert parse_reply("[1,2]")[0] is None + assert parse_reply(FENCE + "json\n[{\"action\":\"lookup\"}]\n" + FENCE)[0] is None + assert parse_reply("42")[0] is None + + +def test_nan_and_infinity_are_rejected_like_json_parse_does() -> None: + assert parse_reply('{"x":NaN}')[0] is None + assert parse_reply('{"x":Infinity}')[0] is None + + +def test_unbalanced_or_absent_braces() -> None: + assert parse_reply('{"action":"lookup"')[0] is None + assert parse_reply("no braces at all")[0] is None + assert parse_reply("")[0] is None + assert parse_reply(None)[0] is None + + +def test_nested_object_is_the_outer_one() -> None: + obj, _ = parse_reply('{"outer":{"action":"lookup"}}') + assert obj == {"outer": {"action": "lookup"}} diff --git a/envs/alert_triage/tests/test_triage_reward.py b/envs/alert_triage/tests/test_triage_reward.py new file mode 100644 index 0000000..4dafa68 --- /dev/null +++ b/envs/alert_triage/tests/test_triage_reward.py @@ -0,0 +1,94 @@ +"""Worked examples of the reward, and the marker region the page quotes.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from alert_triage.reward import ( + FALSE_ESCALATION_CREDIT, + WEIGHTS, + Episode, + escalate_break_even, + metrics, + score, + score_exact, + total, +) + + +def ep(**kw) -> Episode: + base = dict(label="benign", planted=[["kyc.a"]], disposition="close", cites=["kyc.a"], minutes_spent=33, + reference_minutes=33, turns_spent=1, rejected=0) + base.update(kw) + return Episode(**base) + + +def test_weights_sum_to_one_and_have_a_counterweight() -> None: + assert abs(sum(WEIGHTS.values()) - 1.0) < 1e-12 + from alert_triage.reward import ROLES + assert "counterweight" in ROLES.values() + + +def test_perfect_close() -> None: + assert score(ep()) == {"caught": 1.0, "hours": 1.0, "evidence": 1.0} + assert abs(total(ep()) - 1.0) < 1e-12 + + +def test_a_miss_scores_exactly_zero() -> None: + e = ep(label="suspicious", planted=[["T-1", "T-2"]], disposition="close", cites=["kyc.a"]) + assert score(e) == {"caught": 0.0, "hours": 0.0, "evidence": 0.0} + assert total(e) == 0.0 + + +def test_a_false_escalation_takes_half_the_counterweight_and_nothing_else() -> None: + e = ep(disposition="escalate", cites=["kyc.a"]) + assert score(e) == {"caught": FALSE_ESCALATION_CREDIT, "hours": 0.0, "evidence": 0.0} + + +def test_no_disposition_is_zero_and_truncated_is_none() -> None: + assert score(ep(disposition=None, cites=[])) == {"caught": 0.0, "hours": 0.0, "evidence": 0.0} + assert score(ep(truncated=True)) == {"caught": None, "hours": None, "evidence": None} + assert total(ep(truncated=True)) is None + + +def test_hours_is_capped_and_ratio_against_the_reference() -> None: + assert score(ep(minutes_spent=20))["hours"] == 1.0 + assert abs(score(ep(minutes_spent=66))["hours"] - 0.5) < 1e-12 + assert score(ep(reference_minutes=None))["hours"] is None + assert total(ep(reference_minutes=None)) is None + + +def test_evidence_is_f1_over_the_best_alternate() -> None: + e = ep(planted=[["a", "b"], ["a"]], cites=["a"]) + assert score(e)["evidence"] == 1.0 + e = ep(planted=[["a", "b"]], cites=["a", "x", "y"]) + assert abs(score(e)["evidence"] - 0.4) < 1e-12 # 2·1 / (3+2) + e = ep(planted=[["a"]], cites=["a", "b", "c", "d", "e", "f", "g", "h"]) + assert abs(score(e)["evidence"] - 2 / 9) < 1e-12 # one planted among eight cited + assert score_exact(e)["evidence"] == [2, 9] + + +def test_exact_and_float_scores_agree() -> None: + for e in (ep(), ep(minutes_spent=66), ep(planted=[["a", "b"]], cites=["a", "x", "y"]), ep(disposition="escalate")): + exact = score_exact(e) + approx = score(e) + for k in approx: + assert abs(exact[k][0] / exact[k][1] - approx[k]) < 1e-12 + + +def test_break_even_is_between_zero_and_one() -> None: + assert 0.3 < escalate_break_even() < 0.7 + + +def test_metrics_are_diagnostics() -> None: + m = metrics(ep(disposition="escalate", typology="UNKNOWN", true_typology="UNKNOWN", label="suspicious")) + assert m["typology_match"] == 1.0 and m["false_escalation"] == 0.0 + assert metrics(ep())["typology_match"] is None + + +def test_exactly_one_marker_pair() -> None: + text = (Path(__file__).parent.parent / "alert_triage" / "reward.py").read_text() + assert text.count("# region: pig-demo/reward") == 1 + assert text.count("# endregion: pig-demo/reward") == 1 diff --git a/envs/alert_triage/tests/test_triage_rng.py b/envs/alert_triage/tests/test_triage_rng.py new file mode 100644 index 0000000..a54b11d --- /dev/null +++ b/envs/alert_triage/tests/test_triage_rng.py @@ -0,0 +1,52 @@ +"""The portable primitives, pinned so the TypeScript port has vectors to hit.""" + +from __future__ import annotations + +from alert_triage.rng import ( + XorShift32, + civil_from_days, + days_from_civil, + days_in_month, + fnv1a32, + iso_date, +) + +# fnv1a32 over the decimal seed — identical to wordle_five's and to engine.ts. +FNV_VECTORS = {"0": 0x350CA8AF, "1": 0x340CA71C, "42": 0x87E38583, "4095": 0x43875F5F} + + +def test_fnv1a_vectors() -> None: + for text, expect in FNV_VECTORS.items(): + assert fnv1a32(text) == expect, text + + +def test_xorshift_sequence_is_pinned() -> None: + rng = XorShift32(fnv1a32("0")) + first = [rng.next() for _ in range(5)] + assert first == XORSHIFT_FROM_SEED_0 + + +XORSHIFT_FROM_SEED_0 = [2738490563, 3068243922, 3765331391, 3085691315, 2439018365] + + +def test_zero_seed_is_replaced() -> None: + assert XorShift32(0).state != 0 + + +def test_sample_is_distinct_and_partial_shuffle() -> None: + rng = XorShift32(7) + out = rng.sample(list(range(10)), 4) + assert len(out) == 4 and len(set(out)) == 4 + + +def test_civil_dates_round_trip() -> None: + for day in range(days_from_civil(1999, 12, 25), days_from_civil(2030, 3, 2)): + y, m, d = civil_from_days(day) + assert days_from_civil(y, m, d) == day + + +def test_known_dates() -> None: + assert days_from_civil(1970, 1, 1) == 0 + assert iso_date(days_from_civil(2026, 3, 1)) == "2026-03-01" + assert days_in_month(2024, 2) == 29 and days_in_month(2026, 2) == 28 + assert days_in_month(2026, 12) == 31 diff --git a/envs/alert_triage/tests/test_triage_screen_leak.py b/envs/alert_triage/tests/test_triage_screen_leak.py new file mode 100644 index 0000000..e62947c --- /dev/null +++ b/envs/alert_triage/tests/test_triage_screen_leak.py @@ -0,0 +1,87 @@ +"""The hidden tier is hidden only if the generator keeps it so. This is the gate. + +Two checks. The first is structural and exact: for every seed the free screen +is byte-identical with the overlay on or off, for every overlay the screen can +carry. The second is distributional: across seeds, the screen features a +policy could key on do not separate hidden from benign beyond noise. The +generator draws the tier BEFORE the template and the screen, so the two +populations are the same draw; this test is what would catch a regression that +made them differ. +""" + +from __future__ import annotations + +import math + +import pytest + +from alert_triage.generator import canonical_json, compatible_overlays, generate, screen_of + +ALL = range(4096) + + +@pytest.fixture(scope="module") +def pairs(): + out = [] + for s in ALL: + base = generate(s, overlay=None) + if base["template"].startswith("S"): + continue + out.append((s, base)) + return out + + +def test_overlay_never_touches_the_screen(pairs) -> None: + for seed, base in pairs: + before = canonical_json(screen_of(base)) + for kind in compatible_overlays(base): + overlaid = generate(seed, overlay=kind) + assert overlaid["label"] == "suspicious" and overlaid["overlay"] == kind + assert canonical_json(screen_of(overlaid)) == before, (seed, kind) + + +def test_the_seed_overlay_is_one_of_the_compatible_kinds(pairs) -> None: + for seed, base in pairs: + w = generate(seed) + if w["overlay"]: + assert w["overlay"] in compatible_overlays(base) + + +def _features(w) -> list[float]: + c = w["customer"] + screen = screen_of(w) + trig = screen["triggering_transactions"] + amounts = [t["amount"] for t in trig] + return [ + float(c["prior_alerts"]), + float(c["prior_sars"]), + float(c["pep"]), + float(sum(1 for cp in screen["screen_counterparties"] if cp["direction"] == "out")), + float(sum(1 for a in amounts if a % 100000 == 0)) / max(1, len(amounts)), + float(len(trig)), + float(len({t["branch"] for t in trig if t["branch"]})), + float(sum(r["wires_out"] > 0 for r in w["summary"])), + float(screen["alert"]["rule"] == "R-STR-01"), + float(c["expected_monthly_cash"] > 0), + ] + + +def test_screen_features_do_not_separate_hidden_from_benign() -> None: + """Per-feature z-test between the two populations; nothing beyond 4 sigma.""" + benign, hidden = [], [] + for s in ALL: + w = generate(s) + if w["tier"] == "benign": + benign.append(_features(w)) + elif w["tier"] == "hidden": + hidden.append(_features(w)) + assert len(hidden) > 400 + for j in range(len(benign[0])): + b = [f[j] for f in benign] + h = [f[j] for f in hidden] + mb, mh = sum(b) / len(b), sum(h) / len(h) + vb = sum((x - mb) ** 2 for x in b) / max(1, len(b) - 1) + vh = sum((x - mh) ** 2 for x in h) / max(1, len(h) - 1) + se = math.sqrt(vb / len(b) + vh / len(h)) or 1e-9 + z = abs(mb - mh) / se + assert z < 4.0, f"feature {j}: benign {mb:.3f} vs hidden {mh:.3f}, z={z:.1f}" diff --git a/envs/build_manifest.py b/envs/build_manifest.py index 8aeb7eb..95ed28f 100644 --- a/envs/build_manifest.py +++ b/envs/build_manifest.py @@ -13,11 +13,18 @@ from pathlib import Path TRACES = Path(__file__).parent.parent / "public" / "traces" +# Arm ids are shared across tasksets where they mean the same thing (the two +# model arms); the generated arms are per-taskset and simply do not collide. ARMS = { "base-off": ("Out of the box", "recorded"), "base-on": ("Allowed to think", "intervened"), + # wordle "solver": ("Best-known play", "generated"), "cautious": ("Never wastes a guess", "generated"), + # alert-triage: the shipped scripted analysts from alert_triage/policies.py + "fast": ("Reads the screen", "generated"), + "targeted": ("Checks the hidden tells", "generated"), + "thorough": ("Runs the full procedure", "generated"), } # What was done to the run, for arms that had something done to them. Required @@ -27,7 +34,7 @@ INTERVENTIONS = { "base-on": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning.", } -ORDER = ["base-off", "base-on", "solver", "cautious"] +ORDER = ["base-off", "base-on", "solver", "cautious", "fast", "targeted", "thorough"] def main() -> int: @@ -67,11 +74,13 @@ def main() -> int: by_arm.setdefault(r["id"].rsplit("-s", 1)[0], []).append(r) print(f"{slug}: {len(runs)} runs") for arm, group in by_arm.items(): - solved = 0 + outcomes: dict[str, int] = {} for r in group: data = json.loads((TRACES.parent / r["path"].lstrip("/")).read_text()) - solved += 1 if data["outcome"] == "solved" else 0 - print(f" {arm:<10} {len(group)} runs, solved {solved}/{len(group)}") + outcomes[data["outcome"]] = outcomes.get(data["outcome"], 0) + 1 + solved = outcomes.get("solved", 0) + rest = ", ".join(f"{k} {v}" for k, v in sorted(outcomes.items()) if k != "solved") + print(f" {arm:<10} {len(group)} runs, solved {solved}/{len(group)}" + (f" ({rest})" if rest else "")) return 0 diff --git a/envs/capture.py b/envs/capture.py index 4b1539f..0671dc6 100644 --- a/envs/capture.py +++ b/envs/capture.py @@ -12,6 +12,13 @@ number it shows from the recorded moves, and says so. See verify.ts. Usage: uv run python envs/capture.py --arm base-off --seeds 0-7 uv run python envs/capture.py --arm solver --seeds 0-7 # no model needed + uv run python envs/capture.py --taskset alert-triage --arm base-on --seeds 1-7,9 + uv run python envs/capture.py --taskset alert-triage --arm targeted --seeds 1-7,9 + +spark-1 serves one model, single-stream: run the model arms one after another, +never concurrently. A thinking arm on alert-triage takes minutes per seed — +run it in the background with stdout redirected to a file and poll the +fixtures being written rather than the log. """ from __future__ import annotations @@ -27,9 +34,14 @@ import urllib.request from pathlib import Path sys.path.insert(0, str(Path(__file__).parent / "wordle_five")) +sys.path.insert(0, str(Path(__file__).parent / "alert_triage")) import numpy as np # noqa: E402 +from alert_triage import policies as AT # noqa: E402 +from alert_triage import taskset as AT_taskset # noqa: E402 +from alert_triage.generator import is_held_out, world_for_seed # noqa: E402 + from wordle_five import solver as S # noqa: E402 from wordle_five.engine import MAX_GUESSES, Game, answers # noqa: E402 from wordle_five.protocol import ( # noqa: E402 @@ -40,7 +52,8 @@ from wordle_five.protocol import ( # noqa: E402 ) from wordle_five.reward import Episode, metrics, score # noqa: E402 -OUT = Path(__file__).parent.parent / "public" / "traces" / "wordle" +TRACES = Path(__file__).parent.parent / "public" / "traces" +OUT = TRACES / "wordle" ENDPOINT = os.environ.get("PIG_DEMO_INFERENCE", "http://100.127.247.67:8001/v1/chat/completions") MODEL = os.environ.get("PIG_DEMO_MODEL", "brain-qwen38-dspark") @@ -52,7 +65,7 @@ ARMS = { } -def call_model(messages: list[dict], thinking: bool) -> dict: +def call_model(messages: list[dict], thinking: bool, max_tokens: int | None = None, timeout: int = 300) -> dict: """One completion. Returns reply, reasoning and the real call metrics. Never raises on an upstream failure — a dropped call becomes a turn with a @@ -64,7 +77,7 @@ def call_model(messages: list[dict], thinking: bool) -> dict: "model": MODEL, "messages": messages, "temperature": 0.7, - "max_tokens": 2048 if thinking else 512, + "max_tokens": max_tokens or (2048 if thinking else 512), "chat_template_kwargs": {"enable_thinking": bool(thinking)}, } request = urllib.request.Request( @@ -74,7 +87,7 @@ def call_model(messages: list[dict], thinking: bool) -> dict: ) started = time.time() try: - with urllib.request.urlopen(request, timeout=300) as response: + with urllib.request.urlopen(request, timeout=timeout) as response: payload = json.loads(response.read()) except (urllib.error.URLError, TimeoutError, OSError) as exc: return { @@ -245,33 +258,158 @@ def capture(arm: str, seed: int) -> dict: } +# ------------------------------------------------------------ alert-triage -- +# +# The same fixture shape as wordle — runId, seed, model, capturedAt, rewards, +# metrics, truncated, outcome, turns[{reply, reasoning, call, info}] — driven +# through `alert_triage.taskset.play_episode`, the loop the probe and the +# tests share. The browser regenerates the world from the seed and replays the +# reply strings; nothing else in the fixture is trusted by the page. + +AT_ARMS = { + "base-off": {"label": "Out of the box", "thinking": False, "max_tokens": 1024, "timeout": 300}, + "base-on": {"label": "Allowed to think", "thinking": True, "max_tokens": 4096, "timeout": 900}, + "fast": {"label": "Reads the screen", "thinking": None, "policy": "fast"}, + "targeted": {"label": "Checks the hidden tells", "thinking": None, "policy": "targeted"}, + "thorough": {"label": "Runs the full procedure", "thinking": None, "policy": "thorough"}, +} +AT_MODEL_NAME = {"fast": "fast-analyst", "targeted": "targeted-analyst", "thorough": "thorough-analyst"} + +# A thinking budget the model exhausts is recorded as finishReason "length" +# with whatever content survived (usually none, which the engine rejects). +# That is a thing the model did under the budget it was given, not a capture +# error, and the fixture says so rather than retrying until it looks better. + + +def _messages_from(prompt: str, transcript: list[dict]) -> list[dict]: + """The chat a model sees: the system prompt, the screen, then each reply and what it got back.""" + messages = [{"role": "system", "content": prompt}, {"role": "user", "content": transcript[0]["observation"]}] + for entry in transcript[1:]: + messages.append({"role": "assistant", "content": entry["reply"] or ""}) + messages.append({"role": "user", "content": entry["observation"]}) + return messages + + +def _generated_call() -> dict: + return {"promptTokens": None, "completionTokens": None, "reasoningTokens": None, "durationMs": None, "finishReason": "generated"} + + +def capture_alert_triage(arm: str, seed: int) -> dict: + config = AT_ARMS[arm] + calls: list[dict] = [] + + if config["thinking"] is None: + policy = AT.POLICIES[config["policy"]] + + def respond(prompt: str, transcript: list[dict], view: dict) -> str | None: + reply = policy(view) + calls.append({"reply": reply, "reasoning": None, "call": _generated_call()}) + return reply + else: + def respond(prompt: str, transcript: list[dict], view: dict) -> str | None: + result = call_model(_messages_from(prompt, transcript), bool(config["thinking"]), + max_tokens=config["max_tokens"], timeout=config["timeout"]) + calls.append(result) + return result["reply"] + + world = world_for_seed(seed) + played = AT_taskset.play_episode(seed, respond, world) + steps = played["transcript"][1:] + assert len(steps) == len(calls), "one model call per engine step" + + turns = [ + { + **call, + "info": {"action": step["action"], "rejection": step["rejection"], "observation": step["observation"]}, + } + for call, step in zip(calls, steps) + ] + return { + "runId": f"{arm}-s{seed}", + "seed": seed, + "model": AT_MODEL_NAME.get(arm, MODEL), + "capturedAt": time.strftime("%Y-%m-%d"), + "rewards": played["rewards"], + "metrics": played["metrics"], + "truncated": played["truncated"], + "outcome": played["outcome"], + "info": {**played["info"], "tier": world["tier"], "screen": played["transcript"][0]["observation"]}, + "turns": turns, + } + + +def _at_summary(episode: dict) -> str: + r = episode["rewards"] + fmt = lambda v: " -- " if v is None else f"{v:.2f}" # noqa: E731 + actions = [] + for t in episode["turns"]: + a = t["info"]["action"] + if a is None: + actions.append("REJ") + elif a["action"] == "lookup": + actions.append(a.get("month") or a.get("id") or a["what"]) + else: + actions.append(a["action"].upper()) + return ( + f"{episode['info']['tier']:<8}{episode['info']['template']:<3} {episode['outcome']:<8}" + f" caught={fmt(r['caught'])} hours={fmt(r['hours'])} evid={fmt(r['evidence'])}" + f" {episode['metrics']['hours_spent']:.2f}h {actions}" + ) + + +TASKSETS = { + "wordle-five": {"arms": ARMS, "out": TRACES / "wordle", "capture": capture}, + "alert-triage": {"arms": AT_ARMS, "out": TRACES / "alert-triage", "capture": capture_alert_triage}, +} + + def parse_seeds(spec: str) -> list[int]: - if "-" in spec: - lo, hi = spec.split("-") - return list(range(int(lo), int(hi) + 1)) - return [int(s) for s in spec.split(",")] + """"0-7", "1,3", or a mix: "1-7,9".""" + seeds: list[int] = [] + for part in spec.split(","): + if "-" in part: + lo, hi = part.split("-") + seeds.extend(range(int(lo), int(hi) + 1)) + else: + seeds.append(int(part)) + return seeds def main() -> int: parser = argparse.ArgumentParser() - parser.add_argument("--arm", required=True, choices=sorted(ARMS)) + parser.add_argument("--taskset", default="wordle-five", choices=sorted(TASKSETS)) + parser.add_argument("--arm", required=True) parser.add_argument("--seeds", default="0-7") args = parser.parse_args() - OUT.mkdir(parents=True, exist_ok=True) + taskset = TASKSETS[args.taskset] + if args.arm not in taskset["arms"]: + parser.error(f"--arm must be one of {sorted(taskset['arms'])} for {args.taskset}") + + out: Path = taskset["out"] + out.mkdir(parents=True, exist_ok=True) for seed in parse_seeds(args.seeds): + if args.taskset == "alert-triage" and is_held_out(seed): + # The held-out bucket is never captured, probed or trained on. A + # fixture for one would put a held-out alert behind a permalink. + print(f"{args.arm}-s{seed}: seed {seed} is held out — skipped", flush=True) + continue started = time.time() - episode = capture(args.arm, seed) - path = OUT / f"{episode['runId']}.json" + episode = taskset["capture"](args.arm, seed) + path = out / f"{episode['runId']}.json" path.write_text(json.dumps(episode, indent=2) + "\n") - guesses = [t["info"]["guess"] for t in episode["turns"]] - print( - f"{episode['runId']:>16} {episode['answer']} {episode['outcome']:<7}" - f" solved={episode['rewards']['solved']:.0f}" - f" econ={episode['rewards']['economy']:.2f}" - f" cons={episode['rewards']['consistency']:.2f}" - f" {time.time()-started:5.1f}s {guesses}" - ) + if args.taskset == "wordle-five": + guesses = [t["info"]["guess"] for t in episode["turns"]] + print( + f"{episode['runId']:>16} {episode['answer']} {episode['outcome']:<7}" + f" solved={episode['rewards']['solved']:.0f}" + f" econ={episode['rewards']['economy']:.2f}" + f" cons={episode['rewards']['consistency']:.2f}" + f" {time.time()-started:5.1f}s {guesses}", + flush=True, + ) + else: + print(f"{episode['runId']:>16} {_at_summary(episode)} {time.time()-started:6.1f}s", flush=True) return 0 diff --git a/envs/probe.py b/envs/probe.py index 8492b16..b30aaea 100644 --- a/envs/probe.py +++ b/envs/probe.py @@ -10,8 +10,9 @@ A blended float hides the failure this is built to catch: a component that is flat across every policy is measuring nothing, and it will still move the total because the other components move. -Run: uv run python envs/probe.py -Exits non-zero on any violated assertion. +Run: uv run python envs/probe.py [--taskset wordle-five|alert-triage] +Runs every registered taskset by default. Exits non-zero on any violated +assertion. """ from __future__ import annotations @@ -20,6 +21,7 @@ import sys from pathlib import Path sys.path.insert(0, str(Path(__file__).parent / "wordle_five")) +sys.path.insert(0, str(Path(__file__).parent / "alert_triage")) from wordle_five import solver as S # noqa: E402 from wordle_five.engine import ( # noqa: E402 @@ -32,6 +34,8 @@ from wordle_five.engine import ( # noqa: E402 from wordle_five.protocol import parse_guess # noqa: E402 from wordle_five.reward import ROLES, WEIGHTS, Episode, metrics, score, total # noqa: E402 +from alert_triage import ladder as alert_triage_ladder # noqa: E402 + SEEDS = list(range(16)) COMPONENTS = ("solved", "economy", "consistency") @@ -162,7 +166,7 @@ def play(policy, seed: int) -> Episode: ) -def run() -> int: +def run_wordle() -> int: print(f"Probing wordle-five over {len(SEEDS)} seeds\n") results: dict[str, dict[str, float]] = {} extra: dict[str, dict[str, float]] = {} @@ -243,5 +247,28 @@ def run() -> int: return 0 +# Each taskset registers its ladder here. The alert-triage rungs live in +# envs/alert_triage/alert_triage/ladder.py beside the policies they probe. +TASKSETS = { + "wordle-five": run_wordle, + "alert-triage": alert_triage_ladder.run, +} + + +def main(argv: list[str]) -> int: + import argparse + + parser = argparse.ArgumentParser() + parser.add_argument("--taskset", choices=sorted(TASKSETS), action="append") + args = parser.parse_args(argv) + chosen = args.taskset or list(TASKSETS) + status = 0 + for i, name in enumerate(chosen): + if i: + print("\n" + "=" * 78 + "\n") + status |= TASKSETS[name]() + return status + + if __name__ == "__main__": - raise SystemExit(run()) + raise SystemExit(main(sys.argv[1:])) diff --git a/envs/verify_fixtures.py b/envs/verify_fixtures.py index 84e71ad..92ce189 100644 --- a/envs/verify_fixtures.py +++ b/envs/verify_fixtures.py @@ -16,17 +16,20 @@ import sys from pathlib import Path sys.path.insert(0, str(Path(__file__).parent / "wordle_five")) +sys.path.insert(0, str(Path(__file__).parent / "alert_triage")) +from alert_triage import taskset as AT # noqa: E402 +from alert_triage.generator import is_held_out # noqa: E402 from wordle_five import solver as S # noqa: E402 from wordle_five.engine import Game # noqa: E402 from wordle_five.protocol import parse_guess # noqa: E402 from wordle_five.reward import Episode, score # noqa: E402 -TRACES = Path(__file__).parent.parent / "public" / "traces" / "wordle" +TRACES = Path(__file__).parent.parent / "public" / "traces" TOLERANCE = 1e-9 -def replay(fixture: dict) -> dict[str, float]: +def replay_wordle(fixture: dict) -> dict[str, float]: game = Game(seed=fixture["seed"]) if game.answer != fixture["answer"]: raise AssertionError( @@ -49,27 +52,61 @@ def replay(fixture: dict) -> dict[str, float]: return score(episode) +def replay_alert_triage(fixture: dict) -> dict[str, float | None]: + """The world comes from the seed, the score from the reply strings. Nothing + else in the fixture is trusted — the same rule the browser's verify uses.""" + seed = fixture["seed"] + if is_held_out(seed): + raise AssertionError(f"seed {seed} is in the held-out bucket and must never be captured") + played = AT.replay(seed, [turn["reply"] for turn in fixture["turns"]]) + if played["outcome"] != fixture["outcome"]: + raise AssertionError(f"replay gives outcome {played['outcome']!r}, fixture claims {fixture['outcome']!r}") + return played["rewards"] + + +REPLAY = {"wordle": replay_wordle, "alert-triage": replay_alert_triage} + + +def _delta(recomputed, recorded) -> float: + """None is "not scored". It matches only None; against a number it is a mismatch, not a zero.""" + if recomputed is None or recorded is None: + return 0.0 if recomputed is recorded else float("inf") + return abs(recomputed - recorded) + + def main() -> int: - fixtures = sorted(TRACES.glob("*.json")) - fixtures = [f for f in fixtures if f.name != "manifest.json"] + fixtures = [ + f for demo in sorted(p for p in TRACES.iterdir() if p.is_dir()) + for f in sorted(demo.glob("*.json")) + ] if not fixtures: print("no fixtures found — nothing to verify") return 0 failures = 0 for path in fixtures: + demo = path.parent.name + if demo not in REPLAY: + failures += 1 + print(f"{path.parent.name}/{path.name:<20} NO REPLAYER for demo '{demo}'") + continue data = json.loads(path.read_text()) - recomputed = replay(data) + try: + recomputed = REPLAY[demo](data) + except AssertionError as exc: + failures += 1 + print(f"{path.parent.name}/{path.name:<20} MISMATCH {exc}") + continue recorded = data["rewards"] - deltas = {k: abs(recomputed[k] - recorded[k]) for k in recorded} + deltas = {k: _delta(recomputed[k], recorded[k]) for k in recorded} worst = max(deltas.values()) status = "ok" if worst <= TOLERANCE else "MISMATCH" if worst > TOLERANCE: failures += 1 culprit = max(deltas, key=deltas.get) - print(f"{path.name:>20} {status} worst delta {worst:.3g} on '{culprit}'") + print(f"{path.parent.name}/{path.name:<20} {status} worst delta {worst:.3g} on '{culprit}'") else: - print(f"{path.name:>20} {status} delta {worst:.1e}") + print(f"{path.parent.name}/{path.name:<20} {status} delta {worst:.1e}") print(f"\n{len(fixtures)} fixtures, {failures} mismatched") return 1 if failures else 0 diff --git a/public/og/alert-triage.png b/public/og/alert-triage.png new file mode 100644 index 0000000..a75a912 Binary files /dev/null and b/public/og/alert-triage.png differ diff --git a/public/traces/alert-triage/base-off-s1.json b/public/traces/alert-triage/base-off-s1.json new file mode 100644 index 0000000..bf7c530 --- /dev/null +++ b/public/traces/alert-triage/base-off-s1.json @@ -0,0 +1,92 @@ +{ + "runId": "base-off-s1", + "seed": 1, + "model": "brain-qwen38-dspark", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 0.55, + "evidence": 0.2857142857142857 + }, + "metrics": { + "hours_spent": 1.0, + "reference_hours": 0.55, + "lookups": 1.0, + "turns_used": 2.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": null + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "benign", + "template": "B4", + "overlay": null, + "typology": null, + "planted": [ + [ + "kyc.expected_monthly_cash" + ], + [ + "kyc.expected_monthly_cash", + "kyc.business_type" + ] + ], + "reference_minutes": 33, + "reference_policy": "fast", + "held_out": false, + "tier": "benign", + "screen": "ALERT A-8971 rule R-STR-01 \u2014 Three or more cash deposits just under the reporting threshold within ten days\n fired 2026-04-26\n triggering transactions:\n T-213 2026-04-16 cash_in 9,440.00 USD branch BR-052\n T-217 2026-04-21 cash_in 8,950.00 USD branch BR-052\n T-219 2026-04-22 cash_in 8,880.00 USD branch BR-052\nKYC FILE C-9647 Oakhill Kitchen Group\n kyc.account_opened = 2019-07-11\n kyc.business_type = pizzeria\n kyc.country = United States\n kyc.customer_type = business\n kyc.expected_activity = Cash-intensive food service; daily takings deposited most trading days.\n kyc.expected_monthly_cash = 86,000.00 USD\n kyc.kyc_refreshed = 2026-03-13\n kyc.pep = False\n kyc.prior_alerts = 4\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 492,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 42,400 0 0 0 0 13,935 18\n 2025-06 38,160 0 0 0 0 12,582 19\n 2025-07 44,260 0 0 0 0 15,928 18\n 2025-08 41,680 0 0 0 0 13,940 18\n 2025-09 44,950 0 0 0 0 15,498 17\n 2025-10 35,860 0 0 0 0 13,284 19\n 2025-11 42,370 0 0 0 0 12,792 19\n 2025-12 39,280 0 0 0 0 13,776 19\n 2026-01 39,860 0 0 0 0 13,940 19\n 2026-02 39,280 0 0 0 0 12,718 18\n 2026-03 36,510 0 0 0 0 11,693 19\n 2026-04 71,720 0 0 0 0 14,218 18\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 out 48 txns 103,976.00 USD\n CP-2 out 12 txns 60,331.50 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "{\"action\":\"lookup\",\"what\":\"prior_alerts\"}", + "reasoning": null, + "call": { + "promptTokens": 1531, + "completionTokens": 12, + "reasoningTokens": null, + "durationMs": 1886, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "lookup", + "what": "prior_alerts" + }, + "rejection": null, + "observation": "PRIOR ALERTS:\n A-3217 R-WIRE-04 fired 2024-07-08 closed after 7 day(s) by Beatriz Grimaldi\n Closed: destination and amount match the remittance declared at onboarding; within stated income.\n A-4028 R-DORM-05 fired 2024-10-12 closed after 9 day(s) by Mateo Jensen\n Closed: reactivation explained by a documented source of funds; document reviewed and retained.\n A-2622 R-VEL-02 fired 2025-01-17 closed after 5 day(s) by Samir Pereira\n Closed: pass-through matches the settlement terms in the agreement on file; counterparties verified.\n A-1389 R-STR-01 fired 2026-03-06 closed after 3 day(s) by Marta Eskildsen\n Closed: deposits are documented daily takings; amounts vary with trading days; within kyc.expected_monthly_cash.\n\nHours spent so far: 0.95. You have 7 replies left." + } + }, + { + "reply": "{\"action\":\"close\",\"reason\":\"PREVIOUSLY_REVIEWED\",\"cites\":[\"A-1389\",\"T-213\",\"T-217\",\"T-219\",\"kyc.expected_activity\",\"kyc.expected_monthly_cash\"],\"note\":\"Same rule R-STR-01 was fired on 2026-03-06 (A-1389) and closed by Marta Eskildsen after confirming the deposits were documented daily takings varying with trading days, within expected monthly cash. The current three deposits (T-213, T-217, T-219) follow the identical pattern: cash-intensive pizzeria, amounts just under the 10k threshold consistent with splitting daily takings, all at the same branch, and April's total cash-in of 71,720 is well within the 86,000 expected monthly cash. No new counterparty, no change in activity profile. This is a repeat rule artefact on a known, documented cash-deposit pattern.\"}", + "reasoning": null, + "call": { + "promptTokens": 1827, + "completionTokens": 226, + "reasoningTokens": null, + "durationMs": 8850, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "close", + "reason": "PREVIOUSLY_REVIEWED", + "cites": [ + "A-1389", + "T-213", + "T-217", + "T-219", + "kyc.expected_activity", + "kyc.expected_monthly_cash" + ], + "note": "Same rule R-STR-01 was fired on 2026-03-06 (A-1389) and closed by Marta Eskildsen after confirming the deposits were documented daily takings varying with trading days, within expected monthly cash. The current three deposits (T-213, T-217, T-219) follow the identical pattern: cash-intensive pizzeria, amounts just under the 10k threshold consistent with splitting daily takings, all at the same branch, and April's total cash-in of 71,720 is well within the 86,000 expected monthly cash. No new counterparty, no change in activity profile. This is a repeat rule artefact on a known, documented cash-deposit pattern." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/base-off-s2.json b/public/traces/alert-triage/base-off-s2.json new file mode 100644 index 0000000..17dbe0f --- /dev/null +++ b/public/traces/alert-triage/base-off-s2.json @@ -0,0 +1,145 @@ +{ + "runId": "base-off-s2", + "seed": 2, + "model": "brain-qwen38-dspark", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 0.34146341463414637, + "evidence": 0.4 + }, + "metrics": { + "hours_spent": 2.05, + "reference_hours": 0.7, + "lookups": 4.0, + "turns_used": 5.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": null + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "benign", + "template": "B6", + "overlay": null, + "typology": null, + "planted": [ + [ + "doc.D-aded" + ] + ], + "reference_minutes": 42, + "reference_policy": "fast", + "held_out": false, + "tier": "benign", + "screen": "ALERT A-9760 rule R-DORM-05 \u2014 Large credit to an account with no material activity for nine months\n fired 2026-04-23\n triggering transactions:\n T-4 2026-04-18 wire_in 356,000.00 USD CP-1\n T-5 2026-04-19 wire_out 291,900.00 USD CP-2\nKYC FILE C-7059 Gabriela Kowalczyk\n kyc.account_opened = 2012-03-09\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Retired; low activity expected.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2025-12-12\n kyc.occupation = retired\n kyc.pep = False\n kyc.prior_alerts = 2\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 31,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 0 0 0 0 0 0 0\n 2025-06 0 0 0 0 0 0 0\n 2025-07 0 170 0 0 0 0 1\n 2025-08 0 0 0 0 0 0 0\n 2025-09 0 0 0 0 0 0 0\n 2025-10 0 0 0 0 0 0 0\n 2025-11 0 0 0 0 0 0 0\n 2025-12 0 120 0 0 0 0 1\n 2026-01 0 170 0 0 0 0 1\n 2026-02 0 0 0 0 0 0 0\n 2026-03 0 0 0 0 0 0 0\n 2026-04 0 0 356,000 291,900 0 0 2\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 in 1 txns 356,000.00 USD\n CP-2 out 1 txns 291,900.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "```json\n{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-1\"}\n```", + "reasoning": null, + "call": { + "promptTokens": 1358, + "completionTokens": 22, + "reasoningTokens": null, + "durationMs": 1702, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-1" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-1 Eskildsen & Laurent LLP, client trust account\n type law_firm_trust; jurisdiction United States; first seen 2026-04-18\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-4\n\nHours spent so far: 0.95. You have 7 replies left." + } + }, + { + "reply": "```json\n{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-2\"}\n```", + "reasoning": null, + "call": { + "promptTokens": 1478, + "completionTokens": 22, + "reasoningTokens": null, + "durationMs": 1266, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-2" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-2 Brightwater Securities\n type brokerage; jurisdiction United States; first seen 2026-04-19\n other alerted accounts sharing this counterparty in twelve months: 1\n transactions: T-5\n\nHours spent so far: 1.40. You have 6 replies left." + } + }, + { + "reply": "```json\n{\"action\":\"lookup\",\"what\":\"prior_alerts\"}\n```", + "reasoning": null, + "call": { + "promptTokens": 1587, + "completionTokens": 17, + "reasoningTokens": null, + "durationMs": 509, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "lookup", + "what": "prior_alerts" + }, + "rejection": null, + "observation": "PRIOR ALERTS:\n A-4278 R-DORM-05 fired 2024-12-23 closed after 9 day(s) by Amara Tanaka\n Closed: reactivation explained by a documented source of funds; document reviewed and retained.\n A-4841 R-WIRE-04 fired 2026-02-02 closed after 5 day(s) by Alicia Haddad\n Closed: destination and amount match the remittance declared at onboarding; within stated income.\n\nHours spent so far: 1.85. You have 5 replies left." + } + }, + { + "reply": "```json\n{\"action\":\"lookup\",\"what\":\"documents\"}\n```", + "reasoning": null, + "call": { + "promptTokens": 1762, + "completionTokens": 15, + "reasoningTokens": null, + "durationMs": 553, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "lookup", + "what": "documents" + }, + "rejection": null, + "observation": "DOCUMENTS ON FILE:\n doc.D-3211 [identity] Government-issued identity document, verified at onboarding\n doc.D-aded [executor_letter] Letter from Alicia Haddad, executor, via Eskildsen & Laurent LLP, client trust account: distribution from the estate of Aaron Eskildsen to the named beneficiary.\n\nHours spent so far: 2.00. You have 4 replies left." + } + }, + { + "reply": "```json\n{\"action\":\"close\",\"reason\":\"DOCUMENTED_SOURCE_OF_FUNDS\",\"cites\":[\"doc.D-aded\",\"T-4\",\"CP-1\",\"CP-2\"],\"note\":\"The $356,000 wire-in from CP-1 (Eskildsen & Laurent LLP client trust account) is an estate distribution to a named beneficiary, documented by doc.D-aded (executor letter). The $291,900 wire-out to CP-2 (Brightwater Securities) is consistent with moving estate proceeds into an investment account. Amounts are consistent with an estate distribution and not with the customer's $31,000 stated income, but the source is documented and legitimate. Prior alert A-4278 was similarly closed for a documented source of funds.\"}\n```", + "reasoning": null, + "call": { + "promptTokens": 1885, + "completionTokens": 170, + "reasoningTokens": null, + "durationMs": 6879, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "close", + "reason": "DOCUMENTED_SOURCE_OF_FUNDS", + "cites": [ + "doc.D-aded", + "T-4", + "CP-1", + "CP-2" + ], + "note": "The $356,000 wire-in from CP-1 (Eskildsen & Laurent LLP client trust account) is an estate distribution to a named beneficiary, documented by doc.D-aded (executor letter). The $291,900 wire-out to CP-2 (Brightwater Securities) is consistent with moving estate proceeds into an investment account. Amounts are consistent with an estate distribution and not with the customer's $31,000 stated income, but the source is documented and legitimate. Prior alert A-4278 was similarly closed for a documented source of funds." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/base-off-s3.json b/public/traces/alert-triage/base-off-s3.json new file mode 100644 index 0000000..37bf167 --- /dev/null +++ b/public/traces/alert-triage/base-off-s3.json @@ -0,0 +1,91 @@ +{ + "runId": "base-off-s3", + "seed": 3, + "model": "brain-qwen38-dspark", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 0.55, + "evidence": 0.3333333333333333 + }, + "metrics": { + "hours_spent": 1.0, + "reference_hours": 0.55, + "lookups": 1.0, + "turns_used": 2.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": null + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "benign", + "template": "B4", + "overlay": null, + "typology": null, + "planted": [ + [ + "kyc.expected_monthly_cash" + ], + [ + "kyc.expected_monthly_cash", + "kyc.business_type" + ] + ], + "reference_minutes": 33, + "reference_policy": "fast", + "held_out": false, + "tier": "benign", + "screen": "ALERT A-9453 rule R-STR-01 \u2014 Three or more cash deposits just under the reporting threshold within ten days\n fired 2026-04-23\n triggering transactions:\n T-200 2026-04-14 cash_in 9,770.00 USD branch BR-089\n T-201 2026-04-15 cash_in 9,430.00 USD branch BR-014\n T-206 2026-04-22 cash_in 8,640.00 USD branch BR-089\nKYC FILE C-4718 Lakeside Kitchen LLC\n kyc.account_opened = 2016-10-03\n kyc.business_type = pizzeria\n kyc.country = United States\n kyc.customer_type = business\n kyc.expected_activity = Cash-intensive food service; daily takings deposited most trading days.\n kyc.expected_monthly_cash = 97,000.00 USD\n kyc.kyc_refreshed = 2025-07-13\n kyc.pep = False\n kyc.prior_alerts = 0\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 588,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 52,750 0 0 0 0 16,816 17\n 2025-06 45,240 0 0 0 0 16,317 17\n 2025-07 49,740 0 0 0 0 16,493 17\n 2025-08 40,470 0 0 0 0 16,944 17\n 2025-09 49,720 0 0 0 0 18,522 19\n 2025-10 44,630 0 0 0 0 17,199 16\n 2025-11 49,060 0 0 0 0 18,992 16\n 2025-12 52,570 0 0 0 0 17,992 19\n 2026-01 50,320 0 0 0 0 15,038 17\n 2026-02 51,710 0 0 0 0 17,826 16\n 2026-03 52,240 0 0 0 0 16,934 17\n 2026-04 78,100 0 0 0 0 18,673 18\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 out 48 txns 132,956.60 USD\n CP-2 out 12 txns 74,793.60 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "{\"action\":\"lookup\",\"what\":\"prior_alerts\"}", + "reasoning": null, + "call": { + "promptTokens": 1531, + "completionTokens": 12, + "reasoningTokens": null, + "durationMs": 2064, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "lookup", + "what": "prior_alerts" + }, + "rejection": null, + "observation": "PRIOR ALERTS: none on file.\n\nHours spent so far: 0.95. You have 7 replies left." + } + }, + { + "reply": "{\"action\":\"close\",\"reason\":\"CONSISTENT_WITH_PROFILE\",\"cites\":[\"T-200\",\"T-201\",\"T-206\",\"kyc.expected_activity\",\"kyc.expected_monthly_cash\"],\"note\":\"Lakeside Kitchen LLC is a cash-intensive pizzeria with a documented expectation of ~$97k/month in cash deposits. The three triggering deposits ($9,770, $9,430, $8,640) are each just under the $10,000 reporting threshold but are consistent with the customer's normal pattern of daily takings deposited across multiple trading days. April's total cash-in of $78,100 is within the expected range and the account has no prior alerts, no SARs, and no PEP flags. The rule fired because three sub-threshold deposits occurred within ten days, but this is a natural consequence of a high-volume cash business depositing daily takings in amounts that individually stay below the CTR threshold. No structuring indicators (e.g., sudden spike, new counterparty, unusual timing) are present.\"}", + "reasoning": null, + "call": { + "promptTokens": 1584, + "completionTokens": 233, + "reasoningTokens": null, + "durationMs": 9906, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "close", + "reason": "CONSISTENT_WITH_PROFILE", + "cites": [ + "T-200", + "T-201", + "T-206", + "kyc.expected_activity", + "kyc.expected_monthly_cash" + ], + "note": "Lakeside Kitchen LLC is a cash-intensive pizzeria with a documented expectation of ~$97k/month in cash deposits. The three triggering deposits ($9,770, $9,430, $8,640) are each just under the $10,000 reporting threshold but are consistent with the customer's normal pattern of daily takings deposited across multiple trading days. April's total cash-in of $78,100 is within the expected range and the account has no prior alerts, no SARs, and no PEP flags. The rule fired because three sub-threshold deposits occurred within ten days, but this is a natural consequence of a high-volume cash business depositing daily takings in amounts that individually stay below the CTR threshold. No structuring indicators (e.g., sudden spike, new counterparty, unusual timing) are present." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/base-off-s4.json b/public/traces/alert-triage/base-off-s4.json new file mode 100644 index 0000000..eca2b5f --- /dev/null +++ b/public/traces/alert-triage/base-off-s4.json @@ -0,0 +1,177 @@ +{ + "runId": "base-off-s4", + "seed": 4, + "model": "brain-qwen38-dspark", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 0.22, + "evidence": 0.7058823529411765 + }, + "metrics": { + "hours_spent": 2.5, + "reference_hours": 0.55, + "lookups": 5.0, + "turns_used": 6.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": 1.0 + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "suspicious", + "template": "S1", + "overlay": null, + "typology": "STRUCTURING", + "planted": [ + [ + "T-46", + "T-47", + "T-49", + "T-50", + "T-52", + "T-53", + "T-54", + "T-56" + ] + ], + "reference_minutes": 33, + "reference_policy": "fast", + "held_out": false, + "tier": "visible", + "screen": "ALERT A-8199 rule R-STR-01 \u2014 Three or more cash deposits just under the reporting threshold within ten days\n fired 2026-06-14\n triggering transactions:\n T-49 2026-06-05 cash_in 9,320.00 USD branch BR-027\n T-50 2026-06-06 cash_in 9,820.00 USD branch BR-031\n T-52 2026-06-07 cash_in 9,880.00 USD branch BR-068\n T-53 2026-06-09 cash_in 9,730.00 USD branch BR-027\n T-54 2026-06-10 cash_in 9,680.00 USD branch BR-031\n T-56 2026-06-11 cash_in 9,850.00 USD branch BR-068\nKYC FILE C-5480 Bilal Quintero\n kyc.account_opened = 2017-03-29\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Salaried individual; payroll deposits and routine spending.\n kyc.expected_monthly_cash = 100.00 USD\n kyc.kyc_refreshed = 2026-05-04\n kyc.occupation = delivery driver\n kyc.pep = False\n kyc.prior_alerts = 3\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 45,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-07 0 220 0 0 3,410 0 3\n 2025-08 0 480 0 0 3,382 0 3\n 2025-09 0 570 0 0 3,444 0 4\n 2025-10 0 570 0 0 3,454 0 5\n 2025-11 0 660 0 0 3,460 0 4\n 2025-12 0 550 0 0 3,403 0 3\n 2026-01 0 680 0 0 3,351 0 5\n 2026-02 0 490 0 0 3,445 0 4\n 2026-03 0 540 0 0 3,433 0 4\n 2026-04 0 430 0 0 3,387 0 4\n 2026-05 0 660 0 0 3,411 0 5\n 2026-06 77,210 420 0 66,400 3,348 0 13\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-2 out 1 txns 66,400.00 USD\n CP-1 in 12 txns 40,928.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "{\"action\":\"lookup\",\"what\":\"history\",\"month\":\"2026-06\"}", + "reasoning": null, + "call": { + "promptTokens": 1588, + "completionTokens": 20, + "reasoningTokens": null, + "durationMs": 1398, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "lookup", + "what": "history", + "month": "2026-06" + }, + "rejection": null, + "observation": "HISTORY 2026-06: 13 transactions\n T-45 2026-06-02 ach_in 3,348.00 USD CP-1\n T-46 2026-06-02 cash_in 9,400.00 USD branch BR-031\n T-47 2026-06-03 cash_in 9,530.00 USD branch BR-068\n T-48 2026-06-04 cash_out 110.00 USD branch BR-027\n T-49 2026-06-05 cash_in 9,320.00 USD branch BR-027\n T-50 2026-06-06 cash_in 9,820.00 USD branch BR-031\n T-51 2026-06-07 cash_out 150.00 USD branch BR-027\n T-52 2026-06-07 cash_in 9,880.00 USD branch BR-068\n T-53 2026-06-09 cash_in 9,730.00 USD branch BR-027\n T-54 2026-06-10 cash_in 9,680.00 USD branch BR-031\n T-55 2026-06-11 cash_out 160.00 USD branch BR-068\n T-56 2026-06-11 cash_in 9,850.00 USD branch BR-068\n T-57 2026-06-13 wire_out 66,400.00 USD CP-2\n\nHours spent so far: 0.95. You have 7 replies left." + } + }, + { + "reply": "{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-2\"}", + "reasoning": null, + "call": { + "promptTokens": 2151, + "completionTokens": 17, + "reasoningTokens": null, + "durationMs": 1282, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-2" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-2 Redfern Holdings Co.\n type business; jurisdiction United States; first seen 2026-06-13\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-57\n\nHours spent so far: 1.40. You have 6 replies left." + } + }, + { + "reply": "{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-1\"}", + "reasoning": null, + "call": { + "promptTokens": 2258, + "completionTokens": 17, + "reasoningTokens": null, + "durationMs": 677, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-1" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-1 Stonebridge Diner Group\n type business; jurisdiction United States; first seen 2025-07-03\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-1, T-5, T-7, T-11, T-16, T-21, T-23, T-28, T-32, T-36, T-40, T-45\n\nHours spent so far: 1.85. You have 5 replies left." + } + }, + { + "reply": "{\"action\":\"lookup\",\"what\":\"prior_alerts\"}", + "reasoning": null, + "call": { + "promptTokens": 2417, + "completionTokens": 12, + "reasoningTokens": null, + "durationMs": 563, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "lookup", + "what": "prior_alerts" + }, + "rejection": null, + "observation": "PRIOR ALERTS:\n A-1684 R-STR-01 fired 2025-01-17 closed after 5 day(s) by Andre Abara\n Closed: deposits are documented daily takings; amounts vary with trading days; within kyc.expected_monthly_cash.\n A-3624 R-STR-01 fired 2025-03-06 closed after 8 day(s) by Bilal Laurent\n Closed: deposits are documented daily takings; amounts vary with trading days; within kyc.expected_monthly_cash.\n A-2017 R-CASH-03 fired 2025-05-09 closed after 7 day(s) by Sofia Grimaldi\n Closed: seasonal peak consistent with kyc.expected_activity; volume within stated expectation.\n\nHours spent so far: 2.30. You have 4 replies left." + } + }, + { + "reply": "{\"action\":\"lookup\",\"what\":\"documents\"}", + "reasoning": null, + "call": { + "promptTokens": 2655, + "completionTokens": 10, + "reasoningTokens": null, + "durationMs": 640, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "lookup", + "what": "documents" + }, + "rejection": null, + "observation": "DOCUMENTS ON FILE:\n doc.D-c608 [identity] Government-issued identity document, verified at onboarding\n\nHours spent so far: 2.45. You have 3 replies left." + } + }, + { + "reply": "{\"action\":\"escalate\",\"typology\":\"STRUCTURING\",\"cites\":[\"T-49\",\"T-50\",\"T-52\",\"T-53\",\"T-54\",\"T-56\",\"kyc.expected_monthly_cash\",\"CP-2\",\"T-57\"],\"narrative\":\"Six cash deposits of 9,320\u20139,880 USD each over 06-05 to 06-11, all just under the 10,000 reporting threshold, at three rotating branches (BR-027, BR-031, BR-068). This is a stark departure from the customer's profile: a salaried delivery driver with kyc.expected_monthly_cash of 100 USD and a 12-month history of zero cash-in until June 2026, when 77,210 USD in cash appeared. On 06-13 the customer wired the full 66,400 USD to CP-2 Redfern Holdings Co., a business first seen on that same date \u2014 a rapid movement of the aggregated cash. The pattern of just-under-threshold deposits across multiple branches followed by a single outflow to a newly-seen counterparty is classic structuring/layering, not documented daily takings. Prior R-STR-01 closures (A-1684, A-3624) referenced amounts within expected monthly cash, which does not apply here given the 770x jump and the immediate wire-out. Escalating for investigation.\"}", + "reasoning": null, + "call": { + "promptTokens": 2725, + "completionTokens": 338, + "reasoningTokens": null, + "durationMs": 15080, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "escalate", + "typology": "STRUCTURING", + "cites": [ + "T-49", + "T-50", + "T-52", + "T-53", + "T-54", + "T-56", + "kyc.expected_monthly_cash", + "CP-2", + "T-57" + ], + "narrative": "Six cash deposits of 9,320\u20139,880 USD each over 06-05 to 06-11, all just under the 10,000 reporting threshold, at three rotating branches (BR-027, BR-031, BR-068). This is a stark departure from the customer's profile: a salaried delivery driver with kyc.expected_monthly_cash of 100 USD and a 12-month history of zero cash-in until June 2026, when 77,210 USD in cash appeared. On 06-13 the customer wired the full 66,400 USD to CP-2 Redfern Holdings Co., a business first seen on that same date \u2014 a rapid movement of the aggregated cash. The pattern of just-under-threshold deposits across multiple branches followed by a single outflow to a newly-seen counterparty is classic structuring/layering, not documented daily takings. Prior R-STR-01 closures (A-1684, A-3624) referenced amounts within expected monthly cash, which does not apply here given the 770x jump and the immediate wire-out. Escalating for investigation." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/base-off-s5.json b/public/traces/alert-triage/base-off-s5.json new file mode 100644 index 0000000..15bb274 --- /dev/null +++ b/public/traces/alert-triage/base-off-s5.json @@ -0,0 +1,150 @@ +{ + "runId": "base-off-s5", + "seed": 5, + "model": "brain-qwen38-dspark", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 0.34146341463414637, + "evidence": 0.3333333333333333 + }, + "metrics": { + "hours_spent": 2.05, + "reference_hours": 0.7, + "lookups": 4.0, + "turns_used": 5.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": null + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "benign", + "template": "B3", + "overlay": null, + "typology": null, + "planted": [ + [ + "doc.D-8578" + ], + [ + "doc.D-8578", + "kyc.occupation" + ] + ], + "reference_minutes": 42, + "reference_policy": "fast", + "held_out": false, + "tier": "benign", + "screen": "ALERT A-6081 rule R-VEL-02 \u2014 Funds received and moved out again within 48 hours\n fired 2026-03-22\n triggering transactions:\n T-52 2026-03-18 wire_in 252,000.00 USD CP-2\n T-54 2026-03-19 wire_out 219,200.00 USD CP-3\nKYC FILE C-7077 Andre Nakamura\n kyc.account_opened = 2022-01-15\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Salaried individual; payroll deposits and routine spending.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2025-04-22\n kyc.occupation = teacher\n kyc.pep = False\n kyc.prior_alerts = 3\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 106,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-04 0 590 0 0 8,866 0 4\n 2025-05 0 820 0 0 8,921 0 4\n 2025-06 0 920 0 0 8,952 0 5\n 2025-07 0 680 0 0 8,870 0 3\n 2025-08 0 1,140 0 0 8,906 0 5\n 2025-09 0 390 0 0 8,870 0 4\n 2025-10 0 850 0 0 8,922 0 4\n 2025-11 0 1,020 0 0 8,916 0 5\n 2025-12 0 950 0 0 8,901 0 5\n 2026-01 0 880 0 0 8,854 0 4\n 2026-02 0 1,170 0 0 8,916 0 5\n 2026-03 0 720 252,000 219,200 8,956 0 6\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-2 in 1 txns 252,000.00 USD\n CP-3 out 1 txns 219,200.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-2\"}", + "reasoning": null, + "call": { + "promptTokens": 1433, + "completionTokens": 17, + "reasoningTokens": null, + "durationMs": 1207, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-2" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-2 Iwata Title & Escrow\n type title_company; jurisdiction United States; first seen 2026-03-18\n other alerted accounts sharing this counterparty in twelve months: 1\n transactions: T-52\n\nHours spent so far: 0.95. You have 7 replies left." + } + }, + { + "reply": "{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-3\"}", + "reasoning": null, + "call": { + "promptTokens": 1543, + "completionTokens": 17, + "reasoningTokens": null, + "durationMs": 552, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-3" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-3 Harbor Securities\n type brokerage; jurisdiction United States; first seen 2026-03-19\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-54\n\nHours spent so far: 1.40. You have 6 replies left." + } + }, + { + "reply": "{\"action\":\"lookup\",\"what\":\"prior_alerts\"}", + "reasoning": null, + "call": { + "promptTokens": 1647, + "completionTokens": 12, + "reasoningTokens": null, + "durationMs": 540, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "lookup", + "what": "prior_alerts" + }, + "rejection": null, + "observation": "PRIOR ALERTS:\n A-4824 R-VEL-02 fired 2025-03-19 closed after 4 day(s) by Jamal Whitlock\n Closed: pass-through matches the settlement terms in the agreement on file; counterparties verified.\n A-1410 R-WIRE-04 fired 2025-04-24 closed after 2 day(s) by Lars Farrow\n Closed: destination and amount match the remittance declared at onboarding; within stated income.\n A-2317 R-CASH-03 fired 2025-09-02 closed after 6 day(s) by Chidi Zapata\n Closed: seasonal peak consistent with kyc.expected_activity; volume within stated expectation.\n\nHours spent so far: 1.85. You have 5 replies left." + } + }, + { + "reply": "{\"action\":\"lookup\",\"what\":\"documents\"}", + "reasoning": null, + "call": { + "promptTokens": 1875, + "completionTokens": 10, + "reasoningTokens": null, + "durationMs": 636, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "lookup", + "what": "documents" + }, + "rejection": null, + "observation": "DOCUMENTS ON FILE:\n doc.D-5045 [identity] Government-issued identity document, verified at onboarding\n doc.D-8578 [closing_statement] Closing statement \u2014 sale of 266 Birch Avenue; net proceeds to seller disbursed by Iwata Title & Escrow.\n\nHours spent so far: 2.00. You have 4 replies left." + } + }, + { + "reply": "{\"action\":\"close\",\"reason\":\"DOCUMENTED_SOURCE_OF_FUNDS\",\"cites\":[\"doc.D-8578\",\"CP-2\",\"CP-3\",\"T-52\",\"T-54\"],\"note\":\"Wire-in from Iwata Title & Escrow (CP-2) matches the closing statement on file (doc.D-8578) for the sale of 266 Birch Avenue; next-day wire-out to Harbor Securities (CP-3, brokerage) is consistent with investment of real-estate proceeds. Source of funds documented; amounts within the property sale context.\"}", + "reasoning": null, + "call": { + "promptTokens": 1988, + "completionTokens": 124, + "reasoningTokens": null, + "durationMs": 5392, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "close", + "reason": "DOCUMENTED_SOURCE_OF_FUNDS", + "cites": [ + "doc.D-8578", + "CP-2", + "CP-3", + "T-52", + "T-54" + ], + "note": "Wire-in from Iwata Title & Escrow (CP-2) matches the closing statement on file (doc.D-8578) for the sale of 266 Birch Avenue; next-day wire-out to Harbor Securities (CP-3, brokerage) is consistent with investment of real-estate proceeds. Source of funds documented; amounts within the property sale context." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/base-off-s6.json b/public/traces/alert-triage/base-off-s6.json new file mode 100644 index 0000000..069417f --- /dev/null +++ b/public/traces/alert-triage/base-off-s6.json @@ -0,0 +1,90 @@ +{ + "runId": "base-off-s6", + "seed": 6, + "model": "brain-qwen38-dspark", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 0.55, + "evidence": 0.6666666666666666 + }, + "metrics": { + "hours_spent": 1.0, + "reference_hours": 0.55, + "lookups": 1.0, + "turns_used": 2.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": null + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "benign", + "template": "B1", + "overlay": null, + "typology": null, + "planted": [ + [ + "kyc.expected_activity", + "kyc.expected_monthly_cash" + ], + [ + "kyc.expected_monthly_cash" + ] + ], + "reference_minutes": 33, + "reference_policy": "fast", + "held_out": false, + "tier": "benign", + "screen": "ALERT A-7014 rule R-CASH-03 \u2014 Monthly cash deposits materially above the trailing six-month average\n fired 2026-05-15\n triggering transactions:\n T-83 2026-05-02 cash_in 2,140.00 USD branch BR-102\n T-84 2026-05-03 cash_in 2,930.00 USD branch BR-102\n T-85 2026-05-04 cash_in 2,610.00 USD branch BR-031\n T-86 2026-05-05 cash_in 2,290.00 USD branch BR-031\n T-87 2026-05-06 cash_in 2,930.00 USD branch BR-031\n T-88 2026-05-08 cash_in 2,190.00 USD branch BR-031\n T-89 2026-05-09 cash_in 2,270.00 USD branch BR-031\n T-90 2026-05-10 cash_in 1,970.00 USD branch BR-102\n T-91 2026-05-11 cash_in 2,540.00 USD branch BR-031\n T-92 2026-05-12 cash_in 2,540.00 USD branch BR-031\n T-93 2026-05-13 cash_in 1,970.00 USD branch BR-102\n T-94 2026-05-14 cash_in 2,930.00 USD branch BR-031\nKYC FILE C-3047 Lakeside Landscaping Inc.\n kyc.account_opened = 2013-08-05\n kyc.business_type = landscaping services\n kyc.country = United States\n kyc.customer_type = business\n kyc.expected_activity = Seasonal business; cash receipts peak April to September.\n kyc.expected_monthly_cash = 38,000.00 USD\n kyc.kyc_refreshed = 2025-04-30\n kyc.pep = False\n kyc.prior_alerts = 2\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 266,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-06 28,130 0 0 2,900 0 7,716 10\n 2025-07 24,190 0 0 0 0 8,652 11\n 2025-08 27,600 0 0 0 0 7,538 10\n 2025-09 22,400 0 0 6,300 0 6,691 13\n 2025-10 5,480 0 0 0 0 1,686 4\n 2025-11 7,530 0 0 0 0 1,673 4\n 2025-12 6,520 0 0 3,900 0 2,119 5\n 2026-01 8,260 0 0 0 0 1,811 5\n 2026-02 5,740 0 0 0 0 1,989 4\n 2026-03 6,100 0 0 3,300 0 2,094 6\n 2026-04 24,540 0 0 0 0 5,537 10\n 2026-05 29,310 0 0 0 0 9,777 13\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 out 12 txns 57,289.16 USD\n CP-2 out 4 txns 16,400.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "Looking at this alert, the key question is whether the May cash deposits are genuinely out of line with the customer's profile.\n\nLet me note the facts I've already been shown:\n- Business: landscaping, seasonal cash peaks April\u2013September (kyc.expected_activity)\n- Expected monthly cash: 38,000 USD (kyc.expected_monthly_cash)\n- May 2026 cash in: 29,310 \u2014 below the stated expectation\n- The rule R-CASH-03 fired because May is \"materially above the trailing six-month average\" \u2014 but the trailing six months (Nov\u2013Apr) include the dead winter season, so a low base mechanically trips the rule at the start of season.\n- April 2026 already showed the seasonal uptick (24,540).\n- No wires in, all outflows to CP-1/CP-2 are ordinary business payables; no PEP, no prior SARs.\n\nThis looks like a rule artefact (seasonal peak vs. low winter base), not structuring. The individual deposits are all well under the 10,000 reporting threshold and there's no rapid movement or funneling pattern.\n\nLet me confirm there's no prior alert that already covered this seasonal pattern, which would strengthen a clean close.\n\n{\"action\":\"lookup\",\"what\":\"prior_alerts\"}", + "reasoning": null, + "call": { + "promptTokens": 1867, + "completionTokens": 295, + "reasoningTokens": null, + "durationMs": 13229, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "lookup", + "what": "prior_alerts" + }, + "rejection": null, + "observation": "PRIOR ALERTS:\n A-4237 R-CASH-03 fired 2024-08-10 closed after 7 day(s) by Bilal Rasmussen\n Closed: seasonal peak consistent with kyc.expected_activity; volume within stated expectation.\n A-2077 R-WIRE-04 fired 2025-04-23 closed after 3 day(s) by Dana Pereira\n Closed: destination and amount match the remittance declared at onboarding; within stated income.\n\nHours spent so far: 0.95. You have 7 replies left." + } + }, + { + "reply": "The prior alert A-4237 is decisive. It is the same rule (R-CASH-03), fired in the same seasonal window (August), and was closed by a prior analyst as \"seasonal peak consistent with kyc.expected_activity; volume within stated expectation.\" This exact pattern has already been reviewed and dispositioned.\n\nThe current May volume (29,310) is below the stated monthly expectation (38,000), the trigger is purely mechanical (seasonal peak vs. low winter base), there are no PEP flags, no prior SARs, and the outflows are ordinary business payables. This is a rule artefact that has been previously reviewed.\n\n{\"action\":\"close\",\"reason\":\"PREVIOUSLY_REVIEWED\",\"cites\":[\"A-4237\",\"kyc.expected_activity\",\"kyc.expected_monthly_cash\",\"T-84\"],\"note\":\"Same rule R-CASH-03, same seasonal window; prior alert A-4237 already closed as seasonal peak consistent with profile. May cash (29,310) is below stated monthly expectation (38,000). Trigger is mechanical (low winter base). No PEP, no prior SARs, outflows are ordinary business payables.\"}", + "reasoning": null, + "call": { + "promptTokens": 2318, + "completionTokens": 269, + "reasoningTokens": null, + "durationMs": 20616, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "close", + "reason": "PREVIOUSLY_REVIEWED", + "cites": [ + "A-4237", + "kyc.expected_activity", + "kyc.expected_monthly_cash", + "T-84" + ], + "note": "Same rule R-CASH-03, same seasonal window; prior alert A-4237 already closed as seasonal peak consistent with profile. May cash (29,310) is below stated monthly expectation (38,000). Trigger is mechanical (low winter base). No PEP, no prior SARs, outflows are ordinary business payables." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/base-off-s7.json b/public/traces/alert-triage/base-off-s7.json new file mode 100644 index 0000000..4acdd87 --- /dev/null +++ b/public/traces/alert-triage/base-off-s7.json @@ -0,0 +1,144 @@ +{ + "runId": "base-off-s7", + "seed": 7, + "model": "brain-qwen38-dspark", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 0.34146341463414637, + "evidence": 0.5 + }, + "metrics": { + "hours_spent": 2.05, + "reference_hours": 0.7, + "lookups": 4.0, + "turns_used": 5.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": null + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "benign", + "template": "B6", + "overlay": null, + "typology": null, + "planted": [ + [ + "doc.D-b931" + ] + ], + "reference_minutes": 42, + "reference_policy": "fast", + "held_out": false, + "tier": "benign", + "screen": "ALERT A-5644 rule R-DORM-05 \u2014 Large credit to an account with no material activity for nine months\n fired 2026-06-14\n triggering transactions:\n T-4 2026-06-07 wire_in 249,000.00 USD CP-1\n T-5 2026-06-10 wire_out 134,400.00 USD CP-2\nKYC FILE C-1843 Leila Bergstrom\n kyc.account_opened = 2017-03-20\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Retired; low activity expected.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2026-01-13\n kyc.occupation = retired\n kyc.pep = False\n kyc.prior_alerts = 0\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 33,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-07 0 0 0 0 0 0 0\n 2025-08 0 0 0 0 0 0 0\n 2025-09 0 60 0 0 0 0 1\n 2025-10 0 0 0 0 0 0 0\n 2025-11 0 150 0 0 0 0 1\n 2025-12 0 0 0 0 0 0 0\n 2026-01 0 0 0 0 0 0 0\n 2026-02 0 0 0 0 0 0 0\n 2026-03 0 50 0 0 0 0 1\n 2026-04 0 0 0 0 0 0 0\n 2026-05 0 0 0 0 0 0 0\n 2026-06 0 0 249,000 134,400 0 0 2\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 in 1 txns 249,000.00 USD\n CP-2 out 1 txns 134,400.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-1\"}", + "reasoning": null, + "call": { + "promptTokens": 1354, + "completionTokens": 17, + "reasoningTokens": null, + "durationMs": 1503, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-1" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-1 Xiang & Farrow LLP, client trust account\n type law_firm_trust; jurisdiction United States; first seen 2026-06-07\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-4\n\nHours spent so far: 0.95. You have 7 replies left." + } + }, + { + "reply": "{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-2\"}", + "reasoning": null, + "call": { + "promptTokens": 1469, + "completionTokens": 17, + "reasoningTokens": null, + "durationMs": 685, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-2" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-2 Oakhill Securities\n type brokerage; jurisdiction United States; first seen 2026-06-10\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-5\n\nHours spent so far: 1.40. You have 6 replies left." + } + }, + { + "reply": "{\"action\":\"lookup\",\"what\":\"prior_alerts\"}", + "reasoning": null, + "call": { + "promptTokens": 1573, + "completionTokens": 12, + "reasoningTokens": null, + "durationMs": 5002, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "lookup", + "what": "prior_alerts" + }, + "rejection": null, + "observation": "PRIOR ALERTS: none on file.\n\nHours spent so far: 1.85. You have 5 replies left." + } + }, + { + "reply": "{\"action\":\"lookup\",\"what\":\"documents\"}", + "reasoning": null, + "call": { + "promptTokens": 1626, + "completionTokens": 10, + "reasoningTokens": null, + "durationMs": 737, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "lookup", + "what": "documents" + }, + "rejection": null, + "observation": "DOCUMENTS ON FILE:\n doc.D-1dc2 [identity] Government-issued identity document, verified at onboarding\n doc.D-b931 [executor_letter] Letter from Hana Sandoval, executor, via Xiang & Farrow LLP, client trust account: distribution from the estate of Elena Okafor to the named beneficiary.\n\nHours spent so far: 2.00. You have 4 replies left." + } + }, + { + "reply": "{\"action\":\"close\",\"reason\":\"DOCUMENTED_SOURCE_OF_FUNDS\",\"cites\":[\"doc.D-b931\",\"T-4\",\"CP-1\"],\"note\":\"The $249,000 wire-in from Xiang & Farrow LLP client trust account (CP-1) is documented as a distribution from the estate of Elena Okafor to the named beneficiary, Leila Bergstrom, per executor letter doc.D-b931. The subsequent wire-out of $134,400 to Oakhill Securities (CP-2, a US brokerage) is consistent with a retired individual moving funds into investment accounts. No PEP status, no prior alerts or SARs, and the customer profile (retired, low activity) aligns with a one-time estate distribution. Both counterparties are US-based, legitimate financial/legal entities with no shared alert history. Closing as documented source of funds.\"}", + "reasoning": null, + "call": { + "promptTokens": 1746, + "completionTokens": 189, + "reasoningTokens": null, + "durationMs": 7295, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "close", + "reason": "DOCUMENTED_SOURCE_OF_FUNDS", + "cites": [ + "doc.D-b931", + "T-4", + "CP-1" + ], + "note": "The $249,000 wire-in from Xiang & Farrow LLP client trust account (CP-1) is documented as a distribution from the estate of Elena Okafor to the named beneficiary, Leila Bergstrom, per executor letter doc.D-b931. The subsequent wire-out of $134,400 to Oakhill Securities (CP-2, a US brokerage) is consistent with a retired individual moving funds into investment accounts. No PEP status, no prior alerts or SARs, and the customer profile (retired, low activity) aligns with a one-time estate distribution. Both counterparties are US-based, legitimate financial/legal entities with no shared alert history. Closing as documented source of funds." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/base-off-s9.json b/public/traces/alert-triage/base-off-s9.json new file mode 100644 index 0000000..392cc69 --- /dev/null +++ b/public/traces/alert-triage/base-off-s9.json @@ -0,0 +1,171 @@ +{ + "runId": "base-off-s9", + "seed": 9, + "model": "brain-qwen38-dspark", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 0.0, + "hours": 0.0, + "evidence": 0.0 + }, + "metrics": { + "hours_spent": 2.5, + "reference_hours": 2.05, + "lookups": 5.0, + "turns_used": 6.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 1.0, + "typology_match": null + }, + "truncated": false, + "outcome": "failed", + "info": { + "label": "suspicious", + "template": "B3", + "overlay": "funnel", + "typology": "FUNNEL_ACCOUNT", + "planted": [ + [ + "CP-3", + "T-52" + ], + [ + "T-52" + ] + ], + "reference_minutes": 123, + "reference_policy": "targeted", + "held_out": false, + "tier": "hidden", + "screen": "ALERT A-9537 rule R-VEL-02 \u2014 Funds received and moved out again within 48 hours\n fired 2026-04-19\n triggering transactions:\n T-51 2026-04-16 wire_in 462,000.00 USD CP-2\n T-52 2026-04-17 wire_out 448,100.00 USD CP-3\nKYC FILE C-9673 Samir Tanaka\n kyc.account_opened = 2019-12-21\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Salaried individual; payroll deposits and routine spending.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2025-04-04\n kyc.occupation = electrician\n kyc.pep = False\n kyc.prior_alerts = 4\n kyc.prior_sars = 1\n kyc.remittance_declared = none declared\n kyc.stated_income = 84,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 0 1,090 0 0 6,958 0 5\n 2025-06 0 630 0 0 7,009 0 5\n 2025-07 0 780 0 0 6,957 0 4\n 2025-08 0 470 0 0 6,989 0 3\n 2025-09 0 1,040 0 0 6,959 0 5\n 2025-10 0 740 0 0 6,996 0 4\n 2025-11 0 860 0 0 6,947 0 5\n 2025-12 0 900 0 0 6,940 0 5\n 2026-01 0 650 0 0 7,025 0 3\n 2026-02 0 460 0 0 7,026 0 3\n 2026-03 0 620 0 0 6,962 0 3\n 2026-04 0 960 462,000 448,100 6,944 0 7\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-2 in 1 txns 462,000.00 USD\n CP-3 out 1 txns 448,100.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "I need to investigate this alert. The customer is a salaried electrician earning $84,000/year (~$6,992/month) with a very consistent pattern of ACH deposits around $7,000/month. Suddenly in April 2026, a $462,000 wire came in and $448,100 went out within a day. This is dramatically inconsistent with their profile. Let me gather more information.\n\n{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-2\"}", + "reasoning": null, + "call": { + "promptTokens": 1432, + "completionTokens": 120, + "reasoningTokens": null, + "durationMs": 4843, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-2" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-2 Okafor Title & Escrow\n type title_company; jurisdiction United States; first seen 2026-04-16\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-51\n\nHours spent so far: 0.95. You have 7 replies left." + } + }, + { + "reply": "{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-3\"}", + "reasoning": null, + "call": { + "promptTokens": 1645, + "completionTokens": 17, + "reasoningTokens": null, + "durationMs": 504, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-3" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-3 Summit Securities\n type brokerage; jurisdiction United States; first seen 2026-04-17\n other alerted accounts sharing this counterparty in twelve months: 8\n transactions: T-52\n\nHours spent so far: 1.40. You have 6 replies left." + } + }, + { + "reply": "{\"action\":\"lookup\",\"what\":\"prior_alerts\"}", + "reasoning": null, + "call": { + "promptTokens": 1749, + "completionTokens": 12, + "reasoningTokens": null, + "durationMs": 514, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "lookup", + "what": "prior_alerts" + }, + "rejection": null, + "observation": "PRIOR ALERTS:\n A-2906 R-VEL-02 fired 2025-01-09 closed after 4 day(s) by Hana Xiang\n Closed: pass-through matches the settlement terms in the agreement on file; counterparties verified.\n A-2547 R-DORM-05 fired 2025-07-01 escalated \u2014 report filed after 2 day(s) by Jamal Ferreira\n Escalated: pattern not explained by the file at the time.\n A-2855 R-DORM-05 fired 2025-07-17 closed after 8 day(s) by Carlos Zapata\n Closed: reactivation explained by a documented source of funds; document reviewed and retained.\n A-2904 R-VEL-02 fired 2025-12-05 closed after 5 day(s) by Dana Grimaldi\n Closed: pass-through matches the settlement terms in the agreement on file; counterparties verified.\n\nHours spent so far: 1.85. You have 5 replies left." + } + }, + { + "reply": "{\"action\":\"lookup\",\"what\":\"documents\"}", + "reasoning": null, + "call": { + "promptTokens": 2037, + "completionTokens": 10, + "reasoningTokens": null, + "durationMs": 589, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "lookup", + "what": "documents" + }, + "rejection": null, + "observation": "DOCUMENTS ON FILE:\n doc.D-8b75 [identity] Government-issued identity document, verified at onboarding\n doc.D-838c [closing_statement] Closing statement \u2014 sale of 82 Alder Street; net proceeds to seller disbursed by Okafor Title & Escrow.\n\nHours spent so far: 2.00. You have 4 replies left." + } + }, + { + "reply": "{\"action\":\"lookup\",\"what\":\"history\",\"month\":\"2026-04\"}", + "reasoning": null, + "call": { + "promptTokens": 2150, + "completionTokens": 20, + "reasoningTokens": null, + "durationMs": 634, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "lookup", + "what": "history", + "month": "2026-04" + }, + "rejection": null, + "observation": "HISTORY 2026-04: 7 transactions\n T-46 2026-04-04 ach_in 6,944.00 USD CP-1\n T-47 2026-04-04 cash_out 220.00 USD branch BR-068\n T-48 2026-04-06 cash_out 280.00 USD branch BR-068\n T-49 2026-04-10 cash_out 340.00 USD branch BR-068\n T-50 2026-04-12 cash_out 120.00 USD branch BR-068\n T-51 2026-04-16 wire_in 462,000.00 USD CP-2\n T-52 2026-04-17 wire_out 448,100.00 USD CP-3\n\nHours spent so far: 2.45. You have 3 replies left." + } + }, + { + "reply": "Now I have a complete picture. Let me assess:\n\n**The case:**\n- Customer is a salaried electrician earning ~$7,000/month, with a 12-month pattern of consistent ACH payroll deposits and small cash withdrawals.\n- A $462,000 wire arrived from Okafor Title & Escrow (CP-2), a US title company, first seen 2026-04-16.\n- $448,100 went out the next day to Summit Securities (CP-3), a US brokerage, first seen 2026-04-17.\n- A closing statement on file (doc.D-838c) documents the sale of 82 Alder Street, with net proceeds disbursed by Okafor Title & Escrow.\n- Prior alerts A-2906 and A-2904 (both R-VEL-02, same rule) were closed as pass-throughs matching settlement terms with verified counterparties.\n- No other alerted accounts share CP-2; CP-3 (brokerage) is shared with 8 other accounts, which is typical for a major brokerage.\n\n**Assessment:** This is a documented real estate sale. The customer sold a property, received the net proceeds via a title company (standard for US real estate closings), and wired the funds to a brokerage account (likely to invest the proceeds). The source of funds is documented on file. The counterparty (title company) is a legitimate US entity with no other alerted accounts sharing it. The outflow to a brokerage is a routine investment activity.\n\nThis is consistent with a documented source of funds.\n\n{\"action\":\"close\",\"reason\":\"DOCUMENTED_SOURCE_OF_FUNDS\",\"cites\":[\"doc.D-838c\",\"T-51\",\"CP-2\",\"CP-3\",\"kyc.occupation\",\"kyc.stated_income\"],\"note\":\"The $462,000 wire-in from Okafor Title & Escrow (CP-2) is the net proceeds from the sale of 82 Alder Street, as documented in the closing statement on file (doc.D-838c). The $448,100 wire-out to Summit Securities (CP-3), a US brokerage, is consistent with investing real estate sale proceeds. The customer's profile as a salaried electrician ($84,000/yr) does not account for this amount, but the documented property sale does. CP-2 is a title company with no other alerted accounts sharing it; CP-3 is a major brokerage shared by 8 accounts, typical for a large financial institution. Prior R-VEL-02 alerts on this customer (A-2906, A-2904) were similarly closed as settlement pass-throughs. No structuring, funnel, mule, or rapid-movement typology is indicated.\"}", + "reasoning": null, + "call": { + "promptTokens": 2476, + "completionTokens": 626, + "reasoningTokens": null, + "durationMs": 38579, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "close", + "reason": "DOCUMENTED_SOURCE_OF_FUNDS", + "cites": [ + "doc.D-838c", + "T-51", + "CP-2", + "CP-3", + "kyc.occupation", + "kyc.stated_income" + ], + "note": "The $462,000 wire-in from Okafor Title & Escrow (CP-2) is the net proceeds from the sale of 82 Alder Street, as documented in the closing statement on file (doc.D-838c). The $448,100 wire-out to Summit Securities (CP-3), a US brokerage, is consistent with investing real estate sale proceeds. The customer's profile as a salaried electrician ($84,000/yr) does not account for this amount, but the documented property sale does. CP-2 is a title company with no other alerted accounts sharing it; CP-3 is a major brokerage shared by 8 accounts, typical for a large financial institution. Prior R-VEL-02 alerts on this customer (A-2906, A-2904) were similarly closed as settlement pass-throughs. No structuring, funnel, mule, or rapid-movement typology is indicated." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/base-on-s1.json b/public/traces/alert-triage/base-on-s1.json new file mode 100644 index 0000000..736d3d8 --- /dev/null +++ b/public/traces/alert-triage/base-on-s1.json @@ -0,0 +1,107 @@ +{ + "runId": "base-on-s1", + "seed": 1, + "model": "brain-qwen38-dspark", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 0.5238095238095238, + "evidence": 0.0 + }, + "metrics": { + "hours_spent": 1.05, + "reference_hours": 0.55, + "lookups": 1.0, + "turns_used": 3.0, + "rejected_replies": 1.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": null + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "benign", + "template": "B4", + "overlay": null, + "typology": null, + "planted": [ + [ + "kyc.expected_monthly_cash" + ], + [ + "kyc.expected_monthly_cash", + "kyc.business_type" + ] + ], + "reference_minutes": 33, + "reference_policy": "fast", + "held_out": false, + "tier": "benign", + "screen": "ALERT A-8971 rule R-STR-01 \u2014 Three or more cash deposits just under the reporting threshold within ten days\n fired 2026-04-26\n triggering transactions:\n T-213 2026-04-16 cash_in 9,440.00 USD branch BR-052\n T-217 2026-04-21 cash_in 8,950.00 USD branch BR-052\n T-219 2026-04-22 cash_in 8,880.00 USD branch BR-052\nKYC FILE C-9647 Oakhill Kitchen Group\n kyc.account_opened = 2019-07-11\n kyc.business_type = pizzeria\n kyc.country = United States\n kyc.customer_type = business\n kyc.expected_activity = Cash-intensive food service; daily takings deposited most trading days.\n kyc.expected_monthly_cash = 86,000.00 USD\n kyc.kyc_refreshed = 2026-03-13\n kyc.pep = False\n kyc.prior_alerts = 4\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 492,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 42,400 0 0 0 0 13,935 18\n 2025-06 38,160 0 0 0 0 12,582 19\n 2025-07 44,260 0 0 0 0 15,928 18\n 2025-08 41,680 0 0 0 0 13,940 18\n 2025-09 44,950 0 0 0 0 15,498 17\n 2025-10 35,860 0 0 0 0 13,284 19\n 2025-11 42,370 0 0 0 0 12,792 19\n 2025-12 39,280 0 0 0 0 13,776 19\n 2026-01 39,860 0 0 0 0 13,940 19\n 2026-02 39,280 0 0 0 0 12,718 18\n 2026-03 36,510 0 0 0 0 11,693 19\n 2026-04 71,720 0 0 0 0 14,218 18\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 out 48 txns 103,976.00 USD\n CP-2 out 12 txns 60,331.50 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "\n\n{\"action\":\"lookup\",\"what\":\"prior_alerts\"}", + "reasoning": "Let me analyze this alert carefully.\n\n**Alert A-8971**: Rule R-STR-01 \u2014 Three or more cash deposits just under the reporting threshold within ten days.\n- T-213: 2026-04-16, cash_in, 9,440.00 USD\n- T-217: 2026-04-21, cash_in, 8,950.00 USD\n- T-219: 2026-04-22, cash_in, 8,880.00 USD\n\nAll three are just under the 10,000 USD reporting threshold. They're within ten days (April 16 to April 22 = 6 days). This is the classic structuring pattern.\n\n**KYC Profile**: Oakhill Kitchen Group, a pizzeria, cash-intensive food service, expected monthly cash 86,000 USD, daily takings deposited most trading days. Account opened 2019. PEP false. Prior alerts 4, prior SARs 0.\n\n**Twelve-month summary**: Cash in ranges from ~35,860 to ~44,950 per month, with 2026-04 at 71,720 (higher, likely due to the alert transactions plus normal activity). No wires, no cash out, only ACH out. 17-19 transactions per month.\n\nWait, this is interesting. The expected monthly cash is 86,000 USD, but the actual monthly cash in is only ~36,000-45,000. That's less than half of expected. Hmm.\n\nBut the key question: is this structuring?\n\nLet me think about the pattern. The rule fired because three cash deposits just under 10,000 within ten days. For a pizzeria that deposits daily takings, having deposits of 8,880-9,440 is actually... let me think. If they deposit daily takings, and expected monthly cash is 86,000, that's about 2,867 per day (86,000/30). But the actual monthly cash in is ~40,000, which is about 1,333 per day.\n\nWait, but the transactions shown are individual deposits of 8,880-9,440. If they deposit daily, each daily deposit would be ~1,300-1,500. But these are ~9,000 each. So these are NOT daily takings deposits \u2014 these are larger lump sums.\n\nHmm, but wait. Let me reconsider. The twelve-month summary shows cash in of ~40,000 per month with 17-19 transactions. So average transaction is ~40,000/18 \u2248 2,222 per transaction. That's consistent with daily takings deposits of a few thousand each.\n\nBut the alert transactions are 8,880-9,440 each \u2014 much larger than the typical ~2,200 deposit. And they're all just under 10,000. That's suspicious.\n\nActually, let me reconsider. Are these deposits consistent with the profile? The profile says \"daily takings deposited most trading days.\" If they deposit daily, a single day's takings for a pizzeria would be maybe a few thousand. A deposit of 9,440 in one go is a lot more than a single day's takings.\n\nBut actually, the expected monthly cash is 86,000, and actual is ~40,000. So the actual deposits are LOWER than expected. The alert transactions at ~9,000 each are HIGHER than the typical ~2,200 deposit.\n\nSo the pattern: normally the pizzeria deposits ~2,200 per transaction (daily takings), 18 times a month = ~40,000. But in April, there are three deposits of ~9,000 each that are just under the 10,000 threshold. These stand out as abnormally large for this account AND just under the reporting threshold.\n\nThis is a classic structuring indicator: breaking up what would be a single larger deposit into multiple sub-threshold deposits.\n\nLet me think about whether this is consistent with profile. The profile expects 86,000 monthly cash. The actual is ~40,000. The three alert transactions total 27,270. If we add those to a \"normal\" month... actually the April total is 71,720 which is higher than other months. So April had more cash.\n\nHmm, let me reconsider. April cash in = 71,720 with 18 transactions. Other months are ~40,000 with 18 transactions. So April's average transaction is ~3,984 vs ~2,222 for other months. The three alert transactions (8,880-9,440) are pulling the average up.\n\nLet me estimate: if April had 18 transactions totaling 71,720, and 3 of them are 8,880+8,950+9,440 = 27,270, then the other 15 transactions total 71,720-27,270 = 44,450, average ~2,963. That's a bit higher than the typical ~2,222 but in the right ballpark.\n\nSo the question is: are these three ~9,000 deposits consistent with a pizzeria's cash deposits, or are they structured?\n\nThe strong structuring signal:\n1. Three deposits all just under 10,000 (9,440, 8,950, 8,880) \u2014 all within ~1,120 of the threshold.\n2. Within ten days.\n3. The customer's normal deposits are ~2,200, so ~9,000 deposits are 4x the norm.\n\nBut counter-signals:\n1. Cash-intensive business, expected to deposit cash.\n2. No prior SARs (4 prior alerts but 0 SARs \u2014 so previous alerts were presumably closed without escalation).\n3. The amounts are plausible for a busy pizzeria's weekly takings.\n\nLet me look at the prior alerts. I should check prior_alerts to see if this has been reviewed before. The KYC says prior_alerts = 4. If these 4 prior alerts were for the same rule R-STR-01 and were closed as consistent with profile, that would be a strong signal to close again.\n\nActually, this is a key consideration. Let me look up prior alerts.\n\nAlso, I should consider the counterparty angle. The counterparties shown are only \"out\" (ACH out presumably). CP-1 out 48 txns 103,976, CP-2 out 12 txns 60,331. These are payments out, probably suppliers (flour, ingredients, rent, etc.). No counterparties for cash in (cash deposits don't have counterparties). So counterparty lookup isn't very relevant here since the alert is about cash deposits.\n\nLet me think about the decision framework:\n- Close: CONSISTENT_WITH_PROFILE, DOCUMENTED_SOURCE_OF_FUNDS, PREVIOUSLY_REVIEWED, RULE_ARTEFACT\n- Escalate: STRUCTURING, FUNNEL_ACCOUNT, MONEY_MULE, RAPID_MOVEMENT, UNKNOWN\n\nThe rule is literally R-STR-01 (structuring). The pattern of three sub-threshold deposits within ten days is the textbook definition of structuring.\n\nBut I need to weigh: is this genuinely suspicious structuring, or is it a legitimate business pattern that happens to trigger the rule?\n\nKey factors for a pizzeria:\n- A pizzeria with expected monthly cash of 86,000 would have daily takings. If they deposit daily, each deposit is small. But if they deposit less frequently (e.g., weekly), each deposit would be larger.\n- Actually, \"daily takings deposited most trading days\" suggests daily deposits. So ~9,000 single deposits would be unusual.\n\nHmm, but wait. Let me reconsider the expected vs actual. Expected monthly cash 86,000 but actual ~40,000. That's a big gap. Maybe the expected is aspirational or the business is underperforming. Regardless, the actual pattern is ~40,000/month in ~18 deposits = ~2,200 each.\n\nThe three ~9,000 deposits are 4x the typical deposit. This is a deviation from the established pattern. Combined with being just under the threshold, this is a strong structuring signal.\n\nHowever, I want to check the prior alerts first. If the customer has 4 prior alerts and 0 SARs, it's very likely these prior alerts were the same rule firing repeatedly and being closed as consistent with profile. That would change my analysis significantly.\n\nLet me look up prior_alerts. Cost 0.40h. I have plenty of budget (8 replies, 0.05 each = 0.40h for replies, plus lookups).\n\nActually, let me reconsider the budget. I've spent 0.50h. Each reply is 0.05h. Lookups: history 0.40, counterparty 0.40, prior_alerts 0.40, documents 0.10. I have 8 replies.\n\nLet me be strategic. The most informative lookup here is prior_alerts, because:\n1. If prior alerts are the same rule and were closed, that strongly suggests this is a recurring benign pattern (or a recurring pattern that was already judged benign).\n2. It tells me the disposition history.\n\nLet me also consider: should I look up a month's history to see the full transaction pattern in April? That would show me whether the ~9,000 deposits are part of a pattern of larger deposits or truly anomalous. But the twelve-month summary already gives me the monthly totals.\n\nActually, the most decisive piece of evidence is probably the prior alerts. Let me get those.\n\nLet me think about what a careful analyst would conclude:\n\nThe rule R-STR-01 is specifically designed to catch structuring. The three deposits are all just under 10,000 within 10 days. For a business whose NORMAL deposits are ~2,200, these ~9,000 deposits are anomalous in size AND suspicious in being just under the threshold.\n\nBut: cash-intensive business, and the total April cash (71,720) is still well within the expected monthly cash (86,000). So the total cash volume is not alarming.\n\nThe structuring concern is really about the SIZE and THRESHOLD-PROXIMITY of individual deposits, not the total volume. A pizzeria could legitimately have a big weekend and deposit ~9,000. But three such deposits in 10 days, each just under 10,000, when the norm is ~2,200... that's the pattern the rule targets.\n\nI think the right move is to look up prior alerts to see if this is a recurring pattern that's been reviewed. This is the highest-value lookup.\n\nLet me do that.\n\nActually, let me reconsider whether I even need to look anything up. The task says \"Cite the facts that decide the case.\" I have the alert, KYC, and twelve-month summary on screen. \n\nThe deciding facts:\n- For escalation (structuring): three sub-threshold deposits within 10 days (T-213, T-217, T-219), all just under 10,000, in a business whose normal deposits are ~2,200 (4x deviation), all at same branch BR-052.\n- For close (consistent with profile): cash-intensive pizzeria, expected monthly cash 86,000, April total 71,720 within expected, 0 prior SARs.\n\nHmm. This is genuinely a judgment call. Let me think about which way the balance tips.\n\nThe strongest argument for escalation: The deposits are 4x the customer's normal deposit size and all sit just under the reporting threshold within 10 days. This is the exact pattern R-STR-01 is built to catch. A pizzeria depositing daily takings wouldn't normally produce three ~9,000 deposits in 10 days.\n\nThe strongest argument for close: It's a cash business, the total is within expected, and there are no red flags otherwise. But \"no prior SARs\" is weak evidence because SARs are only filed on escalation; 4 prior alerts with 0 SARs means 4 prior alerts were closed.\n\nI really want to see the prior alerts. Let me look them up. That's the single most decisive piece of information.\n\nDecision: Look up prior_alerts.\n\nLet me write the JSON for the lookup.\n\n{\"action\":\"lookup\",\"what\":\"prior_alerts\"}\n\nThis costs 0.40h. After this I'll have spent 0.50 + 0.05 (this reply) + 0.40 = 0.95h. I have 7 replies left.\n\nLet me do it.\n", + "call": { + "promptTokens": 1567, + "completionTokens": 2985, + "reasoningTokens": null, + "durationMs": 108274, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "lookup", + "what": "prior_alerts" + }, + "rejection": null, + "observation": "PRIOR ALERTS:\n A-3217 R-WIRE-04 fired 2024-07-08 closed after 7 day(s) by Beatriz Grimaldi\n Closed: destination and amount match the remittance declared at onboarding; within stated income.\n A-4028 R-DORM-05 fired 2024-10-12 closed after 9 day(s) by Mateo Jensen\n Closed: reactivation explained by a documented source of funds; document reviewed and retained.\n A-2622 R-VEL-02 fired 2025-01-17 closed after 5 day(s) by Samir Pereira\n Closed: pass-through matches the settlement terms in the agreement on file; counterparties verified.\n A-1389 R-STR-01 fired 2026-03-06 closed after 3 day(s) by Marta Eskildsen\n Closed: deposits are documented daily takings; amounts vary with trading days; within kyc.expected_monthly_cash.\n\nHours spent so far: 0.95. You have 7 replies left." + } + }, + { + "reply": "", + "reasoning": "Let me analyze this alert carefully.\n\n**Alert A-8971**: Rule R-STR-01 \u2014 \"Three or more cash deposits just under the reporting threshold within ten days\"\n- T-213: 2026-04-16, cash_in, 9,440.00 USD\n- T-217: 2026-04-21, cash_in, 8,950.00 USD\n- T-219: 2026-04-22, cash_in, 8,880.00 USD\n\nAll three are just under 10,000 (the reporting threshold). Three cash deposits under 10k within 10 days (April 16 to April 22 = 6 days).\n\n**Customer**: Oakhill Kitchen Group, a pizzeria, cash-intensive food service, expected monthly cash of 86,000 USD.\n\n**Twelve-month summary**: Monthly cash in ranges from ~35,860 to ~44,950, with April 2026 at 71,720. So the typical month is around 40k cash in. The expected monthly cash is 86,000.\n\nWait, the expected monthly cash is 86,000 but the actual monthly cash in is around 40k. That's interesting \u2014 the actual is lower than expected. But that's not necessarily suspicious; it just means they deposit less than expected.\n\n**Key consideration**: This is a structuring rule. Three deposits just under the threshold. But the customer is a pizzeria that deposits daily takings. The prior alert A-1389 (same rule R-STR-01, fired 2026-03-06) was closed by Marta Eskildsen with rationale: \"deposits are documented daily takings; amounts vary with trading days; within kyc.expected_monthly_cash.\"\n\nSo this is a repeat of a previously reviewed and closed alert on the same rule. The prior closure rationale is sound: daily takings for a pizzeria, amounts vary with trading days.\n\nLet me think about whether this is a genuine structuring concern or a rule artefact / consistent with profile.\n\nThe amounts: 9,440, 8,950, 8,880. These are all under 10k but not suspiciously close to 10k (like 9,990). They range from 8,880 to 9,440. For a pizzeria doing ~40k/month cash, daily takings would be around 40k/22 trading days \u2248 1,800/day. Hmm, but these deposits are 8,880-9,440, which is much larger than a single day's takings.\n\nWait, let me reconsider. If monthly cash is ~40k and they deposit \"most trading days,\" then over ~22 days, that's ~1,800/day. But these individual deposits are ~9k. That's about 5 days of takings per deposit. That's plausible if they deposit weekly or a few times a week.\n\nActually, the rule is about \"three or more cash deposits just under the reporting threshold within ten days.\" The total of these three is 9,440 + 8,950 + 8,880 = 27,270. That's within the expected monthly cash of 86,000, and actually the April total cash in is 71,720, so these three deposits are a subset of that.\n\nHmm, but is the total of 27,270 over 10 days a structuring concern? The rule specifically flags \"just under the reporting threshold\" deposits. The concern with structuring is that someone breaks up a large deposit into smaller ones to avoid the 10k CTR threshold.\n\nBut for a legitimate cash business, daily/periodic takings naturally come in as deposits that may individually be under 10k. This is exactly what the prior alert A-1389 concluded: \"deposits are documented daily takings; amounts vary with trading days; within kyc.expected_monthly_cash.\"\n\nSo this looks like a PREVIOUSLY_REVIEWED situation, or CONSISTENT_WITH_PROFILE, or RULE_ARTEFACT.\n\nLet me think about which close reason is most appropriate:\n- CONSISTENT_WITH_PROFILE: The activity matches the expected profile (cash-intensive food service, daily takings).\n- DOCUMENTED_SOURCE_OF_FUNDS: There's documentation of where the cash comes from.\n- PREVIOUSLY_REVIEWED: A prior alert on the same rule was reviewed and closed.\n- RULE_ARTEFACT: The rule fired but it's a false positive / artefact of the rule design.\n\nThe strongest case here: This is a repeat of R-STR-01 that was already reviewed and closed 3 weeks ago (A-1389, fired 2026-03-06, closed by Marta Eskildsen). The rationale was that deposits are documented daily takings, amounts vary with trading days, within expected monthly cash.\n\nNow, is the current alert materially different? Let me check:\n- Prior alert fired 2026-03-06.\n- Current alert fired 2026-04-26.\n- The triggering transactions are in April (16, 21, 22).\n\nThe prior alert A-1389 was about March. The current is about April. So it's a new occurrence of the same pattern.\n\nGiven the customer is a pizzeria with cash-intensive food service and daily takings, and the prior same-rule alert was closed as documented daily takings, this current alert is very likely also just the same legitimate pattern.\n\nNow, should I escalate? The rule R-STR-01 is a structuring rule. Three deposits just under threshold within 10 days is a classic structuring indicator. But the context (cash business, daily takings, amounts within expected monthly cash, prior closure) strongly suggests this is legitimate.\n\nLet me consider the amounts more carefully. For structuring, you'd typically see amounts clustered just below the threshold (e.g., 9,900, 9,800, 9,950) that sum to a round or significant number. Here:\n- 9,440\n- 8,950\n- 8,880\n\nThese vary, and they're not all tightly clustered at ~9,900. They range from 8,880 to 9,440. This variation is consistent with \"amounts vary with trading days\" as the prior analyst noted.\n\nThe total over the period is 27,270, which is well within the expected monthly cash of 86,000.\n\nI think the right call is to CLOSE this alert. The question is which reason.\n\nGiven that:\n1. The customer is a cash-intensive pizzeria (matches profile).\n2. Daily takings deposited most trading days (matches profile).\n3. Amounts are within expected monthly cash.\n4. A prior alert on the EXACT same rule (R-STR-01) was reviewed and closed with sound rationale.\n\nI think PREVIOUSLY_REVIEWED is a strong candidate because there's a directly relevant prior alert on the same rule. But CONSISTENT_WITH_PROFILE is also strong because the activity genuinely matches the expected profile.\n\nActually, let me think about the distinction. PREVIOUSLY_REVIEWED typically means \"this exact issue was already reviewed and no action needed.\" CONSISTENT_WITH_PROFILE means \"the activity is consistent with what we expect for this customer.\"\n\nThe most decisive fact here is that this is a repeat of a previously reviewed and closed alert on the same rule, AND the activity is consistent with the customer's profile.\n\nI could cite both. Let me think about which is the primary reason.\n\nHmm. The prior alert A-1389 was closed as \"documented daily takings; amounts vary with trading days; within kyc.expected_monthly_cash.\" That's essentially a CONSISTENT_WITH_PROFILE / DOCUMENTED_SOURCE_OF_FUNDS rationale.\n\nI think the cleanest close is CONSISTENT_WITH_PROFILE, citing:\n- kyc.business_type (pizzeria)\n- kyc.expected_activity (Cash-intensive food service; daily takings deposited most trading days)\n- kyc.expected_monthly_cash (86,000)\n- A-1389 (prior alert on same rule, closed)\n- T-213, T-217, T-219 (the triggering transactions, to show they're within range)\n\nActually, let me reconsider whether I need to do more lookups. I've already seen:\n- The alert and triggering transactions.\n- The KYC file.\n- The twelve-month summary.\n- The prior alerts.\n- Counterparties on screen (CP-1, CP-2) \u2014 these are OUT counterparties, likely suppliers/vendors for the pizzeria.\n\nDo I need to look up history for a month? The twelve-month summary already shows the monthly cash in. April 2026 is 71,720. The three triggering deposits (27,270) are a subset. The pattern is consistent with prior months.\n\nDo I need to look up the counterparty? The counterparties are \"out\" (payments out), likely to suppliers. Not relevant to a cash-in structuring alert. The cash is coming in from customers (walk-ins), not from a counterparty. So counterparty lookup isn't needed.\n\nDo I need to look up documents? The prior alert mentioned \"documented daily takings\" and \"documented source of funds.\" But I don't have specific document IDs on screen. The documents lookup would show what's on file. However, I'm not sure I need it \u2014 the prior closure already established the documented basis.\n\nLet me think about whether I have enough to close confidently.\n\nThe case for closing:\n1. Pizzeria = cash-intensive, daily takings. \u2713\n2. Expected monthly cash 86,000; actual monthly ~40k (below expected, not above). \u2713\n3. Three deposits totaling 27,270 over 6 days = well within a month's takings. \u2713\n4. Amounts vary (8,880-9,440), consistent with varying trading days. \u2713\n5. Prior alert on SAME rule (R-STR-01) closed 3 weeks ago with sound rationale. \u2713\n\nThe case for escalating:\n1. Three deposits just under 10k within 10 days is a textbook structuring pattern.\n2. But the amounts aren't tightly clustered at the threshold, and the context strongly supports legitimate business.\n\nI'm fairly confident this should be closed. The risk of a false positive (escalating a clean customer) is real here, and the context strongly supports the customer being clean.\n\nNow, which close reason? Let me reconsider the options:\n- CONSISTENT_WITH_PROFILE: Best when the activity matches the expected profile. This fits well.\n- DOCUMENTED_SOURCE_OF_FUNDS: Best when there's specific documentation of the source. The prior alert referenced \"documented daily takings\" but I don't have a specific doc ID.\n- PREVIOUSLY_REVIEWED: Best when this exact issue was already reviewed. A-1389 is on the same rule but a different month.\n- RULE_ARTEFACT: Best when the rule fired due to a technicality/false positive.\n\nI think the strongest, most defensible reason is CONSISTENT_WITH_PROFILE, because the core reason the alert is a false positive is that this cash business legitimately deposits daily takings that happen to fall under the threshold. The prior alert (A-1389) reinforces this but the primary basis is the profile match.\n\nActually, let me reconsider PREVIOUSLY_REVIEWED. The prior alert A-1389 is on the EXACT same rule (R-STR-01) and was closed 3 weeks ago. This is a very strong signal that the compliance team has already vetted this customer's cash deposit pattern under this exact rule. That's a strong basis for PREVIOUSLY_REVIEWED.\n\nBut PREVIOUSLY_REVIEWED usually applies when the SAME alert/issue was already reviewed \u2014 i.e., a duplicate. Here it's a new occurrence (April vs March). So it's not exactly the same alert.\n\nHmm. I think I'll go with CONSISTENT_WITH_PROFILE as the primary reason, and I can cite A-1389 as supporting evidence. Actually, let me reconsider \u2014 the instructions say \"Cite the facts that decide the case, not everything you saw.\"\n\nThe facts that decide the case:\n1. The customer is a cash-intensive pizzeria with daily takings (kyc.business_type, kyc.expected_activity).\n2. The deposits are within expected monthly cash (kyc.expected_monthly_cash, and the twelve-month summary).\n3. A prior alert on the same rule was reviewed and closed (A-1389).\n\nI think CONSISTENT_WITH_PROFILE is the best fit. Let me cite:\n- kyc.business_type\n- kyc.expected_activity\n- kyc.expected_monthly_cash\n- A-1389\n- T-213, T-217, T-219 (to show the amounts)\n\nWait, but should I cite the transactions? They're the triggering transactions, and showing they're within range supports the case. Yes, I'll cite them.\n\nActually, let me reconsider whether to do one more lookup to strengthen the case. Let me think about what would make this airtight.\n\nOne concern: The amounts 9,440, 8,950, 8,880 \u2014 are these really \"daily takings\"? For a pizzeria doing ~40k/month, that's ~1,800/day. A 9,440 deposit would be ~5 days of takings. So these are more like \"weekly\" or \"periodic\" deposits, not daily. But the KYC says \"daily takings deposited most trading days.\" Hmm.\n\nWait, let me recompute. If monthly cash is 40k and they deposit most trading days (say 22 days), that's ~1,800/day. But the expected monthly cash in KYC is 86,000, which would be ~3,900/day. The actual is lower than expected.\n\nBut the individual deposits of ~9k are larger than a single day's takings. So they're depositing accumulated takings every few days. That's still consistent with a cash business \u2014 they accumulate cash over a few days and deposit.\n\nActually, the rule fired on \"three or more cash deposits just under the reporting threshold within ten days.\" The fact that they're depositing ~9k every few days, and these happen to be under 10k, is just how a cash business's deposits naturally look. It's not that they're deliberately keeping each deposit under 10k to avoid reporting \u2014 it's that their natural deposit amounts happen to be in that range.\n\nThis is a classic case where a structuring rule fires on a legitimate cash business. The prior analyst recognized this and closed it.\n\nI'm confident in closing. Let me decide on the reason.\n\nI'll go with CONSISTENT_WITH_PROFILE. The narrative: The customer is a cash-intensive pizzeria whose daily/periodic takings are deposited and naturally fall just under the CTR threshold. The three triggering deposits total 27,270 USD over 6 days, well within the expected monthly cash of 86,000 USD and consistent with the twelve-month cash-in pattern (~35k-45k/month, April 71,720). A prior alert on the identical rule (R-STR-01, A-1389) was reviewed and closed on the same basis (documented daily takings, amounts vary with trading days, within expected monthly cash). No structuring indicator beyond the rule's mechanical trigger.\n\nHmm, but let me reconsider PREVIOUSLY_REVIEWED vs CONSISTENT_WITH_PROFILE one more time.\n\nThe key question: what is the DECISIVE fact?\n\nIf I say CONSISTENT_WITH_PROFILE, I'm saying \"this is normal for this customer.\" That's true and well-supported.\n\nIf I say PREVIOUSLY_REVIEWED, I'm saying \"we already looked at this exact pattern and closed it.\" That's also true (A-1389, same rule).\n\nI think CONSISTENT_WITH_PROFILE is more fundamentally correct because the reason it's a false positive is the profile match, not merely that we reviewed it before. The prior review is supporting evidence.\n\nBut actually, you know what, let me reconsider. Both are defensible. The instructions say cite the facts that decide the case. I'll use CONSISTENT_WITH_PROFILE and cite the profile fields plus the prior alert plus the transactions.\n\nWait, I want to make sure I'm not missing something that would make this a real escalation. Let me double check the numbers.\n\nReporting threshold: 10,000.\nT-213: 9,440 (560 under)\nT-217: 8,950 (1,050 under)\nT-219: 8,880 (1,120 under)\n\nThese are \"just under\" in the sense of under 10k, but 8,880 and 8,950 are not that close to 10k. Only 9,440 is fairly close. So it's a mix. Not a tight cluster at 9,9xx.\n\nTotal: 27,270 over 6 days (April 16-22).\n\nIf this were structuring, you'd expect the total to be a \"round\" amount that they're trying to hide, like 30,000 or 25,000, broken into sub-10k pieces. 27,270 isn't particularly round. And the individual amounts vary, which is more consistent with genuine takings.\n\nAlso, the customer deposits cash regularly (12 months of ~40k/month cash in). This is an established pattern, not a sudden change. April is 71,720 which is higher than usual but still within a reasonable range for a pizzeria (maybe a busy month, or they did a bigger deposit).\n\nActually, April being 71,720 vs ~40k typical is a notable jump", + "call": { + "promptTokens": 1863, + "completionTokens": 4096, + "reasoningTokens": null, + "durationMs": 140542, + "finishReason": "length" + }, + "info": { + "action": null, + "rejection": "no JSON object found in the reply", + "observation": "That reply was not accepted: no JSON object found in the reply. It still cost a turn. You have 6 replies left." + } + }, + { + "reply": "\n\n{\"action\":\"close\",\"reason\":\"PREVIOUSLY_REVIEWED\",\"cites\":[\"A-1389\",\"kyc.expected_activity\",\"T-213\",\"T-217\",\"T-219\"],\"note\":\"Same rule R-STR-01 fired 2026-03-06 (A-1389) and was closed: deposits are documented daily takings, amounts vary with trading days, within expected monthly cash. Current deposits (9,440 / 8,950 / 8,880) show the same organic variation consistent with a cash-intensive pizzeria's daily takings. KYC refreshed 2026-03-13; no PEP; zero prior SARs. Pattern unchanged from the previously reviewed baseline.\"}", + "reasoning": "Let me analyze this alert carefully.\n\nAlert A-8971: Rule R-STR-01 \u2014 Three or more cash deposits just under the reporting threshold within ten days.\n\nTriggering transactions:\n- T-213: 2026-04-16, cash_in, 9,440.00 USD, branch BR-052\n- T-217: 2026-04-21, cash_in, 8,950.00 USD, branch BR-052\n- T-219: 2026-04-22, cash_in, 8,880.00 USD, branch BR-052\n\nReporting threshold: 10,000.00 USD\n\nAll three deposits are just under 10,000. That's the rule firing.\n\nNow let me look at the customer profile:\n- Oakhill Kitchen Group, pizzeria, cash-intensive food service\n- Daily takings deposited most trading days\n- Expected monthly cash: 86,000 USD\n- Stated income: 492,000 USD\n- KYC refreshed 2026-03-13 (recent)\n- PEP: False\n- Prior alerts: 4, prior SARs: 0\n\nTwelve-month summary:\n- Monthly cash in ranges from ~35,860 to ~44,950 for most months\n- 2026-04: 71,720 cash in (higher than usual, but April may be partial - the alert fired 2026-04-26, so April isn't complete)\n\nWait, let me think about this. The monthly cash in for typical months is around 36k-45k. The expected monthly cash is 86,000. That's interesting - the actual monthly cash in is much lower than the expected 86,000. But that could be because the summary shows what was actually deposited, and maybe they don't deposit all cash, or the expected is a target.\n\nActually, let me reconsider. The 12-month summary shows cash in of ~36k-45k per month, but kyc.expected_monthly_cash = 86,000. That's a discrepancy, but it's the customer's stated expectation vs actual. Not necessarily a red flag for this alert.\n\nPrior alerts:\n- A-1389: R-STR-01 (SAME RULE) fired 2026-03-06, closed by Marta Eskildsen: \"deposits are documented daily takings; amounts vary with trading days; within kyc.expected_monthly_cash.\"\n\nThis is very relevant! The same structuring rule fired in March 2026 and was closed as consistent with profile - documented daily takings, amounts vary with trading days, within expected monthly cash.\n\nNow, the current alert is April 2026, same rule, three deposits just under 10k.\n\nLet me think about whether this is consistent with profile or whether it should be escalated.\n\nArguments for close (CONSISTENT_WITH_PROFILE or PREVIOUSLY_REVIEWED):\n1. Customer is a pizzeria - cash-intensive food service\n2. Daily takings deposited most trading days\n3. Same rule (R-STR-01) was just closed in March 2026 as consistent - documented daily takings, amounts vary with trading days, within expected monthly cash\n4. KYC recently refreshed (2026-03-13)\n5. No PEP\n6. No prior SARs\n7. The amounts (9,440, 8,950, 8,880) vary - they're not all the same amount, which is consistent with daily takings varying\n8. April cash in is 71,720 which is higher than typical but April is partial month (alert fired 4/26)\n\nWait, but let me look more carefully. The three triggering transactions are 9,440, 8,950, 8,880. These are all just under 10,000. The rule is about \"just under the reporting threshold.\"\n\nHowever, the key question is: is this actually structuring (deliberately keeping deposits under threshold to avoid CTR reporting), or is it just the natural pattern of a cash business depositing daily takings that happen to be under 10k?\n\nFor a pizzeria with daily takings, if they deposit daily, the daily amounts would naturally vary. If a typical day's takings are around 8,000-9,500, then deposits would naturally fall just under 10,000. This is the \"amounts vary with trading days\" rationale from the March close.\n\nThe fact that the same rule fired in March and was closed with this exact rationale strongly suggests this is a recurring pattern that has been reviewed and found consistent.\n\nLet me check: March had R-STR-01 fire on 2026-03-06 and was closed. Now April has R-STR-01 fire on 2026-04-26. \n\nThe amounts in the triggering transactions:\n- T-213: 9,440\n- T-217: 8,950\n- T-219: 8,880\n\nThese vary, which is consistent with daily takings.\n\nNow, the concern with structuring is that amounts are deliberately kept just under threshold. But for a cash business, if their daily takings are in the 8,000-9,500 range, they'd naturally deposit amounts just under 10,000. The variation (9,440 vs 8,950 vs 8,880) supports this being organic rather than deliberately structured (if someone were structuring, they might keep amounts very close to but under threshold, like 9,900, 9,950, 9,990 - all very close to 10,000).\n\nActually, 8,880 and 8,950 are not that close to 10,000. They're about 1,000-1,100 under. 9,440 is about 560 under. So there's variation in how close to threshold they are, which supports organic daily takings.\n\nLet me think about the disposition options:\n- CONSISTENT_WITH_PROFILE: The activity matches the expected profile of a cash-intensive pizzeria\n- PREVIOUSLY_REVIEWED: The same rule was just reviewed and closed in March\n- DOCUMENTED_SOURCE_OF_FUNDS: Daily takings are documented\n- RULE_ARTEFACT: The rule is firing due to the natural pattern of cash deposits\n\nI think the best disposition here is either CONSISTENT_WITH_PROFILE or PREVIOUSLY_REVIEWED.\n\nGiven that:\n1. The same rule (R-STR-01) was just closed in March 2026 with a clear rationale\n2. The customer is a cash-intensive pizzeria\n3. The amounts vary consistent with daily takings\n4. KYC recently refreshed\n5. No PEP, no prior SARs\n\nI'll go with PREVIOUSLY_REVIEWED, citing the prior alert A-1389 (same rule, same customer, recently closed as consistent) and the KYC profile fields.\n\nActually, let me reconsider. The options are:\n- CONSISTENT_WITH_PROFILE\n- DOCUMENTED_SOURCE_OF_FUNDS\n- PREVIOUSLY_REVIEWED\n- RULE_ARTEFACT\n\nPREVIOUSLY_REVIEWED seems most appropriate because the exact same rule was just reviewed and closed for this customer in March 2026. The rationale was \"deposits are documented daily takings; amounts vary with trading days; within kyc.expected_monthly_cash.\"\n\nLet me cite:\n- A-1389 (prior alert, same rule, closed as consistent)\n- kyc.business_type (pizzeria)\n- kyc.expected_activity (cash-intensive food service; daily takings deposited most trading days)\n- T-213, T-217, T-219 (the triggering transactions showing varying amounts)\n\nActually, I should cite the facts that decide the case. The key facts are:\n1. Prior alert A-1389 - same rule, same customer, closed as consistent with daily takings\n2. KYC profile showing cash-intensive business with daily takings\n3. The varying amounts in the triggering transactions\n\nLet me write the close.\n\nI'll cite: A-1389, kyc.business_type, kyc.expected_activity, T-213, T-217, T-219\n\nActually, let me be more focused. The deciding facts are:\n- A-1389: prior alert on same rule, closed as consistent\n- kyc.expected_activity: cash-intensive, daily takings\n- The transactions themselves showing variation\n\nLet me write this up.\n", + "call": { + "promptTokens": 1904, + "completionTokens": 2191, + "reasoningTokens": null, + "durationMs": 98431, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "close", + "reason": "PREVIOUSLY_REVIEWED", + "cites": [ + "A-1389", + "kyc.expected_activity", + "T-213", + "T-217", + "T-219" + ], + "note": "Same rule R-STR-01 fired 2026-03-06 (A-1389) and was closed: deposits are documented daily takings, amounts vary with trading days, within expected monthly cash. Current deposits (9,440 / 8,950 / 8,880) show the same organic variation consistent with a cash-intensive pizzeria's daily takings. KYC refreshed 2026-03-13; no PEP; zero prior SARs. Pattern unchanged from the previously reviewed baseline." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/base-on-s2.json b/public/traces/alert-triage/base-on-s2.json new file mode 100644 index 0000000..1f8ef17 --- /dev/null +++ b/public/traces/alert-triage/base-on-s2.json @@ -0,0 +1,126 @@ +{ + "runId": "base-on-s2", + "seed": 2, + "model": "brain-qwen38-dspark", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 0.4375, + "evidence": 0.4 + }, + "metrics": { + "hours_spent": 1.6, + "reference_hours": 0.7, + "lookups": 3.0, + "turns_used": 4.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": null + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "benign", + "template": "B6", + "overlay": null, + "typology": null, + "planted": [ + [ + "doc.D-aded" + ] + ], + "reference_minutes": 42, + "reference_policy": "fast", + "held_out": false, + "tier": "benign", + "screen": "ALERT A-9760 rule R-DORM-05 \u2014 Large credit to an account with no material activity for nine months\n fired 2026-04-23\n triggering transactions:\n T-4 2026-04-18 wire_in 356,000.00 USD CP-1\n T-5 2026-04-19 wire_out 291,900.00 USD CP-2\nKYC FILE C-7059 Gabriela Kowalczyk\n kyc.account_opened = 2012-03-09\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Retired; low activity expected.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2025-12-12\n kyc.occupation = retired\n kyc.pep = False\n kyc.prior_alerts = 2\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 31,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 0 0 0 0 0 0 0\n 2025-06 0 0 0 0 0 0 0\n 2025-07 0 170 0 0 0 0 1\n 2025-08 0 0 0 0 0 0 0\n 2025-09 0 0 0 0 0 0 0\n 2025-10 0 0 0 0 0 0 0\n 2025-11 0 0 0 0 0 0 0\n 2025-12 0 120 0 0 0 0 1\n 2026-01 0 170 0 0 0 0 1\n 2026-02 0 0 0 0 0 0 0\n 2026-03 0 0 0 0 0 0 0\n 2026-04 0 0 356,000 291,900 0 0 2\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 in 1 txns 356,000.00 USD\n CP-2 out 1 txns 291,900.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "\n\n{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-1\"}", + "reasoning": "Let me analyze this alert carefully.\n\nAlert A-9760: R-DORM-05 \u2014 Large credit to an account with no material activity for nine months\nFired 2026-04-23\nTriggering transactions:\n- T-4: 2026-04-18 wire_in 356,000.00 USD from CP-1\n- T-5: 2026-04-19 wire_out 291,900.00 USD to CP-2\n\nKYC:\n- Gabriela Kowalczyk, individual, US\n- Account opened 2012-03-09\n- Retired, low activity expected\n- Expected monthly cash 0\n- KYC refreshed 2025-12-12\n- Stated income 31,000 USD\n- PEP false\n- Prior alerts: 2\n- Prior SARs: 0\n- Remittance declared: none\n\nTwelve-month summary:\n- Account essentially dormant (only small cash out of 170, 120, 170 in Jul, Dec, Jan)\n- April 2026: wire in 356,000, wire out 291,900\n\nSo a retired person with stated income of 31k receives a 356k wire and the next day wires out 291,900 to another party. This is a classic rapid movement / layering pattern. The account was dormant for ~9 months, then a large credit comes in and most of it moves out next day. This looks like a possible mule/funnel account or rapid movement.\n\nKey considerations:\n- Large credit inconsistent with profile (retired, low income)\n- Rapid movement: 291,900 out next day (82% of inflow)\n- CP-1 and CP-2 unknown - need to look up\n- Prior alerts = 2 - need to check if previously reviewed\n\nLet me think about what to do. I have 8 replies. Each lookup costs 0.40h except documents 0.10h.\n\nThe most decisive facts:\n1. The rapid movement (in then out next day, 82%)\n2. The counterparties - are they shared with other alerted accounts? (funnel/mule indicator)\n3. Prior alerts - were they escalated or closed?\n\nLet me look up the counterparties first. CP-1 and CP-2. The counterparty lookup tells me \"how many OTHER alerted accounts share it\" which is a strong mule/funnel indicator.\n\nActually, let me be strategic. The pattern here strongly suggests escalation (RAPID_MOVEMENT or MONEY_MULE or FUNNEL_ACCOUNT). But I should confirm with lookups to cite proper facts.\n\nLet me think about which disposition:\n- CONSISTENT_WITH_PROFILE: No, a 356k wire to a retired person with 31k income and dormant account is NOT consistent.\n- DOCUMENTED_SOURCE_OF_FUNDS: Would need documents.\n- PREVIOUSLY_REVIEWED: If prior alerts cover this.\n- RULE_ARTEFACT: If the rule fired in error.\n- Escalate: RAPID_MOVEMENT, MONEY_MULE, FUNNEL_ACCOUNT, etc.\n\nThe strongest signal: rapid movement of funds (in 356k, out 291.9k next day) to an account inconsistent with profile. This is a strong escalation candidate.\n\nBut I should check:\n1. Counterparty lookups to see if these are known/shared (mule/funnel)\n2. Prior alerts to see if this was already reviewed\n\nLet me start with counterparty lookups. Actually, let me look up both CP-1 and CP-2, and prior alerts. That's 3 lookups = 1.2h. Plus I've spent 0.5h. Total 1.7h. I have plenty of budget (8 replies * 0.05 = 0.4h base cost, but the \"hours\" seem to be a separate resource - actually the cost is in analyst-hours and each reply costs 0.05h, lookups cost 0.40h).\n\nWait, let me re-read. \"You have 8 replies. Each reply must be exactly ONE JSON object... and every reply costs 0.05 hours whether or not it is accepted.\" And lookups cost +0.40h or +0.10h.\n\nSo the constraint is 8 replies, not hours. The hours are just a cost metric. Let me count replies: I have 8. Each lookup is a reply. So I can do up to 8 lookups/actions.\n\nLet me be efficient. I'll do:\n1. Lookup counterparty CP-1\n2. Lookup counterparty CP-2\n3. Lookup prior_alerts\n\nThen decide.\n\nActually, the most decisive single fact might be the counterparty sharing. Let me look up CP-1 and CP-2 to see \"how many OTHER alerted accounts share it.\" If CP-1 or CP-2 is shared with many other alerted accounts, that's a strong mule/funnel indicator.\n\nLet me do the lookups. I'll batch my thinking but each reply is one JSON.\n\nLet me start with CP-1.\n\nActually, let me reconsider. The pattern of \"large credit to dormant account, then rapid outflow\" is the textbook RAPID_MOVEMENT typology. The counterparty info will tell me if it's a mule/funnel network.\n\nLet me look up both counterparties and prior alerts. I'll do them one at a time.\n\nReply 1: lookup counterparty CP-1\nReply 2: lookup counterparty CP-2\nReply 3: lookup prior_alerts\n\nThen I'll have enough to decide.\n\nLet me start.\n", + "call": { + "promptTokens": 1394, + "completionTokens": 1306, + "reasoningTokens": null, + "durationMs": 56324, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-1" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-1 Eskildsen & Laurent LLP, client trust account\n type law_firm_trust; jurisdiction United States; first seen 2026-04-18\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-4\n\nHours spent so far: 0.95. You have 7 replies left." + } + }, + { + "reply": "\n\n{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-2\"}", + "reasoning": "Let me analyze what I have so far.\n\nThe alert is R-DORM-05 \u2014 Large credit to an account with no material activity for nine months.\n\nKey facts:\n- Customer: Gabriela Kowalczyk, retired, low activity expected\n- Account opened 2012-03-09\n- Country: US\n- KYC refreshed 2025-12-12\n- Stated income: 31,000 USD\n- Prior alerts: 2, prior SARs: 0\n- Remittance declared: none declared\n\nTriggering transactions:\n- T-4: 2026-04-18 wire_in 356,000 USD from CP-1\n- T-5: 2026-04-19 wire_out 291,900 USD to CP-2\n\nTwelve-month summary shows the account was essentially dormant (only tiny cash out of 170, 120, 170 in a few months) until April 2026 when the big wires hit.\n\nCP-1 = Eskildsen & Laurent LLP, client trust account (law_firm_trust), US jurisdiction, first seen 2026-04-18, 0 other alerted accounts sharing it.\n\nThis is interesting. A law firm trust account sending 356,000 USD to a retired individual. This could be a legitimate settlement or inheritance/lawyer distribution. But then 291,900 went out to CP-2 the next day.\n\nLet me think about the typology. The account was dormant for 9+ months, then a large credit arrives, and the next day most of it (291,900 out of 356,000, ~82%) is wired out to another party. This is a classic rapid movement / funnel pattern.\n\nThe question is whether this is:\n1. A legitimate source of funds (law firm trust = settlement/inheritance) \u2192 could be DOCUMENTED_SOURCE_OF_FUNDS or CONSISTENT_WITH_PROFILE\n2. A mule/funnel account \u2192 RAPID_MOVEMENT, MONEY_MULE, or FUNNEL_ACCOUNT\n\nLet me consider. The customer is retired with stated income of 31,000. A 356,000 wire from a law firm trust account is a large amount but could be a settlement or estate distribution. However, the rapid outflow the next day to a different counterparty (CP-2) is a red flag for layering / rapid movement / mule activity.\n\nI need to look at CP-2 to understand where the money went. Let me also consider prior alerts (2 prior alerts on this customer).\n\nLet me look up CP-2 and prior alerts. Let me be strategic with my 7 remaining replies.\n\nActually, let me think about what decides the case. The key tension:\n- Law firm trust account as source = potentially documented/legitimate\n- But rapid movement out to CP-2 next day = layering\n\nI should look at CP-2 to see who receives the funds. If CP-2 is another law firm or a related entity, it might be legitimate. If CP-2 is a high-risk jurisdiction or unknown, it's a red flag.\n\nLet me look up CP-2.\n", + "call": { + "promptTokens": 1509, + "completionTokens": 746, + "reasoningTokens": null, + "durationMs": 23849, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-2" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-2 Brightwater Securities\n type brokerage; jurisdiction United States; first seen 2026-04-19\n other alerted accounts sharing this counterparty in twelve months: 1\n transactions: T-5\n\nHours spent so far: 1.40. You have 6 replies left." + } + }, + { + "reply": "\n\n{\"action\":\"lookup\",\"what\":\"documents\"}", + "reasoning": "Let me analyze this alert carefully.\n\n**Alert A-9760**: Rule R-DORM-05 \u2014 Large credit to an account with no material activity for nine months\n- T-4: 2026-04-18 wire_in 356,000 USD from CP-1\n- T-5: 2026-04-19 wire_out 291,900 USD to CP-2\n\n**Customer**: Gabriela Kowalczyk, C-7059\n- Account opened 2012-03-09\n- US individual, retired\n- Expected activity: low\n- Stated income: 31,000 USD\n- KYC refreshed 2025-12-12\n- 2 prior alerts, 0 prior SARs\n\n**Twelve-month summary**: Essentially dormant. Only tiny cash-outs (170, 120, 170) in 2025-07, 2025-12, 2026-01. Then in 2026-04: 356,000 wires in, 291,900 wires out.\n\n**Counterparties**:\n- CP-1: Eskildsen & Laurent LLP, client trust account, law_firm_trust, US, first seen 2026-04-18, 0 other alerted accounts share it. This is a law firm trust account \u2014 consistent with a legal settlement/inheritance/claim payout to a retired individual.\n- CP-2: Brightwater Securities, brokerage, US, first seen 2026-04-19, 1 other alerted account shares it. This is a brokerage \u2014 consistent with depositing funds into an investment account.\n\n**Analysis**:\n- The customer is a long-standing (2012) US retiree with low activity, consistent with profile.\n- The large credit of 356,000 comes from a law firm trust account (client trust) \u2014 a very common and legitimate source for large one-off funds (legal settlement, inheritance, class action payout, etc.).\n- The next day, 291,900 (about 82% of it) is wired out to a brokerage (Brightwater Securities) \u2014 consistent with a retiree moving money into investments.\n- The remaining ~64,100 stays in the account.\n- The counterparty CP-1 is a law firm trust with 0 other alerted accounts \u2014 not a shared/suspicious entity.\n- CP-2 is a brokerage with 1 other alerted account \u2014 that's mild, but a brokerage is a normal destination.\n\nThis looks like a legitimate pattern: a retired customer receives a legal payout (settlement/inheritance from a law firm trust) and immediately moves most of it into a brokerage account. This is CONSISTENT_WITH_PROFILE or DOCUMENTED_SOURCE_OF_FUNDS.\n\nThe key question: is this a rule artefact or consistent with profile? The rule fired because the account was dormant for 9 months and then received a large credit. But the source is a law firm trust account (legitimate), and the outflow is to a brokerage (legitimate). \n\nLet me think about whether I need more info. The prior alerts (2) and documents could help. Let me consider:\n- Prior alerts: knowing the disposition of the 2 prior alerts would tell me if this customer has been reviewed before (PREVIOUSLY_REVIEWED option) and what the rationale was.\n- Documents: could show a legal settlement letter or source of funds documentation (DOCUMENTED_SOURCE_OF_FUNDS).\n\nThe strongest close here would be either:\n1. CONSISTENT_WITH_PROFILE \u2014 the pattern (law firm trust \u2192 brokerage) is consistent with a retiree receiving a legal payout and investing it.\n2. DOCUMENTED_SOURCE_OF_FUNDS \u2014 if documents show the source.\n3. PREVIOUSLY_REVIEWED \u2014 if prior alerts were already reviewed and cleared.\n\nLet me check prior alerts and documents to strengthen my case. Actually, let me be strategic. I have 6 replies left. Let me look at prior alerts and documents.\n\nActually, the most decisive evidence for a clean close is documents showing the source of funds (e.g., a legal settlement agreement or inheritance document). Let me check documents first, and prior alerts.\n\nLet me look at documents (0.10) and prior_alerts (0.40).\n\nActually, let me think about which is more decisive. The transaction pattern already strongly suggests legitimacy:\n- In from law firm trust (CP-1, 0 shared alerts)\n- Out to brokerage (CP-2)\n\nThis is a classic \"legal settlement \u2192 investment\" pattern for a retiree. The KYC says retired, low activity. A large one-off legal payout is entirely consistent with a retired person's profile (e.g., they may have had a personal injury claim, inheritance, or class action).\n\nI think the best close reason is CONSISTENT_WITH_PROFILE, citing the KYC (retired, low activity), T-4 (from law firm trust), T-5 (to brokerage), and the counterparties.\n\nBut let me check documents to see if there's a documented source, which would make it a stronger DOCUMENTED_SOURCE_OF_FUNDS close. And prior alerts to see if it's PREVIOUSLY_REVIEWED.\n\nLet me do both lookups to be thorough, then decide. I have enough replies.\n\nLet me look at documents first (cheapest, 0.10) and prior alerts.\n\nActually, I'll do both in sequence since I need to see results. Let me start with documents.\n", + "call": { + "promptTokens": 1613, + "completionTokens": 1246, + "reasoningTokens": null, + "durationMs": 42157, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "lookup", + "what": "documents" + }, + "rejection": null, + "observation": "DOCUMENTS ON FILE:\n doc.D-3211 [identity] Government-issued identity document, verified at onboarding\n doc.D-aded [executor_letter] Letter from Alicia Haddad, executor, via Eskildsen & Laurent LLP, client trust account: distribution from the estate of Aaron Eskildsen to the named beneficiary.\n\nHours spent so far: 1.55. You have 5 replies left." + } + }, + { + "reply": "\n\n{\"action\":\"close\",\"reason\":\"DOCUMENTED_SOURCE_OF_FUNDS\",\"cites\":[\"doc.D-aded\",\"T-4\",\"T-5\",\"kyc.expected_activity\"],\"note\":\"Wire-in of 356K from Eskildsen & Laurent LLP client trust account is documented by executor letter (doc.D-aded) as an estate distribution to the named beneficiary. Next-day wire-out of 291.9K to Brightwater Securities (brokerage) is a normal disposition of estate funds. Customer is retired with low activity expected; the dormant period is consistent with profile. CP-1 (trust account) has zero other alerted accounts. No PEP, no prior SARs, KYC refreshed 2025-12-12.\"}", + "reasoning": "Let me analyze what I have:\n\n- Alert A-9760: Large credit to an account with no material activity for nine months\n- Customer: Gabriela Kowalczyk, retired, low activity expected\n- Triggering transactions:\n - T-4: 2026-04-18 wire_in 356,000 USD from CP-1 (Eskildsen & Laurent LLP, client trust account)\n - T-5: 2026-04-19 wire_out 291,900 USD to CP-2 (Brightwater Securities, brokerage)\n\n- CP-1: Law firm trust account, US jurisdiction, first seen 2026-04-18, 0 other alerted accounts sharing it\n- CP-2: Brokerage, US jurisdiction, first seen 2026-04-19, 1 other alerted account sharing it\n- Documents:\n - doc.D-3211: Identity document\n - doc.D-aded: Executor letter from Alicia Haddad, via Eskildsen & Laurent LLP, stating distribution from the estate of Aaron Eskildsen to the named beneficiary\n\nSo the picture is:\n- Retired customer with low activity (consistent with profile)\n- Large wire IN from a law firm trust account (estate distribution)\n- Next day, wire OUT to a brokerage account\n- There's documentation: an executor letter explaining the source of funds (estate distribution)\n- CP-1 has 0 other alerted accounts sharing it\n- CP-2 has 1 other alerted account sharing it (slightly concerning but not definitive)\n\nThe key question: Is this a legitimate estate distribution being moved to a brokerage account, or is this suspicious?\n\nArguments for closing (DOCUMENTED_SOURCE_OF_FUNDS):\n- The customer is retired with low activity expected\n- The source is a law firm trust account (estate distribution)\n- There's an executor letter documenting the source of funds\n- The outflow is to a brokerage (Brightwater Securities), which is a normal place to put estate money\n- CP-1 has no other alerted accounts\n- The amounts are consistent (356K in, 291.9K out - the difference could be fees or partial distribution)\n- KYC was refreshed recently (2025-12-12)\n- No PEP status\n- No prior SARs\n\nArguments for escalating:\n- Rapid movement (next day outflow)\n- Large amount relative to stated income (31K)\n- CP-2 has 1 other alerted account\n- Account was dormant for 9 months\n\nHowever, the rapid movement to a brokerage is actually quite normal for estate distributions - people often deposit estate money into investment accounts. The documentation clearly explains the source. The law firm trust account is a legitimate source. The executor letter is a strong document.\n\nI think this is a DOCUMENTED_SOURCE_OF_FUNDS case. The executor letter from the law firm explains the source of the 356K wire. The outflow to a brokerage is a normal disposition of estate funds. The customer profile (retired, low activity) is consistent with receiving a large lump sum after being dormant.\n\nLet me cite:\n- doc.D-aded (executor letter documenting source)\n- T-4 (the wire in from trust account)\n- T-5 (the wire out to brokerage)\n- kyc.expected_activity (retired, low activity expected)\n- kyc.occupation (retired)\n\nI'll close with DOCUMENTED_SOURCE_OF_FUNDS.\n", + "call": { + "promptTokens": 1731, + "completionTokens": 930, + "reasoningTokens": null, + "durationMs": 48857, + "finishReason": "stop" + }, + "info": { + "action": { + "action": "close", + "reason": "DOCUMENTED_SOURCE_OF_FUNDS", + "cites": [ + "doc.D-aded", + "T-4", + "T-5", + "kyc.expected_activity" + ], + "note": "Wire-in of 356K from Eskildsen & Laurent LLP client trust account is documented by executor letter (doc.D-aded) as an estate distribution to the named beneficiary. Next-day wire-out of 291.9K to Brightwater Securities (brokerage) is a normal disposition of estate funds. Customer is retired with low activity expected; the dormant period is consistent with profile. CP-1 (trust account) has zero other alerted accounts. No PEP, no prior SARs, KYC refreshed 2025-12-12." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/fast-s1.json b/public/traces/alert-triage/fast-s1.json new file mode 100644 index 0000000..442bfa6 --- /dev/null +++ b/public/traces/alert-triage/fast-s1.json @@ -0,0 +1,68 @@ +{ + "runId": "fast-s1", + "seed": 1, + "model": "fast-analyst", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 1.0, + "evidence": 1.0 + }, + "metrics": { + "hours_spent": 0.55, + "reference_hours": 0.55, + "lookups": 0.0, + "turns_used": 1.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": null + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "benign", + "template": "B4", + "overlay": null, + "typology": null, + "planted": [ + [ + "kyc.expected_monthly_cash" + ], + [ + "kyc.expected_monthly_cash", + "kyc.business_type" + ] + ], + "reference_minutes": 33, + "reference_policy": "fast", + "held_out": false, + "tier": "benign", + "screen": "ALERT A-8971 rule R-STR-01 \u2014 Three or more cash deposits just under the reporting threshold within ten days\n fired 2026-04-26\n triggering transactions:\n T-213 2026-04-16 cash_in 9,440.00 USD branch BR-052\n T-217 2026-04-21 cash_in 8,950.00 USD branch BR-052\n T-219 2026-04-22 cash_in 8,880.00 USD branch BR-052\nKYC FILE C-9647 Oakhill Kitchen Group\n kyc.account_opened = 2019-07-11\n kyc.business_type = pizzeria\n kyc.country = United States\n kyc.customer_type = business\n kyc.expected_activity = Cash-intensive food service; daily takings deposited most trading days.\n kyc.expected_monthly_cash = 86,000.00 USD\n kyc.kyc_refreshed = 2026-03-13\n kyc.pep = False\n kyc.prior_alerts = 4\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 492,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 42,400 0 0 0 0 13,935 18\n 2025-06 38,160 0 0 0 0 12,582 19\n 2025-07 44,260 0 0 0 0 15,928 18\n 2025-08 41,680 0 0 0 0 13,940 18\n 2025-09 44,950 0 0 0 0 15,498 17\n 2025-10 35,860 0 0 0 0 13,284 19\n 2025-11 42,370 0 0 0 0 12,792 19\n 2025-12 39,280 0 0 0 0 13,776 19\n 2026-01 39,860 0 0 0 0 13,940 19\n 2026-02 39,280 0 0 0 0 12,718 18\n 2026-03 36,510 0 0 0 0 11,693 19\n 2026-04 71,720 0 0 0 0 14,218 18\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 out 48 txns 103,976.00 USD\n CP-2 out 12 txns 60,331.50 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "{\"action\": \"close\", \"reason\": \"CONSISTENT_WITH_PROFILE\", \"cites\": [\"kyc.expected_monthly_cash\"], \"note\": \"Cash volume within the expectation on file; band deposits are ordinary takings.\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "close", + "reason": "CONSISTENT_WITH_PROFILE", + "cites": [ + "kyc.expected_monthly_cash" + ], + "note": "Cash volume within the expectation on file; band deposits are ordinary takings." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/fast-s2.json b/public/traces/alert-triage/fast-s2.json new file mode 100644 index 0000000..ca632c3 --- /dev/null +++ b/public/traces/alert-triage/fast-s2.json @@ -0,0 +1,83 @@ +{ + "runId": "fast-s2", + "seed": 2, + "model": "fast-analyst", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 1.0, + "evidence": 1.0 + }, + "metrics": { + "hours_spent": 0.7, + "reference_hours": 0.7, + "lookups": 1.0, + "turns_used": 2.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": null + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "benign", + "template": "B6", + "overlay": null, + "typology": null, + "planted": [ + [ + "doc.D-aded" + ] + ], + "reference_minutes": 42, + "reference_policy": "fast", + "held_out": false, + "tier": "benign", + "screen": "ALERT A-9760 rule R-DORM-05 \u2014 Large credit to an account with no material activity for nine months\n fired 2026-04-23\n triggering transactions:\n T-4 2026-04-18 wire_in 356,000.00 USD CP-1\n T-5 2026-04-19 wire_out 291,900.00 USD CP-2\nKYC FILE C-7059 Gabriela Kowalczyk\n kyc.account_opened = 2012-03-09\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Retired; low activity expected.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2025-12-12\n kyc.occupation = retired\n kyc.pep = False\n kyc.prior_alerts = 2\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 31,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 0 0 0 0 0 0 0\n 2025-06 0 0 0 0 0 0 0\n 2025-07 0 170 0 0 0 0 1\n 2025-08 0 0 0 0 0 0 0\n 2025-09 0 0 0 0 0 0 0\n 2025-10 0 0 0 0 0 0 0\n 2025-11 0 0 0 0 0 0 0\n 2025-12 0 120 0 0 0 0 1\n 2026-01 0 170 0 0 0 0 1\n 2026-02 0 0 0 0 0 0 0\n 2026-03 0 0 0 0 0 0 0\n 2026-04 0 0 356,000 291,900 0 0 2\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 in 1 txns 356,000.00 USD\n CP-2 out 1 txns 291,900.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "{\"action\": \"lookup\", \"what\": \"documents\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "documents" + }, + "rejection": null, + "observation": "DOCUMENTS ON FILE:\n doc.D-3211 [identity] Government-issued identity document, verified at onboarding\n doc.D-aded [executor_letter] Letter from Alicia Haddad, executor, via Eskildsen & Laurent LLP, client trust account: distribution from the estate of Aaron Eskildsen to the named beneficiary.\n\nHours spent so far: 0.65. You have 7 replies left." + } + }, + { + "reply": "{\"action\": \"close\", \"reason\": \"DOCUMENTED_SOURCE_OF_FUNDS\", \"cites\": [\"doc.D-aded\"], \"note\": \"Source of funds documented: executor_letter.\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "close", + "reason": "DOCUMENTED_SOURCE_OF_FUNDS", + "cites": [ + "doc.D-aded" + ], + "note": "Source of funds documented: executor_letter." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/fast-s3.json b/public/traces/alert-triage/fast-s3.json new file mode 100644 index 0000000..7558e00 --- /dev/null +++ b/public/traces/alert-triage/fast-s3.json @@ -0,0 +1,68 @@ +{ + "runId": "fast-s3", + "seed": 3, + "model": "fast-analyst", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 1.0, + "evidence": 1.0 + }, + "metrics": { + "hours_spent": 0.55, + "reference_hours": 0.55, + "lookups": 0.0, + "turns_used": 1.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": null + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "benign", + "template": "B4", + "overlay": null, + "typology": null, + "planted": [ + [ + "kyc.expected_monthly_cash" + ], + [ + "kyc.expected_monthly_cash", + "kyc.business_type" + ] + ], + "reference_minutes": 33, + "reference_policy": "fast", + "held_out": false, + "tier": "benign", + "screen": "ALERT A-9453 rule R-STR-01 \u2014 Three or more cash deposits just under the reporting threshold within ten days\n fired 2026-04-23\n triggering transactions:\n T-200 2026-04-14 cash_in 9,770.00 USD branch BR-089\n T-201 2026-04-15 cash_in 9,430.00 USD branch BR-014\n T-206 2026-04-22 cash_in 8,640.00 USD branch BR-089\nKYC FILE C-4718 Lakeside Kitchen LLC\n kyc.account_opened = 2016-10-03\n kyc.business_type = pizzeria\n kyc.country = United States\n kyc.customer_type = business\n kyc.expected_activity = Cash-intensive food service; daily takings deposited most trading days.\n kyc.expected_monthly_cash = 97,000.00 USD\n kyc.kyc_refreshed = 2025-07-13\n kyc.pep = False\n kyc.prior_alerts = 0\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 588,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 52,750 0 0 0 0 16,816 17\n 2025-06 45,240 0 0 0 0 16,317 17\n 2025-07 49,740 0 0 0 0 16,493 17\n 2025-08 40,470 0 0 0 0 16,944 17\n 2025-09 49,720 0 0 0 0 18,522 19\n 2025-10 44,630 0 0 0 0 17,199 16\n 2025-11 49,060 0 0 0 0 18,992 16\n 2025-12 52,570 0 0 0 0 17,992 19\n 2026-01 50,320 0 0 0 0 15,038 17\n 2026-02 51,710 0 0 0 0 17,826 16\n 2026-03 52,240 0 0 0 0 16,934 17\n 2026-04 78,100 0 0 0 0 18,673 18\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 out 48 txns 132,956.60 USD\n CP-2 out 12 txns 74,793.60 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "{\"action\": \"close\", \"reason\": \"CONSISTENT_WITH_PROFILE\", \"cites\": [\"kyc.expected_monthly_cash\"], \"note\": \"Cash volume within the expectation on file; band deposits are ordinary takings.\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "close", + "reason": "CONSISTENT_WITH_PROFILE", + "cites": [ + "kyc.expected_monthly_cash" + ], + "note": "Cash volume within the expectation on file; band deposits are ordinary takings." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/fast-s4.json b/public/traces/alert-triage/fast-s4.json new file mode 100644 index 0000000..ca29c78 --- /dev/null +++ b/public/traces/alert-triage/fast-s4.json @@ -0,0 +1,76 @@ +{ + "runId": "fast-s4", + "seed": 4, + "model": "fast-analyst", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 1.0, + "evidence": 0.8571428571428571 + }, + "metrics": { + "hours_spent": 0.55, + "reference_hours": 0.55, + "lookups": 0.0, + "turns_used": 1.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": 1.0 + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "suspicious", + "template": "S1", + "overlay": null, + "typology": "STRUCTURING", + "planted": [ + [ + "T-46", + "T-47", + "T-49", + "T-50", + "T-52", + "T-53", + "T-54", + "T-56" + ] + ], + "reference_minutes": 33, + "reference_policy": "fast", + "held_out": false, + "tier": "visible", + "screen": "ALERT A-8199 rule R-STR-01 \u2014 Three or more cash deposits just under the reporting threshold within ten days\n fired 2026-06-14\n triggering transactions:\n T-49 2026-06-05 cash_in 9,320.00 USD branch BR-027\n T-50 2026-06-06 cash_in 9,820.00 USD branch BR-031\n T-52 2026-06-07 cash_in 9,880.00 USD branch BR-068\n T-53 2026-06-09 cash_in 9,730.00 USD branch BR-027\n T-54 2026-06-10 cash_in 9,680.00 USD branch BR-031\n T-56 2026-06-11 cash_in 9,850.00 USD branch BR-068\nKYC FILE C-5480 Bilal Quintero\n kyc.account_opened = 2017-03-29\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Salaried individual; payroll deposits and routine spending.\n kyc.expected_monthly_cash = 100.00 USD\n kyc.kyc_refreshed = 2026-05-04\n kyc.occupation = delivery driver\n kyc.pep = False\n kyc.prior_alerts = 3\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 45,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-07 0 220 0 0 3,410 0 3\n 2025-08 0 480 0 0 3,382 0 3\n 2025-09 0 570 0 0 3,444 0 4\n 2025-10 0 570 0 0 3,454 0 5\n 2025-11 0 660 0 0 3,460 0 4\n 2025-12 0 550 0 0 3,403 0 3\n 2026-01 0 680 0 0 3,351 0 5\n 2026-02 0 490 0 0 3,445 0 4\n 2026-03 0 540 0 0 3,433 0 4\n 2026-04 0 430 0 0 3,387 0 4\n 2026-05 0 660 0 0 3,411 0 5\n 2026-06 77,210 420 0 66,400 3,348 0 13\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-2 out 1 txns 66,400.00 USD\n CP-1 in 12 txns 40,928.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "{\"action\": \"escalate\", \"typology\": \"STRUCTURING\", \"cites\": [\"T-49\", \"T-50\", \"T-52\", \"T-53\", \"T-54\", \"T-56\"], \"narrative\": \"Repeated cash deposits just under the threshold across branches.\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "escalate", + "typology": "STRUCTURING", + "cites": [ + "T-49", + "T-50", + "T-52", + "T-53", + "T-54", + "T-56" + ], + "narrative": "Repeated cash deposits just under the threshold across branches." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/fast-s5.json b/public/traces/alert-triage/fast-s5.json new file mode 100644 index 0000000..79ef415 --- /dev/null +++ b/public/traces/alert-triage/fast-s5.json @@ -0,0 +1,87 @@ +{ + "runId": "fast-s5", + "seed": 5, + "model": "fast-analyst", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 1.0, + "evidence": 1.0 + }, + "metrics": { + "hours_spent": 0.7, + "reference_hours": 0.7, + "lookups": 1.0, + "turns_used": 2.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": null + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "benign", + "template": "B3", + "overlay": null, + "typology": null, + "planted": [ + [ + "doc.D-8578" + ], + [ + "doc.D-8578", + "kyc.occupation" + ] + ], + "reference_minutes": 42, + "reference_policy": "fast", + "held_out": false, + "tier": "benign", + "screen": "ALERT A-6081 rule R-VEL-02 \u2014 Funds received and moved out again within 48 hours\n fired 2026-03-22\n triggering transactions:\n T-52 2026-03-18 wire_in 252,000.00 USD CP-2\n T-54 2026-03-19 wire_out 219,200.00 USD CP-3\nKYC FILE C-7077 Andre Nakamura\n kyc.account_opened = 2022-01-15\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Salaried individual; payroll deposits and routine spending.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2025-04-22\n kyc.occupation = teacher\n kyc.pep = False\n kyc.prior_alerts = 3\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 106,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-04 0 590 0 0 8,866 0 4\n 2025-05 0 820 0 0 8,921 0 4\n 2025-06 0 920 0 0 8,952 0 5\n 2025-07 0 680 0 0 8,870 0 3\n 2025-08 0 1,140 0 0 8,906 0 5\n 2025-09 0 390 0 0 8,870 0 4\n 2025-10 0 850 0 0 8,922 0 4\n 2025-11 0 1,020 0 0 8,916 0 5\n 2025-12 0 950 0 0 8,901 0 5\n 2026-01 0 880 0 0 8,854 0 4\n 2026-02 0 1,170 0 0 8,916 0 5\n 2026-03 0 720 252,000 219,200 8,956 0 6\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-2 in 1 txns 252,000.00 USD\n CP-3 out 1 txns 219,200.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "{\"action\": \"lookup\", \"what\": \"documents\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "documents" + }, + "rejection": null, + "observation": "DOCUMENTS ON FILE:\n doc.D-5045 [identity] Government-issued identity document, verified at onboarding\n doc.D-8578 [closing_statement] Closing statement \u2014 sale of 266 Birch Avenue; net proceeds to seller disbursed by Iwata Title & Escrow.\n\nHours spent so far: 0.65. You have 7 replies left." + } + }, + { + "reply": "{\"action\": \"close\", \"reason\": \"DOCUMENTED_SOURCE_OF_FUNDS\", \"cites\": [\"doc.D-8578\"], \"note\": \"Source of funds documented: closing_statement.\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "close", + "reason": "DOCUMENTED_SOURCE_OF_FUNDS", + "cites": [ + "doc.D-8578" + ], + "note": "Source of funds documented: closing_statement." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/fast-s6.json b/public/traces/alert-triage/fast-s6.json new file mode 100644 index 0000000..bd4369d --- /dev/null +++ b/public/traces/alert-triage/fast-s6.json @@ -0,0 +1,69 @@ +{ + "runId": "fast-s6", + "seed": 6, + "model": "fast-analyst", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 1.0, + "evidence": 1.0 + }, + "metrics": { + "hours_spent": 0.55, + "reference_hours": 0.55, + "lookups": 0.0, + "turns_used": 1.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": null + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "benign", + "template": "B1", + "overlay": null, + "typology": null, + "planted": [ + [ + "kyc.expected_activity", + "kyc.expected_monthly_cash" + ], + [ + "kyc.expected_monthly_cash" + ] + ], + "reference_minutes": 33, + "reference_policy": "fast", + "held_out": false, + "tier": "benign", + "screen": "ALERT A-7014 rule R-CASH-03 \u2014 Monthly cash deposits materially above the trailing six-month average\n fired 2026-05-15\n triggering transactions:\n T-83 2026-05-02 cash_in 2,140.00 USD branch BR-102\n T-84 2026-05-03 cash_in 2,930.00 USD branch BR-102\n T-85 2026-05-04 cash_in 2,610.00 USD branch BR-031\n T-86 2026-05-05 cash_in 2,290.00 USD branch BR-031\n T-87 2026-05-06 cash_in 2,930.00 USD branch BR-031\n T-88 2026-05-08 cash_in 2,190.00 USD branch BR-031\n T-89 2026-05-09 cash_in 2,270.00 USD branch BR-031\n T-90 2026-05-10 cash_in 1,970.00 USD branch BR-102\n T-91 2026-05-11 cash_in 2,540.00 USD branch BR-031\n T-92 2026-05-12 cash_in 2,540.00 USD branch BR-031\n T-93 2026-05-13 cash_in 1,970.00 USD branch BR-102\n T-94 2026-05-14 cash_in 2,930.00 USD branch BR-031\nKYC FILE C-3047 Lakeside Landscaping Inc.\n kyc.account_opened = 2013-08-05\n kyc.business_type = landscaping services\n kyc.country = United States\n kyc.customer_type = business\n kyc.expected_activity = Seasonal business; cash receipts peak April to September.\n kyc.expected_monthly_cash = 38,000.00 USD\n kyc.kyc_refreshed = 2025-04-30\n kyc.pep = False\n kyc.prior_alerts = 2\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 266,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-06 28,130 0 0 2,900 0 7,716 10\n 2025-07 24,190 0 0 0 0 8,652 11\n 2025-08 27,600 0 0 0 0 7,538 10\n 2025-09 22,400 0 0 6,300 0 6,691 13\n 2025-10 5,480 0 0 0 0 1,686 4\n 2025-11 7,530 0 0 0 0 1,673 4\n 2025-12 6,520 0 0 3,900 0 2,119 5\n 2026-01 8,260 0 0 0 0 1,811 5\n 2026-02 5,740 0 0 0 0 1,989 4\n 2026-03 6,100 0 0 3,300 0 2,094 6\n 2026-04 24,540 0 0 0 0 5,537 10\n 2026-05 29,310 0 0 0 0 9,777 13\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 out 12 txns 57,289.16 USD\n CP-2 out 4 txns 16,400.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "{\"action\": \"close\", \"reason\": \"CONSISTENT_WITH_PROFILE\", \"cites\": [\"kyc.expected_activity\", \"kyc.expected_monthly_cash\"], \"note\": \"Seasonal peak the file anticipated; within expected monthly cash.\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "close", + "reason": "CONSISTENT_WITH_PROFILE", + "cites": [ + "kyc.expected_activity", + "kyc.expected_monthly_cash" + ], + "note": "Seasonal peak the file anticipated; within expected monthly cash." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/fast-s7.json b/public/traces/alert-triage/fast-s7.json new file mode 100644 index 0000000..f0b2956 --- /dev/null +++ b/public/traces/alert-triage/fast-s7.json @@ -0,0 +1,83 @@ +{ + "runId": "fast-s7", + "seed": 7, + "model": "fast-analyst", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 1.0, + "evidence": 1.0 + }, + "metrics": { + "hours_spent": 0.7, + "reference_hours": 0.7, + "lookups": 1.0, + "turns_used": 2.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": null + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "benign", + "template": "B6", + "overlay": null, + "typology": null, + "planted": [ + [ + "doc.D-b931" + ] + ], + "reference_minutes": 42, + "reference_policy": "fast", + "held_out": false, + "tier": "benign", + "screen": "ALERT A-5644 rule R-DORM-05 \u2014 Large credit to an account with no material activity for nine months\n fired 2026-06-14\n triggering transactions:\n T-4 2026-06-07 wire_in 249,000.00 USD CP-1\n T-5 2026-06-10 wire_out 134,400.00 USD CP-2\nKYC FILE C-1843 Leila Bergstrom\n kyc.account_opened = 2017-03-20\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Retired; low activity expected.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2026-01-13\n kyc.occupation = retired\n kyc.pep = False\n kyc.prior_alerts = 0\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 33,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-07 0 0 0 0 0 0 0\n 2025-08 0 0 0 0 0 0 0\n 2025-09 0 60 0 0 0 0 1\n 2025-10 0 0 0 0 0 0 0\n 2025-11 0 150 0 0 0 0 1\n 2025-12 0 0 0 0 0 0 0\n 2026-01 0 0 0 0 0 0 0\n 2026-02 0 0 0 0 0 0 0\n 2026-03 0 50 0 0 0 0 1\n 2026-04 0 0 0 0 0 0 0\n 2026-05 0 0 0 0 0 0 0\n 2026-06 0 0 249,000 134,400 0 0 2\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 in 1 txns 249,000.00 USD\n CP-2 out 1 txns 134,400.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "{\"action\": \"lookup\", \"what\": \"documents\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "documents" + }, + "rejection": null, + "observation": "DOCUMENTS ON FILE:\n doc.D-1dc2 [identity] Government-issued identity document, verified at onboarding\n doc.D-b931 [executor_letter] Letter from Hana Sandoval, executor, via Xiang & Farrow LLP, client trust account: distribution from the estate of Elena Okafor to the named beneficiary.\n\nHours spent so far: 0.65. You have 7 replies left." + } + }, + { + "reply": "{\"action\": \"close\", \"reason\": \"DOCUMENTED_SOURCE_OF_FUNDS\", \"cites\": [\"doc.D-b931\"], \"note\": \"Source of funds documented: executor_letter.\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "close", + "reason": "DOCUMENTED_SOURCE_OF_FUNDS", + "cites": [ + "doc.D-b931" + ], + "note": "Source of funds documented: executor_letter." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/fast-s9.json b/public/traces/alert-triage/fast-s9.json new file mode 100644 index 0000000..7ab625d --- /dev/null +++ b/public/traces/alert-triage/fast-s9.json @@ -0,0 +1,87 @@ +{ + "runId": "fast-s9", + "seed": 9, + "model": "fast-analyst", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 0.0, + "hours": 0.0, + "evidence": 0.0 + }, + "metrics": { + "hours_spent": 0.7, + "reference_hours": 2.05, + "lookups": 1.0, + "turns_used": 2.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 1.0, + "typology_match": null + }, + "truncated": false, + "outcome": "failed", + "info": { + "label": "suspicious", + "template": "B3", + "overlay": "funnel", + "typology": "FUNNEL_ACCOUNT", + "planted": [ + [ + "CP-3", + "T-52" + ], + [ + "T-52" + ] + ], + "reference_minutes": 123, + "reference_policy": "targeted", + "held_out": false, + "tier": "hidden", + "screen": "ALERT A-9537 rule R-VEL-02 \u2014 Funds received and moved out again within 48 hours\n fired 2026-04-19\n triggering transactions:\n T-51 2026-04-16 wire_in 462,000.00 USD CP-2\n T-52 2026-04-17 wire_out 448,100.00 USD CP-3\nKYC FILE C-9673 Samir Tanaka\n kyc.account_opened = 2019-12-21\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Salaried individual; payroll deposits and routine spending.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2025-04-04\n kyc.occupation = electrician\n kyc.pep = False\n kyc.prior_alerts = 4\n kyc.prior_sars = 1\n kyc.remittance_declared = none declared\n kyc.stated_income = 84,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 0 1,090 0 0 6,958 0 5\n 2025-06 0 630 0 0 7,009 0 5\n 2025-07 0 780 0 0 6,957 0 4\n 2025-08 0 470 0 0 6,989 0 3\n 2025-09 0 1,040 0 0 6,959 0 5\n 2025-10 0 740 0 0 6,996 0 4\n 2025-11 0 860 0 0 6,947 0 5\n 2025-12 0 900 0 0 6,940 0 5\n 2026-01 0 650 0 0 7,025 0 3\n 2026-02 0 460 0 0 7,026 0 3\n 2026-03 0 620 0 0 6,962 0 3\n 2026-04 0 960 462,000 448,100 6,944 0 7\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-2 in 1 txns 462,000.00 USD\n CP-3 out 1 txns 448,100.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "{\"action\": \"lookup\", \"what\": \"documents\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "documents" + }, + "rejection": null, + "observation": "DOCUMENTS ON FILE:\n doc.D-8b75 [identity] Government-issued identity document, verified at onboarding\n doc.D-838c [closing_statement] Closing statement \u2014 sale of 82 Alder Street; net proceeds to seller disbursed by Okafor Title & Escrow.\n\nHours spent so far: 0.65. You have 7 replies left." + } + }, + { + "reply": "{\"action\": \"close\", \"reason\": \"DOCUMENTED_SOURCE_OF_FUNDS\", \"cites\": [\"doc.D-838c\"], \"note\": \"Source of funds documented: closing_statement.\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "close", + "reason": "DOCUMENTED_SOURCE_OF_FUNDS", + "cites": [ + "doc.D-838c" + ], + "note": "Source of funds documented: closing_statement." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/targeted-s1.json b/public/traces/alert-triage/targeted-s1.json new file mode 100644 index 0000000..efcec54 --- /dev/null +++ b/public/traces/alert-triage/targeted-s1.json @@ -0,0 +1,146 @@ +{ + "runId": "targeted-s1", + "seed": 1, + "model": "targeted-analyst", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 0.2682926829268293, + "evidence": 1.0 + }, + "metrics": { + "hours_spent": 2.05, + "reference_hours": 0.55, + "lookups": 4.0, + "turns_used": 5.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": null + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "benign", + "template": "B4", + "overlay": null, + "typology": null, + "planted": [ + [ + "kyc.expected_monthly_cash" + ], + [ + "kyc.expected_monthly_cash", + "kyc.business_type" + ] + ], + "reference_minutes": 33, + "reference_policy": "fast", + "held_out": false, + "tier": "benign", + "screen": "ALERT A-8971 rule R-STR-01 \u2014 Three or more cash deposits just under the reporting threshold within ten days\n fired 2026-04-26\n triggering transactions:\n T-213 2026-04-16 cash_in 9,440.00 USD branch BR-052\n T-217 2026-04-21 cash_in 8,950.00 USD branch BR-052\n T-219 2026-04-22 cash_in 8,880.00 USD branch BR-052\nKYC FILE C-9647 Oakhill Kitchen Group\n kyc.account_opened = 2019-07-11\n kyc.business_type = pizzeria\n kyc.country = United States\n kyc.customer_type = business\n kyc.expected_activity = Cash-intensive food service; daily takings deposited most trading days.\n kyc.expected_monthly_cash = 86,000.00 USD\n kyc.kyc_refreshed = 2026-03-13\n kyc.pep = False\n kyc.prior_alerts = 4\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 492,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 42,400 0 0 0 0 13,935 18\n 2025-06 38,160 0 0 0 0 12,582 19\n 2025-07 44,260 0 0 0 0 15,928 18\n 2025-08 41,680 0 0 0 0 13,940 18\n 2025-09 44,950 0 0 0 0 15,498 17\n 2025-10 35,860 0 0 0 0 13,284 19\n 2025-11 42,370 0 0 0 0 12,792 19\n 2025-12 39,280 0 0 0 0 13,776 19\n 2026-01 39,860 0 0 0 0 13,940 19\n 2026-02 39,280 0 0 0 0 12,718 18\n 2026-03 36,510 0 0 0 0 11,693 19\n 2026-04 71,720 0 0 0 0 14,218 18\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 out 48 txns 103,976.00 USD\n CP-2 out 12 txns 60,331.50 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "{\"action\": \"lookup\", \"what\": \"documents\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "documents" + }, + "rejection": null, + "observation": "DOCUMENTS ON FILE:\n doc.D-e0ed [registration] Business registration \u2014 Oakhill Kitchen Group\n doc.D-000c [licence] Food service licence, renewed annually\n\nHours spent so far: 0.65. You have 7 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "prior_alerts" + }, + "rejection": null, + "observation": "PRIOR ALERTS:\n A-3217 R-WIRE-04 fired 2024-07-08 closed after 7 day(s) by Beatriz Grimaldi\n Closed: destination and amount match the remittance declared at onboarding; within stated income.\n A-4028 R-DORM-05 fired 2024-10-12 closed after 9 day(s) by Mateo Jensen\n Closed: reactivation explained by a documented source of funds; document reviewed and retained.\n A-2622 R-VEL-02 fired 2025-01-17 closed after 5 day(s) by Samir Pereira\n Closed: pass-through matches the settlement terms in the agreement on file; counterparties verified.\n A-1389 R-STR-01 fired 2026-03-06 closed after 3 day(s) by Marta Eskildsen\n Closed: deposits are documented daily takings; amounts vary with trading days; within kyc.expected_monthly_cash.\n\nHours spent so far: 1.10. You have 6 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-1\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-1" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-1 Highland Produce Co.\n type business; jurisdiction United States; first seen 2025-05-06\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-5, T-7, T-11, T-12, T-20, T-22, T-29, T-33, T-42, T-50, T-51, T-53, T-64, T-66, T-69, T-71, T-75, T-81, T-85, T-90, T-94, T-95, T-97, T-101, T-114, T-116, T-120, T-121, T-130, T-133, T-135, T-141, T-150, T-152, T-157, T-163, T-168, T-176, T-178, T-182, T-192, T-195, T-198, T-201, T-206, T-211, T-212, T-216\n\nHours spent so far: 1.55. You have 5 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-2" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-2 Highland Foods Co.\n type business; jurisdiction United States; first seen 2025-05-03\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-3, T-25, T-38, T-61, T-76, T-92, T-113, T-131, T-153, T-171, T-189, T-208\n\nHours spent so far: 2.00. You have 4 replies left." + } + }, + { + "reply": "{\"action\": \"close\", \"reason\": \"CONSISTENT_WITH_PROFILE\", \"cites\": [\"kyc.expected_monthly_cash\"], \"note\": \"Cash volume within the expectation on file; band deposits are ordinary takings.\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "close", + "reason": "CONSISTENT_WITH_PROFILE", + "cites": [ + "kyc.expected_monthly_cash" + ], + "note": "Cash volume within the expectation on file; band deposits are ordinary takings." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/targeted-s2.json b/public/traces/alert-triage/targeted-s2.json new file mode 100644 index 0000000..63d6ea7 --- /dev/null +++ b/public/traces/alert-triage/targeted-s2.json @@ -0,0 +1,142 @@ +{ + "runId": "targeted-s2", + "seed": 2, + "model": "targeted-analyst", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 0.34146341463414637, + "evidence": 1.0 + }, + "metrics": { + "hours_spent": 2.05, + "reference_hours": 0.7, + "lookups": 4.0, + "turns_used": 5.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": null + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "benign", + "template": "B6", + "overlay": null, + "typology": null, + "planted": [ + [ + "doc.D-aded" + ] + ], + "reference_minutes": 42, + "reference_policy": "fast", + "held_out": false, + "tier": "benign", + "screen": "ALERT A-9760 rule R-DORM-05 \u2014 Large credit to an account with no material activity for nine months\n fired 2026-04-23\n triggering transactions:\n T-4 2026-04-18 wire_in 356,000.00 USD CP-1\n T-5 2026-04-19 wire_out 291,900.00 USD CP-2\nKYC FILE C-7059 Gabriela Kowalczyk\n kyc.account_opened = 2012-03-09\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Retired; low activity expected.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2025-12-12\n kyc.occupation = retired\n kyc.pep = False\n kyc.prior_alerts = 2\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 31,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 0 0 0 0 0 0 0\n 2025-06 0 0 0 0 0 0 0\n 2025-07 0 170 0 0 0 0 1\n 2025-08 0 0 0 0 0 0 0\n 2025-09 0 0 0 0 0 0 0\n 2025-10 0 0 0 0 0 0 0\n 2025-11 0 0 0 0 0 0 0\n 2025-12 0 120 0 0 0 0 1\n 2026-01 0 170 0 0 0 0 1\n 2026-02 0 0 0 0 0 0 0\n 2026-03 0 0 0 0 0 0 0\n 2026-04 0 0 356,000 291,900 0 0 2\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 in 1 txns 356,000.00 USD\n CP-2 out 1 txns 291,900.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "{\"action\": \"lookup\", \"what\": \"documents\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "documents" + }, + "rejection": null, + "observation": "DOCUMENTS ON FILE:\n doc.D-3211 [identity] Government-issued identity document, verified at onboarding\n doc.D-aded [executor_letter] Letter from Alicia Haddad, executor, via Eskildsen & Laurent LLP, client trust account: distribution from the estate of Aaron Eskildsen to the named beneficiary.\n\nHours spent so far: 0.65. You have 7 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "prior_alerts" + }, + "rejection": null, + "observation": "PRIOR ALERTS:\n A-4278 R-DORM-05 fired 2024-12-23 closed after 9 day(s) by Amara Tanaka\n Closed: reactivation explained by a documented source of funds; document reviewed and retained.\n A-4841 R-WIRE-04 fired 2026-02-02 closed after 5 day(s) by Alicia Haddad\n Closed: destination and amount match the remittance declared at onboarding; within stated income.\n\nHours spent so far: 1.10. You have 6 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-1\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-1" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-1 Eskildsen & Laurent LLP, client trust account\n type law_firm_trust; jurisdiction United States; first seen 2026-04-18\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-4\n\nHours spent so far: 1.55. You have 5 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-2" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-2 Brightwater Securities\n type brokerage; jurisdiction United States; first seen 2026-04-19\n other alerted accounts sharing this counterparty in twelve months: 1\n transactions: T-5\n\nHours spent so far: 2.00. You have 4 replies left." + } + }, + { + "reply": "{\"action\": \"close\", \"reason\": \"DOCUMENTED_SOURCE_OF_FUNDS\", \"cites\": [\"doc.D-aded\"], \"note\": \"Source of funds documented: executor_letter.\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "close", + "reason": "DOCUMENTED_SOURCE_OF_FUNDS", + "cites": [ + "doc.D-aded" + ], + "note": "Source of funds documented: executor_letter." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/targeted-s3.json b/public/traces/alert-triage/targeted-s3.json new file mode 100644 index 0000000..c5b8bb8 --- /dev/null +++ b/public/traces/alert-triage/targeted-s3.json @@ -0,0 +1,146 @@ +{ + "runId": "targeted-s3", + "seed": 3, + "model": "targeted-analyst", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 0.2682926829268293, + "evidence": 1.0 + }, + "metrics": { + "hours_spent": 2.05, + "reference_hours": 0.55, + "lookups": 4.0, + "turns_used": 5.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": null + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "benign", + "template": "B4", + "overlay": null, + "typology": null, + "planted": [ + [ + "kyc.expected_monthly_cash" + ], + [ + "kyc.expected_monthly_cash", + "kyc.business_type" + ] + ], + "reference_minutes": 33, + "reference_policy": "fast", + "held_out": false, + "tier": "benign", + "screen": "ALERT A-9453 rule R-STR-01 \u2014 Three or more cash deposits just under the reporting threshold within ten days\n fired 2026-04-23\n triggering transactions:\n T-200 2026-04-14 cash_in 9,770.00 USD branch BR-089\n T-201 2026-04-15 cash_in 9,430.00 USD branch BR-014\n T-206 2026-04-22 cash_in 8,640.00 USD branch BR-089\nKYC FILE C-4718 Lakeside Kitchen LLC\n kyc.account_opened = 2016-10-03\n kyc.business_type = pizzeria\n kyc.country = United States\n kyc.customer_type = business\n kyc.expected_activity = Cash-intensive food service; daily takings deposited most trading days.\n kyc.expected_monthly_cash = 97,000.00 USD\n kyc.kyc_refreshed = 2025-07-13\n kyc.pep = False\n kyc.prior_alerts = 0\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 588,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 52,750 0 0 0 0 16,816 17\n 2025-06 45,240 0 0 0 0 16,317 17\n 2025-07 49,740 0 0 0 0 16,493 17\n 2025-08 40,470 0 0 0 0 16,944 17\n 2025-09 49,720 0 0 0 0 18,522 19\n 2025-10 44,630 0 0 0 0 17,199 16\n 2025-11 49,060 0 0 0 0 18,992 16\n 2025-12 52,570 0 0 0 0 17,992 19\n 2026-01 50,320 0 0 0 0 15,038 17\n 2026-02 51,710 0 0 0 0 17,826 16\n 2026-03 52,240 0 0 0 0 16,934 17\n 2026-04 78,100 0 0 0 0 18,673 18\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 out 48 txns 132,956.60 USD\n CP-2 out 12 txns 74,793.60 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "{\"action\": \"lookup\", \"what\": \"documents\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "documents" + }, + "rejection": null, + "observation": "DOCUMENTS ON FILE:\n doc.D-f244 [registration] Business registration \u2014 Lakeside Kitchen LLC\n doc.D-ee54 [licence] Food service licence, renewed annually\n\nHours spent so far: 0.65. You have 7 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "prior_alerts" + }, + "rejection": null, + "observation": "PRIOR ALERTS: none on file.\n\nHours spent so far: 1.10. You have 6 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-1\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-1" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-1 Blue Ridge Produce LLC\n type business; jurisdiction United States; first seen 2025-05-08\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-4, T-9, T-10, T-13, T-23, T-25, T-28, T-30, T-35, T-41, T-48, T-51, T-55, T-57, T-63, T-66, T-70, T-74, T-75, T-79, T-92, T-95, T-99, T-101, T-104, T-106, T-111, T-113, T-125, T-127, T-128, T-133, T-145, T-146, T-150, T-153, T-160, T-165, T-168, T-171, T-174, T-176, T-179, T-184, T-194, T-196, T-199, T-204\n\nHours spent so far: 1.55. You have 5 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-2" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-2 Copperfield Foods Group\n type business; jurisdiction United States; first seen 2025-05-03\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-2, T-21, T-39, T-56, T-76, T-90, T-108, T-122, T-139, T-157, T-177, T-190\n\nHours spent so far: 2.00. You have 4 replies left." + } + }, + { + "reply": "{\"action\": \"close\", \"reason\": \"CONSISTENT_WITH_PROFILE\", \"cites\": [\"kyc.expected_monthly_cash\"], \"note\": \"Cash volume within the expectation on file; band deposits are ordinary takings.\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "close", + "reason": "CONSISTENT_WITH_PROFILE", + "cites": [ + "kyc.expected_monthly_cash" + ], + "note": "Cash volume within the expectation on file; band deposits are ordinary takings." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/targeted-s4.json b/public/traces/alert-triage/targeted-s4.json new file mode 100644 index 0000000..1c00ff7 --- /dev/null +++ b/public/traces/alert-triage/targeted-s4.json @@ -0,0 +1,76 @@ +{ + "runId": "targeted-s4", + "seed": 4, + "model": "targeted-analyst", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 1.0, + "evidence": 0.8571428571428571 + }, + "metrics": { + "hours_spent": 0.55, + "reference_hours": 0.55, + "lookups": 0.0, + "turns_used": 1.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": 1.0 + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "suspicious", + "template": "S1", + "overlay": null, + "typology": "STRUCTURING", + "planted": [ + [ + "T-46", + "T-47", + "T-49", + "T-50", + "T-52", + "T-53", + "T-54", + "T-56" + ] + ], + "reference_minutes": 33, + "reference_policy": "fast", + "held_out": false, + "tier": "visible", + "screen": "ALERT A-8199 rule R-STR-01 \u2014 Three or more cash deposits just under the reporting threshold within ten days\n fired 2026-06-14\n triggering transactions:\n T-49 2026-06-05 cash_in 9,320.00 USD branch BR-027\n T-50 2026-06-06 cash_in 9,820.00 USD branch BR-031\n T-52 2026-06-07 cash_in 9,880.00 USD branch BR-068\n T-53 2026-06-09 cash_in 9,730.00 USD branch BR-027\n T-54 2026-06-10 cash_in 9,680.00 USD branch BR-031\n T-56 2026-06-11 cash_in 9,850.00 USD branch BR-068\nKYC FILE C-5480 Bilal Quintero\n kyc.account_opened = 2017-03-29\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Salaried individual; payroll deposits and routine spending.\n kyc.expected_monthly_cash = 100.00 USD\n kyc.kyc_refreshed = 2026-05-04\n kyc.occupation = delivery driver\n kyc.pep = False\n kyc.prior_alerts = 3\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 45,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-07 0 220 0 0 3,410 0 3\n 2025-08 0 480 0 0 3,382 0 3\n 2025-09 0 570 0 0 3,444 0 4\n 2025-10 0 570 0 0 3,454 0 5\n 2025-11 0 660 0 0 3,460 0 4\n 2025-12 0 550 0 0 3,403 0 3\n 2026-01 0 680 0 0 3,351 0 5\n 2026-02 0 490 0 0 3,445 0 4\n 2026-03 0 540 0 0 3,433 0 4\n 2026-04 0 430 0 0 3,387 0 4\n 2026-05 0 660 0 0 3,411 0 5\n 2026-06 77,210 420 0 66,400 3,348 0 13\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-2 out 1 txns 66,400.00 USD\n CP-1 in 12 txns 40,928.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "{\"action\": \"escalate\", \"typology\": \"STRUCTURING\", \"cites\": [\"T-49\", \"T-50\", \"T-52\", \"T-53\", \"T-54\", \"T-56\"], \"narrative\": \"Repeated cash deposits just under the threshold across branches.\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "escalate", + "typology": "STRUCTURING", + "cites": [ + "T-49", + "T-50", + "T-52", + "T-53", + "T-54", + "T-56" + ], + "narrative": "Repeated cash deposits just under the threshold across branches." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/targeted-s5.json b/public/traces/alert-triage/targeted-s5.json new file mode 100644 index 0000000..c8fe347 --- /dev/null +++ b/public/traces/alert-triage/targeted-s5.json @@ -0,0 +1,146 @@ +{ + "runId": "targeted-s5", + "seed": 5, + "model": "targeted-analyst", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 0.34146341463414637, + "evidence": 1.0 + }, + "metrics": { + "hours_spent": 2.05, + "reference_hours": 0.7, + "lookups": 4.0, + "turns_used": 5.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": null + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "benign", + "template": "B3", + "overlay": null, + "typology": null, + "planted": [ + [ + "doc.D-8578" + ], + [ + "doc.D-8578", + "kyc.occupation" + ] + ], + "reference_minutes": 42, + "reference_policy": "fast", + "held_out": false, + "tier": "benign", + "screen": "ALERT A-6081 rule R-VEL-02 \u2014 Funds received and moved out again within 48 hours\n fired 2026-03-22\n triggering transactions:\n T-52 2026-03-18 wire_in 252,000.00 USD CP-2\n T-54 2026-03-19 wire_out 219,200.00 USD CP-3\nKYC FILE C-7077 Andre Nakamura\n kyc.account_opened = 2022-01-15\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Salaried individual; payroll deposits and routine spending.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2025-04-22\n kyc.occupation = teacher\n kyc.pep = False\n kyc.prior_alerts = 3\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 106,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-04 0 590 0 0 8,866 0 4\n 2025-05 0 820 0 0 8,921 0 4\n 2025-06 0 920 0 0 8,952 0 5\n 2025-07 0 680 0 0 8,870 0 3\n 2025-08 0 1,140 0 0 8,906 0 5\n 2025-09 0 390 0 0 8,870 0 4\n 2025-10 0 850 0 0 8,922 0 4\n 2025-11 0 1,020 0 0 8,916 0 5\n 2025-12 0 950 0 0 8,901 0 5\n 2026-01 0 880 0 0 8,854 0 4\n 2026-02 0 1,170 0 0 8,916 0 5\n 2026-03 0 720 252,000 219,200 8,956 0 6\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-2 in 1 txns 252,000.00 USD\n CP-3 out 1 txns 219,200.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "{\"action\": \"lookup\", \"what\": \"documents\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "documents" + }, + "rejection": null, + "observation": "DOCUMENTS ON FILE:\n doc.D-5045 [identity] Government-issued identity document, verified at onboarding\n doc.D-8578 [closing_statement] Closing statement \u2014 sale of 266 Birch Avenue; net proceeds to seller disbursed by Iwata Title & Escrow.\n\nHours spent so far: 0.65. You have 7 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "prior_alerts" + }, + "rejection": null, + "observation": "PRIOR ALERTS:\n A-4824 R-VEL-02 fired 2025-03-19 closed after 4 day(s) by Jamal Whitlock\n Closed: pass-through matches the settlement terms in the agreement on file; counterparties verified.\n A-1410 R-WIRE-04 fired 2025-04-24 closed after 2 day(s) by Lars Farrow\n Closed: destination and amount match the remittance declared at onboarding; within stated income.\n A-2317 R-CASH-03 fired 2025-09-02 closed after 6 day(s) by Chidi Zapata\n Closed: seasonal peak consistent with kyc.expected_activity; volume within stated expectation.\n\nHours spent so far: 1.10. You have 6 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-2" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-2 Iwata Title & Escrow\n type title_company; jurisdiction United States; first seen 2026-03-18\n other alerted accounts sharing this counterparty in twelve months: 1\n transactions: T-52\n\nHours spent so far: 1.55. You have 5 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-3\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-3" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-3 Harbor Securities\n type brokerage; jurisdiction United States; first seen 2026-03-19\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-54\n\nHours spent so far: 2.00. You have 4 replies left." + } + }, + { + "reply": "{\"action\": \"close\", \"reason\": \"DOCUMENTED_SOURCE_OF_FUNDS\", \"cites\": [\"doc.D-8578\"], \"note\": \"Source of funds documented: closing_statement.\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "close", + "reason": "DOCUMENTED_SOURCE_OF_FUNDS", + "cites": [ + "doc.D-8578" + ], + "note": "Source of funds documented: closing_statement." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/targeted-s6.json b/public/traces/alert-triage/targeted-s6.json new file mode 100644 index 0000000..2ed5991 --- /dev/null +++ b/public/traces/alert-triage/targeted-s6.json @@ -0,0 +1,147 @@ +{ + "runId": "targeted-s6", + "seed": 6, + "model": "targeted-analyst", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 0.2682926829268293, + "evidence": 1.0 + }, + "metrics": { + "hours_spent": 2.05, + "reference_hours": 0.55, + "lookups": 4.0, + "turns_used": 5.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": null + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "benign", + "template": "B1", + "overlay": null, + "typology": null, + "planted": [ + [ + "kyc.expected_activity", + "kyc.expected_monthly_cash" + ], + [ + "kyc.expected_monthly_cash" + ] + ], + "reference_minutes": 33, + "reference_policy": "fast", + "held_out": false, + "tier": "benign", + "screen": "ALERT A-7014 rule R-CASH-03 \u2014 Monthly cash deposits materially above the trailing six-month average\n fired 2026-05-15\n triggering transactions:\n T-83 2026-05-02 cash_in 2,140.00 USD branch BR-102\n T-84 2026-05-03 cash_in 2,930.00 USD branch BR-102\n T-85 2026-05-04 cash_in 2,610.00 USD branch BR-031\n T-86 2026-05-05 cash_in 2,290.00 USD branch BR-031\n T-87 2026-05-06 cash_in 2,930.00 USD branch BR-031\n T-88 2026-05-08 cash_in 2,190.00 USD branch BR-031\n T-89 2026-05-09 cash_in 2,270.00 USD branch BR-031\n T-90 2026-05-10 cash_in 1,970.00 USD branch BR-102\n T-91 2026-05-11 cash_in 2,540.00 USD branch BR-031\n T-92 2026-05-12 cash_in 2,540.00 USD branch BR-031\n T-93 2026-05-13 cash_in 1,970.00 USD branch BR-102\n T-94 2026-05-14 cash_in 2,930.00 USD branch BR-031\nKYC FILE C-3047 Lakeside Landscaping Inc.\n kyc.account_opened = 2013-08-05\n kyc.business_type = landscaping services\n kyc.country = United States\n kyc.customer_type = business\n kyc.expected_activity = Seasonal business; cash receipts peak April to September.\n kyc.expected_monthly_cash = 38,000.00 USD\n kyc.kyc_refreshed = 2025-04-30\n kyc.pep = False\n kyc.prior_alerts = 2\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 266,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-06 28,130 0 0 2,900 0 7,716 10\n 2025-07 24,190 0 0 0 0 8,652 11\n 2025-08 27,600 0 0 0 0 7,538 10\n 2025-09 22,400 0 0 6,300 0 6,691 13\n 2025-10 5,480 0 0 0 0 1,686 4\n 2025-11 7,530 0 0 0 0 1,673 4\n 2025-12 6,520 0 0 3,900 0 2,119 5\n 2026-01 8,260 0 0 0 0 1,811 5\n 2026-02 5,740 0 0 0 0 1,989 4\n 2026-03 6,100 0 0 3,300 0 2,094 6\n 2026-04 24,540 0 0 0 0 5,537 10\n 2026-05 29,310 0 0 0 0 9,777 13\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 out 12 txns 57,289.16 USD\n CP-2 out 4 txns 16,400.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "{\"action\": \"lookup\", \"what\": \"documents\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "documents" + }, + "rejection": null, + "observation": "DOCUMENTS ON FILE:\n doc.D-ab7d [registration] Business registration \u2014 Lakeside Landscaping Inc.\n doc.D-f811 [licence] Trade licence, landscaping services, renewed annually\n\nHours spent so far: 0.65. You have 7 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "prior_alerts" + }, + "rejection": null, + "observation": "PRIOR ALERTS:\n A-4237 R-CASH-03 fired 2024-08-10 closed after 7 day(s) by Bilal Rasmussen\n Closed: seasonal peak consistent with kyc.expected_activity; volume within stated expectation.\n A-2077 R-WIRE-04 fired 2025-04-23 closed after 3 day(s) by Dana Pereira\n Closed: destination and amount match the remittance declared at onboarding; within stated income.\n\nHours spent so far: 1.10. You have 6 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-1\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-1" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-1 Lakeside Supply LLC\n type business; jurisdiction United States; first seen 2025-06-25\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-9, T-17, T-28, T-40, T-48, T-52, T-55, T-62, T-64, T-70, T-80, T-95\n\nHours spent so far: 1.55. You have 5 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-2" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-2 Harbor Equipment Group\n type business; jurisdiction United States; first seen 2025-06-06\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-3, T-33, T-53, T-67\n\nHours spent so far: 2.00. You have 4 replies left." + } + }, + { + "reply": "{\"action\": \"close\", \"reason\": \"CONSISTENT_WITH_PROFILE\", \"cites\": [\"kyc.expected_activity\", \"kyc.expected_monthly_cash\"], \"note\": \"Seasonal peak the file anticipated; within expected monthly cash.\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "close", + "reason": "CONSISTENT_WITH_PROFILE", + "cites": [ + "kyc.expected_activity", + "kyc.expected_monthly_cash" + ], + "note": "Seasonal peak the file anticipated; within expected monthly cash." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/targeted-s7.json b/public/traces/alert-triage/targeted-s7.json new file mode 100644 index 0000000..a11472e --- /dev/null +++ b/public/traces/alert-triage/targeted-s7.json @@ -0,0 +1,142 @@ +{ + "runId": "targeted-s7", + "seed": 7, + "model": "targeted-analyst", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 0.34146341463414637, + "evidence": 1.0 + }, + "metrics": { + "hours_spent": 2.05, + "reference_hours": 0.7, + "lookups": 4.0, + "turns_used": 5.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": null + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "benign", + "template": "B6", + "overlay": null, + "typology": null, + "planted": [ + [ + "doc.D-b931" + ] + ], + "reference_minutes": 42, + "reference_policy": "fast", + "held_out": false, + "tier": "benign", + "screen": "ALERT A-5644 rule R-DORM-05 \u2014 Large credit to an account with no material activity for nine months\n fired 2026-06-14\n triggering transactions:\n T-4 2026-06-07 wire_in 249,000.00 USD CP-1\n T-5 2026-06-10 wire_out 134,400.00 USD CP-2\nKYC FILE C-1843 Leila Bergstrom\n kyc.account_opened = 2017-03-20\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Retired; low activity expected.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2026-01-13\n kyc.occupation = retired\n kyc.pep = False\n kyc.prior_alerts = 0\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 33,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-07 0 0 0 0 0 0 0\n 2025-08 0 0 0 0 0 0 0\n 2025-09 0 60 0 0 0 0 1\n 2025-10 0 0 0 0 0 0 0\n 2025-11 0 150 0 0 0 0 1\n 2025-12 0 0 0 0 0 0 0\n 2026-01 0 0 0 0 0 0 0\n 2026-02 0 0 0 0 0 0 0\n 2026-03 0 50 0 0 0 0 1\n 2026-04 0 0 0 0 0 0 0\n 2026-05 0 0 0 0 0 0 0\n 2026-06 0 0 249,000 134,400 0 0 2\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 in 1 txns 249,000.00 USD\n CP-2 out 1 txns 134,400.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "{\"action\": \"lookup\", \"what\": \"documents\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "documents" + }, + "rejection": null, + "observation": "DOCUMENTS ON FILE:\n doc.D-1dc2 [identity] Government-issued identity document, verified at onboarding\n doc.D-b931 [executor_letter] Letter from Hana Sandoval, executor, via Xiang & Farrow LLP, client trust account: distribution from the estate of Elena Okafor to the named beneficiary.\n\nHours spent so far: 0.65. You have 7 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "prior_alerts" + }, + "rejection": null, + "observation": "PRIOR ALERTS: none on file.\n\nHours spent so far: 1.10. You have 6 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-1\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-1" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-1 Xiang & Farrow LLP, client trust account\n type law_firm_trust; jurisdiction United States; first seen 2026-06-07\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-4\n\nHours spent so far: 1.55. You have 5 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-2" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-2 Oakhill Securities\n type brokerage; jurisdiction United States; first seen 2026-06-10\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-5\n\nHours spent so far: 2.00. You have 4 replies left." + } + }, + { + "reply": "{\"action\": \"close\", \"reason\": \"DOCUMENTED_SOURCE_OF_FUNDS\", \"cites\": [\"doc.D-b931\"], \"note\": \"Source of funds documented: executor_letter.\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "close", + "reason": "DOCUMENTED_SOURCE_OF_FUNDS", + "cites": [ + "doc.D-b931" + ], + "note": "Source of funds documented: executor_letter." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/targeted-s9.json b/public/traces/alert-triage/targeted-s9.json new file mode 100644 index 0000000..873cdd5 --- /dev/null +++ b/public/traces/alert-triage/targeted-s9.json @@ -0,0 +1,147 @@ +{ + "runId": "targeted-s9", + "seed": 9, + "model": "targeted-analyst", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 1.0, + "evidence": 1.0 + }, + "metrics": { + "hours_spent": 2.05, + "reference_hours": 2.05, + "lookups": 4.0, + "turns_used": 5.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": 1.0 + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "suspicious", + "template": "B3", + "overlay": "funnel", + "typology": "FUNNEL_ACCOUNT", + "planted": [ + [ + "CP-3", + "T-52" + ], + [ + "T-52" + ] + ], + "reference_minutes": 123, + "reference_policy": "targeted", + "held_out": false, + "tier": "hidden", + "screen": "ALERT A-9537 rule R-VEL-02 \u2014 Funds received and moved out again within 48 hours\n fired 2026-04-19\n triggering transactions:\n T-51 2026-04-16 wire_in 462,000.00 USD CP-2\n T-52 2026-04-17 wire_out 448,100.00 USD CP-3\nKYC FILE C-9673 Samir Tanaka\n kyc.account_opened = 2019-12-21\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Salaried individual; payroll deposits and routine spending.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2025-04-04\n kyc.occupation = electrician\n kyc.pep = False\n kyc.prior_alerts = 4\n kyc.prior_sars = 1\n kyc.remittance_declared = none declared\n kyc.stated_income = 84,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 0 1,090 0 0 6,958 0 5\n 2025-06 0 630 0 0 7,009 0 5\n 2025-07 0 780 0 0 6,957 0 4\n 2025-08 0 470 0 0 6,989 0 3\n 2025-09 0 1,040 0 0 6,959 0 5\n 2025-10 0 740 0 0 6,996 0 4\n 2025-11 0 860 0 0 6,947 0 5\n 2025-12 0 900 0 0 6,940 0 5\n 2026-01 0 650 0 0 7,025 0 3\n 2026-02 0 460 0 0 7,026 0 3\n 2026-03 0 620 0 0 6,962 0 3\n 2026-04 0 960 462,000 448,100 6,944 0 7\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-2 in 1 txns 462,000.00 USD\n CP-3 out 1 txns 448,100.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "{\"action\": \"lookup\", \"what\": \"documents\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "documents" + }, + "rejection": null, + "observation": "DOCUMENTS ON FILE:\n doc.D-8b75 [identity] Government-issued identity document, verified at onboarding\n doc.D-838c [closing_statement] Closing statement \u2014 sale of 82 Alder Street; net proceeds to seller disbursed by Okafor Title & Escrow.\n\nHours spent so far: 0.65. You have 7 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "prior_alerts" + }, + "rejection": null, + "observation": "PRIOR ALERTS:\n A-2906 R-VEL-02 fired 2025-01-09 closed after 4 day(s) by Hana Xiang\n Closed: pass-through matches the settlement terms in the agreement on file; counterparties verified.\n A-2547 R-DORM-05 fired 2025-07-01 escalated \u2014 report filed after 2 day(s) by Jamal Ferreira\n Escalated: pattern not explained by the file at the time.\n A-2855 R-DORM-05 fired 2025-07-17 closed after 8 day(s) by Carlos Zapata\n Closed: reactivation explained by a documented source of funds; document reviewed and retained.\n A-2904 R-VEL-02 fired 2025-12-05 closed after 5 day(s) by Dana Grimaldi\n Closed: pass-through matches the settlement terms in the agreement on file; counterparties verified.\n\nHours spent so far: 1.10. You have 6 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-2" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-2 Okafor Title & Escrow\n type title_company; jurisdiction United States; first seen 2026-04-16\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-51\n\nHours spent so far: 1.55. You have 5 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-3\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-3" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-3 Summit Securities\n type brokerage; jurisdiction United States; first seen 2026-04-17\n other alerted accounts sharing this counterparty in twelve months: 8\n transactions: T-52\n\nHours spent so far: 2.00. You have 4 replies left." + } + }, + { + "reply": "{\"action\": \"escalate\", \"typology\": \"FUNNEL_ACCOUNT\", \"cites\": [\"CP-3\", \"T-52\"], \"narrative\": \"CP-3 is shared with 8 other alerted accounts; funds converge on it.\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "escalate", + "typology": "FUNNEL_ACCOUNT", + "cites": [ + "CP-3", + "T-52" + ], + "narrative": "CP-3 is shared with 8 other alerted accounts; funds converge on it." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/thorough-s1.json b/public/traces/alert-triage/thorough-s1.json new file mode 100644 index 0000000..e7b58e2 --- /dev/null +++ b/public/traces/alert-triage/thorough-s1.json @@ -0,0 +1,186 @@ +{ + "runId": "thorough-s1", + "seed": 1, + "model": "thorough-analyst", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 0.1864406779661017, + "evidence": 1.0 + }, + "metrics": { + "hours_spent": 2.95, + "reference_hours": 0.55, + "lookups": 6.0, + "turns_used": 7.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": null + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "benign", + "template": "B4", + "overlay": null, + "typology": null, + "planted": [ + [ + "kyc.expected_monthly_cash" + ], + [ + "kyc.expected_monthly_cash", + "kyc.business_type" + ] + ], + "reference_minutes": 33, + "reference_policy": "fast", + "held_out": false, + "tier": "benign", + "screen": "ALERT A-8971 rule R-STR-01 \u2014 Three or more cash deposits just under the reporting threshold within ten days\n fired 2026-04-26\n triggering transactions:\n T-213 2026-04-16 cash_in 9,440.00 USD branch BR-052\n T-217 2026-04-21 cash_in 8,950.00 USD branch BR-052\n T-219 2026-04-22 cash_in 8,880.00 USD branch BR-052\nKYC FILE C-9647 Oakhill Kitchen Group\n kyc.account_opened = 2019-07-11\n kyc.business_type = pizzeria\n kyc.country = United States\n kyc.customer_type = business\n kyc.expected_activity = Cash-intensive food service; daily takings deposited most trading days.\n kyc.expected_monthly_cash = 86,000.00 USD\n kyc.kyc_refreshed = 2026-03-13\n kyc.pep = False\n kyc.prior_alerts = 4\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 492,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 42,400 0 0 0 0 13,935 18\n 2025-06 38,160 0 0 0 0 12,582 19\n 2025-07 44,260 0 0 0 0 15,928 18\n 2025-08 41,680 0 0 0 0 13,940 18\n 2025-09 44,950 0 0 0 0 15,498 17\n 2025-10 35,860 0 0 0 0 13,284 19\n 2025-11 42,370 0 0 0 0 12,792 19\n 2025-12 39,280 0 0 0 0 13,776 19\n 2026-01 39,860 0 0 0 0 13,940 19\n 2026-02 39,280 0 0 0 0 12,718 18\n 2026-03 36,510 0 0 0 0 11,693 19\n 2026-04 71,720 0 0 0 0 14,218 18\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 out 48 txns 103,976.00 USD\n CP-2 out 12 txns 60,331.50 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "{\"action\": \"lookup\", \"what\": \"documents\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "documents" + }, + "rejection": null, + "observation": "DOCUMENTS ON FILE:\n doc.D-e0ed [registration] Business registration \u2014 Oakhill Kitchen Group\n doc.D-000c [licence] Food service licence, renewed annually\n\nHours spent so far: 0.65. You have 7 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "prior_alerts" + }, + "rejection": null, + "observation": "PRIOR ALERTS:\n A-3217 R-WIRE-04 fired 2024-07-08 closed after 7 day(s) by Beatriz Grimaldi\n Closed: destination and amount match the remittance declared at onboarding; within stated income.\n A-4028 R-DORM-05 fired 2024-10-12 closed after 9 day(s) by Mateo Jensen\n Closed: reactivation explained by a documented source of funds; document reviewed and retained.\n A-2622 R-VEL-02 fired 2025-01-17 closed after 5 day(s) by Samir Pereira\n Closed: pass-through matches the settlement terms in the agreement on file; counterparties verified.\n A-1389 R-STR-01 fired 2026-03-06 closed after 3 day(s) by Marta Eskildsen\n Closed: deposits are documented daily takings; amounts vary with trading days; within kyc.expected_monthly_cash.\n\nHours spent so far: 1.10. You have 6 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-1\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-1" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-1 Highland Produce Co.\n type business; jurisdiction United States; first seen 2025-05-06\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-5, T-7, T-11, T-12, T-20, T-22, T-29, T-33, T-42, T-50, T-51, T-53, T-64, T-66, T-69, T-71, T-75, T-81, T-85, T-90, T-94, T-95, T-97, T-101, T-114, T-116, T-120, T-121, T-130, T-133, T-135, T-141, T-150, T-152, T-157, T-163, T-168, T-176, T-178, T-182, T-192, T-195, T-198, T-201, T-206, T-211, T-212, T-216\n\nHours spent so far: 1.55. You have 5 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-2" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-2 Highland Foods Co.\n type business; jurisdiction United States; first seen 2025-05-03\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-3, T-25, T-38, T-61, T-76, T-92, T-113, T-131, T-153, T-171, T-189, T-208\n\nHours spent so far: 2.00. You have 4 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-04\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "history", + "month": "2026-04" + }, + "rejection": null, + "observation": "HISTORY 2026-04: 18 transactions\n T-204 2026-04-01 cash_in 3,380.00 USD branch BR-052\n T-205 2026-04-04 cash_in 3,420.00 USD branch BR-052\n T-206 2026-04-07 ach_out 2,091.00 USD CP-1\n T-207 2026-04-08 cash_in 4,850.00 USD branch BR-052\n T-208 2026-04-10 ach_out 5,436.60 USD CP-2\n T-209 2026-04-11 cash_in 5,220.00 USD branch BR-052\n T-210 2026-04-13 cash_in 4,850.00 USD branch BR-052\n T-211 2026-04-13 ach_out 2,509.20 USD CP-1\n T-212 2026-04-14 ach_out 1,672.80 USD CP-1\n T-213 2026-04-16 cash_in 9,440.00 USD branch BR-052\n T-214 2026-04-16 cash_in 5,010.00 USD branch BR-052\n T-215 2026-04-19 cash_in 3,970.00 USD branch BR-052\n T-216 2026-04-19 ach_out 2,509.20 USD CP-1\n T-217 2026-04-21 cash_in 8,950.00 USD branch BR-052\n T-218 2026-04-21 cash_in 4,640.00 USD branch BR-052\n T-219 2026-04-22 cash_in 8,880.00 USD branch BR-052\n T-220 2026-04-24 cash_in 4,720.00 USD branch BR-052\n T-221 2026-04-25 cash_in 4,390.00 USD branch BR-014\n\nHours spent so far: 2.45. You have 3 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-03\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "history", + "month": "2026-03" + }, + "rejection": null, + "observation": "HISTORY 2026-03: 19 transactions\n T-185 2026-03-01 cash_in 2,020.00 USD branch BR-052\n T-186 2026-03-02 cash_in 2,850.00 USD branch BR-014\n T-187 2026-03-03 cash_in 2,990.00 USD branch BR-014\n T-188 2026-03-04 cash_in 2,500.00 USD branch BR-014\n T-189 2026-03-04 ach_out 4,526.40 USD CP-2\n T-190 2026-03-08 cash_in 2,900.00 USD branch BR-014\n T-191 2026-03-11 cash_in 2,610.00 USD branch BR-052\n T-192 2026-03-11 ach_out 1,886.00 USD CP-1\n T-193 2026-03-12 cash_in 2,150.00 USD branch BR-052\n T-194 2026-03-13 cash_in 2,230.00 USD branch BR-014\n T-195 2026-03-15 ach_out 1,886.00 USD CP-1\n T-196 2026-03-17 cash_in 2,310.00 USD branch BR-014\n T-197 2026-03-20 cash_in 2,690.00 USD branch BR-014\n T-198 2026-03-20 ach_out 1,886.00 USD CP-1\n T-199 2026-03-21 cash_in 3,170.00 USD branch BR-052\n T-200 2026-03-22 cash_in 2,900.00 USD branch BR-014\n T-201 2026-03-23 ach_out 1,508.80 USD CP-1\n T-202 2026-03-29 cash_in 2,740.00 USD branch BR-052\n T-203 2026-03-31 cash_in 2,450.00 USD branch BR-014\n\nHours spent so far: 2.90. You have 2 replies left." + } + }, + { + "reply": "{\"action\": \"close\", \"reason\": \"CONSISTENT_WITH_PROFILE\", \"cites\": [\"kyc.expected_monthly_cash\"], \"note\": \"Cash volume within the expectation on file; band deposits are ordinary takings.\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "close", + "reason": "CONSISTENT_WITH_PROFILE", + "cites": [ + "kyc.expected_monthly_cash" + ], + "note": "Cash volume within the expectation on file; band deposits are ordinary takings." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/thorough-s2.json b/public/traces/alert-triage/thorough-s2.json new file mode 100644 index 0000000..638d243 --- /dev/null +++ b/public/traces/alert-triage/thorough-s2.json @@ -0,0 +1,182 @@ +{ + "runId": "thorough-s2", + "seed": 2, + "model": "thorough-analyst", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 0.23728813559322035, + "evidence": 1.0 + }, + "metrics": { + "hours_spent": 2.95, + "reference_hours": 0.7, + "lookups": 6.0, + "turns_used": 7.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": null + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "benign", + "template": "B6", + "overlay": null, + "typology": null, + "planted": [ + [ + "doc.D-aded" + ] + ], + "reference_minutes": 42, + "reference_policy": "fast", + "held_out": false, + "tier": "benign", + "screen": "ALERT A-9760 rule R-DORM-05 \u2014 Large credit to an account with no material activity for nine months\n fired 2026-04-23\n triggering transactions:\n T-4 2026-04-18 wire_in 356,000.00 USD CP-1\n T-5 2026-04-19 wire_out 291,900.00 USD CP-2\nKYC FILE C-7059 Gabriela Kowalczyk\n kyc.account_opened = 2012-03-09\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Retired; low activity expected.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2025-12-12\n kyc.occupation = retired\n kyc.pep = False\n kyc.prior_alerts = 2\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 31,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 0 0 0 0 0 0 0\n 2025-06 0 0 0 0 0 0 0\n 2025-07 0 170 0 0 0 0 1\n 2025-08 0 0 0 0 0 0 0\n 2025-09 0 0 0 0 0 0 0\n 2025-10 0 0 0 0 0 0 0\n 2025-11 0 0 0 0 0 0 0\n 2025-12 0 120 0 0 0 0 1\n 2026-01 0 170 0 0 0 0 1\n 2026-02 0 0 0 0 0 0 0\n 2026-03 0 0 0 0 0 0 0\n 2026-04 0 0 356,000 291,900 0 0 2\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 in 1 txns 356,000.00 USD\n CP-2 out 1 txns 291,900.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "{\"action\": \"lookup\", \"what\": \"documents\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "documents" + }, + "rejection": null, + "observation": "DOCUMENTS ON FILE:\n doc.D-3211 [identity] Government-issued identity document, verified at onboarding\n doc.D-aded [executor_letter] Letter from Alicia Haddad, executor, via Eskildsen & Laurent LLP, client trust account: distribution from the estate of Aaron Eskildsen to the named beneficiary.\n\nHours spent so far: 0.65. You have 7 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "prior_alerts" + }, + "rejection": null, + "observation": "PRIOR ALERTS:\n A-4278 R-DORM-05 fired 2024-12-23 closed after 9 day(s) by Amara Tanaka\n Closed: reactivation explained by a documented source of funds; document reviewed and retained.\n A-4841 R-WIRE-04 fired 2026-02-02 closed after 5 day(s) by Alicia Haddad\n Closed: destination and amount match the remittance declared at onboarding; within stated income.\n\nHours spent so far: 1.10. You have 6 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-1\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-1" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-1 Eskildsen & Laurent LLP, client trust account\n type law_firm_trust; jurisdiction United States; first seen 2026-04-18\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-4\n\nHours spent so far: 1.55. You have 5 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-2" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-2 Brightwater Securities\n type brokerage; jurisdiction United States; first seen 2026-04-19\n other alerted accounts sharing this counterparty in twelve months: 1\n transactions: T-5\n\nHours spent so far: 2.00. You have 4 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-04\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "history", + "month": "2026-04" + }, + "rejection": null, + "observation": "HISTORY 2026-04: 2 transactions\n T-4 2026-04-18 wire_in 356,000.00 USD CP-1\n T-5 2026-04-19 wire_out 291,900.00 USD CP-2\n\nHours spent so far: 2.45. You have 3 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-03\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "history", + "month": "2026-03" + }, + "rejection": null, + "observation": "HISTORY 2026-03: 0 transactions\n\nHours spent so far: 2.90. You have 2 replies left." + } + }, + { + "reply": "{\"action\": \"close\", \"reason\": \"DOCUMENTED_SOURCE_OF_FUNDS\", \"cites\": [\"doc.D-aded\"], \"note\": \"Source of funds documented: executor_letter.\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "close", + "reason": "DOCUMENTED_SOURCE_OF_FUNDS", + "cites": [ + "doc.D-aded" + ], + "note": "Source of funds documented: executor_letter." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/thorough-s3.json b/public/traces/alert-triage/thorough-s3.json new file mode 100644 index 0000000..afddff9 --- /dev/null +++ b/public/traces/alert-triage/thorough-s3.json @@ -0,0 +1,186 @@ +{ + "runId": "thorough-s3", + "seed": 3, + "model": "thorough-analyst", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 0.1864406779661017, + "evidence": 1.0 + }, + "metrics": { + "hours_spent": 2.95, + "reference_hours": 0.55, + "lookups": 6.0, + "turns_used": 7.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": null + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "benign", + "template": "B4", + "overlay": null, + "typology": null, + "planted": [ + [ + "kyc.expected_monthly_cash" + ], + [ + "kyc.expected_monthly_cash", + "kyc.business_type" + ] + ], + "reference_minutes": 33, + "reference_policy": "fast", + "held_out": false, + "tier": "benign", + "screen": "ALERT A-9453 rule R-STR-01 \u2014 Three or more cash deposits just under the reporting threshold within ten days\n fired 2026-04-23\n triggering transactions:\n T-200 2026-04-14 cash_in 9,770.00 USD branch BR-089\n T-201 2026-04-15 cash_in 9,430.00 USD branch BR-014\n T-206 2026-04-22 cash_in 8,640.00 USD branch BR-089\nKYC FILE C-4718 Lakeside Kitchen LLC\n kyc.account_opened = 2016-10-03\n kyc.business_type = pizzeria\n kyc.country = United States\n kyc.customer_type = business\n kyc.expected_activity = Cash-intensive food service; daily takings deposited most trading days.\n kyc.expected_monthly_cash = 97,000.00 USD\n kyc.kyc_refreshed = 2025-07-13\n kyc.pep = False\n kyc.prior_alerts = 0\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 588,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 52,750 0 0 0 0 16,816 17\n 2025-06 45,240 0 0 0 0 16,317 17\n 2025-07 49,740 0 0 0 0 16,493 17\n 2025-08 40,470 0 0 0 0 16,944 17\n 2025-09 49,720 0 0 0 0 18,522 19\n 2025-10 44,630 0 0 0 0 17,199 16\n 2025-11 49,060 0 0 0 0 18,992 16\n 2025-12 52,570 0 0 0 0 17,992 19\n 2026-01 50,320 0 0 0 0 15,038 17\n 2026-02 51,710 0 0 0 0 17,826 16\n 2026-03 52,240 0 0 0 0 16,934 17\n 2026-04 78,100 0 0 0 0 18,673 18\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 out 48 txns 132,956.60 USD\n CP-2 out 12 txns 74,793.60 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "{\"action\": \"lookup\", \"what\": \"documents\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "documents" + }, + "rejection": null, + "observation": "DOCUMENTS ON FILE:\n doc.D-f244 [registration] Business registration \u2014 Lakeside Kitchen LLC\n doc.D-ee54 [licence] Food service licence, renewed annually\n\nHours spent so far: 0.65. You have 7 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "prior_alerts" + }, + "rejection": null, + "observation": "PRIOR ALERTS: none on file.\n\nHours spent so far: 1.10. You have 6 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-1\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-1" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-1 Blue Ridge Produce LLC\n type business; jurisdiction United States; first seen 2025-05-08\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-4, T-9, T-10, T-13, T-23, T-25, T-28, T-30, T-35, T-41, T-48, T-51, T-55, T-57, T-63, T-66, T-70, T-74, T-75, T-79, T-92, T-95, T-99, T-101, T-104, T-106, T-111, T-113, T-125, T-127, T-128, T-133, T-145, T-146, T-150, T-153, T-160, T-165, T-168, T-171, T-174, T-176, T-179, T-184, T-194, T-196, T-199, T-204\n\nHours spent so far: 1.55. You have 5 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-2" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-2 Copperfield Foods Group\n type business; jurisdiction United States; first seen 2025-05-03\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-2, T-21, T-39, T-56, T-76, T-90, T-108, T-122, T-139, T-157, T-177, T-190\n\nHours spent so far: 2.00. You have 4 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-04\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "history", + "month": "2026-04" + }, + "rejection": null, + "observation": "HISTORY 2026-04: 18 transactions\n T-189 2026-04-01 cash_in 4,840.00 USD branch BR-014\n T-190 2026-04-03 ach_out 7,065.80 USD CP-2\n T-191 2026-04-04 cash_in 5,650.00 USD branch BR-014\n T-192 2026-04-05 cash_in 6,050.00 USD branch BR-014\n T-193 2026-04-07 cash_in 4,790.00 USD branch BR-089\n T-194 2026-04-07 ach_out 2,523.50 USD CP-1\n T-195 2026-04-10 cash_in 3,880.00 USD branch BR-089\n T-196 2026-04-10 ach_out 3,028.20 USD CP-1\n T-197 2026-04-11 cash_in 5,340.00 USD branch BR-089\n T-198 2026-04-13 cash_in 5,240.00 USD branch BR-089\n T-199 2026-04-13 ach_out 3,028.20 USD CP-1\n T-200 2026-04-14 cash_in 9,770.00 USD branch BR-089\n T-201 2026-04-15 cash_in 9,430.00 USD branch BR-014\n T-202 2026-04-16 cash_in 3,930.00 USD branch BR-089\n T-203 2026-04-17 cash_in 4,390.00 USD branch BR-014\n T-204 2026-04-20 ach_out 3,028.20 USD CP-1\n T-205 2026-04-21 cash_in 6,150.00 USD branch BR-014\n T-206 2026-04-22 cash_in 8,640.00 USD branch BR-089\n\nHours spent so far: 2.45. You have 3 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-03\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "history", + "month": "2026-03" + }, + "rejection": null, + "observation": "HISTORY 2026-03: 17 transactions\n T-172 2026-03-01 cash_in 5,420.00 USD branch BR-089\n T-173 2026-03-02 cash_in 5,330.00 USD branch BR-089\n T-174 2026-03-04 ach_out 3,175.20 USD CP-1\n T-175 2026-03-05 cash_in 3,740.00 USD branch BR-089\n T-176 2026-03-06 ach_out 3,704.40 USD CP-1\n T-177 2026-03-06 ach_out 5,292.00 USD CP-2\n T-178 2026-03-08 cash_in 4,490.00 USD branch BR-014\n T-179 2026-03-08 ach_out 2,646.00 USD CP-1\n T-180 2026-03-11 cash_in 4,180.00 USD branch BR-089\n T-181 2026-03-12 cash_in 3,660.00 USD branch BR-089\n T-182 2026-03-13 cash_in 3,520.00 USD branch BR-089\n T-183 2026-03-15 cash_in 4,540.00 USD branch BR-089\n T-184 2026-03-15 ach_out 2,116.80 USD CP-1\n T-185 2026-03-21 cash_in 4,320.00 USD branch BR-089\n T-186 2026-03-22 cash_in 5,380.00 USD branch BR-089\n T-187 2026-03-24 cash_in 3,830.00 USD branch BR-089\n T-188 2026-03-26 cash_in 3,830.00 USD branch BR-089\n\nHours spent so far: 2.90. You have 2 replies left." + } + }, + { + "reply": "{\"action\": \"close\", \"reason\": \"CONSISTENT_WITH_PROFILE\", \"cites\": [\"kyc.expected_monthly_cash\"], \"note\": \"Cash volume within the expectation on file; band deposits are ordinary takings.\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "close", + "reason": "CONSISTENT_WITH_PROFILE", + "cites": [ + "kyc.expected_monthly_cash" + ], + "note": "Cash volume within the expectation on file; band deposits are ordinary takings." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/thorough-s4.json b/public/traces/alert-triage/thorough-s4.json new file mode 100644 index 0000000..bc31683 --- /dev/null +++ b/public/traces/alert-triage/thorough-s4.json @@ -0,0 +1,196 @@ +{ + "runId": "thorough-s4", + "seed": 4, + "model": "thorough-analyst", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 0.1864406779661017, + "evidence": 1.0 + }, + "metrics": { + "hours_spent": 2.95, + "reference_hours": 0.55, + "lookups": 6.0, + "turns_used": 7.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": 1.0 + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "suspicious", + "template": "S1", + "overlay": null, + "typology": "STRUCTURING", + "planted": [ + [ + "T-46", + "T-47", + "T-49", + "T-50", + "T-52", + "T-53", + "T-54", + "T-56" + ] + ], + "reference_minutes": 33, + "reference_policy": "fast", + "held_out": false, + "tier": "visible", + "screen": "ALERT A-8199 rule R-STR-01 \u2014 Three or more cash deposits just under the reporting threshold within ten days\n fired 2026-06-14\n triggering transactions:\n T-49 2026-06-05 cash_in 9,320.00 USD branch BR-027\n T-50 2026-06-06 cash_in 9,820.00 USD branch BR-031\n T-52 2026-06-07 cash_in 9,880.00 USD branch BR-068\n T-53 2026-06-09 cash_in 9,730.00 USD branch BR-027\n T-54 2026-06-10 cash_in 9,680.00 USD branch BR-031\n T-56 2026-06-11 cash_in 9,850.00 USD branch BR-068\nKYC FILE C-5480 Bilal Quintero\n kyc.account_opened = 2017-03-29\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Salaried individual; payroll deposits and routine spending.\n kyc.expected_monthly_cash = 100.00 USD\n kyc.kyc_refreshed = 2026-05-04\n kyc.occupation = delivery driver\n kyc.pep = False\n kyc.prior_alerts = 3\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 45,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-07 0 220 0 0 3,410 0 3\n 2025-08 0 480 0 0 3,382 0 3\n 2025-09 0 570 0 0 3,444 0 4\n 2025-10 0 570 0 0 3,454 0 5\n 2025-11 0 660 0 0 3,460 0 4\n 2025-12 0 550 0 0 3,403 0 3\n 2026-01 0 680 0 0 3,351 0 5\n 2026-02 0 490 0 0 3,445 0 4\n 2026-03 0 540 0 0 3,433 0 4\n 2026-04 0 430 0 0 3,387 0 4\n 2026-05 0 660 0 0 3,411 0 5\n 2026-06 77,210 420 0 66,400 3,348 0 13\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-2 out 1 txns 66,400.00 USD\n CP-1 in 12 txns 40,928.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "{\"action\": \"lookup\", \"what\": \"documents\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "documents" + }, + "rejection": null, + "observation": "DOCUMENTS ON FILE:\n doc.D-c608 [identity] Government-issued identity document, verified at onboarding\n\nHours spent so far: 0.65. You have 7 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "prior_alerts" + }, + "rejection": null, + "observation": "PRIOR ALERTS:\n A-1684 R-STR-01 fired 2025-01-17 closed after 5 day(s) by Andre Abara\n Closed: deposits are documented daily takings; amounts vary with trading days; within kyc.expected_monthly_cash.\n A-3624 R-STR-01 fired 2025-03-06 closed after 8 day(s) by Bilal Laurent\n Closed: deposits are documented daily takings; amounts vary with trading days; within kyc.expected_monthly_cash.\n A-2017 R-CASH-03 fired 2025-05-09 closed after 7 day(s) by Sofia Grimaldi\n Closed: seasonal peak consistent with kyc.expected_activity; volume within stated expectation.\n\nHours spent so far: 1.10. You have 6 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-2" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-2 Redfern Holdings Co.\n type business; jurisdiction United States; first seen 2026-06-13\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-57\n\nHours spent so far: 1.55. You have 5 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-1\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-1" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-1 Stonebridge Diner Group\n type business; jurisdiction United States; first seen 2025-07-03\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-1, T-5, T-7, T-11, T-16, T-21, T-23, T-28, T-32, T-36, T-40, T-45\n\nHours spent so far: 2.00. You have 4 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-06\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "history", + "month": "2026-06" + }, + "rejection": null, + "observation": "HISTORY 2026-06: 13 transactions\n T-45 2026-06-02 ach_in 3,348.00 USD CP-1\n T-46 2026-06-02 cash_in 9,400.00 USD branch BR-031\n T-47 2026-06-03 cash_in 9,530.00 USD branch BR-068\n T-48 2026-06-04 cash_out 110.00 USD branch BR-027\n T-49 2026-06-05 cash_in 9,320.00 USD branch BR-027\n T-50 2026-06-06 cash_in 9,820.00 USD branch BR-031\n T-51 2026-06-07 cash_out 150.00 USD branch BR-027\n T-52 2026-06-07 cash_in 9,880.00 USD branch BR-068\n T-53 2026-06-09 cash_in 9,730.00 USD branch BR-027\n T-54 2026-06-10 cash_in 9,680.00 USD branch BR-031\n T-55 2026-06-11 cash_out 160.00 USD branch BR-068\n T-56 2026-06-11 cash_in 9,850.00 USD branch BR-068\n T-57 2026-06-13 wire_out 66,400.00 USD CP-2\n\nHours spent so far: 2.45. You have 3 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-05\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "history", + "month": "2026-05" + }, + "rejection": null, + "observation": "HISTORY 2026-05: 5 transactions\n T-40 2026-05-02 ach_in 3,411.00 USD CP-1\n T-41 2026-05-02 cash_out 190.00 USD branch BR-068\n T-42 2026-05-15 cash_out 120.00 USD branch BR-027\n T-43 2026-05-18 cash_out 150.00 USD branch BR-068\n T-44 2026-05-29 cash_out 200.00 USD branch BR-027\n\nHours spent so far: 2.90. You have 2 replies left." + } + }, + { + "reply": "{\"action\": \"escalate\", \"typology\": \"STRUCTURING\", \"cites\": [\"T-49\", \"T-50\", \"T-52\", \"T-53\", \"T-54\", \"T-56\", \"T-46\", \"T-47\"], \"narrative\": \"Repeated cash deposits just under the threshold across branches.\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "escalate", + "typology": "STRUCTURING", + "cites": [ + "T-49", + "T-50", + "T-52", + "T-53", + "T-54", + "T-56", + "T-46", + "T-47" + ], + "narrative": "Repeated cash deposits just under the threshold across branches." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/thorough-s5.json b/public/traces/alert-triage/thorough-s5.json new file mode 100644 index 0000000..b18b8c9 --- /dev/null +++ b/public/traces/alert-triage/thorough-s5.json @@ -0,0 +1,186 @@ +{ + "runId": "thorough-s5", + "seed": 5, + "model": "thorough-analyst", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 0.23728813559322035, + "evidence": 1.0 + }, + "metrics": { + "hours_spent": 2.95, + "reference_hours": 0.7, + "lookups": 6.0, + "turns_used": 7.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": null + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "benign", + "template": "B3", + "overlay": null, + "typology": null, + "planted": [ + [ + "doc.D-8578" + ], + [ + "doc.D-8578", + "kyc.occupation" + ] + ], + "reference_minutes": 42, + "reference_policy": "fast", + "held_out": false, + "tier": "benign", + "screen": "ALERT A-6081 rule R-VEL-02 \u2014 Funds received and moved out again within 48 hours\n fired 2026-03-22\n triggering transactions:\n T-52 2026-03-18 wire_in 252,000.00 USD CP-2\n T-54 2026-03-19 wire_out 219,200.00 USD CP-3\nKYC FILE C-7077 Andre Nakamura\n kyc.account_opened = 2022-01-15\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Salaried individual; payroll deposits and routine spending.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2025-04-22\n kyc.occupation = teacher\n kyc.pep = False\n kyc.prior_alerts = 3\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 106,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-04 0 590 0 0 8,866 0 4\n 2025-05 0 820 0 0 8,921 0 4\n 2025-06 0 920 0 0 8,952 0 5\n 2025-07 0 680 0 0 8,870 0 3\n 2025-08 0 1,140 0 0 8,906 0 5\n 2025-09 0 390 0 0 8,870 0 4\n 2025-10 0 850 0 0 8,922 0 4\n 2025-11 0 1,020 0 0 8,916 0 5\n 2025-12 0 950 0 0 8,901 0 5\n 2026-01 0 880 0 0 8,854 0 4\n 2026-02 0 1,170 0 0 8,916 0 5\n 2026-03 0 720 252,000 219,200 8,956 0 6\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-2 in 1 txns 252,000.00 USD\n CP-3 out 1 txns 219,200.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "{\"action\": \"lookup\", \"what\": \"documents\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "documents" + }, + "rejection": null, + "observation": "DOCUMENTS ON FILE:\n doc.D-5045 [identity] Government-issued identity document, verified at onboarding\n doc.D-8578 [closing_statement] Closing statement \u2014 sale of 266 Birch Avenue; net proceeds to seller disbursed by Iwata Title & Escrow.\n\nHours spent so far: 0.65. You have 7 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "prior_alerts" + }, + "rejection": null, + "observation": "PRIOR ALERTS:\n A-4824 R-VEL-02 fired 2025-03-19 closed after 4 day(s) by Jamal Whitlock\n Closed: pass-through matches the settlement terms in the agreement on file; counterparties verified.\n A-1410 R-WIRE-04 fired 2025-04-24 closed after 2 day(s) by Lars Farrow\n Closed: destination and amount match the remittance declared at onboarding; within stated income.\n A-2317 R-CASH-03 fired 2025-09-02 closed after 6 day(s) by Chidi Zapata\n Closed: seasonal peak consistent with kyc.expected_activity; volume within stated expectation.\n\nHours spent so far: 1.10. You have 6 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-2" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-2 Iwata Title & Escrow\n type title_company; jurisdiction United States; first seen 2026-03-18\n other alerted accounts sharing this counterparty in twelve months: 1\n transactions: T-52\n\nHours spent so far: 1.55. You have 5 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-3\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-3" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-3 Harbor Securities\n type brokerage; jurisdiction United States; first seen 2026-03-19\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-54\n\nHours spent so far: 2.00. You have 4 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-03\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "history", + "month": "2026-03" + }, + "rejection": null, + "observation": "HISTORY 2026-03: 6 transactions\n T-49 2026-03-02 cash_out 160.00 USD branch BR-052\n T-50 2026-03-04 cash_out 320.00 USD branch BR-052\n T-51 2026-03-05 ach_in 8,956.00 USD CP-1\n T-52 2026-03-18 wire_in 252,000.00 USD CP-2\n T-53 2026-03-19 cash_out 240.00 USD branch BR-089\n T-54 2026-03-19 wire_out 219,200.00 USD CP-3\n\nHours spent so far: 2.45. You have 3 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-02\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "history", + "month": "2026-02" + }, + "rejection": null, + "observation": "HISTORY 2026-02: 5 transactions\n T-44 2026-02-01 ach_in 8,916.00 USD CP-1\n T-45 2026-02-16 cash_out 150.00 USD branch BR-089\n T-46 2026-02-18 cash_out 390.00 USD branch BR-052\n T-47 2026-02-21 cash_out 380.00 USD branch BR-052\n T-48 2026-02-26 cash_out 250.00 USD branch BR-089\n\nHours spent so far: 2.90. You have 2 replies left." + } + }, + { + "reply": "{\"action\": \"close\", \"reason\": \"DOCUMENTED_SOURCE_OF_FUNDS\", \"cites\": [\"doc.D-8578\"], \"note\": \"Source of funds documented: closing_statement.\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "close", + "reason": "DOCUMENTED_SOURCE_OF_FUNDS", + "cites": [ + "doc.D-8578" + ], + "note": "Source of funds documented: closing_statement." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/thorough-s6.json b/public/traces/alert-triage/thorough-s6.json new file mode 100644 index 0000000..ee477c1 --- /dev/null +++ b/public/traces/alert-triage/thorough-s6.json @@ -0,0 +1,187 @@ +{ + "runId": "thorough-s6", + "seed": 6, + "model": "thorough-analyst", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 0.1864406779661017, + "evidence": 1.0 + }, + "metrics": { + "hours_spent": 2.95, + "reference_hours": 0.55, + "lookups": 6.0, + "turns_used": 7.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": null + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "benign", + "template": "B1", + "overlay": null, + "typology": null, + "planted": [ + [ + "kyc.expected_activity", + "kyc.expected_monthly_cash" + ], + [ + "kyc.expected_monthly_cash" + ] + ], + "reference_minutes": 33, + "reference_policy": "fast", + "held_out": false, + "tier": "benign", + "screen": "ALERT A-7014 rule R-CASH-03 \u2014 Monthly cash deposits materially above the trailing six-month average\n fired 2026-05-15\n triggering transactions:\n T-83 2026-05-02 cash_in 2,140.00 USD branch BR-102\n T-84 2026-05-03 cash_in 2,930.00 USD branch BR-102\n T-85 2026-05-04 cash_in 2,610.00 USD branch BR-031\n T-86 2026-05-05 cash_in 2,290.00 USD branch BR-031\n T-87 2026-05-06 cash_in 2,930.00 USD branch BR-031\n T-88 2026-05-08 cash_in 2,190.00 USD branch BR-031\n T-89 2026-05-09 cash_in 2,270.00 USD branch BR-031\n T-90 2026-05-10 cash_in 1,970.00 USD branch BR-102\n T-91 2026-05-11 cash_in 2,540.00 USD branch BR-031\n T-92 2026-05-12 cash_in 2,540.00 USD branch BR-031\n T-93 2026-05-13 cash_in 1,970.00 USD branch BR-102\n T-94 2026-05-14 cash_in 2,930.00 USD branch BR-031\nKYC FILE C-3047 Lakeside Landscaping Inc.\n kyc.account_opened = 2013-08-05\n kyc.business_type = landscaping services\n kyc.country = United States\n kyc.customer_type = business\n kyc.expected_activity = Seasonal business; cash receipts peak April to September.\n kyc.expected_monthly_cash = 38,000.00 USD\n kyc.kyc_refreshed = 2025-04-30\n kyc.pep = False\n kyc.prior_alerts = 2\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 266,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-06 28,130 0 0 2,900 0 7,716 10\n 2025-07 24,190 0 0 0 0 8,652 11\n 2025-08 27,600 0 0 0 0 7,538 10\n 2025-09 22,400 0 0 6,300 0 6,691 13\n 2025-10 5,480 0 0 0 0 1,686 4\n 2025-11 7,530 0 0 0 0 1,673 4\n 2025-12 6,520 0 0 3,900 0 2,119 5\n 2026-01 8,260 0 0 0 0 1,811 5\n 2026-02 5,740 0 0 0 0 1,989 4\n 2026-03 6,100 0 0 3,300 0 2,094 6\n 2026-04 24,540 0 0 0 0 5,537 10\n 2026-05 29,310 0 0 0 0 9,777 13\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 out 12 txns 57,289.16 USD\n CP-2 out 4 txns 16,400.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "{\"action\": \"lookup\", \"what\": \"documents\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "documents" + }, + "rejection": null, + "observation": "DOCUMENTS ON FILE:\n doc.D-ab7d [registration] Business registration \u2014 Lakeside Landscaping Inc.\n doc.D-f811 [licence] Trade licence, landscaping services, renewed annually\n\nHours spent so far: 0.65. You have 7 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "prior_alerts" + }, + "rejection": null, + "observation": "PRIOR ALERTS:\n A-4237 R-CASH-03 fired 2024-08-10 closed after 7 day(s) by Bilal Rasmussen\n Closed: seasonal peak consistent with kyc.expected_activity; volume within stated expectation.\n A-2077 R-WIRE-04 fired 2025-04-23 closed after 3 day(s) by Dana Pereira\n Closed: destination and amount match the remittance declared at onboarding; within stated income.\n\nHours spent so far: 1.10. You have 6 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-1\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-1" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-1 Lakeside Supply LLC\n type business; jurisdiction United States; first seen 2025-06-25\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-9, T-17, T-28, T-40, T-48, T-52, T-55, T-62, T-64, T-70, T-80, T-95\n\nHours spent so far: 1.55. You have 5 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-2" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-2 Harbor Equipment Group\n type business; jurisdiction United States; first seen 2025-06-06\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-3, T-33, T-53, T-67\n\nHours spent so far: 2.00. You have 4 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-05\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "history", + "month": "2026-05" + }, + "rejection": null, + "observation": "HISTORY 2026-05: 13 transactions\n T-83 2026-05-02 cash_in 2,140.00 USD branch BR-102\n T-84 2026-05-03 cash_in 2,930.00 USD branch BR-102\n T-85 2026-05-04 cash_in 2,610.00 USD branch BR-031\n T-86 2026-05-05 cash_in 2,290.00 USD branch BR-031\n T-87 2026-05-06 cash_in 2,930.00 USD branch BR-031\n T-88 2026-05-08 cash_in 2,190.00 USD branch BR-031\n T-89 2026-05-09 cash_in 2,270.00 USD branch BR-031\n T-90 2026-05-10 cash_in 1,970.00 USD branch BR-102\n T-91 2026-05-11 cash_in 2,540.00 USD branch BR-031\n T-92 2026-05-12 cash_in 2,540.00 USD branch BR-031\n T-93 2026-05-13 cash_in 1,970.00 USD branch BR-102\n T-94 2026-05-14 cash_in 2,930.00 USD branch BR-031\n T-95 2026-05-14 ach_out 9,777.43 USD CP-1\n\nHours spent so far: 2.45. You have 3 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-04\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "history", + "month": "2026-04" + }, + "rejection": null, + "observation": "HISTORY 2026-04: 10 transactions\n T-73 2026-04-04 cash_in 3,270.00 USD branch BR-102\n T-74 2026-04-09 cash_in 2,740.00 USD branch BR-031\n T-75 2026-04-12 cash_in 3,180.00 USD branch BR-102\n T-76 2026-04-13 cash_in 2,990.00 USD branch BR-102\n T-77 2026-04-16 cash_in 2,430.00 USD branch BR-102\n T-78 2026-04-18 cash_in 2,570.00 USD branch BR-102\n T-79 2026-04-24 cash_in 2,370.00 USD branch BR-031\n T-80 2026-04-24 ach_out 5,537.66 USD CP-1\n T-81 2026-04-27 cash_in 2,370.00 USD branch BR-031\n T-82 2026-04-30 cash_in 2,620.00 USD branch BR-102\n\nHours spent so far: 2.90. You have 2 replies left." + } + }, + { + "reply": "{\"action\": \"close\", \"reason\": \"CONSISTENT_WITH_PROFILE\", \"cites\": [\"kyc.expected_activity\", \"kyc.expected_monthly_cash\"], \"note\": \"Seasonal peak the file anticipated; within expected monthly cash.\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "close", + "reason": "CONSISTENT_WITH_PROFILE", + "cites": [ + "kyc.expected_activity", + "kyc.expected_monthly_cash" + ], + "note": "Seasonal peak the file anticipated; within expected monthly cash." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/thorough-s7.json b/public/traces/alert-triage/thorough-s7.json new file mode 100644 index 0000000..69c0a2a --- /dev/null +++ b/public/traces/alert-triage/thorough-s7.json @@ -0,0 +1,182 @@ +{ + "runId": "thorough-s7", + "seed": 7, + "model": "thorough-analyst", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 0.23728813559322035, + "evidence": 1.0 + }, + "metrics": { + "hours_spent": 2.95, + "reference_hours": 0.7, + "lookups": 6.0, + "turns_used": 7.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": null + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "benign", + "template": "B6", + "overlay": null, + "typology": null, + "planted": [ + [ + "doc.D-b931" + ] + ], + "reference_minutes": 42, + "reference_policy": "fast", + "held_out": false, + "tier": "benign", + "screen": "ALERT A-5644 rule R-DORM-05 \u2014 Large credit to an account with no material activity for nine months\n fired 2026-06-14\n triggering transactions:\n T-4 2026-06-07 wire_in 249,000.00 USD CP-1\n T-5 2026-06-10 wire_out 134,400.00 USD CP-2\nKYC FILE C-1843 Leila Bergstrom\n kyc.account_opened = 2017-03-20\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Retired; low activity expected.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2026-01-13\n kyc.occupation = retired\n kyc.pep = False\n kyc.prior_alerts = 0\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 33,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-07 0 0 0 0 0 0 0\n 2025-08 0 0 0 0 0 0 0\n 2025-09 0 60 0 0 0 0 1\n 2025-10 0 0 0 0 0 0 0\n 2025-11 0 150 0 0 0 0 1\n 2025-12 0 0 0 0 0 0 0\n 2026-01 0 0 0 0 0 0 0\n 2026-02 0 0 0 0 0 0 0\n 2026-03 0 50 0 0 0 0 1\n 2026-04 0 0 0 0 0 0 0\n 2026-05 0 0 0 0 0 0 0\n 2026-06 0 0 249,000 134,400 0 0 2\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 in 1 txns 249,000.00 USD\n CP-2 out 1 txns 134,400.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "{\"action\": \"lookup\", \"what\": \"documents\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "documents" + }, + "rejection": null, + "observation": "DOCUMENTS ON FILE:\n doc.D-1dc2 [identity] Government-issued identity document, verified at onboarding\n doc.D-b931 [executor_letter] Letter from Hana Sandoval, executor, via Xiang & Farrow LLP, client trust account: distribution from the estate of Elena Okafor to the named beneficiary.\n\nHours spent so far: 0.65. You have 7 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "prior_alerts" + }, + "rejection": null, + "observation": "PRIOR ALERTS: none on file.\n\nHours spent so far: 1.10. You have 6 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-1\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-1" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-1 Xiang & Farrow LLP, client trust account\n type law_firm_trust; jurisdiction United States; first seen 2026-06-07\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-4\n\nHours spent so far: 1.55. You have 5 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-2" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-2 Oakhill Securities\n type brokerage; jurisdiction United States; first seen 2026-06-10\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-5\n\nHours spent so far: 2.00. You have 4 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-06\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "history", + "month": "2026-06" + }, + "rejection": null, + "observation": "HISTORY 2026-06: 2 transactions\n T-4 2026-06-07 wire_in 249,000.00 USD CP-1\n T-5 2026-06-10 wire_out 134,400.00 USD CP-2\n\nHours spent so far: 2.45. You have 3 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-05\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "history", + "month": "2026-05" + }, + "rejection": null, + "observation": "HISTORY 2026-05: 0 transactions\n\nHours spent so far: 2.90. You have 2 replies left." + } + }, + { + "reply": "{\"action\": \"close\", \"reason\": \"DOCUMENTED_SOURCE_OF_FUNDS\", \"cites\": [\"doc.D-b931\"], \"note\": \"Source of funds documented: executor_letter.\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "close", + "reason": "DOCUMENTED_SOURCE_OF_FUNDS", + "cites": [ + "doc.D-b931" + ], + "note": "Source of funds documented: executor_letter." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/alert-triage/thorough-s9.json b/public/traces/alert-triage/thorough-s9.json new file mode 100644 index 0000000..12ba94a --- /dev/null +++ b/public/traces/alert-triage/thorough-s9.json @@ -0,0 +1,187 @@ +{ + "runId": "thorough-s9", + "seed": 9, + "model": "thorough-analyst", + "capturedAt": "2026-08-28", + "rewards": { + "caught": 1.0, + "hours": 0.6949152542372882, + "evidence": 1.0 + }, + "metrics": { + "hours_spent": 2.95, + "reference_hours": 2.05, + "lookups": 6.0, + "turns_used": 7.0, + "rejected_replies": 0.0, + "false_escalation": 0.0, + "missed": 0.0, + "typology_match": 1.0 + }, + "truncated": false, + "outcome": "solved", + "info": { + "label": "suspicious", + "template": "B3", + "overlay": "funnel", + "typology": "FUNNEL_ACCOUNT", + "planted": [ + [ + "CP-3", + "T-52" + ], + [ + "T-52" + ] + ], + "reference_minutes": 123, + "reference_policy": "targeted", + "held_out": false, + "tier": "hidden", + "screen": "ALERT A-9537 rule R-VEL-02 \u2014 Funds received and moved out again within 48 hours\n fired 2026-04-19\n triggering transactions:\n T-51 2026-04-16 wire_in 462,000.00 USD CP-2\n T-52 2026-04-17 wire_out 448,100.00 USD CP-3\nKYC FILE C-9673 Samir Tanaka\n kyc.account_opened = 2019-12-21\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Salaried individual; payroll deposits and routine spending.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2025-04-04\n kyc.occupation = electrician\n kyc.pep = False\n kyc.prior_alerts = 4\n kyc.prior_sars = 1\n kyc.remittance_declared = none declared\n kyc.stated_income = 84,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 0 1,090 0 0 6,958 0 5\n 2025-06 0 630 0 0 7,009 0 5\n 2025-07 0 780 0 0 6,957 0 4\n 2025-08 0 470 0 0 6,989 0 3\n 2025-09 0 1,040 0 0 6,959 0 5\n 2025-10 0 740 0 0 6,996 0 4\n 2025-11 0 860 0 0 6,947 0 5\n 2025-12 0 900 0 0 6,940 0 5\n 2026-01 0 650 0 0 7,025 0 3\n 2026-02 0 460 0 0 7,026 0 3\n 2026-03 0 620 0 0 6,962 0 3\n 2026-04 0 960 462,000 448,100 6,944 0 7\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-2 in 1 txns 462,000.00 USD\n CP-3 out 1 txns 448,100.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left." + }, + "turns": [ + { + "reply": "{\"action\": \"lookup\", \"what\": \"documents\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "documents" + }, + "rejection": null, + "observation": "DOCUMENTS ON FILE:\n doc.D-8b75 [identity] Government-issued identity document, verified at onboarding\n doc.D-838c [closing_statement] Closing statement \u2014 sale of 82 Alder Street; net proceeds to seller disbursed by Okafor Title & Escrow.\n\nHours spent so far: 0.65. You have 7 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "prior_alerts" + }, + "rejection": null, + "observation": "PRIOR ALERTS:\n A-2906 R-VEL-02 fired 2025-01-09 closed after 4 day(s) by Hana Xiang\n Closed: pass-through matches the settlement terms in the agreement on file; counterparties verified.\n A-2547 R-DORM-05 fired 2025-07-01 escalated \u2014 report filed after 2 day(s) by Jamal Ferreira\n Escalated: pattern not explained by the file at the time.\n A-2855 R-DORM-05 fired 2025-07-17 closed after 8 day(s) by Carlos Zapata\n Closed: reactivation explained by a documented source of funds; document reviewed and retained.\n A-2904 R-VEL-02 fired 2025-12-05 closed after 5 day(s) by Dana Grimaldi\n Closed: pass-through matches the settlement terms in the agreement on file; counterparties verified.\n\nHours spent so far: 1.10. You have 6 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-2" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-2 Okafor Title & Escrow\n type title_company; jurisdiction United States; first seen 2026-04-16\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-51\n\nHours spent so far: 1.55. You have 5 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-3\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "counterparty", + "id": "CP-3" + }, + "rejection": null, + "observation": "COUNTERPARTY CP-3 Summit Securities\n type brokerage; jurisdiction United States; first seen 2026-04-17\n other alerted accounts sharing this counterparty in twelve months: 8\n transactions: T-52\n\nHours spent so far: 2.00. You have 4 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-04\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "history", + "month": "2026-04" + }, + "rejection": null, + "observation": "HISTORY 2026-04: 7 transactions\n T-46 2026-04-04 ach_in 6,944.00 USD CP-1\n T-47 2026-04-04 cash_out 220.00 USD branch BR-068\n T-48 2026-04-06 cash_out 280.00 USD branch BR-068\n T-49 2026-04-10 cash_out 340.00 USD branch BR-068\n T-50 2026-04-12 cash_out 120.00 USD branch BR-068\n T-51 2026-04-16 wire_in 462,000.00 USD CP-2\n T-52 2026-04-17 wire_out 448,100.00 USD CP-3\n\nHours spent so far: 2.45. You have 3 replies left." + } + }, + { + "reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-03\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "lookup", + "what": "history", + "month": "2026-03" + }, + "rejection": null, + "observation": "HISTORY 2026-03: 3 transactions\n T-43 2026-03-02 ach_in 6,962.00 USD CP-1\n T-44 2026-03-13 cash_out 370.00 USD branch BR-027\n T-45 2026-03-18 cash_out 250.00 USD branch BR-027\n\nHours spent so far: 2.90. You have 2 replies left." + } + }, + { + "reply": "{\"action\": \"escalate\", \"typology\": \"FUNNEL_ACCOUNT\", \"cites\": [\"CP-3\", \"T-52\"], \"narrative\": \"CP-3 is shared with 8 other alerted accounts; funds converge on it.\"}", + "reasoning": null, + "call": { + "promptTokens": null, + "completionTokens": null, + "reasoningTokens": null, + "durationMs": null, + "finishReason": "generated" + }, + "info": { + "action": { + "action": "escalate", + "typology": "FUNNEL_ACCOUNT", + "cites": [ + "CP-3", + "T-52" + ], + "narrative": "CP-3 is shared with 8 other alerted accounts; funds converge on it." + }, + "rejection": null, + "observation": "Disposition recorded. The episode is over." + } + } + ] +} diff --git a/public/traces/manifest.json b/public/traces/manifest.json index 9223a60..d15b73b 100644 --- a/public/traces/manifest.json +++ b/public/traces/manifest.json @@ -1,4 +1,314 @@ { + "alert-triage": [ + { + "id": "base-off-s1", + "label": "Out of the box", + "path": "/traces/alert-triage/base-off-s1.json", + "kind": "recorded", + "model": "brain-qwen38-dspark", + "capturedAt": "2026-08-28", + "seed": 1 + }, + { + "id": "base-off-s2", + "label": "Out of the box", + "path": "/traces/alert-triage/base-off-s2.json", + "kind": "recorded", + "model": "brain-qwen38-dspark", + "capturedAt": "2026-08-28", + "seed": 2 + }, + { + "id": "base-off-s3", + "label": "Out of the box", + "path": "/traces/alert-triage/base-off-s3.json", + "kind": "recorded", + "model": "brain-qwen38-dspark", + "capturedAt": "2026-08-28", + "seed": 3 + }, + { + "id": "base-off-s4", + "label": "Out of the box", + "path": "/traces/alert-triage/base-off-s4.json", + "kind": "recorded", + "model": "brain-qwen38-dspark", + "capturedAt": "2026-08-28", + "seed": 4 + }, + { + "id": "base-off-s5", + "label": "Out of the box", + "path": "/traces/alert-triage/base-off-s5.json", + "kind": "recorded", + "model": "brain-qwen38-dspark", + "capturedAt": "2026-08-28", + "seed": 5 + }, + { + "id": "base-off-s6", + "label": "Out of the box", + "path": "/traces/alert-triage/base-off-s6.json", + "kind": "recorded", + "model": "brain-qwen38-dspark", + "capturedAt": "2026-08-28", + "seed": 6 + }, + { + "id": "base-off-s7", + "label": "Out of the box", + "path": "/traces/alert-triage/base-off-s7.json", + "kind": "recorded", + "model": "brain-qwen38-dspark", + "capturedAt": "2026-08-28", + "seed": 7 + }, + { + "id": "base-off-s9", + "label": "Out of the box", + "path": "/traces/alert-triage/base-off-s9.json", + "kind": "recorded", + "model": "brain-qwen38-dspark", + "capturedAt": "2026-08-28", + "seed": 9 + }, + { + "id": "base-on-s1", + "label": "Allowed to think", + "path": "/traces/alert-triage/base-on-s1.json", + "kind": "intervened", + "model": "brain-qwen38-dspark", + "capturedAt": "2026-08-28", + "seed": 1, + "intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning." + }, + { + "id": "base-on-s2", + "label": "Allowed to think", + "path": "/traces/alert-triage/base-on-s2.json", + "kind": "intervened", + "model": "brain-qwen38-dspark", + "capturedAt": "2026-08-28", + "seed": 2, + "intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning." + }, + { + "id": "fast-s1", + "label": "Reads the screen", + "path": "/traces/alert-triage/fast-s1.json", + "kind": "generated", + "model": "fast-analyst", + "capturedAt": "2026-08-28", + "seed": 1 + }, + { + "id": "fast-s2", + "label": "Reads the screen", + "path": "/traces/alert-triage/fast-s2.json", + "kind": "generated", + "model": "fast-analyst", + "capturedAt": "2026-08-28", + "seed": 2 + }, + { + "id": "fast-s3", + "label": "Reads the screen", + "path": "/traces/alert-triage/fast-s3.json", + "kind": "generated", + "model": "fast-analyst", + "capturedAt": "2026-08-28", + "seed": 3 + }, + { + "id": "fast-s4", + "label": "Reads the screen", + "path": "/traces/alert-triage/fast-s4.json", + "kind": "generated", + "model": "fast-analyst", + "capturedAt": "2026-08-28", + "seed": 4 + }, + { + "id": "fast-s5", + "label": "Reads the screen", + "path": "/traces/alert-triage/fast-s5.json", + "kind": "generated", + "model": "fast-analyst", + "capturedAt": "2026-08-28", + "seed": 5 + }, + { + "id": "fast-s6", + "label": "Reads the screen", + "path": "/traces/alert-triage/fast-s6.json", + "kind": "generated", + "model": "fast-analyst", + "capturedAt": "2026-08-28", + "seed": 6 + }, + { + "id": "fast-s7", + "label": "Reads the screen", + "path": "/traces/alert-triage/fast-s7.json", + "kind": "generated", + "model": "fast-analyst", + "capturedAt": "2026-08-28", + "seed": 7 + }, + { + "id": "fast-s9", + "label": "Reads the screen", + "path": "/traces/alert-triage/fast-s9.json", + "kind": "generated", + "model": "fast-analyst", + "capturedAt": "2026-08-28", + "seed": 9 + }, + { + "id": "targeted-s1", + "label": "Checks the hidden tells", + "path": "/traces/alert-triage/targeted-s1.json", + "kind": "generated", + "model": "targeted-analyst", + "capturedAt": "2026-08-28", + "seed": 1 + }, + { + "id": "targeted-s2", + "label": "Checks the hidden tells", + "path": "/traces/alert-triage/targeted-s2.json", + "kind": "generated", + "model": "targeted-analyst", + "capturedAt": "2026-08-28", + "seed": 2 + }, + { + "id": "targeted-s3", + "label": "Checks the hidden tells", + "path": "/traces/alert-triage/targeted-s3.json", + "kind": "generated", + "model": "targeted-analyst", + "capturedAt": "2026-08-28", + "seed": 3 + }, + { + "id": "targeted-s4", + "label": "Checks the hidden tells", + "path": "/traces/alert-triage/targeted-s4.json", + "kind": "generated", + "model": "targeted-analyst", + "capturedAt": "2026-08-28", + "seed": 4 + }, + { + "id": "targeted-s5", + "label": "Checks the hidden tells", + "path": "/traces/alert-triage/targeted-s5.json", + "kind": "generated", + "model": "targeted-analyst", + "capturedAt": "2026-08-28", + "seed": 5 + }, + { + "id": "targeted-s6", + "label": "Checks the hidden tells", + "path": "/traces/alert-triage/targeted-s6.json", + "kind": "generated", + "model": "targeted-analyst", + "capturedAt": "2026-08-28", + "seed": 6 + }, + { + "id": "targeted-s7", + "label": "Checks the hidden tells", + "path": "/traces/alert-triage/targeted-s7.json", + "kind": "generated", + "model": "targeted-analyst", + "capturedAt": "2026-08-28", + "seed": 7 + }, + { + "id": "targeted-s9", + "label": "Checks the hidden tells", + "path": "/traces/alert-triage/targeted-s9.json", + "kind": "generated", + "model": "targeted-analyst", + "capturedAt": "2026-08-28", + "seed": 9 + }, + { + "id": "thorough-s1", + "label": "Runs the full procedure", + "path": "/traces/alert-triage/thorough-s1.json", + "kind": "generated", + "model": "thorough-analyst", + "capturedAt": "2026-08-28", + "seed": 1 + }, + { + "id": "thorough-s2", + "label": "Runs the full procedure", + "path": "/traces/alert-triage/thorough-s2.json", + "kind": "generated", + "model": "thorough-analyst", + "capturedAt": "2026-08-28", + "seed": 2 + }, + { + "id": "thorough-s3", + "label": "Runs the full procedure", + "path": "/traces/alert-triage/thorough-s3.json", + "kind": "generated", + "model": "thorough-analyst", + "capturedAt": "2026-08-28", + "seed": 3 + }, + { + "id": "thorough-s4", + "label": "Runs the full procedure", + "path": "/traces/alert-triage/thorough-s4.json", + "kind": "generated", + "model": "thorough-analyst", + "capturedAt": "2026-08-28", + "seed": 4 + }, + { + "id": "thorough-s5", + "label": "Runs the full procedure", + "path": "/traces/alert-triage/thorough-s5.json", + "kind": "generated", + "model": "thorough-analyst", + "capturedAt": "2026-08-28", + "seed": 5 + }, + { + "id": "thorough-s6", + "label": "Runs the full procedure", + "path": "/traces/alert-triage/thorough-s6.json", + "kind": "generated", + "model": "thorough-analyst", + "capturedAt": "2026-08-28", + "seed": 6 + }, + { + "id": "thorough-s7", + "label": "Runs the full procedure", + "path": "/traces/alert-triage/thorough-s7.json", + "kind": "generated", + "model": "thorough-analyst", + "capturedAt": "2026-08-28", + "seed": 7 + }, + { + "id": "thorough-s9", + "label": "Runs the full procedure", + "path": "/traces/alert-triage/thorough-s9.json", + "kind": "generated", + "model": "thorough-analyst", + "capturedAt": "2026-08-28", + "seed": 9 + } + ], "wordle": [ { "id": "base-off-s0", diff --git a/scripts/conformance.mjs b/scripts/conformance.mjs index 35ac560..1785a13 100644 --- a/scripts/conformance.mjs +++ b/scripts/conformance.mjs @@ -2,31 +2,47 @@ /** * The cross-language conformance gate. * - * `envs/wordle_five/wordle_five/engine.py` is the reference implementation and - * `src/demos/wordle/engine.ts` is a port of it. The site's entire argument is - * that the board you play in the browser is the same environment the model was - * evaluated in, so the two implementations agreeing is not a nice-to-have — it - * is the claim. + * Every environment under `envs/` is the reference implementation, and the + * matching `src/demos//engine.ts` is a port of it. The site's entire + * argument is that the board you play in the browser is the same environment + * the model was evaluated in, so the two implementations agreeing is not a + * nice-to-have — it is the claim. Each environment commits the digest(s) its + * Python computes to `envs//CONFORMANCE.txt`; this script recomputes them + * from the TypeScript and fails if any differ. * + * ── wordle ────────────────────────────────────────────────────────────────── * A hand-written vector file would only ever catch the cases somebody thought * of, and the case nobody thinks of is always the same one: repeated letters, * where a green must claim its letter before any yellow is assigned. So the - * gate is exhaustive instead. For every answer A in `words/answers.json`, in - * list order, concatenate `score(G, A)` for every guess G in that same list, in - * that same order, and stream the whole thing through SHA-256. That is the - * digest `engine.py`'s `conformance_digest()` computes, and it is the digest - * this script computes from the TypeScript. + * gate is exhaustive. For every answer A in `words/answers.json`, in list + * order, concatenate `score(G, A)` for every guess G in that same list, in that + * same order, and stream the whole thing through SHA-256. Both the list order + * and the two nested loop orders are part of the definition. * - * Both the answer list order and the two nested loop orders are part of the - * definition. Sorting the list, deduplicating it, or swapping the loops all - * produce a different, equally valid, completely useless number. + * ── alert-triage ──────────────────────────────────────────────────────────── + * Three digests, because three things can drift: + * world canonical JSON of the generated world for seeds 0–4095, with the + * reference minutes the shipped policies compute. One RNG draw out + * of order anywhere in the generator changes every seed after it. + * scorer `scoreExact` over a fixed grid of synthetic episodes, as exact + * fractions so no float formatting is hashed. + * protocol parse + engine over a committed corpus of reply strings — + * accepted actions, rejection reasons and the minute meter after + * every turn. JSON-in-prose with fences and a brace scan is where + * two runtimes disagree, and the rejection reasons quote the + * offending value with Python's `repr`, so this is the digest that + * catches a port that "mostly works". * * Usage: - * node scripts/conformance.mjs full digest, compared to CONFORMANCE.txt - * node scripts/conformance.mjs --limit 50 first 50 answers only, no comparison + * node scripts/conformance.mjs every environment, compared to its CONFORMANCE.txt + * node scripts/conformance.mjs --only wordle one environment + * node scripts/conformance.mjs --limit 50 wordle: first 50 answers; alert-triage: first 50 seeds + * of the world digest. No comparison. * - * Exits 0 with a note (not an error) when `engine.ts` does not exist yet — the - * TypeScript port and this script are allowed to arrive in either order. + * An environment whose `engine.ts` does not exist yet is noted and skipped + * (exit 0) — the TypeScript port and this gate are allowed to arrive in + * either order. An environment whose CONFORMANCE.txt does not exist yet has + * its digests printed with the command that writes the file. */ import fs from 'node:fs'; @@ -36,142 +52,224 @@ import { spawnSync } from 'node:child_process'; import { abs, cyan, die, dim, exists, green, read, red, rel } from './_lib.mjs'; -const ANSWERS = abs('envs', 'wordle_five', 'words', 'answers.json'); -const EXPECTED_FILE = abs('envs', 'wordle_five', 'CONFORMANCE.txt'); +/* ------------------------------------------------------------------ flags */ -const limitFlag = process.argv.indexOf('--limit'); -const limit = limitFlag === -1 ? 0 : Number(process.argv[limitFlag + 1] ?? 0); +const argv = process.argv.slice(2); +const limitFlag = argv.indexOf('--limit'); +const limit = limitFlag === -1 ? 0 : Number(argv[limitFlag + 1] ?? 0); if (limitFlag !== -1 && (!Number.isInteger(limit) || limit <= 0)) { die('--limit takes a positive integer, e.g. `--limit 50`.'); } +const onlyFlag = argv.indexOf('--only'); +const only = onlyFlag === -1 ? null : argv[onlyFlag + 1]; -/* ------------------------------------------------------ find the TS engine */ - -function findEngine() { - const preferred = abs('src', 'demos', 'wordle', 'engine.ts'); - if (exists(preferred)) return preferred; - const demosDir = abs('src', 'demos'); - if (!exists(demosDir)) return null; - const found = fs - .readdirSync(demosDir, { withFileTypes: true }) - .filter((e) => e.isDirectory() && !e.name.startsWith('_')) - .map((e) => path.join(demosDir, e.name, 'engine.ts')) - .filter(exists); - // More than one engine.ts and no `wordle` directory means there is nothing to - // guess between; say so rather than silently conforming the wrong one. - return found.length === 1 ? found[0] : null; -} - -const engine = findEngine(); -if (!engine) { - console.log(dim('conformance: engine not present yet — no src/demos/wordle/engine.ts to check.')); - console.log(dim('The TypeScript port and this gate may land in either order; re-run once it exists.')); - process.exit(0); -} - -if (!exists(ANSWERS)) die(`${rel(ANSWERS)} does not exist. It defines both the pairs and their order.`); -if (!exists(abs('node_modules', '.bin', 'tsx'))) die('tsx is not installed, and the engine is TypeScript. Run `pnpm install`.'); - -/* ------------------------------------------------------------------- run it */ - -const probeDir = fs.mkdtempSync(path.join(os.tmpdir(), 'pig-conformance-')); -const probe = path.join(probeDir, 'digest.mts'); -fs.writeFileSync(probe, probeSource(), 'utf8'); - -const started = Date.now(); -console.log(dim(`conformance: hashing every (guess, answer) pair through ${rel(engine)}...`)); - -// The tsx bin is a POSIX shell shim, not a JS file, so it is the executable -// here rather than an argument to node. -const run = spawnSync(abs('node_modules', '.bin', 'tsx'), [probe, engine, ANSWERS, String(limit)], { - cwd: abs('.'), - encoding: 'utf8', - maxBuffer: 16 * 1024 * 1024, - timeout: 15 * 60_000, - // The probe lives in a temp directory, where tsx would never find the - // repo's tsconfig — and without it the `@/` alias in engine.ts does not - // resolve and the import fails for a reason that looks nothing like the cause. - env: { ...process.env, TSX_TSCONFIG_PATH: abs('tsconfig.json') }, - stdio: ['ignore', 'pipe', 'inherit'], -}); -fs.rmSync(probeDir, { recursive: true, force: true }); - -if (run.error) die(`could not run the digest probe: ${run.error.message}`); -if (run.status !== 0) die(`the digest probe exited ${run.status}. See the error above.`); - -let result; -try { - result = JSON.parse(String(run.stdout).trim().split('\n').pop()); -} catch { - die(`the digest probe produced no parseable result. Output was:\n${run.stdout}`); -} -if (result.error) { - die( - `${rel(engine)}: ${result.error}\n` + - ' The gate needs a scoring function it can recognise. Export one named `scoreGuess` (or `score`), ' + - 'taking (guess, answer) and returning the pattern — either a "GYXXG" string or an array of ' + - "'exact' | 'present' | 'absent' tiles. Exporting `conformanceDigest()` directly also works.", - ); -} - -const seconds = ((Date.now() - started) / 1000).toFixed(1); -console.log(` scored ${result.pairs.toLocaleString('en-US')} pairs from ${result.answers.toLocaleString('en-US')} answers in ${seconds}s`); -console.log(` via ${cyan(result.via)}`); -console.log(` digest ${cyan(result.digest)}`); - -if (limit) { - console.log(''); - console.log(dim(`Partial run (--limit ${limit}): not compared to ${rel(EXPECTED_FILE)}. Run without --limit for the real gate.`)); - process.exit(0); -} - -/* ----------------------------------------------------------- compare it */ - -if (!exists(EXPECTED_FILE)) { - console.log(''); - console.log(green('conformance: computed, but there is nothing to compare against yet.')); - console.log(`Write the digest to ${rel(EXPECTED_FILE)} once the Python agrees:`); - console.log(dim(` python -c "from wordle_five.engine import conformance_digest; print(conformance_digest())"`)); - process.exit(0); -} - -const expected = read(EXPECTED_FILE).match(/\b[0-9a-f]{64}\b/)?.[0]; -if (!expected) { - die(`${rel(EXPECTED_FILE)} contains no 64-character hex digest.`); -} - -if (expected !== result.digest) { - console.error(''); - console.error(`${red('FAIL')} the TypeScript engine and ${rel(EXPECTED_FILE)} disagree.`); - console.error(` expected ${expected}`); - console.error(` got ${result.digest}`); - console.error(''); - console.error( - ' One of the two implementations is wrong, and the board on the site is therefore not the\n' + - ' environment the model was evaluated in. The usual culprit is the two-pass scoring: every\n' + - ' green must claim its letter out of the pool BEFORE any yellow is assigned, or a guess like\n' + - ' SASSY against BASIS marks an S yellow that the greens have already spent.\n' + - ' If the Python changed on purpose, re-run its own `conformance_digest()` and update the file.', - ); - process.exit(1); -} - -console.log(''); -console.log(green(`conformance: the TypeScript engine matches ${rel(EXPECTED_FILE)} exactly.`)); - -/* ------------------------------------------------------------------ probe */ +/* ----------------------------------------------------------- environments */ /** - * Runs under tsx so it can import the TypeScript engine directly. + * @typedef {{ + * name: string, + * engine: string, + * expected: string, + * args: string[], + * probe: () => string, + * writeHint: string, + * diagnose: string, + * describe: (result: any, seconds: string) => string[], + * }} Environment + */ + +/** @type {Environment[]} */ +const ENVIRONMENTS = [ + { + name: 'wordle', + engine: abs('src', 'demos', 'wordle', 'engine.ts'), + expected: abs('envs', 'wordle_five', 'CONFORMANCE.txt'), + args: [abs('envs', 'wordle_five', 'words', 'answers.json')], + probe: wordleProbeSource, + writeHint: 'python -c "from wordle_five.engine import conformance_digest; print(conformance_digest())"', + diagnose: + ' The usual culprit is the two-pass scoring: every green must claim its letter out of the pool\n' + + ' BEFORE any yellow is assigned, or a guess like SASSY against BASIS marks an S yellow that the\n' + + ' greens have already spent.', + describe: (result, seconds) => [ + ` scored ${result.pairs.toLocaleString('en-US')} pairs from ${result.answers.toLocaleString('en-US')} answers in ${seconds}s`, + ` via ${cyan(result.via)}`, + ], + }, + { + name: 'alert-triage', + engine: abs('src', 'demos', 'alert-triage', 'engine.ts'), + expected: abs('envs', 'alert_triage', 'CONFORMANCE.txt'), + args: [abs('envs', 'alert_triage', 'conformance', 'replies.json')], + probe: triageProbeSource, + writeHint: 'uv run python -m alert_triage.conformance --write (from envs/alert_triage)', + diagnose: + ' world: an RNG draw made out of order, or a `//` that became a `/`. Bisect by seed: dump\n' + + ' canonicalJson(worldForSeed(s)) on both sides and diff the first seed that differs.\n' + + ' scorer: scoreExact() — check the fraction forms and the best-alternate tie-break (first wins).\n' + + ' protocol: the parse rules (fence wins; first balanced span; no rescanning) or a rejection\n' + + ' reason whose repr() of the offending value differs from Python.', + describe: (result, seconds) => [ + ` hashed ${result.seeds.toLocaleString('en-US')} worlds, ${result.grid.toLocaleString('en-US')} grid episodes and ${result.cases} corpus cases in ${seconds}s`, + ], + }, +]; + +/* ------------------------------------------------------------------ run it */ + +if (!exists(abs('node_modules', '.bin', 'tsx'))) die('tsx is not installed, and the engines are TypeScript. Run `pnpm install`.'); + +const selected = ENVIRONMENTS.filter((env) => only === null || env.name === only); +if (only !== null && selected.length === 0) { + die(`--only ${only}: no such environment. Known: ${ENVIRONMENTS.map((e) => e.name).join(', ')}.`); +} + +let failed = false; +for (const env of selected) { + if (!runEnvironment(env)) failed = true; + console.log(''); +} +process.exit(failed ? 1 : 0); + +/** @param {Environment} env */ +function runEnvironment(env) { + if (!exists(env.engine)) { + console.log(dim(`conformance[${env.name}]: engine not present yet — no ${rel(env.engine)} to check.`)); + console.log(dim('The TypeScript port and this gate may land in either order; re-run once it exists.')); + return true; + } + for (const arg of env.args) { + if (!exists(arg)) die(`${rel(arg)} does not exist. It is part of the definition of the ${env.name} digest.`); + } + + const probeDir = fs.mkdtempSync(path.join(os.tmpdir(), 'pig-conformance-')); + const probe = path.join(probeDir, 'digest.mts'); + fs.writeFileSync(probe, env.probe(), 'utf8'); + + const started = Date.now(); + console.log(dim(`conformance[${env.name}]: hashing through ${rel(env.engine)}...`)); + + // The tsx bin is a POSIX shell shim, not a JS file, so it is the executable + // here rather than an argument to node. + const run = spawnSync(abs('node_modules', '.bin', 'tsx'), [probe, env.engine, ...env.args, String(limit)], { + cwd: abs('.'), + encoding: 'utf8', + maxBuffer: 16 * 1024 * 1024, + timeout: 15 * 60_000, + // The probe lives in a temp directory, where tsx would never find the + // repo's tsconfig — and without it the `@/` alias in an engine does not + // resolve and the import fails for a reason that looks nothing like the cause. + env: { ...process.env, TSX_TSCONFIG_PATH: abs('tsconfig.json') }, + stdio: ['ignore', 'pipe', 'inherit'], + }); + fs.rmSync(probeDir, { recursive: true, force: true }); + + if (run.error) die(`could not run the ${env.name} digest probe: ${run.error.message}`); + if (run.status !== 0) die(`the ${env.name} digest probe exited ${run.status}. See the error above.`); + + let result; + try { + result = JSON.parse(String(run.stdout).trim().split('\n').pop()); + } catch { + die(`the ${env.name} digest probe produced no parseable result. Output was:\n${run.stdout}`); + } + if (result.error) { + die(`${rel(env.engine)}: ${result.error}`); + } + + const seconds = ((Date.now() - started) / 1000).toFixed(1); + for (const line of env.describe(result, seconds)) console.log(line); + /** @type {Record} */ + const digests = result.digests; + const names = Object.keys(digests); + for (const name of names) console.log(` ${names.length > 1 ? name.padEnd(9) : 'digest'} ${cyan(digests[name])}`); + + if (limit) { + console.log(''); + console.log(dim(`Partial run (--limit ${limit}): not compared to ${rel(env.expected)}. Run without --limit for the real gate.`)); + return true; + } + + /* ----------------------------------------------------------- compare */ + + if (!exists(env.expected)) { + console.log(''); + console.log(green(`conformance[${env.name}]: computed, but there is nothing to compare against yet.`)); + console.log(`Write the digest(s) to ${rel(env.expected)} once the Python agrees:`); + console.log(dim(` ${env.writeHint}`)); + return true; + } + + const expected = parseExpected(read(env.expected), names); + if (!expected) { + die(`${rel(env.expected)} contains no 64-character hex digest${names.length > 1 ? ` for each of: ${names.join(', ')}` : ''}.`); + } + + const mismatched = names.filter((name) => expected[name] !== digests[name]); + if (mismatched.length) { + console.error(''); + console.error(`${red('FAIL')} the TypeScript engine and ${rel(env.expected)} disagree${names.length > 1 ? ` on: ${mismatched.join(', ')}` : ''}.`); + for (const name of mismatched) { + const label = names.length > 1 ? ` (${name})` : ''; + console.error(` expected${label} ${expected[name]}`); + console.error(` got${label} ${digests[name]}`); + } + console.error(''); + console.error( + ' One of the two implementations is wrong, and the board on the site is therefore not the\n' + + ' environment the model was evaluated in.\n' + + env.diagnose + + '\n If the Python changed on purpose, re-run its own digest and update the file:\n' + + ` ${env.writeHint}`, + ); + return false; + } + + console.log(''); + console.log(green(`conformance[${env.name}]: the TypeScript engine matches ${rel(env.expected)} exactly${names.length > 1 ? ` (${names.join(', ')})` : ''}.`)); + return true; +} + +/** + * Reads the committed digests. A single-digest file is one hex string on a + * line; a multi-digest file is `hex name description` per line, and every + * name the probe produced must be present. * - * Kept here as a string rather than as a checked-in `.mts`: it is an - * implementation detail of this script, and a stray TypeScript file in + * @param {string} text + * @param {string[]} names + * @returns {Record | null} + */ +function parseExpected(text, names) { + /** @type {Record} */ + const out = {}; + if (names.length === 1) { + const hex = text.match(/\b[0-9a-f]{64}\b/)?.[0]; + if (!hex) return null; + out[names[0]] = hex; + return out; + } + for (const line of text.split('\n')) { + const m = line.match(/^\s*([0-9a-f]{64})\s+(\S+)/); + if (m) out[m[2]] = m[1]; + } + return names.every((n) => n in out) ? out : null; +} + +/* ------------------------------------------------------------------ probes */ + +/** + * Each probe runs under tsx so it can import the TypeScript engine directly, + * and prints one JSON line: `{ digests: { : hex }, ...counts }` or + * `{ error }`. + * + * Kept here as strings rather than as checked-in `.mts` files: they are + * implementation details of this script, and a stray TypeScript file in * `scripts/` would be swept into a typecheck it is deliberately outside of. * - * A function declaration, not a `const`, so it is hoisted above the code that - * writes it to disk near the top of this file. + * Function declarations, not `const`s, so they are hoisted above the table + * that references them near the top of this file. */ -function probeSource() { +function wordleProbeSource() { return ` import { createHash } from 'node:crypto'; import { readFileSync } from 'node:fs'; @@ -195,7 +293,7 @@ const pool = limit > 0 ? answers.slice(0, limit) : answers; // here — it is the implementation's own statement of the definition. if (typeof mod.conformanceDigest === 'function' && limit === 0) { const digest = await mod.conformanceDigest(); - say({ digest, via: 'engine.conformanceDigest()', pairs: pool.length * pool.length, answers: pool.length }); + say({ digests: { wordle: digest }, via: 'engine.conformanceDigest()', pairs: pool.length * pool.length, answers: pool.length }); process.exit(0); } @@ -209,7 +307,13 @@ const name = (typeof mod.default === 'function' ? 'default' : null); if (!name) { - say({ error: 'no scoring function is exported. Found: ' + Object.keys(mod).join(', ') }); + say({ + error: + 'no scoring function is exported. Found: ' + Object.keys(mod).join(', ') + + '. The gate needs a scoring function it can recognise. Export one named scoreGuess (or score), ' + + 'taking (guess, answer) and returning the pattern — either a "GYXXG" string or an array of ' + + "'exact' | 'present' | 'absent' tiles. Exporting conformanceDigest() directly also works.", + }); process.exit(0); } const score = mod[name]; @@ -271,6 +375,65 @@ for (let i = 0; i < pool.length; i += 1) { } if (process.stderr.isTTY) process.stderr.write(' '.repeat(40) + '\\r'); -say({ digest: hash.digest('hex'), via: 'engine.' + name + '(guess, answer)', pairs: pool.length * pool.length, answers: pool.length }); +say({ digests: { wordle: hash.digest('hex') }, via: 'engine.' + name + '(guess, answer)', pairs: pool.length * pool.length, answers: pool.length }); +`; +} + +/** + * The alert-triage probe. Three digests, each SHA-256 over newline-terminated + * canonical JSON lines, exactly as \`alert_triage/conformance.py\` defines + * them. The engine exports the lines; the probe only hashes them, so the + * engine stays free of node:crypto and browser-clean. + */ +function triageProbeSource() { + return ` +import { createHash } from 'node:crypto'; +import { readFileSync } from 'node:fs'; + +const [enginePath, corpusPath, limitRaw] = process.argv.slice(2); +const limit = Number(limitRaw ?? 0); +const say = (value: unknown) => console.log(JSON.stringify(value)); + +let mod: any; +try { + mod = await import(enginePath); +} catch (error: any) { + say({ error: 'could not import it: ' + String(error?.message ?? error).split('\\n')[0] }); + process.exit(0); +} + +for (const name of ['worldLines', 'scorerLines', 'protocolLines']) { + if (typeof mod[name] !== 'function') { + say({ error: 'does not export ' + name + '(). Found: ' + Object.keys(mod).join(', ') }); + process.exit(0); + } +} + +function digestOf(lines: Iterable, label: string, total: number): [string, number] { + const hash = createHash('sha256'); + let count = 0; + for (const line of lines) { + hash.update(line, 'utf8'); + hash.update('\\n'); + count += 1; + if (count % 256 === 0 && process.stderr.isTTY) process.stderr.write(' ' + label + ' ' + count + '/' + total + '\\r'); + } + if (process.stderr.isTTY) process.stderr.write(' '.repeat(40) + '\\r'); + return [hash.digest('hex'), count]; +} + +const SEEDS = 4096; +const seedCount = limit > 0 ? Math.min(limit, SEEDS) : SEEDS; +const seeds = Array.from({ length: seedCount }, (_, i) => i); +const corpus = JSON.parse(readFileSync(corpusPath, 'utf8')); + +try { + const [world] = digestOf(mod.worldLines(seeds), 'world', seedCount); + const [scorer, grid] = digestOf(mod.scorerLines(), 'scorer', 0); + const [protocol, cases] = digestOf(mod.protocolLines(corpus), 'protocol', corpus.length); + say({ digests: { world, scorer, protocol }, seeds: seedCount, grid, cases }); +} catch (error: any) { + say({ error: String(error?.stack ?? error?.message ?? error).split('\\n').slice(0, 3).join(' | ') }); +} `; } diff --git a/src/components/demo/DemoShell.tsx b/src/components/demo/DemoShell.tsx index b1097b5..1ab7f57 100644 --- a/src/components/demo/DemoShell.tsx +++ b/src/components/demo/DemoShell.tsx @@ -744,7 +744,7 @@ function ShellSkeleton() { * The run a visitor sees before choosing one. * * `runs[0]` is the manifest's first entry — the weakest agent on seed 0, which - * for wordle is a failed game with thinking off. So the Watch tab opened on a + * for the first demo shipped was a failed game with thinking off. So the Watch tab opened on a * loss with an empty reasoning panel and the Reward tab on a row of zeros: * the model's least interesting attempt, chosen by accident of sort order. * diff --git a/src/demos/alert-triage/__tests__/engine.test.ts b/src/demos/alert-triage/__tests__/engine.test.ts new file mode 100644 index 0000000..bdd28c8 --- /dev/null +++ b/src/demos/alert-triage/__tests__/engine.test.ts @@ -0,0 +1,275 @@ +import assert from 'node:assert/strict'; +import { readFileSync } from 'node:fs'; +import { test } from 'node:test'; + +import { + Engine, + XorShift32, + canonicalJson, + civilFromDays, + daysFromCivil, + daysInMonth, + extractCandidate, + fnv1a32, + generate, + isHeldOut, + isoDate, + parseReply, + protocolTrace, + pyJsonDumps, + pyJsonLoads, + pyRepr, + referenceFor, + runPolicy, + screenOf, + scoreExact, + worldForSeed, + POLICIES, +} from '../engine'; +import type { CorpusCase } from '../engine'; + +/* + * The cross-language digest in envs/alert_triage/CONFORMANCE.txt is the real + * gate (`pnpm conformance`). These vectors exist so that a break is LOCATED, + * not merely detected: each one is pinned identically in + * envs/alert_triage/tests/, or was produced by the Python on the day the port + * landed, and names the primitive that drifted. + */ + +/* ---------------------------------------------------------------- rng.py */ + +test('fnv1a32 vectors match test_triage_rng.py', () => { + assert.equal(fnv1a32('0'), 0x350ca8af); + assert.equal(fnv1a32('1'), 0x340ca71c); + assert.equal(fnv1a32('42'), 0x87e38583); + assert.equal(fnv1a32('4095'), 0x43875f5f); +}); + +test('xorshift32 from seed 0 is pinned', () => { + const rng = new XorShift32(fnv1a32('0')); + assert.deepEqual(Array.from({ length: 5 }, () => rng.next()), [2738490563, 3068243922, 3765331391, 3085691315, 2439018365]); + assert.notEqual(new XorShift32(0).state, 0); +}); + +test('sample is a distinct partial shuffle', () => { + const out = new XorShift32(7).sample([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], 4); + assert.equal(out.length, 4); + assert.equal(new Set(out).size, 4); +}); + +test('civil dates round-trip and hit the known ones', () => { + for (let day = daysFromCivil(1999, 12, 25); day < daysFromCivil(2030, 3, 2); day += 1) { + const [y, m, d] = civilFromDays(day); + assert.equal(daysFromCivil(y, m, d), day); + } + assert.equal(daysFromCivil(1970, 1, 1), 0); + assert.equal(daysFromCivil(2026, 3, 1), 20513); + assert.equal(daysFromCivil(2000, 2, 29), 11016); + assert.equal(isoDate(20000), '2024-10-04'); + assert.equal(isoDate(0), '1970-01-01'); + assert.equal(isoDate(-1), '1969-12-31'); + assert.equal(daysInMonth(2024, 2), 29); + assert.equal(daysInMonth(2026, 2), 28); + assert.equal(daysInMonth(2026, 12), 31); +}); + +/* ---------------------------------------------------------- generator.py */ + +test('the held-out slice is fnv1a(seed) % 8 == 7', () => { + const heldOut = Array.from({ length: 64 }, (_, s) => s).filter(isHeldOut); + assert.deepEqual(heldOut, [0, 8, 11, 19, 20, 28, 37, 46, 55]); +}); + +// Produced by `world_for_seed` in Python the day the port landed. A different +// template or alert id here means an RNG draw moved; a different reference +// means a policy diverged. +const WORLD_VECTORS: [number, string, string, string | null, string, string, string, number, string, string][] = [ + [0, 'B6', 'hidden', 'funnel', 'suspicious', 'A-9571', '2026-04-18', 123, 'targeted', 'Emeka Zapata'], + [1, 'B4', 'benign', null, 'benign', 'A-8971', '2026-04-26', 33, 'fast', 'Oakhill Kitchen Group'], + [2, 'B6', 'benign', null, 'benign', 'A-9760', '2026-04-23', 42, 'fast', 'Gabriela Kowalczyk'], + [3, 'B4', 'benign', null, 'benign', 'A-9453', '2026-04-23', 33, 'fast', 'Lakeside Kitchen LLC'], + [4, 'S1', 'visible', null, 'suspicious', 'A-8199', '2026-06-14', 33, 'fast', 'Bilal Quintero'], + [5, 'B3', 'benign', null, 'benign', 'A-6081', '2026-03-22', 42, 'fast', 'Andre Nakamura'], + [6, 'B1', 'benign', null, 'benign', 'A-7014', '2026-05-15', 33, 'fast', 'Lakeside Landscaping Inc.'], + [7, 'B6', 'benign', null, 'benign', 'A-5644', '2026-06-14', 42, 'fast', 'Leila Bergstrom'], + [8, 'S3', 'visible', null, 'suspicious', 'A-5828', '2026-04-24', 33, 'fast', 'Dana Yilmaz'], + [9, 'B3', 'hidden', 'funnel', 'suspicious', 'A-9537', '2026-04-19', 123, 'targeted', 'Samir Tanaka'], + [10, 'S1', 'visible', null, 'suspicious', 'A-5846', '2026-06-26', 33, 'fast', 'Malik Marchetti'], + [11, 'B3', 'benign', null, 'benign', 'A-9882', '2026-05-16', 42, 'fast', 'Naomi Abara'], +]; + +test('world vectors match the Python generator', () => { + for (const [seed, template, tier, overlay, label, alertId, fired, minutes, policy, name] of WORLD_VECTORS) { + const w = worldForSeed(seed); + assert.deepEqual( + [w.template, w.tier, w.overlay, w.label, w.alert.id, w.alert.fired, w.reference_minutes, w.reference_policy, w.customer.name], + [template, tier, overlay, label, alertId, fired, minutes, policy, name], + `seed ${seed}`, + ); + } +}); + +test('the screen is byte-identical with a hidden overlay on or off', () => { + for (let seed = 0; seed < 64; seed += 1) { + const auto = generate(seed, 'auto'); + if (auto.tier === 'visible') continue; + const twin = generate(seed, null); + assert.equal(canonicalJson(screenOf(auto)), canonicalJson(screenOf(twin)), `seed ${seed}`); + } +}); + +test('canonical JSON is sorted, compact and drops undefined', () => { + assert.equal(canonicalJson({ b: 1, a: [null, true, 'x'], c: undefined }), '{"a":[null,true,"x"],"b":1}'); + assert.equal(canonicalJson('dash — “quotes”'), '"dash — “quotes”"'); + assert.throws(() => canonicalJson(1.5)); +}); + +/* ----------------------------------------------------------- protocol.py */ + +test('python repr vectors', () => { + const strings: [string, string][] = [ + ['a', "'a'"], + ["it's", '"it\'s"'], + ['say "hi"', "'say \"hi\"'"], + ['it\'s "both"', "'it\\'s \"both\"'"], + ['tab\there', "'tab\\there'"], + ['nl\n', "'nl\\n'"], + ['\x01\x7f\x80\xa0', "'\\x01\\x7f\\x80\\xa0'"], + ['café 
 ​', "'café \\u2028 \\u200b'"], + ['😀', "'😀'"], + ['back\\slash', "'back\\\\slash'"], + ]; + for (const [input, expected] of strings) assert.equal(pyRepr(input), expected, JSON.stringify(input)); + + const numbers: [string, string][] = [ + ['1', '1'], + ['-0', '0'], + ['1.0', '1.0'], + ['1.5', '1.5'], + ['1e5', '100000.0'], + ['1E16', '1e+16'], + ['1e-5', '1e-05'], + ['0.0001', '0.0001'], + ['12345678901234567890', '12345678901234567890'], + ['-2.5e-7', '-2.5e-07'], + ['1e22', '1e+22'], + ['123456789012345.6', '123456789012345.6'], + ['1e400', 'inf'], + ['0.1', '0.1'], + ['100.0', '100.0'], + ]; + for (const [raw, expected] of numbers) assert.equal(pyRepr(pyJsonLoads(raw)), expected, raw); + + assert.equal(pyRepr(null), 'None'); + assert.equal(pyRepr(undefined), 'None'); + assert.equal(pyRepr(true), 'True'); + assert.equal(pyRepr(pyJsonLoads('[1, "a", null]')), "[1, 'a', None]"); + assert.equal(pyRepr(pyJsonLoads('{"k": 2.0, "z": [true]}')), "{'k': 2.0, 'z': [True]}"); +}); + +test('the JSON reader accepts what json.loads accepts and nothing more', () => { + assert.throws(() => pyJsonLoads('{"a":1,}')); + assert.throws(() => pyJsonLoads('{"a":NaN}')); + assert.throws(() => pyJsonLoads('{"a":Infinity}')); + assert.throws(() => pyJsonLoads('{"a":"x\ny"}')); + assert.throws(() => pyJsonLoads('{"a":1} {"b":2}')); + assert.throws(() => pyJsonLoads("{'a':1}")); + assert.equal(pyJsonLoads(' \t"\\ud83d\\ude00\\u00e9" \n'), '😀é'); + assert.equal(pyJsonLoads('"\\ud800"'), '\ud800'); + const obj = pyJsonLoads('{"__proto__": 1, "a": {"b": [1e2]}}'); + assert.ok(obj instanceof Map); + assert.equal(pyRepr(obj), "{'__proto__': 1, 'a': {'b': [100.0]}}"); +}); + +test('the pinned candidate rules', () => { + assert.equal(extractCandidate('{"a":1} ```json\n{"b":2}\n```'), '{"b":2}\n'); + assert.equal(extractCandidate('text {"a":"}"} {"b":2}'), '{"a":"}"}'); + assert.equal(extractCandidate('text {"a":"\\"}"} x'), '{"a":"\\"}"}'); + assert.equal(extractCandidate('{"a":1'), null); + assert.equal(extractCandidate('no braces'), null); + assert.equal(extractCandidate('[{"a":1}]'), '{"a":1}'); + assert.deepEqual(parseReply(''), [null, 'no JSON object found in the reply']); + assert.deepEqual(parseReply(null), [null, 'no JSON object found in the reply']); + assert.deepEqual(parseReply('{"a":1,} {"a":1}'), [null, 'the JSON object could not be parsed']); + assert.deepEqual(parseReply('```json\n[1]\n```'), [null, 'the reply parsed but is not a JSON object']); +}); + +test('pyJsonDumps writes what json.dumps writes', () => { + assert.equal(pyJsonDumps({ action: 'close', cites: ['a', 'b'], note: 'é\x7f"' }), '{"action": "close", "cites": ["a", "b"], "note": "\\u00e9\\u007f\\""}'); +}); + +/* ------------------------------------------------------------- engine.py */ + +test('the rejection reasons quote Python repr', () => { + const engine = new Engine(worldForSeed(1)); + assert.equal(engine.step('{"action":"Lookup"}').reason, "unknown action 'Lookup'; use lookup, close or escalate"); + assert.equal(engine.step('{"what":"documents"}').reason, 'unknown action None; use lookup, close or escalate'); + assert.equal(engine.step('{"action":"lookup","what":3.0}').reason, "unknown lookup 3.0; use history, counterparty, prior_alerts or documents"); + assert.equal(engine.step('{"action":"lookup","what":"history","month":["x"]}').reason, "month ['x'] is not in the twelve-month window"); + assert.equal(engine.step('{"action":"close","reason":"CONSISTENT_WITH_PROFILE","cites":["doc.x"]}').reason, 'cites an id you have not been shown: doc.x'); + assert.equal(engine.turns, 5); + assert.equal(engine.rejected, 5); + assert.equal(engine.minutes, 30 + 5 * 3); +}); + +test('aborted is exactly eight turns; fewer is truncated and unscored', () => { + const aborted = new Engine(worldForSeed(3)); + for (let i = 0; i < 8; i += 1) aborted.step('thinking'); + assert.equal(aborted.outcome, 'aborted'); + assert.deepEqual(scoreExact(aborted.episode()), { caught: [0, 1], hours: [0, 1], evidence: [0, 1] }); + + const truncated = new Engine(worldForSeed(3)); + truncated.step('thinking'); + assert.equal(truncated.outcome, null); + assert.deepEqual(scoreExact(truncated.episode()), { caught: null, hours: null, evidence: null }); +}); + +test('the reference is the cheapest correct shipped policy and never null', () => { + for (let seed = 0; seed < 96; seed += 1) { + const world = generate(seed); + const [minutes, policy] = referenceFor(world); + assert.notEqual(minutes, null, `seed ${seed}`); + assert.equal(runPolicy(POLICIES[policy!]!, world).minutes, minutes); + if (world.tier === 'hidden') { + assert.notEqual(runPolicy(POLICIES.fast!, world).outcome, 'solved', `fast cannot see the hidden signal on seed ${seed}`); + assert.equal(runPolicy(POLICIES.targeted!, world).outcome, 'solved', `targeted catches seed ${seed}`); + } + } +}); + +test('the committed corpus replays to the same rejection reasons the Python recorded', () => { + const corpus: CorpusCase[] = JSON.parse( + readFileSync(new URL('../../../../envs/alert_triage/conformance/replies.json', import.meta.url), 'utf8'), + ); + const reasons = protocolTrace(corpus) + .flatMap((entry) => entry.turns as { reason: string | null }[]) + .map((t) => t.reason) + .filter((r): r is string => r !== null); + assert.deepEqual(reasons, [ + "month 'BADMONTH' is not in the twelve-month window", + 'the JSON object could not be parsed', + "unknown close reason 'consistent_with_profile'", + "unknown action 'Lookup'; use lookup, close or escalate", + 'cites must be a non-empty list of ids', + 'cites must be a non-empty list of ids', + "unknown typology 'structuring'", + 'cites must be a non-empty list of ids', + 'prior_alerts has already been looked up', + "counterparty 'CP-99' has not appeared on your screen or in a fetched month", + "counterparty 'cp-1' has not appeared on your screen or in a fetched month", + "month '2024-01' is not in the twelve-month window", + "month '2026-1' is not in the twelve-month window", + 'cites an id you have not been shown: doc.probate_letter', + 'no JSON object found in the reply', + 'no JSON object found in the reply', + 'no JSON object found in the reply', + 'the JSON object could not be parsed', + 'the reply parsed but is not a JSON object', + 'unknown action None; use lookup, close or escalate', + 'no JSON object found in the reply', + 'unknown action None; use lookup, close or escalate', + "counterparty 'CP-3' has not appeared on your screen or in a fetched month", + 'no JSON object found in the reply', + "unknown lookup 'emails'; use history, counterparty, prior_alerts or documents", + ]); +}); diff --git a/src/demos/alert-triage/__tests__/game.test.ts b/src/demos/alert-triage/__tests__/game.test.ts new file mode 100644 index 0000000..c61a113 --- /dev/null +++ b/src/demos/alert-triage/__tests__/game.test.ts @@ -0,0 +1,112 @@ +import assert from 'node:assert/strict'; +import { test } from 'node:test'; + +import { describe } from '../adapter'; +import { Engine, isHeldOut, worldForSeed } from '../engine'; +import { initState, play, replay } from '../game'; +import { thorough, closeBlind, inaction } from '../policies'; + +/** + * The seam, exercised the way the shell exercises it: a fresh board per + * seed, one reply at a time through `play()`, and a recording through + * `replay()`. The engine's own conformance is the port's business; these + * check that the board never shows what it must not and always agrees with + * the engine it wraps. + */ + +const SEEDS = Array.from({ length: 48 }, (_, i) => i); + +test('a fresh board shows the screen, the free-screen hours, and no answer', () => { + for (const seed of SEEDS) { + const state = initState(seed); + assert.equal(state.seed, seed); + assert.equal(state.reveal, null, `seed ${seed} leaked the label before a disposition`); + assert.equal(state.status, 'playing'); + assert.equal(state.turnsSpent, 0); + assert.equal(state.minutesSpent, 30); + assert.equal(state.lookups.length, 0); + assert.equal(state.screen.summary.length, 12); + assert.ok(state.shown.includes(state.screen.alert.id)); + assert.equal(state.screen.screen_order, isHeldOut(seed) ? 'kyc_first' : 'alert_first'); + } +}); + +test('the thorough analyst, played through play(), reaches the right call on every seed', () => { + for (const seed of SEEDS) { + let state = initState(seed); + let guard = 0; + while (state.status === 'playing' && guard < 12) { + // The policy reads the engine's view; rebuild the same view off the board. + const view = { + screen: state.screen, + lookups: state.lookups, + turns_spent: state.turnsSpent, + turns_left: 8 - state.turnsSpent, + rejected: state.rejected, + minutes_spent: state.minutesSpent, + last_rejection: state.lastRejection, + }; + state = play(state, thorough(view)); + guard += 1; + } + assert.equal(state.status === 'closed' || state.status === 'escalated', true, `seed ${seed} never decided`); + assert.equal(state.outcome, 'solved', `seed ${seed}: thorough got it wrong`); + assert.ok(state.reveal, `seed ${seed} has no reveal after the episode ended`); + assert.equal(state.reveal?.label, worldForSeed(seed).label); + assert.equal(state.reveal?.referenceMinutes, worldForSeed(seed).reference_minutes); + assert.equal(state.rejected, 0); + // Playing on after the end is a no-op, never a new board. + assert.equal(play(state, '{"action":"lookup","what":"documents"}'), state); + } +}); + +test('replay() agrees with the engine step for step and one snapshot per reply', () => { + for (const seed of SEEDS.slice(0, 16)) { + const engine = new Engine(worldForSeed(seed)); + const replies: string[] = []; + while (!engine.done) { + const reply = thorough(engine.view()); + replies.push(reply); + engine.step(reply); + } + const snapshots = replay(seed, replies); + assert.equal(snapshots.length, replies.length); + const final = snapshots[snapshots.length - 1]!; + assert.equal(final.minutesSpent, engine.minutes); + assert.equal(final.turnsSpent, engine.turns); + assert.deepEqual(final.disposition, engine.disposition); + assert.deepEqual(final.shown.sort(), [...engine.shown].sort()); + // A fixture's own reference wins over the world's for the reveal. + assert.equal(replay(seed, replies, { referenceMinutes: 99 }).at(-1)?.reveal?.referenceMinutes, 99); + } +}); + +test('refusals cost a turn, are announced, and eight of them abort', () => { + let state = initState(3); + state = play(state, inaction()); + assert.equal(state.turnsSpent, 1); + assert.equal(state.rejected, 1); + assert.equal(state.minutesSpent, 33); + assert.ok(state.lastRejection); + assert.equal(state.reveal, null); + assert.match(describe(state, initState(3), 0).announce, /refused/); + + for (let i = 1; i < 8; i += 1) state = play(state, inaction()); + assert.equal(state.status, 'aborted'); + assert.equal(state.outcome, 'aborted'); + assert.ok(state.reveal, 'an aborted episode still reveals the answer'); + assert.match(describe(state, null, 7).announce, /stays in the queue/); +}); + +test('citing an id never shown is refused, and a blind close is scored against the label', () => { + const state = play(initState(5), '{"action":"close","reason":"DOCUMENTED_SOURCE_OF_FUNDS","cites":["doc.probate_letter"],"note":""}'); + assert.equal(state.rejected, 1); + assert.match(state.lastRejection ?? '', /not been shown/); + + const blind = play(initState(5), closeBlind()); + assert.equal(blind.status, 'closed'); + assert.equal(blind.outcome, worldForSeed(5).label === 'benign' ? 'solved' : 'failed'); + const step = describe(blind, initState(5), 0); + assert.equal(step.caption, 'closed'); + assert.match(step.announce, /citing 1 id/); +}); diff --git a/src/demos/alert-triage/__tests__/scorer.test.ts b/src/demos/alert-triage/__tests__/scorer.test.ts new file mode 100644 index 0000000..912e20b --- /dev/null +++ b/src/demos/alert-triage/__tests__/scorer.test.ts @@ -0,0 +1,73 @@ +import assert from 'node:assert/strict'; +import { test } from 'node:test'; + +import { bestAlternate } from '../grade'; +import { evidence, f1Exact, score, type Episode } from '../score'; + +/** + * Vectors the page's copy relies on, checked against the browser scorer. The + * cross-language digest proves the port matches the Python; these prove the + * sentences on the Reward tab match the port. + */ + +const base: Episode = { + label: 'benign', + planted: [['kyc.expected_monthly_cash'], ['kyc.expected_monthly_cash', 'kyc.business_type']], + disposition: 'close', + cites: ['kyc.expected_monthly_cash'], + minutes_spent: 33, + reference_minutes: 33, + turns_spent: 1, + rejected: 0, + truncated: false, +}; + +test('a correct, well-cited close at the reference scores 1.0 everywhere', () => { + assert.deepEqual(score(base), { caught: 1, hours: 1, evidence: 1 }); +}); + +test('a missed escalation scores exactly 0.000 on every component', () => { + assert.deepEqual(score({ ...base, label: 'suspicious', planted: [['T-1', 'T-2']] }), { + caught: 0, + hours: 0, + evidence: 0, + }); +}); + +test('a false escalation earns the credit on caught and nothing else', () => { + assert.deepEqual(score({ ...base, disposition: 'escalate', cites: ['T-1'] }), { + caught: 0.5, + hours: 0, + evidence: 0, + }); +}); + +test('no disposition is zero, not null; a truncated run is null, not zero', () => { + assert.deepEqual(score({ ...base, disposition: null, cites: [] }), { caught: 0, hours: 0, evidence: 0 }); + assert.deepEqual(score({ ...base, truncated: true }), { caught: null, hours: null, evidence: null }); +}); + +test('hours is capped at one and null without a reference', () => { + assert.equal(score({ ...base, minutes_spent: 20 }).hours, 1); + assert.equal(score({ ...base, minutes_spent: 66 }).hours, 0.5); + assert.equal(score({ ...base, reference_minutes: null }).hours, null); +}); + +test('precision bites: one planted id among eight cited is F1 0.22', () => { + const cites = ['kyc.expected_monthly_cash', 'T-1', 'T-2', 'T-3', 'T-4', 'T-5', 'T-6', 'T-7']; + assert.deepEqual(f1Exact(cites, ['kyc.expected_monthly_cash']), [2, 9]); + assert.equal(evidence({ ...base, cites }), 2 / 9); +}); + +test('the best alternate is scored, and the board can recover which one', () => { + const cites = ['kyc.expected_monthly_cash', 'kyc.business_type']; + assert.deepEqual(bestAlternate(cites, base.planted), base.planted[1]); + assert.equal(evidence({ ...base, cites }), 1); + // A tie goes to the first alternate, as the scorer's strict `>` does. + assert.deepEqual(bestAlternate(['T-9'], base.planted), base.planted[0]); +}); + +test('planted ids are matched exact-case and an unshown id simply misses', () => { + assert.equal(evidence({ ...base, cites: ['KYC.EXPECTED_MONTHLY_CASH'] }), 0); + assert.equal(evidence({ ...base, cites: ['doc.probate_letter', 'kyc.expected_monthly_cash'] }), 2 / 3); +}); diff --git a/src/demos/alert-triage/adapter.ts b/src/demos/alert-triage/adapter.ts new file mode 100644 index 0000000..33c2511 --- /dev/null +++ b/src/demos/alert-triage/adapter.ts @@ -0,0 +1,138 @@ +import type { DemoEpisode, DemoStep } from '@/lib/demo-kit'; + +import { hours } from './format'; +import { replay } from './game'; +import type { TriageState } from './types'; + +/** + * A recorded episode becomes a list of board snapshots. + * + * Nothing is read off the fixture but the seed and the reply strings. The + * world is regenerated from the seed and every reply is stepped through the + * same engine the environment ran, so the board you scrub through is what the + * engine did with what the model said — not what the capture script wrote + * down about it. A fixture whose `info` disagrees with its own replies cannot + * fool this adapter, because the adapter never looks. + * + * `reference_hours` is the one metric read, and only for the reveal: it comes + * from running the shipped analysts, which the browser has no business doing. + * Its absence leaves the reveal without a reference, never with a zero. + * + * Snapshots, not deltas: the scrubber jumps to any step. + */ +export function adapt(episode: DemoEpisode): DemoStep[] { + const referenceHours = episode.metrics?.['reference_hours']; + const referenceMinutes = typeof referenceHours === 'number' && Number.isFinite(referenceHours) + ? Math.round(referenceHours * 60) + : null; + + const snapshots = replay( + episode.seed, + episode.turns.map((turn) => turn.reply), + { referenceMinutes }, + ); + + let previous: TriageState | null = null; + + return episode.turns.map((turn, index) => { + const state = snapshots[index] ?? snapshots[snapshots.length - 1]; + if (!state) { + // Cannot happen: replay returns one snapshot per reply. Typed defensively + // so a broken port degrades to a readable step rather than a crash. + throw new Error(`replay produced no snapshot for turn ${index + 1}`); + } + const { announce, caption } = describe(state, previous, index); + previous = state; + return { + index, + state, + reply: turn.reply, + reasoning: turn.reasoning, + call: turn.call, + announce, + caption, + }; + }); +} + +/** + * What a screen reader hears when a step lands, and the six words under the + * chip. Derived from the change between snapshots, so it says what the engine + * did rather than what the reply claimed. + */ +export function describe( + state: TriageState, + previous: TriageState | null, + index: number, +): { announce: string; caption: string } { + const turn = `Turn ${index + 1}`; + const meter = `${hours(state.minutesSpent)} hours spent.`; + + const wasDone = previous !== null && previous.status !== 'playing'; + if (wasDone) { + return { announce: `${turn}: the episode was already over.`, caption: 'after the end' }; + } + + if (state.lastRejection !== null && state.turnsSpent === (previous?.turnsSpent ?? 0) + 1 && state.lookups.length === (previous?.lookups.length ?? 0) && state.disposition === null) { + const why = state.lastRejection; + return { + announce: `${turn} refused: ${why}. It still cost a turn. ${meter}`, + caption: `refused: ${shorten(why)}`, + }; + } + + const d = state.disposition; + if (d !== null) { + const what = d.action === 'close' ? `closed, ${d.reason.toLowerCase().replace(/_/g, ' ')}` : `escalated as ${d.typology.toLowerCase().replace(/_/g, ' ')}`; + const cites = `citing ${d.cites.length} ${d.cites.length === 1 ? 'id' : 'ids'}`; + const verdict = + state.outcome === 'solved' + ? 'Right call.' + : state.reveal?.label === 'suspicious' + ? 'Missed: the alert was suspicious.' + : 'A clean customer was escalated.'; + return { + announce: `${turn}: ${what}, ${cites}. ${verdict} ${meter}`, + caption: d.action === 'close' ? 'closed' : 'escalated', + }; + } + + if (state.status === 'aborted') { + return { + announce: `${turn}: no disposition in eight replies. The alert stays in the queue. ${meter}`, + caption: 'aborted', + }; + } + + const lookup = state.lookups[state.lookups.length - 1]; + if (lookup && state.lookups.length > (previous?.lookups.length ?? 0)) { + switch (lookup.what) { + case 'history': + return { + announce: `${turn}: looked up ${lookup.key}, ${lookup.result.transactions.length} transactions. ${meter}`, + caption: `history ${lookup.key}`, + }; + case 'counterparty': + return { + announce: `${turn}: looked up ${lookup.key}, ${lookup.result.name}, shared with ${lookup.result.other_alerted_accounts} other alerted accounts. ${meter}`, + caption: `counterparty ${lookup.key}`, + }; + case 'prior_alerts': + return { + announce: `${turn}: looked up prior alerts, ${lookup.result.alerts.length} on file. ${meter}`, + caption: 'prior alerts', + }; + case 'documents': + return { + announce: `${turn}: looked up documents, ${lookup.result.documents.length} on file. ${meter}`, + caption: 'documents', + }; + } + } + + return { announce: `${turn}: nothing changed. ${meter}`, caption: 'no change' }; +} + +function shorten(text: string): string { + return text.length > 40 ? `${text.slice(0, 37)}…` : text; +} diff --git a/src/demos/alert-triage/controls.tsx b/src/demos/alert-triage/controls.tsx new file mode 100644 index 0000000..6f951f3 --- /dev/null +++ b/src/demos/alert-triage/controls.tsx @@ -0,0 +1,354 @@ +import { useId, useMemo, useState } from 'react'; + +import { Button } from '@/components/ui/button'; +import { cn } from '@/lib/utils'; + +import { enumLabel, fieldLabel, hours } from './format'; +import { MAX_TURNS, PRICES, play } from './game'; +import { DOC_MINUTES, LOOKUP_MINUTES, TURN_MINUTES } from './score'; +import { CLOSE_REASONS, TYPOLOGIES, type CloseReason, type TriageState, type Typology } from './types'; + +/** + * The analyst's controls: four lookups with their prices on the button, and + * a disposition with a citation checklist. + * + * Every control produces exactly the JSON object a model would have to + * produce, and hands it to the same engine. The visitor never sees a control + * the model does not have, and the reply the engine received is shown under + * the controls so the action set is learned by using it. The controls own no + * state the board does not: the next board comes back from `play()` and goes + * to `onChange`, and the only local state here is what is half-typed. + */ + +type Panel = 'none' | 'close' | 'escalate'; + +const price = (minutes: number) => `+${hours(minutes)} h`; + +function Section({ title, children }: { title: string; children: React.ReactNode }) { + return ( +
+ {title} + {children} +
+ ); +} + +const SELECT = + 'tap min-w-0 rounded-md border border-border bg-surface px-2 text-sm text-fg focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-brand disabled:opacity-50'; + +/** The citable ids, grouped the way the screen groups them. */ +function groupShown(shown: readonly string[]) { + const groups: { label: string; ids: string[] }[] = [ + { label: 'KYC file', ids: [] }, + { label: 'Transactions', ids: [] }, + { label: 'Counterparties', ids: [] }, + { label: 'Prior alerts', ids: [] }, + { label: 'Documents', ids: [] }, + { label: 'Other', ids: [] }, + ]; + const byId = (a: string, b: string) => { + const na = Number(a.replace(/^\D+/, '')); + const nb = Number(b.replace(/^\D+/, '')); + return Number.isFinite(na) && Number.isFinite(nb) && na !== nb ? na - nb : a.localeCompare(b); + }; + for (const id of shown) { + const index = id.startsWith('kyc.') + ? 0 + : id.startsWith('T-') + ? 1 + : id.startsWith('CP-') + ? 2 + : id.startsWith('A-') + ? 3 + : id.startsWith('doc.') + ? 4 + : 5; + groups[index]!.ids.push(id); + } + for (const group of groups) group.ids.sort(byId); + return groups.filter((group) => group.ids.length > 0); +} + +export function Controls({ + state, + onChange, +}: { + state: TriageState; + onChange: (next: TriageState) => void; +}) { + const ids = useId(); + const [month, setMonth] = useState(''); + const [cp, setCp] = useState(''); + const [panel, setPanel] = useState('none'); + const [reason, setReason] = useState('CONSISTENT_WITH_PROFILE'); + const [typology, setTypology] = useState('STRUCTURING'); + const [cites, setCites] = useState([]); + const [prose, setProse] = useState(''); + const [lastReply, setLastReply] = useState(null); + + const done = state.status !== 'playing'; + const left = MAX_TURNS - state.turnsSpent; + + const send = (object: Record) => { + const reply = JSON.stringify(object); + setLastReply(reply); + onChange(play(state, reply)); + }; + + const monthsLeft = useMemo( + () => state.screen.months.filter((m) => !state.lookups.some((l) => l.what === 'history' && l.key === m)), + [state.screen.months, state.lookups], + ); + const cpsLeft = useMemo( + () => + state.shown + .filter((id) => id.startsWith('CP-')) + .filter((id) => !state.lookups.some((l) => l.what === 'counterparty' && l.key === id)) + .sort((a, b) => Number(a.slice(3)) - Number(b.slice(3))), + [state.shown, state.lookups], + ); + const priorsMade = state.lookups.some((l) => l.what === 'prior_alerts'); + const docsMade = state.lookups.some((l) => l.what === 'documents'); + const groups = useMemo(() => groupShown(state.shown), [state.shown]); + + const toggleCite = (id: string) => + setCites((current) => (current.includes(id) ? current.filter((c) => c !== id) : [...current, id])); + + const submitDisposition = () => { + if (cites.length === 0 || done) return; + if (panel === 'close') send({ action: 'close', reason, cites, note: prose }); + else if (panel === 'escalate') send({ action: 'escalate', typology, cites, narrative: prose }); + setPanel('none'); + setCites([]); + setProse(''); + }; + + const status = done + ? state.status === 'aborted' + ? `No disposition in ${MAX_TURNS} replies. The alert stays in the queue.` + : `Disposition recorded after ${hours(state.minutesSpent)} hours. The board shows what the generator planted.` + : state.lastRejection + ? `That reply was not accepted: ${state.lastRejection}. It still cost a turn. ${left} ${left === 1 ? 'reply' : 'replies'} left.` + : `Hours spent so far: ${hours(state.minutesSpent)}. ${left} ${left === 1 ? 'reply' : 'replies'} left.`; + + return ( +
+ {/* The board is a group of tables; the outcome of every click is said + here in words, where it announces itself. */} +

+ {status} +

+ +
+
+
+ + + +
+ +
+ + + +
+ +
+ + +
+
+
+ +
+
+ + +
+ + {panel !== 'none' && !done ? ( +
{ + event.preventDefault(); + submitDisposition(); + }} + > + + +
+ + Cite the facts that decide it — only ids you have been shown + +
+ {groups.map((group) => ( +
+

{group.label}

+
+ {group.ids.map((id) => { + const on = cites.includes(id); + return ( + + ); + })} +
+
+ ))} +
+
+ +