PIG/PIG-Demo
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Alert Triage: environment #2, built end to end by the pipeline
ci / web (push) Successful in 2m43s
Details
ci / python (push) Successful in 2m36s
Details

Browse Source 
The first environment shipped through .claude/workflows/new-environment.js:
specification, three adversarial reviews (all 'fixable', none fatal), the
Python environment, the TypeScript port, captured rollouts, and the demo page.
Eleven agents, no errors.

The proof that the platform scales is one line long. Alert Triage has a
completely different shape from Word Five — JSON actions, priced lookups, an
analyst screen instead of a grid — and the only change under
src/components/demo/ is a comment edit, because the isolation lint refused the
word "wordle" there. Zero shell code changed. 415 contract checks now pass
against two demos, up from 206 against one.

The environment is honest by construction. Every alert is synthetic, generated
from the seed, and the banner saying so sits inside the board surface. Two of
the eleven scenario templates are hidden-suspicious: generated by the same code
as their benign twin with the signal overlaid only in lookup data, so the free
screen is identically distributed and a screen-only policy STRUCTURALLY cannot
tell them apart. The probe ladder measures it: `fast` catches 0.0 of hidden
seeds. That is the counterweight made real rather than asserted.

Twelve policies, thirteen ladder assertions, a genuine three-way trade:

  fast      0.846   wins hours (0.85), misses every hidden case
  targeted  0.894   wins the shipped total
  thorough  0.820   wins evidence (1.00), spends 2.9 hours

None dominates. 92 Python tests, 35 TypeScript tests, 65 fixtures replaying at
delta 0, and conformance gated on world + scorer + protocol so the browser shows
the same alert for ?seed= that Python generated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019mt6sHQHEnEYrJZvoMCJSB
This commit is contained in:
karti-ai
2026-08-28 19:48:36 -07:00
parent 1239dc7034
commit 2dfa96939e
91 changed files with 13763 additions and 182 deletions
Download Patch File Download Diff File Expand all files Collapse all files
envs/alert_triage/CONFORMANCE.txt
+3
View File
@@ -0,0 +1,3 @@
505ae48b664aae23aea01208879d899701cc2f6babd8e76ccbe36e75f9ba1b34 world sha256 over canonical world JSON, seeds 0-4095, newline-joined, reference_minutes included
f370d5f6f28964a070e926e7469ffbdea595fcb9ce45b4dbc7e16f607ed4cc47 scorer sha256 over score_exact() of the fixed grid in conformance.py, newline-joined
63dafb1034f19973fd208345f5bec8c7abd9839d8e1c11c2bb1ed8d581f34205 protocol sha256 over protocol_trace() of conformance/replies.json, newline-joined
envs/alert_triage/README.md
+259
View File
@@ -0,0 +1,259 @@
# alert-triage — the environment
One transaction-monitoring alert on one synthetic customer. The agent reads
the analyst's screen, may pay analyst-hours for lookups, and ends the episode
by closing the alert with a cited rationale or escalating it with a typology
and the transactions that constitute it. The grader compares the disposition
and the citations to what the generator planted and charges the hours. No
model, human or prose is read anywhere in it.
`SPEC.md` is the specification as written and reviewed before this code
existed. This file records **what was implemented**, including every change
the reviewers demanded; where the two differ, this file is what ships.
```bash
uv run python envs/probe.py --taskset alert-triage # the ladder
uv run pytest envs/alert_triage/tests # ~30 s; hashes 4096 worlds
uv run python -m alert_triage.conformance # the three digests
```
## Layout
| file | what |
|---|---|
| `alert_triage/rng.py` | FNV-1a seeding, xorshift32, civil dates. No platform RNG, no platform date. |
| `alert_triage/generator.py` | Templates, overlays, the canonical world, `reference_minutes`, digest (a). |
| `alert_triage/engine.py` | Screen, lookups, dispositions, shown-id discipline, the minute meter. |
| `alert_triage/protocol.py` | The pinned parse rules, the system prompt, text renderers. |
| `alert_triage/reward.py` | The three components. The `# region: pig-demo/reward` block is what the page quotes. |
| `alert_triage/policies.py` | `fast`, `targeted`, `thorough` and the seven cheats. Read only the engine's view. |
| `alert_triage/ladder.py` | The probe rungs and their thirteen assertions. |
| `alert_triage/conformance.py` | Digests (b) and (c); writes `CONFORMANCE.txt`. |
| `alert_triage/taskset.py` | `play_episode`, `replay`, and the optional verifiers Env. |
| `conformance/replies.json` | The reply corpus digest (c) runs. |
| `names/` | The synthetic name and word lists, with `PROVENANCE.md`. |
## The world, and what the reviewers changed
**Templates.** Six benign builders (B1 seasonal cash, B2 processor
settlement, B3 property sale, B4 cash restaurant, B5 family remittance, B6
inheritance) and three visible-suspicious ones (S1 structuring, S3 money mule,
S6 smurfing over time). The hidden tier is no longer two fixed twins. A
hidden alert is any benign builder run to completion — screen frozen — with
one of three **overlays** applied to lookup data only:
| overlay | where it lives | compatible with | planted set |
|---|---|---|---|
| `funnel` | the counterparty lookup: the top outbound counterparty on screen is shared with 5–14 other alerted accounts | every benign template | `CP-x` + its transactions; alternate: the transactions |
| `serial_closer` | the prior-alerts lookup: every prior alert on the same rule, closed within a day on one identical boilerplate, by different analysts | screens showing ≥ 3 prior alerts and no prior report | the `A-` ids + the triggering transactions; alternate: the `A-` ids |
| `doc_mismatch` | the documents lookup against the counterparty lookup: the document names a party; the funds came from someone else | B2, B3, B6 | the document + `CP-x` + its triggering credits; alternate: the document + `CP-x` |
The tier is drawn *before* the template and the screen, and the prior-alert
count on the KYC block is drawn before any template code runs, so the screen
distribution is the same for hidden and benign by construction, not by
effort. `tests/test_triage_screen_leak.py` asserts the screen is
byte-identical with every compatible overlay on or off for every seed, and
that no screen feature separates the populations beyond four sigma.
This answers three reviewer findings at once: every rule id now fires on both
labels (`R-WIRE-04` and `R-DORM-05` carry funnel and serial-closer twins);
each hidden signal lands on several templates so there is no template→lookup
mapping to memorise; and no screen-identifiable segment carries a suspicious
prior above the overall 15 / (55 + 15) ≈ 21 %.
**Removed:** B4's standalone `PREVIOUSLY_REVIEWED` alternate. A prior closure
never resolves an alert on its own. The reason stays in the enum as legal and
never planted, so its use renders as a diagnostic.
**Documents** carry per-world ids (`doc.D-<4 hex>`), and the engine rejects
any citation of an id it has not rendered. `fast` therefore pulls
`documents` on the templates a document explains, and the reference on those
seeds is 0.70 h, not 0.55 h.
**The held-out slice** is `fnv1a(str(seed)) % 8 == 7`, about 12.5 % of
seeds. It is generated in a **fictional** currency, KRN, with a fictional
20,000 KRN reporting threshold (amounts scale ×2), rule ids renamed
`TM-A1..A5`, the KYC block before the alert and the summary newest-first. It
is never probed and never used to tune anything; the capture stage runs it
once, after the arms are frozen, for the side-by-side numbers on the page.
No claim about any real jurisdiction's threshold is made anywhere.
## Canonical world
Sorted keys, no whitespace, `null` for absent. Every amount is an integer in
minor units; every date is a `YYYY-MM-DD` string produced by
`rng.iso_date` from a day ordinal. `reference_minutes` and
`reference_policy` are **part of the world** and therefore of digest (a):
the browser recomputes them by running the ported policies rather than
trusting the recorded number, and a mismatch is a verification failure, not
an "unverifiable".
Draw order is the contract. Reading `generator.py` top to bottom in the
order the builder calls things is the port; the port must not reorder a
single `rng` call.
## The action set and the pinned parse rules
One JSON object per reply. A ```` ```json ```` fence anywhere wins; otherwise
the first string-aware balanced `{…}` span; an unparseable or non-object
candidate is a rejection with no further scanning. Enums exact-case. Cites
deduplicated in order, matched exact-case against the shown set. The
twelve-month window is the twelve calendar months ending in the fire month
inclusive. A repeated lookup is rejected. Only accepted lookups add their
price; every turn, accepted or rejected, costs `TURN_MINUTES`.
`aborted` is exactly `MAX_TURNS = 8` turns spent with no accepted
disposition and scores 0.000. A run that stops earlier with no disposition is
`truncated` and every component is `null` — rendered "not scored", never
zero. Correct disposition → `solved`, wrong → `failed`.
## Hours
Integers in minutes, shown as hours. **The generator's config, not
measurements** — nobody timed an analyst:
| constant | minutes | hours |
|---|---|---|
| `SCREEN_MINUTES` | 30 | 0.50 |
| `TURN_MINUTES` | 3 | 0.05 |
| `LOOKUP_MINUTES` | 24 | 0.40 |
| `DOC_MINUTES` | 6 | 0.10 |
`LOOKUP_MINUTES` was raised from the spec's 0.25 h to 0.40 h so that the
hours spent investigating clean screens genuinely exceed the hours a
screen-only analyst loses to misses — the trade-off the reviewers said was
manufactured in the spec is real at these constants and the ladder measures
it (rung 8).
## The reward
| key | weight | role | definition |
|---|---|---|---|
| `caught` | 0.55 | counterweight | 1.0 on a correct call; 0.0 on a miss or no disposition; **0.5 on a false escalation** ("filed on a clean customer"). |
| `hours` | 0.25 | objective | `min(1, reference / spent)` if correct, else 0. `null` if no shipped policy got the seed right (CI asserts this never happens). Capped: faster than the reference earns 1.0, not more. |
| `evidence` | 0.20 | objective | F1 of the cites against the best planted alternate; 0 if the call was wrong. |
The 0.5 credit is the reviewers' fix for segment-blind defensive filing: with
it, a blind escalation only beats a well-cited close when P(benign) is below
**0.52** (`reward.escalate_break_even()`, printed by the probe), and no
benign-looking screen segment is that suspicious.
## The ladder, measured
`uv run python envs/probe.py --taskset alert-triage`, 894 main-slice seeds
(seeds 0–1023 minus the held-out bucket). These are the numbers the page may
quote; the spec's `~0.86 vs ~0.85` text may not.
```
policy caught hours evidence TOTAL hours@0.45
inaction 0.0000 0.0000 0.0000 0.0000 0.0000
malformed 0.0000 0.0000 0.0000 0.0000 0.0000
close_blind 0.5705 0.5705 0.0000 0.4564 0.4868
close_cites_all 0.5705 0.5705 0.0815 0.4727 0.4988
cite_unseen 0.5705 0.5473 0.0000 0.4506 0.4764
escalate_blind 0.7148 0.4295 0.3006 0.5606 0.5257
fast_defensive 0.7148 0.4295 0.3006 0.5606 0.5257
rule_table 0.7550 0.5593 0.3680 0.6287 0.6102
fast 0.8535 0.8535 0.8159 0.8460 0.8480
targeted 1.0000 0.6039 0.9625 0.8935 0.8163
thorough 1.0000 0.2784 1.0000 0.8196 0.6753
thorough_wasteful 1.0000 0.2412 1.0000 0.8103 0.6585
```
Mean hours: fast 0.60, targeted 1.63, thorough 2.92. Caught on hidden seeds:
fast 0.0, targeted 1.0, thorough 1.0.
The three shipped analysts, and the claim the page can make:
- **`fast`** decides from the screen (plus `documents` where a document
should explain the alert). Wins `hours` outright. Misses every hidden
seed, and only those.
- **`targeted`** is `fast` on a screen that shows a typology, and otherwise
pulls exactly the three lookups that could carry a hidden signal. Catches
everything. **Wins at shipped weights, 0.894 to 0.846.** It defines
`reference_minutes` on hidden seeds.
- **`thorough`** always runs the full procedure, including two months of
itemised history, and cites every constituting transaction it saw:
evidence 1.000 exactly, and the lowest hours.
Move `hours` to **0.45** (the other two weights keeping their ratio) and
`fast` wins, 0.848 to 0.816. Nothing else on the page depends on the
direction; rung 8 asserts both margins are at least 0.02.
Note the weakest honest reading: `thorough` — which gets every call and
every citation right — scores *below* `fast` at shipped weights, because
spending five times the hours on the 85 % of alerts the screen settles costs
more than the misses. That is the environment's argument, not a defect.
### Rungs a reviewer named
| rung | the cheat | assertion |
|---|---|---|
| `fast_defensive` | file blind on every benign-looking screen | total < every good policy |
| `cite_unseen` | close citing a document it never opened | rejected on every seed; total < fast |
| `rule_table` | close blind on wire/dormancy rules, file blind elsewhere | total < every good policy |
| `close_cites_all` | cite every id on the screen | evidence < ½ its caught share; < fast's evidence |
| `thorough_wasteful` | pull a month that carries nothing | total < thorough, same caught and evidence |
## The reference analysts see only the view
`policies.py` functions take `Engine.view()` and nothing else — the
screen, the lookups so far, the turn and minute counters. Tests assert the
signature, that the view carries none of `label`, `typology`, `planted`,
`overlay`, `tier`, `template`, `reference_*`, and that every policy emits
identical replies against a world with those fields deleted.
## Conformance
Three digests in `CONFORMANCE.txt`, each a CI gate for the TypeScript port:
1. **world** — canonical world JSON for seeds 0–4095, reference included.
The port must carry the generator, not only the scorer: the Play tab and
`?seed=` need the same alert on both sides.
2. **scorer** — `score_exact()` over a fixed grid of episodes. Exact
fractions, so no float formatting is hashed.
3. **protocol** — parse + engine over `conformance/replies.json` on fixed
worlds: accepted actions, rejection reasons, and the minute meter after
every turn. This is the digest wordle does not need.
## What this is easier than, for the page
1. Every alert has a knowable answer; real SAR decisions are judgements, and a
filed report is not a proven crime. Nine builders and three overlays stand
in for a typology space in the hundreds.
2. The 55/30/15 mix is deliberately not the real base rate. The great
majority of production alerts close as false positives — no figure is
quoted here because none was measured by us. "Hours per thousand alerts"
on the page is Σ `hours_spent` over recorded seeds inside this
environment, with the constants above beside it.
3. The hour constants are assumptions.
4. **The lookups are idealised.** Each returns the decisive fact in one
call: a counterparty's other-alerted-account count, a prior alert's
analyst and rationale, a document's named party. In production those
signals are spread across systems and often absent. This environment
tests whether the agent asks; it does not test whether the answer exists.
5. The narrative prose is unscored.
6. The adversary is static: the launderer shaped the data to pass the screen
and the held-out slice moves the threshold once. Nothing responds to the
agent's policy. The demo must claim exactly that and no more, and the
wordle limit that names this demo as its answer should be softened to
"nobody designed the data to pass the screen".
7. `caught` is 1.0 on every benign alert by construction; the counterweight
is argued from the ladder over seeds, never from one run.
## For the next stages
- **Port**: `rng.py`, `generator.py`, `engine.py`, `protocol.py`,
`reward.py`, `policies.py` (`fast`, `targeted`, `thorough` — verify
recomputes `reference_minutes`). Gate on all three digests.
- **Capture**: `taskset.play_episode(seed, respond)` is the loop;
`taskset.replay(seed, replies)` re-scores a fixture from its reply strings.
Scripted arms: `fast`, `targeted`, `thorough`, and at least one cheat.
Capture the held-out bucket exactly once, after the arms are frozen.
- **Demo**: the board must carry "Synthetic data — no real customer, account
or transaction"; the hour constants as config; the narrative marked "not
scored"; the label reveal only after a disposition. The vertical card's
`reward` line must be amended to say "hours as a ratio against a reference
analyst policy", not "hours per thousand alerts".
- `SPEC.md` is stage 1's document and was not edited; §3, §4, §6, §7 and
§8 are superseded by this file where they differ.
envs/alert_triage/alert_triage/__init__.py
+11
View File
@@ -0,0 +1,11 @@
"""alert-triage: one synthetic transaction-monitoring alert per seed.
Public surface, in the order the next stages need it:
generator.world_for_seed(seed) the world, canonical and portable
engine.Engine(world) lookups, dispositions, the hour meter
protocol.parse_reply(text) one JSON object -> action or rejection
reward.score(episode) the three components, or None
policies.POLICIES the scripted analysts
taskset.play_episode(seed, fn) the loop capture, probe and tests share
"""
envs/alert_triage/alert_triage/conformance.py
+124
View File
@@ -0,0 +1,124 @@
"""The three cross-language digests. `CONFORMANCE.txt` commits them; CI gates them.
(a) world SHA-256 over the canonical JSON world for seeds 0–4095,
reference minutes included. The browser must show the same
alert for ?seed= as Python generated.
(b) scorer SHA-256 over the exact-fraction scores of a fixed grid of
synthetic episodes: label × disposition × cites × planted ×
minutes × reference. No floats are hashed.
(c) protocol SHA-256 over what parse + engine make of a committed corpus of
reply strings (`conformance/replies.json`) on fixed worlds:
accepted actions, rejection reasons and the minute meter after
every turn. This is the digest wordle does not need and this
environment cannot do without — JSON-in-prose with fences and a
brace scan is where two runtimes disagree.
Run: uv run python -m alert_triage.conformance [--write]
"""
from __future__ import annotations
import hashlib
import json
import sys
from pathlib import Path
from .engine import Engine
from .generator import canonical_json, world_digest, world_for_seed
from .reward import Episode, score_exact
ROOT = Path(__file__).parent.parent
CORPUS = ROOT / "conformance" / "replies.json"
COMMITTED = ROOT / "CONFORMANCE.txt"
GRID_LABELS = ("benign", "suspicious")
GRID_DISPOSITIONS = ("close", "escalate", "none", "truncated")
GRID_CITES = (["a"], ["a", "b"], ["a", "b", "c", "d"], ["x"], ["a", "x"], ["a", "b", "x", "y", "z"], ["a", "a", "b"], ["kyc.z"])
GRID_PLANTED = ([["a"]], [["a", "b"]], [["a", "b"], ["a"]], [["c"], ["a", "b", "c", "d"]])
GRID_MINUTES = (33, 42, 60, 96, 150, 204)
GRID_REFERENCE = (33, 42, 96, None)
def scorer_digest() -> str:
digest = hashlib.sha256()
for label in GRID_LABELS:
for disposition in GRID_DISPOSITIONS:
for cites in GRID_CITES:
for planted in GRID_PLANTED:
for minutes in GRID_MINUTES:
for reference in GRID_REFERENCE:
ep = Episode(
label=label,
planted=planted,
disposition=None if disposition in ("none", "truncated") else disposition,
cites=[] if disposition in ("none", "truncated") else cites,
minutes_spent=minutes,
reference_minutes=reference,
turns_spent=1,
rejected=0,
truncated=disposition == "truncated",
)
digest.update(canonical_json(score_exact(ep)).encode())
digest.update(b"\n")
return digest.hexdigest()
def protocol_trace(corpus: list[dict] | None = None) -> list[dict]:
corpus = corpus if corpus is not None else json.loads(CORPUS.read_text())
out = []
for case in corpus:
engine = Engine(world_for_seed(case["seed"]))
turns = []
for reply in case["replies"]:
if engine.done:
break
step = engine.step(reply)
turns.append({"accepted": step.accepted, "action": step.action, "reason": step.reason, "minutes": step.minutes_spent, "turns": step.turns_spent})
ep = engine.episode()
out.append({"seed": case["seed"], "turns": turns, "outcome": engine.outcome, "truncated": ep.truncated, "score": score_exact(ep)})
return out
def protocol_digest() -> str:
digest = hashlib.sha256()
for entry in protocol_trace():
digest.update(canonical_json(entry).encode())
digest.update(b"\n")
return digest.hexdigest()
def all_digests() -> dict[str, str]:
return {"world": world_digest(), "scorer": scorer_digest(), "protocol": protocol_digest()}
def committed() -> dict[str, str]:
out = {}
for line in COMMITTED.read_text().splitlines():
parts = line.split()
if len(parts) >= 2:
out[parts[1]] = parts[0]
return out
def main(argv: list[str]) -> int:
digests = all_digests()
lines = [
f"{digests['world']} world sha256 over canonical world JSON, seeds 0-4095, newline-joined, reference_minutes included",
f"{digests['scorer']} scorer sha256 over score_exact() of the fixed grid in conformance.py, newline-joined",
f"{digests['protocol']} protocol sha256 over protocol_trace() of conformance/replies.json, newline-joined",
]
if "--write" in argv:
COMMITTED.write_text("\n".join(lines) + "\n")
print(f"wrote {COMMITTED}")
for line in lines:
print(line)
if COMMITTED.exists():
mismatched = [k for k, v in committed().items() if digests.get(k) != v]
if mismatched:
print(f"MISMATCH against CONFORMANCE.txt: {mismatched}")
return 1
return 0
if __name__ == "__main__":
raise SystemExit(main(sys.argv[1:]))
envs/alert_triage/alert_triage/engine.py
+229
View File
@@ -0,0 +1,229 @@
"""One episode: the free screen, up to eight replies, and the hour meter.
Pure and deterministic in the world it is given. It never raises on input: a
bad reply is a rejection with a one-line reason, charged a turn.
Two discipline rules the reviewers demanded, both enforced here rather than in
the grader:
* **Shown ids.** A disposition may cite only ids the engine has actually
rendered — KYC fields and triggering transactions from the screen, plus
whatever lookups returned. "Closed per the probate letter on file"
without opening the file is a rejected reply, not a scored one.
* **Abort versus truncate.** `aborted` is exactly MAX_TURNS turns spent
with no accepted disposition, and scores 0.000. Anything that stops
earlier with no disposition is `truncated` and is not scored at all.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any
from .generator import MAX_TURNS, screen_of
from .protocol import (
ACTIONS,
CLOSE_REASONS,
LOOKUPS,
TYPOLOGIES,
_MONTH,
parse_reply,
)
from .reward import DOC_MINUTES, LOOKUP_MINUTES, SCREEN_MINUTES, TURN_MINUTES, Episode
LOOKUP_PRICE = {"history": LOOKUP_MINUTES, "counterparty": LOOKUP_MINUTES, "prior_alerts": LOOKUP_MINUTES, "documents": DOC_MINUTES}
@dataclass
class Step:
accepted: bool
action: dict[str, Any] | None # canonical form of an accepted action
reason: str | None # why a rejected reply was rejected
result: dict[str, Any] | None # what an accepted lookup returned
minutes_spent: int
turns_spent: int
done: bool
@dataclass
class Engine:
world: dict[str, Any]
turns: int = 0
rejected: int = 0
minutes: int = SCREEN_MINUTES
actions: list[dict[str, Any]] = field(default_factory=list)
lookups: list[dict[str, Any]] = field(default_factory=list)
disposition: dict[str, Any] | None = None
shown: set[str] = field(default_factory=set)
last_rejection: str | None = None
def __post_init__(self) -> None:
screen = self.screen()
self.shown.update(f"kyc.{k}" for k in screen["customer"] if k not in ("id", "name"))
self.shown.add(screen["alert"]["id"])
for t in screen["triggering_transactions"]:
self.shown.add(t["id"])
if t["cp"]:
self.shown.add(t["cp"])
for cp in screen["screen_counterparties"]:
self.shown.add(cp["id"])
# ------------------------------------------------------------ views --
def screen(self) -> dict[str, Any]:
screen = screen_of(self.world)
if screen["screen_order"] == "kyc_first":
screen["summary"] = list(reversed(screen["summary"]))
return screen
def view(self) -> dict[str, Any]:
"""Everything the model has been shown, and nothing it has not.
The scripted policies take this and only this. There is no label,
typology, planted set, overlay or reference in it — `tests/` scans
the keys to make sure.
"""
return {
"screen": self.screen(),
"lookups": list(self.lookups),
"turns_spent": self.turns,
"turns_left": MAX_TURNS - self.turns,
"rejected": self.rejected,
"minutes_spent": self.minutes,
"last_rejection": self.last_rejection,
}
@property
def done(self) -> bool:
return self.disposition is not None or self.turns >= MAX_TURNS
@property
def aborted(self) -> bool:
return self.disposition is None and self.turns >= MAX_TURNS
# ------------------------------------------------------------- step --
def step(self, reply: str | None) -> Step:
if self.done:
return Step(False, None, "the episode is over", None, self.minutes, self.turns, True)
obj, reason = parse_reply(reply)
action = None
if obj is not None:
action, reason = self._validate(obj)
self.turns += 1
self.minutes += TURN_MINUTES
if action is None:
self.rejected += 1
self.last_rejection = reason
return Step(False, None, reason, None, self.minutes, self.turns, self.done)
self.last_rejection = None
self.actions.append(action)
result = None
if action["action"] == "lookup":
self.minutes += LOOKUP_PRICE[action["what"]]
result = self._lookup(action)
self.lookups.append({"what": action["what"], "key": action.get("month") or action.get("id"), "result": result})
else:
self.disposition = action
return Step(True, action, None, result, self.minutes, self.turns, self.done)
def _validate(self, obj: dict[str, Any]) -> tuple[dict[str, Any] | None, str | None]:
action = obj.get("action")
if action not in ACTIONS:
return None, f"unknown action {action!r}; use lookup, close or escalate"
if action == "lookup":
what = obj.get("what")
if what not in LOOKUPS:
return None, f"unknown lookup {what!r}; use history, counterparty, prior_alerts or documents"
if what == "history":
month = obj.get("month")
if not isinstance(month, str) or not _MONTH.match(month) or month not in self.world["months"]:
return None, f"month {month!r} is not in the twelve-month window"
if any(l["what"] == "history" and l["key"] == month for l in self.lookups):
return None, f"history for {month} has already been looked up"
return {"action": "lookup", "what": "history", "month": month}, None
if what == "counterparty":
cid = obj.get("id")
if not isinstance(cid, str) or cid not in self.shown or not cid.startswith("CP-"):
return None, f"counterparty {cid!r} has not appeared on your screen or in a fetched month"
if any(l["what"] == "counterparty" and l["key"] == cid for l in self.lookups):
return None, f"counterparty {cid} has already been looked up"
return {"action": "lookup", "what": "counterparty", "id": cid}, None
if any(l["what"] == what for l in self.lookups):
return None, f"{what} has already been looked up"
return {"action": "lookup", "what": what}, None
cites = obj.get("cites")
if not isinstance(cites, list) or not cites:
return None, "cites must be a non-empty list of ids"
seen: list[str] = []
for c in cites:
if not isinstance(c, str):
return None, "cites must be a non-empty list of ids"
if c not in seen:
seen.append(c)
for c in seen:
if c not in self.shown:
return None, f"cites an id you have not been shown: {c}"
if action == "close":
reason = obj.get("reason")
if reason not in CLOSE_REASONS:
return None, f"unknown close reason {reason!r}"
note = obj.get("note")
return {"action": "close", "reason": reason, "cites": seen, "note": note if isinstance(note, str) else ""}, None
typology = obj.get("typology")
if typology not in TYPOLOGIES:
return None, f"unknown typology {typology!r}"
narrative = obj.get("narrative")
return {"action": "escalate", "typology": typology, "cites": seen, "narrative": narrative if isinstance(narrative, str) else ""}, None
def _lookup(self, action: dict[str, Any]) -> dict[str, Any]:
w = self.world
what = action["what"]
if what == "history":
rows = [t for t in w["transactions"] if t["month"] == action["month"]]
for t in rows:
self.shown.add(t["id"])
if t["cp"]:
self.shown.add(t["cp"])
return {"month": action["month"], "transactions": rows}
if what == "counterparty":
cp = next(c for c in w["counterparties"] if c["id"] == action["id"])
self.shown.update(cp["transactions"])
return dict(cp)
if what == "prior_alerts":
self.shown.update(p["id"] for p in w["prior_alerts"])
return {"alerts": list(w["prior_alerts"])}
self.shown.update(d["id"] for d in w["documents"])
return {"documents": list(w["documents"])}
# ---------------------------------------------------------- outcome --
@property
def outcome(self) -> str | None:
"""solved | failed | aborted, or None while running / if truncated."""
if self.disposition is not None:
label = self.world["label"]
right = (label == "benign") == (self.disposition["action"] == "close")
return "solved" if right else "failed"
return "aborted" if self.aborted else None
def episode(self, truncated: bool | None = None) -> Episode:
if truncated is None:
truncated = not self.done
d = self.disposition
return Episode(
label=self.world["label"],
planted=self.world["planted"],
disposition=None if d is None else d["action"],
cites=[] if d is None else list(d["cites"]),
minutes_spent=self.minutes,
reference_minutes=self.world.get("reference_minutes"),
turns_spent=self.turns,
rejected=self.rejected,
truncated=truncated,
typology=None if d is None else d.get("typology"),
true_typology=self.world.get("typology"),
lookups=len(self.lookups),
)
envs/alert_triage/alert_triage/generator.py
+881
View File
@@ -0,0 +1,881 @@
"""The world for a seed: one alert on one synthetic customer.
Everything here is synthetic. No customer, account, transaction, counterparty,
analyst or institution exists behind any of it — see `names/PROVENANCE.md`.
Portability is the whole design. `src/demos/alert-triage/` carries a port of
this file, and CI hashes the canonical JSON of every world for seeds 0–4095 on
both sides. So:
* every random draw goes through `XorShift32` seeded by FNV-1a over the
decimal seed, in the order written here — the order IS the contract;
* every amount is an integer in minor units (cents), never a float;
* every date is a day ordinal turned into YYYY-MM-DD by our own civil-date
function, never a platform date type;
* canonical JSON is sorted keys, no whitespace, `null` for absent.
The hidden tier — a suspicious alert whose free screen is drawn from a benign
template — is produced by running the benign builder to completion, freezing
the screen, and only then overlaying the suspicious signal onto data that a
lookup returns. `screen_of(world)` is byte-identical with the overlay on or
off, and `tests/test_screen_leak.py` asserts it for every seed.
"""
from __future__ import annotations
import hashlib
import json
from pathlib import Path
from typing import Any
from .rng import (
XorShift32,
add_months,
days_from_civil,
days_in_month,
fnv1a32,
iso_date,
month_key,
)
# ------------------------------------------------------------ constants --
MAX_TURNS = 8
WINDOW_MONTHS = 12
SCREEN_COUNTERPARTIES = 2 # top-N by volume shown on the free screen
# Mix by seed. Deliberately not the real false-positive base rate — see
# README.md, "what this is easier than".
BENIGN_PCT = 55
VISIBLE_PCT = 30
# hidden = the remaining 15
FAMILIES = ("STR", "VEL", "CASH", "WIRE", "DORM")
# Rule ids per slice. The held-out slice renames them so a policy that learned
# "9,xxx and R-STR-01" rather than the pattern is caught.
RULE_IDS = {
"main": {"STR": "R-STR-01", "VEL": "R-VEL-02", "CASH": "R-CASH-03", "WIRE": "R-WIRE-04", "DORM": "R-DORM-05"},
"held_out": {"STR": "TM-A1", "VEL": "TM-A2", "CASH": "TM-A3", "WIRE": "TM-A4", "DORM": "TM-A5"},
}
RULE_TEXT = {
"STR": "Three or more cash deposits just under the reporting threshold within ten days",
"VEL": "Funds received and moved out again within 48 hours",
"CASH": "Monthly cash deposits materially above the trailing six-month average",
"WIRE": "Outbound wires to a jurisdiction on the institution's monitored list",
"DORM": "Large credit to an account with no material activity for nine months",
}
# The typology the rule nominally points at — what a defensive filer would
# write. Diagnostic only; `typology` is never scored.
NOMINAL_TYPOLOGY = {"STR": "STRUCTURING", "VEL": "RAPID_MOVEMENT", "CASH": "STRUCTURING", "WIRE": "UNKNOWN", "DORM": "UNKNOWN"}
CURRENCIES = {
# threshold in minor units; scale multiplies every generated amount
"main": {"currency": "USD", "threshold": 1_000_000, "scale": 1},
"held_out": {"currency": "KRN", "threshold": 2_000_000, "scale": 2},
}
BENIGN_TEMPLATES = ("B1", "B2", "B3", "B4", "B5", "B6")
VISIBLE_TEMPLATES = ("S1", "S3", "S6")
OVERLAYS = ("funnel", "serial_closer", "doc_mismatch")
DOC_TEMPLATES = ("B2", "B3", "B6") # where a document is the explaining fact
TEMPLATE_FAMILY = {"B1": "CASH", "B2": "VEL", "B3": "VEL", "B4": "STR", "B5": "WIRE", "B6": "DORM", "S1": "STR", "S3": "VEL", "S6": "CASH"}
INBOUND = ("cash_in", "wire_in", "ach_in", "p2p_in")
_NAMES = Path(__file__).parent.parent / "names"
def _load(name: str):
return json.loads((_NAMES / name).read_text())
FIRST_NAMES: list[str] = _load("first_names.json")
SURNAMES: list[str] = _load("surnames.json")
WORDS: dict[str, list[str]] = _load("words.json")
JURISDICTIONS: dict[str, Any] = _load("jurisdictions.json")
BOILERPLATE = "Reviewed. Activity consistent with customer profile. No further action."
BENIGN_RATIONALE = {
"STR": "Closed: deposits are documented daily takings; amounts vary with trading days; within kyc.expected_monthly_cash.",
"VEL": "Closed: pass-through matches the settlement terms in the agreement on file; counterparties verified.",
"CASH": "Closed: seasonal peak consistent with kyc.expected_activity; volume within stated expectation.",
"WIRE": "Closed: destination and amount match the remittance declared at onboarding; within stated income.",
"DORM": "Closed: reactivation explained by a documented source of funds; document reviewed and retained.",
}
# ----------------------------------------------------------- utilities --
def canonical_json(value: Any) -> str:
return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
def _round_to(cents: int, step: int) -> int:
return (cents // step) * step
class _Ledger:
"""Transactions are collected unordered, then ids are assigned by date.
A stable sort on the day ordinal, with insertion order breaking ties, is
what makes T-ids identical across the two implementations.
"""
def __init__(self) -> None:
self.rows: list[dict[str, Any]] = []
def add(self, day: int, channel: str, amount: int, branch: str | None = None, cp: str | None = None) -> int:
self.rows.append({"day": day, "seq": len(self.rows), "channel": channel, "amount": amount, "branch": branch, "cp": cp})
return len(self.rows) - 1
def finish(self) -> tuple[list[dict[str, Any]], dict[int, str]]:
ordered = sorted(self.rows, key=lambda r: (r["day"], r["seq"]))
id_of: dict[int, str] = {}
out = []
for i, row in enumerate(ordered):
tid = f"T-{i + 1}"
id_of[row["seq"]] = tid
out.append({
"id": tid,
"date": iso_date(row["day"]),
"month": iso_date(row["day"])[:7],
"channel": row["channel"],
"amount": row["amount"],
"branch": row["branch"],
"cp": row["cp"],
})
return out, id_of
class _Ctx:
"""Everything a template builder needs, plus the accumulating world."""
def __init__(self, seed: int, rng: XorShift32, slice_name: str) -> None:
self.seed = seed
self.rng = rng
self.slice = slice_name
cur = CURRENCIES[slice_name]
self.currency = cur["currency"]
self.threshold = cur["threshold"]
self.scale = cur["scale"]
self.rule_ids = RULE_IDS[slice_name]
self.home = JURISDICTIONS["home"][self.currency]
self.ledger = _Ledger()
self.cps: list[dict[str, Any]] = []
self.customer: dict[str, Any] = {}
self.months: list[tuple[int, int]] = []
self.fire_day = 0
self.fire_y = 0
self.fire_m = 0
self.documents: list[dict[str, Any]] = []
self.doc_party_cp: str | None = None # the CP a party-bearing document names
self.branches = rng.sample(WORDS["branches"], 2)
# money: `usd(n)` is n whole units of the slice currency, in minor units.
def usd(self, n: int) -> int:
return n * 100 * self.scale
def money(self, lo: int, hi: int, step: int = 10) -> int:
"""A random whole-unit amount in [lo, hi], rounded down to `step` units."""
return _round_to(self.usd(self.rng.between(lo, hi)), self.usd(step))
def set_fire(self, month_choices: list[int], day_lo: int = 12, day_hi: int = 27) -> None:
self.fire_y = 2026
self.fire_m = self.rng.pick(month_choices)
self.fire_day = days_from_civil(self.fire_y, self.fire_m, self.rng.between(day_lo, day_hi))
self.months = [add_months(self.fire_y, self.fire_m, -(WINDOW_MONTHS - 1 - i)) for i in range(WINDOW_MONTHS)]
def fire_dom(self) -> int:
return self.fire_day - days_from_civil(self.fire_y, self.fire_m, 1) + 1
def day_in(self, i: int, lo: int = 1, hi: int | None = None) -> int:
"""A day ordinal inside window month i. Nothing is ever dated on or after the fire date."""
y, m = self.months[i]
last = days_in_month(y, m) if hi is None else hi
if i == WINDOW_MONTHS - 1:
last = min(last, self.fire_dom() - 1)
lo = min(lo, last)
return days_from_civil(y, m, self.rng.between(lo, last))
def days_in(self, i: int, k: int, lo: int = 1, hi: int | None = None) -> list[int]:
"""k distinct days in month i, ascending."""
y, m = self.months[i]
last = days_in_month(y, m) if hi is None else hi
if i == WINDOW_MONTHS - 1:
last = min(last, self.fire_dom() - 1)
lo = min(lo, last)
first = days_from_civil(y, m, 1)
picks = self.rng.sample(list(range(lo, last + 1)), k)
return sorted(first + d - 1 for d in picks)
def person(self) -> str:
return f"{self.rng.pick(FIRST_NAMES)} {self.rng.pick(SURNAMES)}"
def business(self, noun: str | None = None) -> str:
adj = self.rng.pick(WORDS["adjectives"])
noun = noun or self.rng.pick(WORDS["nouns"])
return f"{adj} {noun} {self.rng.pick(WORDS['suffixes'])}"
def cp(self, name: str, kind: str, jurisdiction: str | None = None) -> str:
cid = f"CP-{len(self.cps) + 1}"
self.cps.append({"id": cid, "name": name, "type": kind, "jurisdiction": jurisdiction or self.home})
return cid
def doc(self, kind: str, description: str, party: str | None = None) -> str:
while True:
did = f"doc.D-{self.rng.hex4()}"
if all(d["id"] != did for d in self.documents):
break
self.documents.append({"id": did, "kind": kind, "description": description, "party": party})
return did
def add(self, day: int, channel: str, amount: int, branch: str | None = None, cp: str | None = None) -> int:
return self.ledger.add(day, channel, amount, branch, cp)
def branch(self) -> str:
return self.rng.pick(self.branches)
def max_month_cash(self) -> int:
"""The largest month of cash actually deposited so far — what the KYC expectation must cover."""
totals: dict[int, int] = {}
for row in self.ledger.rows:
if row["channel"] == "cash_in":
key = iso_date(row["day"])[:7]
idx = int(key[:4]) * 12 + int(key[5:])
totals[idx] = totals.get(idx, 0) + row["amount"]
return max(totals.values()) if totals else 0
def _kyc_common(ctx: _Ctx, customer_type: str) -> None:
"""The KYC draws every template makes, in the same order.
Prior-alert and prior-SAR counts are drawn HERE, before any template or
label-specific code runs, from one distribution. The serial-closer overlay
rewrites what the prior alerts say; it can never change how many there are,
which is the number the screen shows.
"""
rng = ctx.rng
ctx.customer = {
"id": f"C-{rng.between(1000, 9999)}",
"name": "",
"customer_type": customer_type,
"country": ctx.home,
"pep": rng.chance(3),
"prior_alerts": 0,
"prior_sars": 0,
"remittance_declared": "none declared",
}
roll = rng.below(100)
ctx.customer["prior_alerts"] = 0 if roll < 35 else 1 if roll < 55 else 2 if roll < 70 else 3 if roll < 88 else 4
ctx.customer["prior_sars"] = 1 if ctx.customer["prior_alerts"] >= 1 and rng.chance(8) else 0
opened_years = rng.between(2, 15)
ctx.customer["account_opened"] = iso_date(ctx.fire_day - opened_years * 365 - rng.between(0, 300))
ctx.customer["kyc_refreshed"] = iso_date(ctx.fire_day - rng.between(40, 500))
def _salary(ctx: _Ctx, monthly: int, employer: str) -> None:
for i in range(WINDOW_MONTHS):
amount = monthly + ctx.usd(ctx.rng.between(-60, 60))
ctx.add(ctx.day_in(i, 1, 5), "ach_in", amount, cp=employer)
def _small_cash_out(ctx: _Ctx, lo: int, hi: int, per_month_lo: int, per_month_hi: int) -> None:
for i in range(WINDOW_MONTHS):
n = ctx.rng.between(per_month_lo, per_month_hi)
for day in ctx.days_in(i, n):
ctx.add(day, "cash_out", ctx.money(lo, hi), branch=ctx.branch())
def _cash_deposits(ctx: _Ctx, i: int, total: int, n: int, lo_cap: int, hi_cap: int, days: list[int] | None = None) -> list[int]:
"""n cash deposits in month i summing roughly to `total`, each clamped."""
days = days if days is not None else ctx.days_in(i, n)
out = []
for day in days:
amount = total // n * ctx.rng.between(75, 125) // 100
amount = max(lo_cap, min(hi_cap, _round_to(amount, ctx.usd(10))))
out.append(ctx.add(day, "cash_in", amount, branch=ctx.branch()))
return out
# --------------------------------------------------------------- benign --
def _build_B1(ctx: _Ctx) -> dict[str, Any]:
"""Seasonal cash business. CASH fires on the first peak month; the KYC file said it would."""
rng = ctx.rng
kind = rng.pick([
("landscaping services", "Landscaping", [4, 5, 6, 7, 8, 9], [4, 5], "April to September"),
("tax preparation", "Tax Services", [2, 3, 4], [3], "February to April"),
("pool maintenance", "Pool Care", [5, 6, 7, 8], [5, 6], "May to August"),
])
business_type, noun, peak, fire_choices, peak_text = kind
ctx.set_fire(fire_choices)
_kyc_common(ctx, "business")
ctx.customer["name"] = ctx.business(noun)
ctx.customer["business_type"] = business_type
off = ctx.money(3000, 7000, 100)
mult = rng.between(30, 40)
supplier = ctx.cp(ctx.business("Supply"), "business")
equipment = ctx.cp(ctx.business("Equipment"), "business")
for i, (y, m) in enumerate(ctx.months):
in_peak = m in peak
total = off * mult // 10 if in_peak else off
total = total * rng.between(85, 115) // 100
n = rng.between(8, 12) if in_peak else rng.between(3, 5)
_cash_deposits(ctx, i, total, n, ctx.usd(200), ctx.threshold * 8 // 10 - ctx.usd(100))
ctx.add(ctx.day_in(i, 20, 28), "ach_out", total * rng.between(20, 35) // 100, cp=supplier)
if m % 3 == 0:
ctx.add(ctx.day_in(i, 6, 12), "wire_out", ctx.money(2500, 9000, 100), cp=equipment)
expected = _round_to(ctx.max_month_cash() * rng.between(110, 130) // 100, ctx.usd(1000)) + ctx.usd(1000)
ctx.customer["expected_monthly_cash"] = expected
ctx.customer["expected_activity"] = f"Seasonal business; cash receipts peak {peak_text}."
ctx.customer["stated_income"] = _round_to(expected * 7, ctx.usd(1000))
ctx.doc("registration", f"Business registration — {ctx.customer['name']}")
ctx.doc("licence", f"Trade licence, {business_type}, renewed annually")
return {
"family": "CASH",
"triggering": "fire_month_cash",
"planted": [["kyc.expected_activity", "kyc.expected_monthly_cash"], ["kyc.expected_monthly_cash"]],
}
def _build_B4(ctx: _Ctx) -> dict[str, Any]:
"""Cash restaurant. Three of the month's deposits land in the band; the file expected more cash than this."""
rng = ctx.rng
business_type, noun = rng.pick([("restaurant", "Bistro"), ("diner", "Diner"), ("pizzeria", "Kitchen"), ("cafe", "Grill")])
ctx.set_fire([3, 4, 5, 6], 14, 27)
_kyc_common(ctx, "business")
ctx.customer["name"] = ctx.business(noun)
ctx.customer["business_type"] = business_type
base = ctx.money(40000, 75000, 1000)
produce = ctx.cp(ctx.business("Produce"), "business")
foods = ctx.cp(ctx.business("Foods"), "business")
band_lo = ctx.threshold * 8 // 10
for i in range(WINDOW_MONTHS):
total = base * rng.between(90, 110) // 100
n = rng.between(11, 14)
if i == WINDOW_MONTHS - 1:
# Three window deposits in the band, on distinct days; the rest ordinary.
fire_dom = ctx.fire_dom()
window_days = ctx.days_in(i, 3, max(1, fire_dom - 10), fire_dom - 1)
for day in window_days:
ctx.add(day, "cash_in", ctx.money(band_lo // ctx.usd(1) + 100, ctx.threshold // ctx.usd(1) - 100), branch=ctx.branch())
n -= 3
_cash_deposits(ctx, i, total, n, ctx.usd(1500), band_lo - ctx.usd(100))
for day in ctx.days_in(i, 4, 1, 28):
ctx.add(day, "ach_out", total * rng.between(4, 7) // 100, cp=produce)
ctx.add(ctx.day_in(i, 1, 10), "ach_out", total * rng.between(10, 16) // 100, cp=foods)
expected = _round_to(ctx.max_month_cash() * rng.between(110, 130) // 100, ctx.usd(1000)) + ctx.usd(1000)
ctx.customer["expected_monthly_cash"] = expected
ctx.customer["expected_activity"] = "Cash-intensive food service; daily takings deposited most trading days."
ctx.customer["stated_income"] = _round_to(base * 12, ctx.usd(1000))
ctx.doc("registration", f"Business registration — {ctx.customer['name']}")
ctx.doc("licence", "Food service licence, renewed annually")
return {
"family": "STR",
"triggering": "band_window",
"planted": [["kyc.expected_monthly_cash"], ["kyc.expected_monthly_cash", "kyc.business_type"]],
}
def _build_B2(ctx: _Ctx) -> dict[str, Any]:
"""Processor settlement. Wires in from a processor, contractors paid within two days; the agreement is on file."""
rng = ctx.rng
business_type, noun = rng.pick([("staffing agency", "Staffing"), ("freight brokerage", "Freight"), ("event production", "Events"), ("general contractor", "Contracting")])
ctx.set_fire([3, 4, 5, 6], 14, 27)
_kyc_common(ctx, "business")
ctx.customer["name"] = ctx.business(noun)
ctx.customer["business_type"] = business_type
processor_name = f"{rng.pick(WORDS['adjectives'])} Payments LLC"
processor = ctx.cp(processor_name, "processor")
ctx.doc_party_cp = processor
contractors = [ctx.cp(ctx.business(), "business") for _ in range(2)]
revenue = 0
window: list[int] = []
for i in range(WINDOW_MONTHS):
k = rng.between(3, 4)
days = ctx.days_in(i, k, 1, 24)
if i == WINDOW_MONTHS - 1:
# Guarantee one settlement inside the 14-day window with its payouts before the fire date.
fire_dom = ctx.fire_dom()
days = ctx.days_in(i, k - 1, 1, max(1, fire_dom - 8)) + [ctx.fire_day - rng.between(3, 6)]
for day in days:
amount = ctx.money(15000, 60000, 100)
revenue += amount
t_in = ctx.add(day, "wire_in", amount, cp=processor)
share = rng.between(70, 90)
first = amount * share // 100 * rng.between(50, 70) // 100
second = amount * share // 100 - first
t_a = ctx.add(day + 1, "wire_out", _round_to(first, ctx.usd(10)), cp=contractors[0])
t_b = ctx.add(day + rng.between(1, 2), "ach_out", _round_to(second, ctx.usd(10)), cp=contractors[1])
if i == WINDOW_MONTHS - 1 and day >= ctx.fire_day - 14:
window += [t_in, t_a, t_b]
ctx.customer["expected_monthly_cash"] = ctx.usd(0)
ctx.customer["expected_activity"] = "Card settlements from a payment processor; subcontractor payouts within days of settlement."
ctx.customer["stated_income"] = _round_to(revenue, ctx.usd(10000))
ctx.doc("registration", f"Business registration — {ctx.customer['name']}")
ctx.doc("processing_agreement", f"Merchant processing agreement between {ctx.customer['name']} and {processor_name}; settlement to this account on T+1.", party=processor_name)
return {"family": "VEL", "triggering": window, "planted": [["DOC"], ["DOC", "kyc.business_type"]]}
def _build_B3(ctx: _Ctx) -> dict[str, Any]:
"""Property sale. One title-company wire in, moved to a brokerage; closing statement on file."""
rng = ctx.rng
ctx.set_fire([3, 4, 5, 6], 14, 27)
_kyc_common(ctx, "individual")
ctx.customer["name"] = ctx.person()
ctx.customer["occupation"] = rng.pick(["teacher", "software engineer", "pharmacist", "electrician", "accountant"])
salary = ctx.money(4000, 9000, 100)
employer = ctx.cp(ctx.business(), "business")
_salary(ctx, salary, employer)
_small_cash_out(ctx, 100, 400, 2, 4)
title_name = f"{rng.pick(SURNAMES)} Title & Escrow"
title = ctx.cp(title_name, "title_company")
ctx.doc_party_cp = title
brokerage = ctx.cp(f"{rng.pick(WORDS['adjectives'])} Securities", "brokerage")
proceeds = ctx.money(180000, 650000, 1000)
day_in = ctx.fire_day - rng.between(3, 6)
t_in = ctx.add(day_in, "wire_in", proceeds, cp=title)
t_out = ctx.add(day_in + rng.between(1, 2), "wire_out", _round_to(proceeds * rng.between(85, 100) // 100, ctx.usd(100)), cp=brokerage)
ctx.customer["expected_monthly_cash"] = ctx.usd(0)
ctx.customer["expected_activity"] = "Salaried individual; payroll deposits and routine spending."
ctx.customer["stated_income"] = _round_to(salary * 12, ctx.usd(1000))
ctx.doc("identity", "Government-issued identity document, verified at onboarding")
address = f"{rng.between(12, 980)} {rng.pick(WORDS['streets'])}"
ctx.doc("closing_statement", f"Closing statement — sale of {address}; net proceeds to seller disbursed by {title_name}.", party=title_name)
return {"family": "VEL", "triggering": [t_in, t_out], "planted": [["DOC"], ["DOC", "kyc.occupation"]]}
def _build_B5(ctx: _Ctx) -> dict[str, Any]:
"""Family remittance. A monthly wire to one relative in a monitored jurisdiction, declared at onboarding."""
rng = ctx.rng
ctx.set_fire([3, 4, 5, 6])
_kyc_common(ctx, "individual")
surname = rng.pick(SURNAMES)
ctx.customer["name"] = f"{rng.pick(FIRST_NAMES)} {surname}"
ctx.customer["occupation"] = rng.pick(["registered nurse", "civil engineer", "teacher", "pharmacist"])
salary = ctx.money(4500, 8500, 100)
employer = ctx.cp(ctx.business(), "business")
_salary(ctx, salary, employer)
_small_cash_out(ctx, 100, 400, 2, 4)
jurisdiction = rng.pick(JURISDICTIONS["monitored"])
relation = rng.pick(["mother", "father", "brother", "sister"])
relative = ctx.cp(f"{rng.pick(FIRST_NAMES)} {surname}", "individual", jurisdiction)
monthly = ctx.money(800, 2500, 50)
wires = []
for i in range(WINDOW_MONTHS):
wires.append(ctx.add(ctx.day_in(i, 1, 7), "wire_out", monthly + ctx.usd(rng.between(-50, 50)), cp=relative))
ctx.customer["expected_monthly_cash"] = ctx.usd(0)
ctx.customer["expected_activity"] = "Salaried individual; payroll deposits and routine spending."
ctx.customer["remittance_declared"] = f"Monthly support to {relation} in {jurisdiction}, about {monthly // 100 // ctx.scale * ctx.scale:,} {ctx.currency} per month."
ctx.customer["stated_income"] = _round_to(salary * 12, ctx.usd(1000))
ctx.doc("identity", "Government-issued identity document, verified at onboarding")
ctx.doc("address", "Proof of address, utility statement")
return {"family": "WIRE", "triggering": wires[-3:], "planted": [["kyc.remittance_declared"], ["kyc.remittance_declared", "kyc.stated_income"]]}
def _build_B6(ctx: _Ctx) -> dict[str, Any]:
"""Inheritance. A dormant account receives an estate distribution; the executor's letter is on file."""
rng = ctx.rng
ctx.set_fire([3, 4, 5, 6], 14, 27)
_kyc_common(ctx, "individual")
ctx.customer["name"] = ctx.person()
ctx.customer["occupation"] = "retired"
ctx.customer["account_opened"] = iso_date(ctx.fire_day - rng.between(8, 20) * 365 - rng.between(0, 300))
for i in range(WINDOW_MONTHS - 1):
if rng.chance(35):
ctx.add(ctx.day_in(i), "cash_out", ctx.money(40, 200), branch=ctx.branch())
firm = f"{rng.pick(SURNAMES)} & {rng.pick(SURNAMES)} LLP, client trust account"
trust = ctx.cp(firm, "law_firm_trust")
ctx.doc_party_cp = trust
brokerage = ctx.cp(f"{rng.pick(WORDS['adjectives'])} Securities", "brokerage")
amount = ctx.money(60000, 400000, 1000)
day_in = ctx.fire_day - rng.between(4, 7)
t_in = ctx.add(day_in, "wire_in", amount, cp=trust)
t_out = ctx.add(day_in + rng.between(1, 3), "wire_out", _round_to(amount * rng.between(50, 90) // 100, ctx.usd(100)), cp=brokerage)
ctx.customer["expected_monthly_cash"] = ctx.usd(0)
ctx.customer["expected_activity"] = "Retired; low activity expected."
ctx.customer["stated_income"] = ctx.money(20000, 40000, 1000)
ctx.doc("identity", "Government-issued identity document, verified at onboarding")
executor, decedent = ctx.person(), ctx.person()
ctx.doc("executor_letter", f"Letter from {executor}, executor, via {firm}: distribution from the estate of {decedent} to the named beneficiary.", party=firm)
return {"family": "DORM", "triggering": [t_in, t_out], "planted": [["DOC"]]}
# -------------------------------------------------------------- visible --
def _build_S1(ctx: _Ctx) -> dict[str, Any]:
"""Structuring. Eight deposits just under the threshold in twelve days across three branches, on a modest income."""
rng = ctx.rng
ctx.set_fire([3, 4, 5, 6], 14, 27)
_kyc_common(ctx, "individual")
ctx.customer["name"] = ctx.person()
ctx.customer["occupation"] = rng.pick(["warehouse associate", "delivery driver", "retail supervisor", "line cook"])
ctx.branches = rng.sample(WORDS["branches"], 3)
salary = ctx.money(2600, 3600, 100)
employer = ctx.cp(ctx.business(), "business")
_salary(ctx, salary, employer)
_small_cash_out(ctx, 60, 300, 2, 4)
fire_dom = ctx.fire_dom()
days = ctx.days_in(WINDOW_MONTHS - 1, 8, max(1, fire_dom - 12), fire_dom - 2)
lo = ctx.threshold // ctx.usd(1) * 92 // 100
hi = ctx.threshold // ctx.usd(1) * 99 // 100
deposits = []
total = 0
for j, day in enumerate(days):
amount = ctx.money(lo, hi)
total += amount
deposits.append(ctx.add(day, "cash_in", amount, branch=ctx.branches[j % 3]))
holdings = ctx.cp(ctx.business("Holdings"), "business")
ctx.add(ctx.fire_day - 1, "wire_out", _round_to(total * rng.between(85, 95) // 100, ctx.usd(100)), cp=holdings)
ctx.customer["expected_monthly_cash"] = ctx.money(0, 500, 100)
ctx.customer["expected_activity"] = "Salaried individual; payroll deposits and routine spending."
ctx.customer["stated_income"] = _round_to(salary * 12 + ctx.usd(rng.between(0, 6000)), ctx.usd(1000))
ctx.doc("identity", "Government-issued identity document, verified at onboarding")
return {"family": "STR", "triggering": "band_window", "planted": [deposits], "typology": "STRUCTURING"}
def _build_S3(ctx: _Ctx) -> dict[str, Any]:
"""Money mule. Four P2P credits from unrelated individuals, each wired to an exchange within a day."""
rng = ctx.rng
ctx.set_fire([3, 4, 5, 6], 16, 27)
_kyc_common(ctx, "individual")
ctx.customer["name"] = ctx.person()
ctx.customer["occupation"] = rng.pick(["student", "retired", "part-time cashier"])
stipend = ctx.money(900, 1800, 50)
source = ctx.cp(ctx.business(), "business")
_salary(ctx, stipend, source)
_small_cash_out(ctx, 40, 200, 2, 4)
exchange = ctx.cp(f"{rng.pick(WORDS['adjectives'])} Digital Assets", "exchange")
fire_dom = ctx.fire_dom()
days = ctx.days_in(WINDOW_MONTHS - 1, 4, max(1, fire_dom - 14), fire_dom - 2)
pairs = []
for day in days:
sender = ctx.cp(ctx.person(), "individual")
amount = ctx.money(2000, 4900, 10)
t_in = ctx.add(day, "p2p_in", amount, cp=sender)
t_out = ctx.add(day + rng.between(0, 1), "wire_out", _round_to(amount * rng.between(90, 97) // 100, ctx.usd(10)), cp=exchange)
pairs += [t_in, t_out]
ctx.customer["expected_monthly_cash"] = ctx.money(0, 300, 100)
ctx.customer["expected_activity"] = "Low-income individual; small regular credits and routine spending."
ctx.customer["stated_income"] = _round_to(stipend * 12, ctx.usd(1000))
ctx.doc("identity", "Government-issued identity document, verified at onboarding")
return {"family": "VEL", "triggering": pairs, "planted": [pairs], "typology": "MONEY_MULE"}
def _build_S6(ctx: _Ctx) -> dict[str, Any]:
"""Smurfing over time. Monthly cash triples across the year and no single deposit ever reaches the threshold."""
rng = ctx.rng
business_type, noun = rng.pick([("vending route", "Vending"), ("car wash", "Car Wash"), ("laundromat", "Laundry"), ("convenience store", "Market")])
ctx.set_fire([3, 4, 5, 6], 14, 27)
_kyc_common(ctx, "business")
ctx.customer["name"] = ctx.business(noun)
ctx.customer["business_type"] = business_type
c0 = ctx.money(9000, 14000, 500)
supplier = ctx.cp(ctx.business("Supply"), "business")
cap = ctx.threshold - ctx.usd(100)
for i in range(WINDOW_MONTHS):
total = c0 * (11 + 2 * i) // 11 * rng.between(97, 103) // 100
n = rng.between(6, 9)
_cash_deposits(ctx, i, total, n, ctx.usd(2000), cap)
ctx.add(ctx.day_in(i, 20, 28), "ach_out", total * rng.between(15, 25) // 100, cp=supplier)
ctx.customer["expected_monthly_cash"] = _round_to(c0 * 12 // 10, ctx.usd(500))
ctx.customer["expected_activity"] = "Cash-intensive retail; steady volume expected."
ctx.customer["stated_income"] = _round_to(c0 * 14, ctx.usd(1000))
ctx.doc("registration", f"Business registration — {ctx.customer['name']}")
return {"family": "CASH", "triggering": "fire_month_cash", "planted": "last_two_months", "typology": "STRUCTURING"}
BUILDERS = {"B1": _build_B1, "B2": _build_B2, "B3": _build_B3, "B4": _build_B4, "B5": _build_B5, "B6": _build_B6, "S1": _build_S1, "S3": _build_S3, "S6": _build_S6}
# ------------------------------------------------------------- assembly --
def _prior_alerts(ctx: _Ctx, family: str) -> list[dict[str, Any]]:
"""Benign prior alerts: mixed rules, specific rationales, a few days each."""
rng = ctx.rng
out: list[dict[str, Any]] = []
used: set[str] = set()
for j in range(ctx.customer["prior_alerts"]):
while True:
aid = f"A-{rng.between(1000, 4999)}"
if aid not in used:
used.add(aid)
break
fam = rng.pick(list(FAMILIES))
fired = ctx.fire_day - rng.between(30, 720)
escalated = j == 0 and ctx.customer["prior_sars"] >= 1
out.append({
"id": aid,
"rule": ctx.rule_ids[fam],
"fired": iso_date(fired),
"disposition": "escalated — report filed" if escalated else "closed",
"analyst": ctx.person(),
"rationale": "Escalated: pattern not explained by the file at the time." if escalated else BENIGN_RATIONALE[fam],
"elapsed_days": rng.between(2, 9),
})
out.sort(key=lambda a: a["fired"])
return out
def _assemble(ctx: _Ctx, template: str, spec: dict[str, Any]) -> dict[str, Any]:
transactions, id_of = ctx.ledger.finish()
family = spec["family"]
# Triggering set, from the rule where the rule is mechanical.
if spec["triggering"] == "band_window":
lo, hi = ctx.threshold * 8 // 10, ctx.threshold - 1
triggering = [t["id"] for t in transactions
if t["channel"] == "cash_in" and lo <= t["amount"] <= hi
and ctx.fire_day - 10 <= days_from_civil(*map(int, t["date"].split("-"))) <= ctx.fire_day - 1]
elif spec["triggering"] == "fire_month_cash":
fire_month = month_key(ctx.fire_y, ctx.fire_m)
triggering = [t["id"] for t in transactions if t["channel"] == "cash_in" and t["month"] == fire_month]
else:
triggering = sorted((id_of[s] for s in spec["triggering"]), key=lambda x: int(x[2:]))
# Counterparties: transactions and first-seen from the ledger, direction from the channel.
by_cp: dict[str, list[dict[str, Any]]] = {c["id"]: [] for c in ctx.cps}
for t in transactions:
if t["cp"]:
by_cp[t["cp"]].append(t)
counterparties = []
for c in ctx.cps:
rows = by_cp[c["id"]]
counterparties.append({
**c,
"first_seen": rows[0]["date"] if rows else iso_date(ctx.fire_day),
"other_alerted_accounts": 1 if ctx.rng.chance(15) else 0,
"transactions": [t["id"] for t in rows],
})
ranked = sorted(counterparties, key=lambda c: (-sum(t["amount"] for t in by_cp[c["id"]]), int(c["id"][3:])))
screen_cps = []
for c in ranked[:SCREEN_COUNTERPARTIES]:
rows = by_cp[c["id"]]
screen_cps.append({
"id": c["id"],
"direction": "in" if rows and rows[0]["channel"] in INBOUND else "out",
"count": len(rows),
"total": sum(t["amount"] for t in rows),
})
summary = []
for (y, m) in ctx.months:
key = month_key(y, m)
row = {"month": key, "cash_in": 0, "cash_out": 0, "wires_in": 0, "wires_out": 0, "ach_in": 0, "ach_out": 0, "count": 0}
for t in transactions:
if t["month"] != key:
continue
row["count"] += 1
ch = t["channel"]
col = {"cash_in": "cash_in", "cash_out": "cash_out", "wire_in": "wires_in", "wire_out": "wires_out",
"ach_in": "ach_in", "ach_out": "ach_out", "p2p_in": "wires_in", "p2p_out": "wires_out"}[ch]
row[col] += t["amount"]
summary.append(row)
# Planted citation sets. "DOC" is the template's explaining document.
explaining = next((d["id"] for d in ctx.documents if d["party"]), None)
if spec["planted"] == "last_two_months":
last_two = {month_key(*ctx.months[-1]), month_key(*ctx.months[-2])}
planted = [[t["id"] for t in transactions if t["channel"] == "cash_in" and t["month"] in last_two]]
else:
planted = []
for alt in spec["planted"]:
planted.append([
(explaining if x == "DOC" else id_of[x] if isinstance(x, int) else x) for x in alt
])
world = {
"seed": ctx.seed,
"held_out": ctx.slice == "held_out",
"currency": ctx.currency,
"threshold": ctx.threshold,
"screen_order": "kyc_first" if ctx.slice == "held_out" else "alert_first",
"customer": ctx.customer,
"months": [month_key(y, m) for (y, m) in ctx.months],
"transactions": transactions,
"counterparties": counterparties,
"screen_counterparties": screen_cps,
"alert": {
"id": f"A-{ctx.rng.between(5000, 9999)}",
"rule": ctx.rule_ids[family],
"family": family,
"rule_text": RULE_TEXT[family],
"fired": iso_date(ctx.fire_day),
"triggering": triggering,
},
"prior_alerts": _prior_alerts(ctx, family),
"documents": ctx.documents,
"summary": summary,
"template": template,
"overlay": None,
"label": "suspicious" if template.startswith("S") else "benign",
"typology": spec.get("typology"),
"planted": planted,
"doc_party_cp": ctx.doc_party_cp,
}
return world
# -------------------------------------------------------------- overlay --
def compatible_overlays(world: dict[str, Any]) -> list[str]:
"""Which hidden signals this benign world can carry. Depends on the screen, never on the label."""
kinds = ["funnel"]
if world["customer"]["prior_alerts"] >= 3 and world["customer"]["prior_sars"] == 0:
kinds.append("serial_closer")
if world["template"] in DOC_TEMPLATES:
kinds.append("doc_mismatch")
return kinds
def _overlay_urn(kinds: list[str]) -> list[str]:
"""funnel is always available, so the other two are drawn twice as often where they can be."""
urn: list[str] = []
for kind in kinds:
urn += [kind] * (1 if kind == "funnel" else 2)
return urn
def _outbound_target(world: dict[str, Any]) -> str:
"""The highest-volume outbound counterparty on the screen. Every benign template has one."""
for c in world["screen_counterparties"]:
if c["direction"] == "out":
return c["id"]
raise AssertionError(f"template {world['template']} has no outbound counterparty on screen")
def apply_overlay(world: dict[str, Any], kind: str, rng: XorShift32) -> None:
"""Overlay a suspicious signal onto lookup data only. Mutates `world`.
The screen — alert, triggering set, KYC block, summary, screen
counterparties — is not touched, and `tests/test_screen_leak.py` proves it.
"""
family = world["alert"]["family"]
triggering = world["alert"]["triggering"]
world["overlay"] = kind
world["label"] = "suspicious"
if kind == "funnel":
target = _outbound_target(world)
cp = next(c for c in world["counterparties"] if c["id"] == target)
cp["other_alerted_accounts"] = rng.between(5, 14)
world["typology"] = "FUNNEL_ACCOUNT"
world["planted"] = [[target] + cp["transactions"], list(cp["transactions"])]
elif kind == "serial_closer":
priors = world["prior_alerts"]
analysts = []
while len(analysts) < len(priors):
name = f"{rng.pick(FIRST_NAMES)} {rng.pick(SURNAMES)}"
if name not in analysts:
analysts.append(name)
fire_day = days_from_civil(*map(int, world["alert"]["fired"].split("-")))
days = sorted(rng.sample(list(range(fire_day - 330, fire_day - 30)), len(priors)))
for j, prior in enumerate(priors):
prior["rule"] = world["alert"]["rule"]
prior["fired"] = iso_date(days[j])
prior["disposition"] = "closed"
prior["analyst"] = analysts[j]
prior["rationale"] = BOILERPLATE
prior["elapsed_days"] = rng.between(0, 1)
world["typology"] = NOMINAL_TYPOLOGY[family]
ids = [p["id"] for p in priors]
world["planted"] = [ids + triggering, ids]
elif kind == "doc_mismatch":
target = world["doc_party_cp"]
cp = next(c for c in world["counterparties"] if c["id"] == target)
cp["name"] = f"{rng.pick(WORDS['adjectives'])} {rng.pick(['Holdings', 'Ventures', 'Trading'])} {rng.pick(WORDS['suffixes'])}"
doc = next(d["id"] for d in world["documents"] if d["party"])
inbound = [t["id"] for t in world["transactions"] if t["id"] in triggering and t["cp"] == target]
world["typology"] = NOMINAL_TYPOLOGY[family]
world["planted"] = [[doc, target] + inbound, [doc, target]]
else:
raise ValueError(kind)
# -------------------------------------------------------------- public --
def is_held_out(seed: int) -> bool:
return fnv1a32(str(seed)) % 8 == 7
def generate(seed: int, overlay: str | None = "auto") -> dict[str, Any]:
"""The world before reference hours. `overlay` is for tests:
"auto" follows the seed; None forces the benign twin; a kind forces it on."""
h = fnv1a32(str(seed))
slice_name = "held_out" if h % 8 == 7 else "main"
rng = XorShift32(h)
ctx = _Ctx(seed, rng, slice_name)
roll = rng.below(100)
tier = "benign" if roll < BENIGN_PCT else "visible" if roll < BENIGN_PCT + VISIBLE_PCT else "hidden"
if tier == "visible":
template = VISIBLE_TEMPLATES[rng.below(len(VISIBLE_TEMPLATES))]
else:
template = BENIGN_TEMPLATES[rng.below(len(BENIGN_TEMPLATES))]
spec = BUILDERS[template](ctx)
world = _assemble(ctx, template, spec)
world["tier"] = tier
if tier != "visible":
kinds = compatible_overlays(world)
if overlay == "auto":
if tier == "hidden":
apply_overlay(world, rng.pick(_overlay_urn(kinds)), rng)
elif overlay is not None:
apply_overlay(world, overlay, rng)
del world["doc_party_cp"]
return world
def world_for_seed(seed: int) -> dict[str, Any]:
"""The world with `reference_minutes` attached — part of the digest.
The reference is the cheapest shipped policy that reached the correct
disposition on this seed, found by running them through the engine. The
policies read only the engine's view, never this dict, so attaching their
result here is not circular; it is the same move as wordle's
`reference_depth`, made part of the world so the browser recomputes it
instead of trusting the recorded number.
"""
from .policies import reference_for # local import: policies -> engine -> generator
world = generate(seed)
minutes, policy = reference_for(world)
world["reference_minutes"] = minutes
world["reference_policy"] = policy
return world
SCREEN_KEYS = ("seed", "held_out", "currency", "threshold", "screen_order", "customer", "months",
"summary", "screen_counterparties", "alert")
def screen_of(world: dict[str, Any]) -> dict[str, Any]:
"""Exactly what turn 0 shows, including the triggering transactions in full."""
triggering = set(world["alert"]["triggering"])
screen = {k: world[k] for k in SCREEN_KEYS}
screen["triggering_transactions"] = [t for t in world["transactions"] if t["id"] in triggering]
return screen
def world_digest(seeds=range(4096)) -> str:
"""Digest (a): SHA-256 over the canonical world for each seed, newline-joined."""
digest = hashlib.sha256()
for seed in seeds:
digest.update(canonical_json(world_for_seed(seed)).encode())
digest.update(b"\n")
return digest.hexdigest()
envs/alert_triage/alert_triage/ladder.py
+165
View File
@@ -0,0 +1,165 @@
"""The probe ladder for alert-triage. Invoked by `envs/probe.py`.
Every rung is a policy from `policies.py`, played over the main-slice seeds
(held-out seeds are never probed). The assertions are the reward's claims,
checked per weighted component rather than as one blended float. Any cheat a
reviewer named is a rung here with an assertion that it loses — a cheat that
is measured is a cheat that cannot come back unnoticed.
"""
from __future__ import annotations
from .generator import is_held_out, world_for_seed
from .policies import POLICIES, run_policy
from .reward import (
CAUGHT_WEIGHT,
EVIDENCE_WEIGHT,
ROLES,
WEIGHTS,
escalate_break_even,
metrics,
score,
total,
)
# Main-slice seeds only: fnv1a(seed) % 8 == 7 is never probed. 1024 seeds
# because the margins asserted in rung 8 are population claims, and a
# 200-seed sample can draw 11% hidden instead of 15% and turn one into noise.
SEEDS = [s for s in range(1024) if not is_held_out(s)]
COMPONENTS = ("caught", "hours", "evidence")
GOOD = ("fast", "targeted", "thorough")
CHEATS = ("close_blind", "close_cites_all", "cite_unseen", "escalate_blind", "fast_defensive", "rule_table")
# The one-slider counterfactual the page names: move `hours` here and the
# screen-only analyst wins. The other two weights keep their shipped ratio.
COUNTERFACTUAL_HOURS = 0.45
MARGIN = 0.02
def counterfactual_weights(hours_weight: float) -> dict[str, float]:
rest = 1.0 - hours_weight
share = CAUGHT_WEIGHT + EVIDENCE_WEIGHT
return {"caught": rest * CAUGHT_WEIGHT / share, "hours": hours_weight, "evidence": rest * EVIDENCE_WEIGHT / share}
def run(seeds: list[int] | None = None) -> int:
seeds = list(seeds or SEEDS)
worlds = [world_for_seed(s) for s in seeds]
tiers = {w["tier"] for w in worlds}
print(f"Probing alert-triage over {len(seeds)} main-slice seeds "
f"({sum(w['tier'] == 'benign' for w in worlds)} benign, "
f"{sum(w['tier'] == 'visible' for w in worlds)} visible, "
f"{sum(w['tier'] == 'hidden' for w in worlds)} hidden)\n")
results: dict[str, dict[str, float]] = {}
cf: dict[str, float] = {}
per_seed: dict[str, list] = {}
for name, policy in POLICIES.items():
episodes = [run_policy(policy, w).episode() for w in worlds]
scores = [score(e) for e in episodes]
assert all(v is not None for s in scores for v in s.values()), f"{name}: a component was None on a probe seed"
results[name] = {c: sum(s[c] for s in scores) / len(scores) for c in COMPONENTS} # type: ignore[misc]
results[name]["total"] = sum(total(e) for e in episodes) / len(episodes) # type: ignore[misc]
cf[name] = sum(total(e, counterfactual_weights(COUNTERFACTUAL_HOURS)) for e in episodes) / len(episodes) # type: ignore[misc]
per_seed[name] = episodes
width = max(len(n) for n in POLICIES)
header = f"{'policy':<{width}} " + " ".join(f"{c:>10}" for c in COMPONENTS) + f" {'TOTAL':>8} {'hours@' + str(COUNTERFACTUAL_HOURS):>10}"
print(header)
print("-" * len(header))
for name in POLICIES:
row = results[name]
cells = " ".join(f"{row[c]:>10.4f}" for c in COMPONENTS)
print(f"{name:<{width}} {cells} {row['total']:>8.4f} {cf[name]:>10.4f}")
print(f"\nweights: {WEIGHTS}")
print(f"roles: {ROLES}")
print(f"blind-escalation break-even: escalate whenever P(benign) < {escalate_break_even():.2f}")
mean_hours = {n: round(sum(metrics(e)['hours_spent'] for e in per_seed[n]) / len(seeds), 2) for n in GOOD}
print(f"hours spent (mean): {mean_hours}")
hidden = [i for i, w in enumerate(worlds) if w["tier"] == "hidden"]
print("caught on hidden seeds:", {n: round(sum(score(per_seed[n][i])['caught'] for i in hidden) / max(1, len(hidden)), 3) for n in GOOD}) # type: ignore[arg-type]
failures: list[str] = []
r = results
# 0. The seed set must contain every tier, or the assertions below measure noise.
if tiers != {"benign", "visible", "hidden"}:
failures.append(f"probe seeds cover tiers {sorted(tiers)}; need all three")
if any(w["reference_minutes"] is None for w in worlds):
failures.append("a probe seed has no reference_minutes — no shipped policy got it right")
# 1. Doing nothing, and doing it malformed, must measure exactly nothing.
for name in ("inaction", "malformed"):
if abs(r[name]["total"]) > 1e-12:
failures.append(f"{name} scored {r[name]['total']:.6f}, must be 0.000")
if any(e.rejected != 8 or e.disposition is not None for e in per_seed[name]):
failures.append(f"{name} must be rejected on all eight turns and abort")
# 2. The counterweight is attainable: the investigating references catch everything.
for name in ("targeted", "thorough"):
if abs(r[name]["caught"] - 1.0) > 1e-12:
failures.append(f"{name} caught {r[name]['caught']:.4f}, must be exactly 1.0")
# 3. And it bites the policy that defines the hours denominator on most seeds.
if r["fast"]["caught"] >= 1.0 - 1e-12:
failures.append("fast caught 1.0 — the hidden tier leaked into the screen")
# 4. Paying the counterweight's price is a real price.
if not (r["fast"]["hours"] > r["targeted"]["hours"] > r["thorough"]["hours"]):
failures.append("hours must order fast > targeted > thorough")
# 5. No good policy dominates another on every component.
for a, b in (("fast", "targeted"), ("fast", "thorough"), ("targeted", "thorough")):
if all(r[a][c] >= r[b][c] - 1e-12 for c in COMPONENTS):
failures.append(f"{a} dominates {b} on every component — no real trade-off")
if all(r[b][c] >= r[a][c] - 1e-12 for c in COMPONENTS):
failures.append(f"{b} dominates {a} on every component — no real trade-off")
# 6. Every named cheat loses to every good policy.
for cheat in CHEATS:
for good in GOOD:
if r[cheat]["total"] >= r[good]["total"] - 1e-12:
failures.append(f"{cheat} ({r[cheat]['total']:.4f}) >= {good} ({r[good]['total']:.4f})")
# 7. The hour meter binds: a wasted month costs, and nothing else changes.
if r["thorough_wasteful"]["total"] >= r["thorough"]["total"] - 1e-12:
failures.append("thorough_wasteful >= thorough — the hour budget does not bind")
for c in ("caught", "evidence"):
if abs(r["thorough_wasteful"][c] - r["thorough"][c]) > 1e-12:
failures.append(f"thorough_wasteful differs from thorough on {c}")
# 8. The headline claim, measured: targeted wins at shipped weights by a
# stated margin, and the one-slider counterfactual hands it to fast.
if r["targeted"]["total"] - r["fast"]["total"] < MARGIN:
failures.append(f"targeted beats fast by only {r['targeted']['total'] - r['fast']['total']:.4f} at shipped weights (need {MARGIN})")
if cf["fast"] - cf["targeted"] < MARGIN:
failures.append(f"fast beats targeted by only {cf['fast'] - cf['targeted']:.4f} at hours={COUNTERFACTUAL_HOURS} (need {MARGIN})")
# 9. Citing an unopened document is rejected, every time.
if any(e.rejected < 1 for e in per_seed["cite_unseen"]):
failures.append("cite_unseen was not rejected on some seed")
# 10. F1's precision term works: cite-everything earns less than half of its caught share in evidence.
if r["close_cites_all"]["evidence"] >= 0.5 * r["close_cites_all"]["caught"]:
failures.append("close_cites_all earns too much evidence — precision is not biting")
if r["close_cites_all"]["evidence"] >= r["fast"]["evidence"]:
failures.append("close_cites_all evidence >= fast evidence")
# 11. History buys evidence on the visible typologies, and nothing else.
if r["thorough"]["evidence"] <= r["fast"]["evidence"]:
failures.append("thorough evidence <= fast evidence — itemised history buys nothing")
# 12. No weighted component may be flat across the ladder.
for component in COMPONENTS:
values = [r[n][component] for n in POLICIES]
if max(values) - min(values) < 1e-9:
failures.append(f"component '{component}' is flat across the ladder — it measures nothing")
print()
if failures:
for f in failures:
print(f"FAIL: {f}")
return 1
print("All 13 ladder assertions hold.")
return 0
envs/alert_triage/alert_triage/policies.py
+346
View File
@@ -0,0 +1,346 @@
"""The scripted analysts: three shipped references and the probe ladder's cheats.
Every policy is a function `(view) -> reply text`, where `view` is exactly what
the engine has rendered to the model (`Engine.view()`), and nothing else. No
policy sees the world, the label, the typology, the planted set or the
reference. `tests/test_policies.py` enforces the signature and scans the view
for hidden keys. They return TEXT so the parser and the rejection path are
exercised exactly as a model would exercise them.
The decision rules below are heuristics, written down so a reviewer can see
that the reference "analysts" are players, not oracles:
fast decides from the screen; pulls `documents` only where the screen
says a document should explain the alert (a processor, a title
company, an estate). One or two turns. Cannot see a hidden signal.
targeted `fast` on a screen that shows a typology; otherwise the three
lookups that could carry a hidden signal — documents, prior
alerts, each counterparty on screen — then decides. No history.
thorough always the full procedure: documents, prior alerts, each
counterparty on screen, itemised history for the two most recent
months, then decides; cites every constituting transaction it saw.
Hidden-signal rules, shared by targeted and thorough:
* a counterparty shared with ≥ 3 other alerted accounts → FUNNEL_ACCOUNT,
cite the counterparty and its transactions;
* ≥ 3 prior alerts, all closed within a day on one identical rationale
→ the rule's typology,
cite the prior alerts and the triggering transactions;
* a document naming a party whose counterparty lookup carries a different
name → the rule's typology,
cite the document, the counterparty and its triggering credits.
"""
from __future__ import annotations
import json
from typing import Any, Callable
from .generator import INBOUND, NOMINAL_TYPOLOGY, RULE_IDS
Policy = Callable[[dict[str, Any]], str]
_FAMILY_OF = {rid: fam for table in RULE_IDS.values() for fam, rid in table.items()}
EXPLAINING_DOCS = ("processing_agreement", "closing_statement", "executor_letter")
def _emit(obj: dict[str, Any]) -> str:
return json.dumps(obj)
def _family(view: dict[str, Any]) -> str:
return _FAMILY_OF.get(view["screen"]["alert"]["rule"], "STR")
def _looked(view: dict[str, Any], what: str, key: str | None = None) -> dict[str, Any] | None:
for l in view["lookups"]:
if l["what"] == what and (key is None or l["key"] == key):
return l["result"]
return None
def _band(view: dict[str, Any], t: dict[str, Any]) -> bool:
threshold = view["screen"]["threshold"]
return t["channel"] == "cash_in" and threshold * 8 // 10 <= t["amount"] < threshold
def _close(reason: str, cites: list[str], note: str) -> dict[str, Any]:
return {"action": "close", "reason": reason, "cites": cites, "note": note}
def _escalate(typology: str, cites: list[str], narrative: str) -> dict[str, Any]:
return {"action": "escalate", "typology": typology, "cites": cites, "narrative": narrative}
def _dedupe(ids: list[str]) -> list[str]:
out: list[str] = []
for i in ids:
if i not in out:
out.append(i)
return out
# ------------------------------------------------------------ decisions --
def screen_decision(view: dict[str, Any]) -> dict[str, Any] | None:
"""What the screen alone supports. None means "a document should explain this"."""
screen = view["screen"]
fam = _family(view)
trig = screen["triggering_transactions"]
trig_ids = [t["id"] for t in trig]
kyc = screen["customer"]
max_cash = max(r["cash_in"] for r in screen["summary"])
if fam == "STR":
band = [t for t in trig if _band(view, t)]
branches = {t["branch"] for t in band}
if len(band) >= 5 or (len(band) >= 4 and len(branches) >= 3):
return _escalate("STRUCTURING", [t["id"] for t in band], "Repeated cash deposits just under the threshold across branches.")
if kyc["expected_monthly_cash"] >= max_cash:
return _close("CONSISTENT_WITH_PROFILE", ["kyc.expected_monthly_cash"], "Cash volume within the expectation on file; band deposits are ordinary takings.")
return _escalate("STRUCTURING", [t["id"] for t in band] or trig_ids, "Band deposits with no expectation on file to cover them.")
if fam == "CASH":
if kyc["expected_monthly_cash"] >= max_cash:
return _close("CONSISTENT_WITH_PROFILE", ["kyc.expected_activity", "kyc.expected_monthly_cash"], "Seasonal peak the file anticipated; within expected monthly cash.")
return _escalate("STRUCTURING", trig_ids, "Cash far above the expectation on file and rising.")
if fam == "VEL":
senders = {t["cp"] for t in trig if t["channel"] == "p2p_in"}
if len(senders) >= 3:
return _escalate("MONEY_MULE", trig_ids, "Credits from unrelated individuals moved to an exchange within a day.")
return None
if fam == "WIRE":
if kyc.get("remittance_declared", "none declared") != "none declared":
return _close("CONSISTENT_WITH_PROFILE", ["kyc.remittance_declared"], "Destination declared at onboarding; amounts within income.")
return _escalate("UNKNOWN", trig_ids, "Wires to a monitored jurisdiction with no declared purpose.")
return None # DORM: needs the document
def document_decision(view: dict[str, Any]) -> dict[str, Any]:
"""After `documents`: close on the explaining document, or escalate if there is none."""
docs = _looked(view, "documents")
trig_ids = [t["id"] for t in view["screen"]["triggering_transactions"]]
if docs:
for d in docs["documents"]:
if d["kind"] in EXPLAINING_DOCS:
return _close("DOCUMENTED_SOURCE_OF_FUNDS", [d["id"]], f"Source of funds documented: {d['kind']}.")
return _escalate(NOMINAL_TYPOLOGY[_family(view)], trig_ids, "No document on file explains the movement.")
def lookup_decision(view: dict[str, Any]) -> dict[str, Any] | None:
"""A hidden signal in the lookups, or None."""
screen = view["screen"]
fam = _family(view)
trig = screen["triggering_transactions"]
trig_ids = [t["id"] for t in trig]
for l in view["lookups"]:
if l["what"] == "counterparty" and l["result"]["other_alerted_accounts"] >= 3:
r = l["result"]
return _escalate("FUNNEL_ACCOUNT", [r["id"]] + list(r["transactions"]),
f"{r['id']} is shared with {r['other_alerted_accounts']} other alerted accounts; funds converge on it.")
priors = _looked(view, "prior_alerts")
if priors and len(priors["alerts"]) >= 3:
alerts = priors["alerts"]
rationales = {p["rationale"] for p in alerts}
if all(p["disposition"] == "closed" and p["elapsed_days"] <= 1 for p in alerts) and len(rationales) == 1:
return _escalate(NOMINAL_TYPOLOGY[fam], [p["id"] for p in alerts] + trig_ids,
"Repeat alerts closed within a day on one boilerplate rationale; nobody has looked.")
docs = _looked(view, "documents")
if docs:
for d in docs["documents"]:
if not d["party"]:
continue
for cp_id in _dedupe([t["cp"] for t in trig if t["channel"] in INBOUND and t["cp"]]):
cp = _looked(view, "counterparty", cp_id)
if cp and cp["name"] != d["party"]:
credits = [t["id"] for t in trig if t["cp"] == cp_id and t["channel"] in INBOUND]
return _escalate(NOMINAL_TYPOLOGY[fam], [d["id"], cp_id] + credits,
f"The document names {d['party']}; the funds came from {cp['name']}.")
return None
def enriched_decision(view: dict[str, Any]) -> dict[str, Any]:
"""thorough's final call: the screen rules, with citations widened to everything itemised."""
d = screen_decision(view)
fam = _family(view)
screen = view["screen"]
months = screen["months"]
seen: list[dict[str, Any]] = list(screen["triggering_transactions"])
for l in view["lookups"]:
if l["what"] == "history":
seen += l["result"]["transactions"]
if d is not None and d["action"] == "escalate":
if fam == "STR":
d["cites"] = _dedupe([t["id"] for t in seen if _band(view, t) and t["month"] == months[-1]] + d["cites"])
elif fam == "CASH":
last_two = set(months[-2:])
d["cites"] = _dedupe([t["id"] for t in seen if t["channel"] == "cash_in" and t["month"] in last_two] + d["cites"])
return d
if d is None:
return document_decision(view)
return d
def _plan_next(view: dict[str, Any], plan: list[tuple[str, str | None]]) -> str | None:
for what, key in plan:
if _looked(view, what, key) is None:
if what == "history":
return _emit({"action": "lookup", "what": "history", "month": key})
if what == "counterparty":
return _emit({"action": "lookup", "what": "counterparty", "id": key})
return _emit({"action": "lookup", "what": what})
return None
# ------------------------------------------------------------- shipped --
def fast(view: dict[str, Any]) -> str:
d = screen_decision(view)
if d is not None:
return _emit(d)
if _looked(view, "documents") is None:
return _emit({"action": "lookup", "what": "documents"})
return _emit(document_decision(view))
def _investigation_plan(view: dict[str, Any]) -> list[tuple[str, str | None]]:
plan: list[tuple[str, str | None]] = [("documents", None), ("prior_alerts", None)]
plan += [("counterparty", cp["id"]) for cp in view["screen"]["screen_counterparties"]]
return plan
def targeted(view: dict[str, Any]) -> str:
d = screen_decision(view)
if d is not None and d["action"] == "escalate":
return _emit(d)
nxt = _plan_next(view, _investigation_plan(view))
if nxt:
return nxt
found = lookup_decision(view)
if found:
return _emit(found)
return _emit(d if d is not None else document_decision(view))
def thorough(view: dict[str, Any]) -> str:
months = view["screen"]["months"]
plan = _investigation_plan(view) + [("history", months[-1]), ("history", months[-2])]
nxt = _plan_next(view, plan)
if nxt:
return nxt
found = lookup_decision(view)
return _emit(found if found else enriched_decision(view))
def thorough_wasteful(view: dict[str, Any]) -> str:
"""thorough plus a month that carries nothing. Proves the hour meter binds."""
months = view["screen"]["months"]
plan = _investigation_plan(view) + [("history", months[-1]), ("history", months[-2]), ("history", months[-3])]
nxt = _plan_next(view, plan)
if nxt:
return nxt
found = lookup_decision(view)
return _emit(found if found else enriched_decision(view))
# -------------------------------------------------------------- cheats --
def inaction(view: dict[str, Any]) -> str:
return "Let me reason about this alert carefully before committing to a disposition."
def malformed(view: dict[str, Any]) -> str:
return _emit({"action": "file_sar", "cites": []})
def close_blind(view: dict[str, Any]) -> str:
return _emit(_close("CONSISTENT_WITH_PROFILE", ["kyc.customer_type"], "Looks fine."))
def close_cites_all(view: dict[str, Any]) -> str:
screen = view["screen"]
cites = [f"kyc.{k}" for k in screen["customer"] if k not in ("id", "name")]
cites += [t["id"] for t in screen["triggering_transactions"]]
return _emit(_close("CONSISTENT_WITH_PROFILE", cites, "See everything."))
def escalate_blind(view: dict[str, Any]) -> str:
trig_ids = [t["id"] for t in view["screen"]["triggering_transactions"]]
return _emit(_escalate(NOMINAL_TYPOLOGY[_family(view)], trig_ids, "Filing defensively."))
def fast_defensive(view: dict[str, Any]) -> str:
"""fast where the screen shows a typology; a blind filing everywhere else."""
d = screen_decision(view)
if d is not None and d["action"] == "escalate":
return _emit(d)
return escalate_blind(view)
def rule_table(view: dict[str, Any]) -> str:
"""Closes every wire-corridor and dormancy alert blind; files on everything else."""
fam = _family(view)
if fam == "WIRE":
return _emit(_close("CONSISTENT_WITH_PROFILE", ["kyc.remittance_declared"], "Wires are always remittances."))
if fam == "DORM":
return _emit(_close("CONSISTENT_WITH_PROFILE", ["kyc.account_opened"], "Old accounts wake up."))
return escalate_blind(view)
def cite_unseen(view: dict[str, Any]) -> str:
"""Cites a document it never opened; rejected, then falls back to closing blind."""
if view["rejected"] == 0:
return _emit(_close("DOCUMENTED_SOURCE_OF_FUNDS", ["doc.probate_letter"], "Per the letter on file."))
return close_blind(view)
POLICIES: dict[str, Policy] = {
"inaction": inaction,
"malformed": malformed,
"close_blind": close_blind,
"close_cites_all": close_cites_all,
"cite_unseen": cite_unseen,
"escalate_blind": escalate_blind,
"fast_defensive": fast_defensive,
"rule_table": rule_table,
"fast": fast,
"targeted": targeted,
"thorough": thorough,
"thorough_wasteful": thorough_wasteful,
}
SHIPPED = ("fast", "targeted", "thorough")
def run_policy(policy: Policy, world: dict[str, Any]):
"""Play a world to the end under one policy. Returns the finished Engine."""
from .engine import Engine
engine = Engine(world)
while not engine.done:
engine.step(policy(engine.view()))
return engine
def reference_for(world: dict[str, Any]) -> tuple[int | None, str | None]:
"""The cheapest shipped policy that reaches the correct disposition, in minutes.
(None, None) if none does — which `tests/test_reference.py` asserts never
happens for seeds 0–4095, so a null here is a CI failure, not a silent
"not scored" on the page.
"""
best: tuple[int, str] | None = None
for name in SHIPPED:
engine = run_policy(POLICIES[name], world)
if engine.outcome == "solved" and (best is None or engine.minutes < best[0]):
best = (engine.minutes, name)
return best if best else (None, None)
envs/alert_triage/alert_triage/protocol.py
+209
View File
@@ -0,0 +1,209 @@
"""Turning model text into one action, and engine results into text.
The one rule, as in wordle_five: **never raise**. A malformed reply is a
thing the model did, not an error in the harness. It becomes a rejection that
costs a turn and shows up in `rejected_replies`.
The parse rules are pinned here because this is where two runtimes disagree,
and CI hashes a committed corpus of reply strings through this parser and the
engine on both sides (digest (c) in CONFORMANCE.txt):
1. If a ```json fence appears anywhere, its contents are the candidate.
A bare `{` before the fence does not win.
2. Otherwise the candidate is the first string-aware balanced `{…}` span:
braces inside JSON strings do not count, backslash escapes are honoured.
3. If the candidate is not valid JSON, or is JSON but not an object, the
reply is rejected. There is no further scanning for a second candidate.
4. Enum values are matched exact-case. Cites are deduplicated in order and
matched exact-case against what the engine has shown.
"""
from __future__ import annotations
import json
import re
from typing import Any
from .generator import MAX_TURNS
ACTIONS = ("lookup", "close", "escalate")
LOOKUPS = ("history", "counterparty", "prior_alerts", "documents")
CLOSE_REASONS = ("CONSISTENT_WITH_PROFILE", "DOCUMENTED_SOURCE_OF_FUNDS", "PREVIOUSLY_REVIEWED", "RULE_ARTEFACT")
TYPOLOGIES = ("STRUCTURING", "FUNNEL_ACCOUNT", "MONEY_MULE", "RAPID_MOVEMENT", "UNKNOWN")
_FENCE = re.compile(r"```json\s*(.*?)```", re.S)
_MONTH = re.compile(r"^\d{4}-\d{2}$")
def _reject_constant(name: str):
# Python's json accepts NaN/Infinity by default; JSON.parse does not.
raise ValueError(name)
def extract_candidate(text: str) -> str | None:
match = _FENCE.search(text)
if match:
return match.group(1)
start = text.find("{")
if start < 0:
return None
depth = 0
in_string = False
escaped = False
for i in range(start, len(text)):
ch = text[i]
if in_string:
if escaped:
escaped = False
elif ch == "\\":
escaped = True
elif ch == '"':
in_string = False
continue
if ch == '"':
in_string = True
elif ch == "{":
depth += 1
elif ch == "}":
depth -= 1
if depth == 0:
return text[start : i + 1]
return None
def parse_reply(text: str | None) -> tuple[dict[str, Any] | None, str | None]:
"""(object, None) or (None, reason). Validation of fields is the engine's job."""
if not text:
return None, "no JSON object found in the reply"
candidate = extract_candidate(text)
if candidate is None:
return None, "no JSON object found in the reply"
try:
value = json.loads(candidate, parse_constant=_reject_constant)
except (ValueError, RecursionError):
return None, "the JSON object could not be parsed"
if not isinstance(value, dict):
return None, "the reply parsed but is not a JSON object"
return value, None
# -------------------------------------------------------------- render --
SYSTEM_PROMPT = """You are a transaction-monitoring analyst at a bank. One alert has been assigned to you.
All data is synthetic. No real customer, account, transaction or counterparty exists.
Your screen shows the alert, the transactions that triggered it, the customer's KYC file and a twelve-month summary. Reading it has already cost {screen} analyst-hours. You have {turns} replies. Each reply must be exactly ONE JSON object (in a ```json fence, or as the first {{...}} in the reply), and every reply costs {turn} hours whether or not it is accepted.
Lookups (each may be made once):
{{"action":"lookup","what":"history","month":"YYYY-MM"}} itemised transactions for a month in the window +{lookup} h
{{"action":"lookup","what":"counterparty","id":"CP-n"}} name, type, jurisdiction, first seen, how many OTHER alerted accounts share it, its transactions +{lookup} h
{{"action":"lookup","what":"prior_alerts"}} prior alerts on this customer with disposition, analyst and rationale +{lookup} h
{{"action":"lookup","what":"documents"}} documents on file +{doc} h
Dispositions (end the episode):
{{"action":"close","reason":R,"cites":[ids],"note":"..."}}
R is one of CONSISTENT_WITH_PROFILE, DOCUMENTED_SOURCE_OF_FUNDS, PREVIOUSLY_REVIEWED, RULE_ARTEFACT
{{"action":"escalate","typology":T,"cites":[ids],"narrative":"..."}}
T is one of STRUCTURING, FUNNEL_ACCOUNT, MONEY_MULE, RAPID_MOVEMENT, UNKNOWN
`cites` must be a non-empty list of ids you have actually been shown: kyc.<field>, T-<n>, CP-<n>, A-<n> (a prior alert), doc.<id>. Citing an id you have not been shown is rejected. The note or narrative is displayed and never scored.
A close on an alert that should have been escalated scores zero. An escalation on a clean customer scores half on the counterweight and zero on the rest. Cite the facts that decide the case, not everything you saw. If you reach your last reply without a disposition the alert stays in the queue and the episode scores zero."""
def system_prompt() -> str:
from .reward import DOC_MINUTES, LOOKUP_MINUTES, SCREEN_MINUTES, TURN_MINUTES
return SYSTEM_PROMPT.format(
screen=f"{SCREEN_MINUTES / 60:.2f}",
turn=f"{TURN_MINUTES / 60:.2f}",
lookup=f"{LOOKUP_MINUTES / 60:.2f}",
doc=f"{DOC_MINUTES / 60:.2f}",
turns=MAX_TURNS,
)
def money(cents: int, currency: str) -> str:
sign = "-" if cents < 0 else ""
cents = abs(cents)
return f"{sign}{cents // 100:,}.{cents % 100:02d} {currency}"
def _txn_line(t: dict[str, Any], currency: str) -> str:
where = f"branch {t['branch']}" if t["branch"] else f"{t['cp']}"
return f" {t['id']:<6} {t['date']} {t['channel']:<9} {money(t['amount'], currency):>18} {where}"
def render_screen(screen: dict[str, Any]) -> str:
cur = screen["currency"]
c = screen["customer"]
a = screen["alert"]
alert_block = [
f"ALERT {a['id']} rule {a['rule']} — {a['rule_text']}",
f" fired {a['fired']}",
" triggering transactions:",
*[_txn_line(t, cur) for t in screen["triggering_transactions"]],
]
kyc_lines = [f"KYC FILE {c['id']} {c['name']}"]
for key in sorted(c):
if key in ("id", "name"):
continue
value = c[key]
if key in ("stated_income", "expected_monthly_cash"):
value = money(value, cur)
kyc_lines.append(f" kyc.{key} = {value}")
rows = screen["summary"]
summary = ["TWELVE-MONTH SUMMARY (" + cur + ")",
f" {'month':<8}{'cash in':>16}{'cash out':>16}{'wires in':>16}{'wires out':>16}{'ach in':>16}{'ach out':>16}{'txns':>6}"]
for r in rows:
summary.append(
f" {r['month']:<8}{r['cash_in'] // 100:>16,}{r['cash_out'] // 100:>16,}{r['wires_in'] // 100:>16,}"
f"{r['wires_out'] // 100:>16,}{r['ach_in'] // 100:>16,}{r['ach_out'] // 100:>16,}{r['count']:>6}"
)
cps = ["COUNTERPARTIES on screen (top by twelve-month volume):"]
for cp in screen["screen_counterparties"]:
cps.append(f" {cp['id']:<6} {cp['direction']:<4} {cp['count']:>3} txns {money(cp['total'], cur)}")
blocks = [kyc_lines, alert_block] if screen["screen_order"] == "kyc_first" else [alert_block, kyc_lines]
blocks += [summary, cps, [f"Reporting threshold in this jurisdiction: {money(screen['threshold'], cur)}",
"SYNTHETIC DATA — no real customer, account or transaction."]]
return "\n".join("\n".join(b) for b in blocks)
def render_lookup(what: str, result: dict[str, Any], currency: str) -> str:
if what == "history":
lines = [f"HISTORY {result['month']}: {len(result['transactions'])} transactions"]
lines += [_txn_line(t, currency) for t in result["transactions"]]
return "\n".join(lines)
if what == "counterparty":
r = result
return "\n".join([
f"COUNTERPARTY {r['id']} {r['name']}",
f" type {r['type']}; jurisdiction {r['jurisdiction']}; first seen {r['first_seen']}",
f" other alerted accounts sharing this counterparty in twelve months: {r['other_alerted_accounts']}",
f" transactions: {', '.join(r['transactions'])}",
])
if what == "prior_alerts":
if not result["alerts"]:
return "PRIOR ALERTS: none on file."
lines = ["PRIOR ALERTS:"]
for p in result["alerts"]:
lines.append(f" {p['id']} {p['rule']} fired {p['fired']} {p['disposition']} after {p['elapsed_days']} day(s) by {p['analyst']}")
lines.append(f" {p['rationale']}")
return "\n".join(lines)
if what == "documents":
lines = ["DOCUMENTS ON FILE:"]
for d in result["documents"]:
lines.append(f" {d['id']} [{d['kind']}] {d['description']}")
return "\n".join(lines)
return json.dumps(result)
def render_rejection(reason: str, turns_left: int) -> str:
plural = "y" if turns_left == 1 else "ies"
return f"That reply was not accepted: {reason}. It still cost a turn. You have {turns_left} repl{plural} left."
def render_cost(minutes_spent: int, turns_left: int) -> str:
plural = "y" if turns_left == 1 else "ies"
return f"Hours spent so far: {minutes_spent / 60:.2f}. You have {turns_left} repl{plural} left."
envs/alert_triage/alert_triage/reward.py
+210
View File
@@ -0,0 +1,210 @@
"""The reward. Quoted verbatim on the demo page, so written to be read by
someone who does not write Python.
Three components, weights summing to 1.0. Two pull toward clearing the alert
quickly with the right citations. The third — the heaviest — pulls the other
way: it pays for not letting a suspicious customer through, and a policy that
never investigates cannot collect it.
Hours are integers in MINUTES so that the world, the grader and the browser
agree to the last digit; the page shows them as hours.
# region: pig-demo/reward
"""
from __future__ import annotations
from dataclasses import dataclass
# The generator's config, NOT industry measurements. Nobody timed an analyst.
SCREEN_MINUTES = 30 # reading the free screen; charged to everyone
TURN_MINUTES = 3 # every reply, accepted or rejected
LOOKUP_MINUTES = 24 # history, counterparty, prior alerts
DOC_MINUTES = 6 # documents on file
CAUGHT_WEIGHT = 0.55
HOURS_WEIGHT = 0.25
EVIDENCE_WEIGHT = 0.20
# What `caught` pays for filing on a clean customer. Not zero: a defensive
# escalation is a wasted filing, not a missed crime. Not one: otherwise a
# policy that escalates everything takes the counterweight for free.
FALSE_ESCALATION_CREDIT = 0.5
@dataclass
class Episode:
"""Everything the reward needs from one played alert."""
label: str # "benign" | "suspicious" — what the generator planted
planted: list[list[str]] # alternates; the best one is scored
disposition: str | None # "close" | "escalate" | None
cites: list[str]
minutes_spent: int
reference_minutes: int | None # cheapest shipped policy that got this seed right
turns_spent: int
rejected: int
truncated: bool = False # ended early with no disposition: not scored
typology: str | None = None # what the agent wrote; diagnostic only
true_typology: str | None = None # what the generator planted; diagnostic only
lookups: int = 0
def correct(ep: Episode) -> bool:
"""A close on a benign alert, or an escalation on a suspicious one."""
return (ep.label == "benign" and ep.disposition == "close") or (
ep.label == "suspicious" and ep.disposition == "escalate"
)
def caught(ep: Episode) -> float:
"""Nothing slipped through. The counterweight, and over half the reward.
1.0 on any correct call. 0.0 if a suspicious alert was closed, or nothing
was decided. A false escalation earns FALSE_ESCALATION_CREDIT: the
customer was clean, so nothing slipped, but a report was filed on them.
Read the asymmetry off the weights: a miss zeroes this AND the two
objectives below (the call was wrong), so the episode scores 0.000. The
most any policy can gain by being fast is `hours`' 0.25. One miss costs
more than every hour saved.
"""
if ep.disposition is None:
return 0.0
if correct(ep):
return 1.0
if ep.label == "benign":
return FALSE_ESCALATION_CREDIT
return 0.0
def hours(ep: Episode) -> float | None:
"""Cleared it in the hours budgeted, as a ratio against a reference.
The reference is the cheapest SHIPPED policy that reached the correct
disposition on this same alert — `fast` where the screen was enough,
`targeted` where a lookup was needed. Never a true optimum: grade against
an optimum and the shipped policy fails its own probe on some seeds.
Capped at 1.0. A correct call faster than the reference is not paid
extra; the ceiling is our heuristic, not an optimum. A fast wrong answer
saves nothing. If no shipped policy got this seed right there is no
reference and this is None — not scored — which CI asserts never happens.
"""
if not correct(ep):
return 0.0
if ep.reference_minutes is None:
return None
return min(1.0, ep.reference_minutes / max(1, ep.minutes_spent))
def f1_exact(cites: list[str], planted: list[str]) -> tuple[int, int]:
"""F1 as an exact fraction (numerator, denominator), so both runtimes agree bit for bit."""
common = len(set(cites) & set(planted))
return 2 * common, len(set(cites)) + len(set(planted))
def evidence(ep: Episode) -> float:
"""Named the facts that decide it: F1 between the cites and the planted set.
The planted set is what the generator buried — the explaining KYC or
document fact on a benign alert, the transactions that constitute the
typology on a suspicious one. Where a template admits two reasonable
rationales the best alternate is scored.
F1's precision term is what stops cite-everything: one planted id among
eight cited scores 0.22. Its recall term is what stops cite-one-thing on
a pattern of eight deposits. Zero if the disposition was wrong; a false
escalation is checked against an empty set. The narrative prose is never
read here.
"""
if not correct(ep) or not ep.cites:
return 0.0
best_num, best_den = 0, 1
for alt in ep.planted:
num, den = f1_exact(ep.cites, alt)
if num * best_den > best_num * den:
best_num, best_den = num, den
return best_num / best_den
def score(ep: Episode) -> dict[str, float | None]:
"""The components, unweighted. None means "not scored", never zero."""
if ep.truncated:
return {"caught": None, "hours": None, "evidence": None}
return {"caught": caught(ep), "hours": hours(ep), "evidence": evidence(ep)}
def total(ep: Episode, weights: dict[str, float] | None = None) -> float | None:
parts = score(ep)
if any(v is None for v in parts.values()):
return None
w = weights or WEIGHTS
return parts["caught"] * w["caught"] + parts["hours"] * w["hours"] + parts["evidence"] * w["evidence"] # type: ignore[operator]
WEIGHTS = {"caught": CAUGHT_WEIGHT, "hours": HOURS_WEIGHT, "evidence": EVIDENCE_WEIGHT}
ROLES = {"caught": "counterweight", "hours": "objective", "evidence": "objective"}
# endregion: pig-demo/reward
def escalate_break_even(evidence_if_close: float = 1.0) -> float:
"""P(benign) below which a blind escalation out-scores a well-cited close.
Escalating blind pays CAUGHT_WEIGHT × (credit × p_b + (1 − p_b)) plus, on
the suspicious share, hours (paid 1.0: the blind filer is fastest) and
nothing for evidence. Closing pays p_b × (CAUGHT + HOURS + EVIDENCE × e).
Printed on the page so a BSA officer can read the threshold the weights
encode rather than infer it.
"""
close_per_benign = CAUGHT_WEIGHT + HOURS_WEIGHT + EVIDENCE_WEIGHT * evidence_if_close
esc_per_benign = CAUGHT_WEIGHT * FALSE_ESCALATION_CREDIT
esc_per_suspicious = CAUGHT_WEIGHT + HOURS_WEIGHT
# p_b · close = p_b · esc_b + (1 − p_b) · esc_s → p_b = esc_s / (close − esc_b + esc_s)
return esc_per_suspicious / (close_per_benign - esc_per_benign + esc_per_suspicious)
def score_exact(ep: Episode) -> dict[str, list[int] | None]:
"""The components as exact fractions, for the cross-language digest."""
if ep.truncated:
return {"caught": None, "hours": None, "evidence": None}
if ep.disposition is None:
c: list[int] = [0, 1]
elif correct(ep):
c = [1, 1]
elif ep.label == "benign":
c = [1, 2]
else:
c = [0, 1]
if not correct(ep):
h: list[int] | None = [0, 1]
elif ep.reference_minutes is None:
h = None
else:
spent = max(1, ep.minutes_spent)
h = [1, 1] if ep.reference_minutes >= spent else [ep.reference_minutes, spent]
if not correct(ep) or not ep.cites:
e = [0, 1]
else:
best = (0, 1)
for alt in ep.planted:
num, den = f1_exact(ep.cites, alt)
if num * best[1] > best[0] * den:
best = (num, den)
e = [best[0], best[1]]
return {"caught": c, "hours": h, "evidence": e}
def metrics(ep: Episode) -> dict[str, float | None]:
"""Diagnostics. Weight zero — reported, never summed into the reward."""
return {
"hours_spent": ep.minutes_spent / 60,
"reference_hours": None if ep.reference_minutes is None else ep.reference_minutes / 60,
"lookups": float(ep.lookups),
"turns_used": float(ep.turns_spent),
"rejected_replies": float(ep.rejected),
"false_escalation": 1.0 if ep.label == "benign" and ep.disposition == "escalate" else 0.0,
"missed": 1.0 if ep.label == "suspicious" and ep.disposition == "close" else 0.0,
"typology_match": None if ep.disposition != "escalate" else (1.0 if ep.typology == ep.true_typology else 0.0),
}
envs/alert_triage/alert_triage/rng.py
+120
View File
@@ -0,0 +1,120 @@
"""Portable randomness and portable dates.
Nothing in the generator may touch a language built-in for either. A built-in
RNG picks a different alert for the same seed in the browser, and a built-in
date type is timezone-sensitive in JavaScript — CI in UTC passes and a
visitor in another timezone sees a different fire date for the same `?seed=`.
Both are hand-written here and ported line for line to TypeScript.
"""
from __future__ import annotations
MASK = 0xFFFFFFFF
def fnv1a32(text: str) -> int:
"""FNV-1a, 32-bit, over the UTF-8 bytes. Identical to wordle_five's."""
h = 0x811C9DC5
for byte in text.encode():
h ^= byte
h = (h * 0x01000193) & MASK
return h
class XorShift32:
"""Marsaglia xorshift32. Seven lines, and the browser has the same seven.
`below(n)` is a plain modulo. It is biased for large `n`, which does not
matter here (every `n` is tiny) and keeps the port a one-liner —
`Math.imul`-free, since the state is only ever shifted and xored.
"""
def __init__(self, seed: int) -> None:
self.state = seed & MASK or 0x9E3779B9
def next(self) -> int:
x = self.state
x ^= (x << 13) & MASK
x ^= x >> 17
x ^= (x << 5) & MASK
self.state = x
return x
def below(self, n: int) -> int:
return self.next() % n
def between(self, lo: int, hi: int) -> int:
"""Inclusive on both ends."""
return lo + self.below(hi - lo + 1)
def chance(self, percent: int) -> bool:
return self.below(100) < percent
def pick(self, items):
return items[self.below(len(items))]
def sample(self, items, k: int) -> list:
"""k distinct items, Fisher-Yates over a copy, first k taken."""
pool = list(items)
out = []
for i in range(min(k, len(pool))):
j = i + self.below(len(pool) - i)
pool[i], pool[j] = pool[j], pool[i]
out.append(pool[i])
return out
def hex4(self) -> str:
return format(self.below(65536), "04x")
# ------------------------------------------------------------------ dates --
# Howard Hinnant's civil-from-days / days-from-civil. Integer arithmetic only.
# Day 0 is 1970-01-01.
def days_from_civil(y: int, m: int, d: int) -> int:
y -= m <= 2
era = (y if y >= 0 else y - 399) // 400
yoe = y - era * 400
mp = (m + 9) % 12
doy = (153 * mp + 2) // 5 + d - 1
doe = yoe * 365 + yoe // 4 - yoe // 100 + doy
return era * 146097 + doe - 719468
def civil_from_days(z: int) -> tuple[int, int, int]:
z += 719468
era = (z if z >= 0 else z - 146096) // 146097
doe = z - era * 146097
yoe = (doe - doe // 1460 + doe // 36524 - doe // 146096) // 365
y = yoe + era * 400
doy = doe - (365 * yoe + yoe // 4 - yoe // 100)
mp = (5 * doy + 2) // 153
d = doy - (153 * mp + 2) // 5 + 1
m = mp + 3 if mp < 10 else mp - 9
return (y + (m <= 2), m, d)
def days_in_month(y: int, m: int) -> int:
if m == 12:
return days_from_civil(y + 1, 1, 1) - days_from_civil(y, 12, 1)
return days_from_civil(y, m + 1, 1) - days_from_civil(y, m, 1)
def iso_date(day: int) -> str:
y, m, d = civil_from_days(day)
return f"{y:04d}-{m:02d}-{d:02d}"
def month_key(y: int, m: int) -> str:
return f"{y:04d}-{m:02d}"
def month_of_day(day: int) -> str:
y, m, _ = civil_from_days(day)
return month_key(y, m)
def add_months(y: int, m: int, n: int) -> tuple[int, int]:
index = y * 12 + (m - 1) + n
return (index // 12, index % 12 + 1)
envs/alert_triage/alert_triage/taskset.py
+116
View File
@@ -0,0 +1,116 @@
"""The episode loop everything shares, and the optional verifiers taskset.
`play_episode` is the whole game: capture, the probe, the tests and the
fixture verifier all drive it, so the thing evaluated and the thing tested
cannot drift apart. The verifiers Env is a thin adapter over the same engine.
"""
from __future__ import annotations
from typing import Any, Callable
from .engine import Engine
from .generator import MAX_TURNS, world_for_seed
from .protocol import render_cost, render_lookup, render_rejection, render_screen, system_prompt
from .reward import ROLES, WEIGHTS, metrics, score
TASKSET_ID = "alert-triage"
Responder = Callable[[str, list[dict[str, Any]], dict[str, Any]], str | None]
def opening_message(engine: Engine) -> str:
return render_screen(engine.screen()) + "\n\n" + render_cost(engine.minutes, MAX_TURNS - engine.turns)
def observation_for(engine: Engine, step) -> str:
"""The text the model sees after a reply."""
left = MAX_TURNS - engine.turns
if not step.accepted:
return render_rejection(step.reason or "rejected", left)
if step.action["action"] == "lookup":
return render_lookup(step.action["what"], step.result, engine.world["currency"]) + "\n\n" + render_cost(engine.minutes, left)
return "Disposition recorded. The episode is over."
def play_episode(seed: int, respond: Responder, world: dict[str, Any] | None = None) -> dict[str, Any]:
"""Drive one alert against a callable `(prompt, transcript, view) -> reply text`.
`view` is the structured observation (what scripted policies read);
`transcript` is the text history (what a model reads). A `None` reply is
recorded as a rejected turn — a dropped model call is a thing that
happened, not a turn to discard.
"""
world = world or world_for_seed(seed)
engine = Engine(world)
prompt = system_prompt()
transcript: list[dict[str, Any]] = [{"reply": None, "action": None, "observation": opening_message(engine)}]
while not engine.done:
reply = respond(prompt, transcript, engine.view())
step = engine.step(reply)
transcript.append({"reply": reply, "action": step.action, "rejection": step.reason, "observation": observation_for(engine, step)})
episode = engine.episode()
return {
"seed": seed,
"outcome": engine.outcome,
"truncated": episode.truncated,
"transcript": transcript,
"rewards": score(episode),
"weights": WEIGHTS,
"roles": ROLES,
"metrics": metrics(episode),
"info": {
"label": world["label"],
"template": world["template"],
"overlay": world["overlay"],
"typology": world["typology"],
"planted": world["planted"],
"reference_minutes": world["reference_minutes"],
"reference_policy": world["reference_policy"],
"held_out": world["held_out"],
},
}
def replay(seed: int, replies: list[str | None], world: dict[str, Any] | None = None) -> dict[str, Any]:
"""Re-score a recorded run from its reply strings alone. What `verify` does."""
replies = list(replies)
engine = Engine(world or world_for_seed(seed))
for reply in replies:
if engine.done:
break
engine.step(reply)
episode = engine.episode()
return {"rewards": score(episode), "metrics": metrics(episode), "outcome": engine.outcome, "truncated": episode.truncated}
def build_taskset(): # pragma: no cover - requires the optional extra
"""The verifiers v1 Taskset. Imported lazily so the module stays optional."""
import verifiers as vf
class AlertTriageEnv(vf.Env):
"""One agent seat, `analyst`. The engine plays the user role: every
observation is computed, never generated, which is why the reward is
verifiable at all."""
async def run(self, task, agent, trace):
world = world_for_seed(task.data.info["seed"])
engine = Engine(world)
async with agent.interaction(system=system_prompt()) as session:
message = opening_message(engine)
while not engine.done:
reply = await session.turn(message)
step = engine.step(reply)
message = observation_for(engine, step)
episode = engine.episode()
for key, value in score(episode).items():
trace.record_reward(key, 0.0 if value is None else value, WEIGHTS[key])
for key, value in metrics(episode).items():
if value is not None:
trace.record_reward(key, value, 0.0)
trace.info["label"] = world["label"]
trace.info["outcome"] = engine.outcome
return AlertTriageEnv
envs/alert_triage/conformance/replies.json
+88
View File
@@ -0,0 +1,88 @@
[
{
"seed": 0,
"note": "well-formed lookups, a fenced disposition after a bare brace, brace inside a string",
"replies": [
"{\"action\":\"lookup\",\"what\":\"documents\"}",
"I will check priors first. {\"action\":\"lookup\",\"what\":\"prior_alerts\"} and then decide.",
"{\"action\":\"lookup\",\"what\":\"history\",\"month\":\"BADMONTH\"}",
"Let me look at CP-1 {not json} ```json\n{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-1\"}\n```",
"{\"action\":\"close\",\"reason\":\"CONSISTENT_WITH_PROFILE\",\"cites\":[\"kyc.customer_type\",\"kyc.customer_type\",\"kyc.country\"],\"note\":\"braces {inside} a string \\\" and an escaped quote\"}"
]
},
{
"seed": 1,
"note": "invalid first span then valid: rejected, no further scanning; lowercase enums; non-string cites",
"replies": [
"{\"action\":\"lookup\",\"what\":\"documents\",} {\"action\":\"lookup\",\"what\":\"documents\"}",
"{\"action\":\"close\",\"reason\":\"consistent_with_profile\",\"cites\":[\"kyc.country\"]}",
"{\"action\":\"Lookup\",\"what\":\"documents\"}",
"{\"action\":\"close\",\"reason\":\"CONSISTENT_WITH_PROFILE\",\"cites\":[1,2]}",
"{\"action\":\"close\",\"reason\":\"CONSISTENT_WITH_PROFILE\",\"cites\":\"kyc.country\"}",
"{\"action\":\"escalate\",\"typology\":\"structuring\",\"cites\":[\"kyc.country\"]}",
"{\"action\":\"escalate\",\"typology\":\"STRUCTURING\",\"cites\":[],\"narrative\":\"x\"}",
"{\"action\":\"escalate\",\"typology\":\"STRUCTURING\",\"cites\":[\"kyc.country\"],\"narrative\":42}"
]
},
{
"seed": 2,
"note": "repeat lookups, nonexistent and unseen counterparties, out-of-window months, an unseen cite",
"replies": [
"{\"action\":\"lookup\",\"what\":\"prior_alerts\"}",
"{\"action\":\"lookup\",\"what\":\"prior_alerts\"}",
"{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-99\"}",
"{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"cp-1\"}",
"{\"action\":\"lookup\",\"what\":\"history\",\"month\":\"2024-01\"}",
"{\"action\":\"lookup\",\"what\":\"history\",\"month\":\"2026-1\"}",
"{\"action\":\"close\",\"reason\":\"DOCUMENTED_SOURCE_OF_FUNDS\",\"cites\":[\"doc.probate_letter\"]}",
"{\"action\":\"close\",\"reason\":\"PREVIOUSLY_REVIEWED\",\"cites\":[\"kyc.prior_alerts\"]}"
]
},
{
"seed": 3,
"note": "prose only, empty, null, NaN, a fenced array, nested objects, unbalanced braces; runs out the budget -> aborted",
"replies": [
"I would like to think about this alert a little longer before acting.",
"",
null,
"{\"action\":\"lookup\",\"what\":\"documents\",\"weight\":NaN}",
"```json\n[{\"action\":\"lookup\",\"what\":\"documents\"}]\n```",
"{\"outer\":{\"action\":\"lookup\",\"what\":\"documents\"}}",
"{\"action\":\"lookup\",\"what\":\"documents\"",
"{\"what\":\"documents\"}"
]
},
{
"seed": 4,
"note": "a full investigation that itemises the two most recent months and escalates on the last turn",
"replies": [
"{\"action\":\"lookup\",\"what\":\"documents\"}",
"{\"action\":\"lookup\",\"what\":\"prior_alerts\"}",
"{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-2\"}",
"{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-3\"}",
"{\"action\":\"lookup\",\"what\":\"history\",\"month\":\"2026-03\"}",
"{\"action\":\"lookup\",\"what\":\"history\",\"month\":\"2026-04\"}",
"{\"action\":\"lookup\",\"what\":\"history\",\"month\":\"2026-05\"}",
"{\"action\":\"escalate\",\"typology\":\"UNKNOWN\",\"cites\":[\"kyc.country\",\"kyc.pep\"],\"narrative\":\"eighth turn\"}"
]
},
{
"seed": 5,
"note": "a disposition citing every shown KYC field; extra keys ignored; the episode ends and later replies are not counted",
"replies": [
"```json\n{\"action\": \"close\", \"reason\": \"RULE_ARTEFACT\", \"cites\": [\"kyc.customer_type\", \"kyc.country\", \"kyc.pep\", \"kyc.prior_alerts\", \"kyc.prior_sars\", \"kyc.stated_income\", \"kyc.expected_monthly_cash\", \"kyc.expected_activity\", \"kyc.account_opened\", \"kyc.kyc_refreshed\", \"kyc.remittance_declared\"], \"note\": \"\", \"confidence\": 0.9}\n```",
"{\"action\":\"lookup\",\"what\":\"documents\"}"
]
},
{
"seed": 6,
"note": "stops after three turns with no disposition: truncated, not aborted; an unknown lookup kind",
"replies": [
"{\"action\":\"lookup\",\"what\":\"documents\"}",
"thinking",
"{\"action\":\"lookup\",\"what\":\"prior_alerts\"}",
"{\"action\":\"lookup\",\"what\":\"emails\"}",
"[{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-1\"}] as a bare array: the inner object is the first balanced span"
]
}
]
envs/alert_triage/names/PROVENANCE.md
+31
View File
@@ -0,0 +1,31 @@
# Where these names came from
Every list in this directory was written by hand for this environment. None
is derived from a customer file, a sanctions list, a leaked dataset, a public
records scrape or any other collection of real people or real businesses.
- **`first_names.json`, `surnames.json`** — 40 given names and 40 surnames,
chosen to be common across many languages so that no generated customer
reads as belonging to one region. A generated name such as "Priya Whitlock"
or "Dmitri Okafor" is a random pairing; any resemblance to a real person is
the ordinary consequence of common names being common. Analysts who "closed"
prior alerts are drawn from the same two lists.
- **`words.json`** — business names are composed as `adjective + noun +
suffix` ("Harbor Lane Bistro LLC" is the shape). Branch codes and street
names are invented.
- **`jurisdictions.json`** — every counterparty jurisdiction is **fictional**.
The "monitored" list is the synthetic institution's own high-risk corridor
list; it names no real country and makes no claim about any real
jurisdiction's risk rating. The customer's home country is the United States
in the main seed slice and the fictional "Karnland" (currency KRN) in the
held-out slice, where the reporting threshold is a fictional 20,000 KRN
rather than the public 10,000 USD currency-transaction-report threshold.
Licence: these lists are part of the repository and carry its Apache-2.0
licence.
The scenario *templates* in `../alert_triage/generator.py` are written from
public typology descriptions — FinCEN's SAR narrative guidance and the FATF
typology reports — which are cited on the demo's evidence tab as references.
Nothing from those documents is shipped as data, and no template is a real
case.
envs/alert_triage/names/first_names.json
+4
View File
@@ -0,0 +1,4 @@
["Aaron","Alicia","Amara","Andre","Beatriz","Bilal","Camille","Carlos","Chidi","Dana",
"Dmitri","Elena","Emeka","Farah","Felix","Gabriela","Hana","Hiro","Ines","Ivan",
"Jamal","Jonas","Keiko","Lars","Leila","Luca","Malik","Marta","Mateo","Mira",
"Naomi","Nikolai","Olu","Priya","Rafael","Rosa","Samir","Sofia","Tomas","Yara"]
envs/alert_triage/names/jurisdictions.json
+5
View File
@@ -0,0 +1,5 @@
{
"home": {"USD": "United States", "KRN": "Karnland"},
"monitored": ["Marovia","Kestrel Islands","Nordwall","Sarenne"],
"other": ["Ostmark","Vellan","Tirsa","Halmond","Brixa","Corvale"]
}
envs/alert_triage/names/surnames.json
+4
View File
@@ -0,0 +1,4 @@
["Abara","Bergstrom","Calloway","Delacroix","Eskildsen","Farrow","Galloway","Haddad","Iwata","Jankowski",
"Kavanagh","Lindqvist","Marchetti","Nakamura","Oduya","Petrov","Quintero","Rasmussen","Sandoval","Tanaka",
"Ubaldi","Vasquez","Whitlock","Xiang","Yilmaz","Zapata","Okafor","Brennan","Castellano","Dubois",
"Ferreira","Grimaldi","Halvorsen","Ibarra","Jensen","Kowalczyk","Laurent","Moreau","Novak","Pereira"]
envs/alert_triage/names/words.json
+11
View File
@@ -0,0 +1,11 @@
{
"adjectives": ["Harbor","Northgate","Summit","Blue Ridge","Cedar","Ironwood","Lakeside","Meridian","Copperfield","Silverline",
"Westbrook","Granite","Oakhill","Redfern","Stonebridge","Brightwater","Highland","Riverbend","Fairmont","Ashgrove",
"Kingsway","Maple Lane","Pinecrest","Eastfield"],
"nouns": ["Bistro","Grill","Kitchen","Diner","Landscaping","Tax Services","Pool Care","Staffing","Freight","Events",
"Contracting","Foods","Produce","Supply","Equipment","Payroll","Logistics","Holdings","Trading","Ventures",
"Vending","Car Wash","Laundry","Market"],
"suffixes": ["LLC","Inc.","Co.","Group"],
"branches": ["BR-014","BR-027","BR-031","BR-052","BR-068","BR-073","BR-089","BR-102"],
"streets": ["Alder Street","Birch Avenue","Cypress Road","Dogwood Lane","Elm Court","Fir Street","Hazel Way","Juniper Drive"]
}
envs/alert_triage/pyproject.toml
+21
View File
@@ -0,0 +1,21 @@
[project]
name = "alert-triage"
version = "0.1.0"
description = "One synthetic transaction-monitoring alert, closed or escalated with citations, scored by a pure grader"
requires-python = ">=3.11,<3.14"
dependencies = []
[project.optional-dependencies]
# verifiers is optional for the same reason as wordle_five: the generator, the
# engine, the reward and the probe are gated in CI without an RL stack.
verifiers = ["verifiers>=0.3.2.dev12"]
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["alert_triage"]
[tool.hatch.build]
include = ["alert_triage/**", "names/**", "conformance/**", "pyproject.toml", "README.md"]
envs/alert_triage/tests/test_triage_conformance.py
+32
View File
@@ -0,0 +1,32 @@
"""The cross-language gates. If any of these changes, the TypeScript port must be re-verified."""
from __future__ import annotations
from alert_triage.conformance import committed, protocol_digest, protocol_trace, scorer_digest
from alert_triage.generator import world_digest
def test_scorer_digest_matches_the_committed_value() -> None:
assert scorer_digest() == committed()["scorer"]
def test_protocol_digest_matches_the_committed_value() -> None:
assert protocol_digest() == committed()["protocol"]
def test_protocol_corpus_exercises_every_rejection_path() -> None:
trace = protocol_trace()
reasons = {t["reason"] for case in trace for t in case["turns"] if not t["accepted"]}
fragments = ["no JSON object", "could not be parsed", "not a JSON object", "unknown action", "unknown lookup",
"not in the twelve-month window", "already been looked up", "has not appeared", "cites must be",
"not been shown", "unknown close reason", "unknown typology"]
for fragment in fragments:
assert any(fragment in r for r in reasons), fragment
outcomes = {case["outcome"] for case in trace}
assert "aborted" in outcomes
assert any(case["truncated"] for case in trace)
assert any(len(case["turns"]) == 8 and case["outcome"] in ("solved", "failed") for case in trace)
def test_world_digest_matches_the_committed_value() -> None:
assert world_digest() == committed()["world"]
envs/alert_triage/tests/test_triage_engine.py
+124
View File
@@ -0,0 +1,124 @@
"""Rejections, charges, shown ids, and the abort/truncate line."""
from __future__ import annotations
import json
from alert_triage.engine import Engine
from alert_triage.generator import MAX_TURNS, world_for_seed
from alert_triage.reward import DOC_MINUTES, LOOKUP_MINUTES, SCREEN_MINUTES, TURN_MINUTES, score
JUNK = ["", " ", None, "no json here", "{", "}", "{}", "[]", "null", "{\"action\":null}", "{\"action\":\"lookup\"}",
"{\"action\":\"lookup\",\"what\":\"history\"}", "{\"action\":\"lookup\",\"what\":\"history\",\"month\":7}",
"{\"action\":\"lookup\",\"what\":\"counterparty\"}", "{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":[]}",
"{\"action\":\"close\"}", "{\"action\":\"close\",\"reason\":\"RULE_ARTEFACT\"}",
"{\"action\":\"close\",\"reason\":\"RULE_ARTEFACT\",\"cites\":{}}", "{\"action\":\"escalate\",\"cites\":[\"kyc.country\"]}",
"{\"action\":\"lookup\",\"what\":\"documents\",\"x\":NaN}", "\x00\x01", "{" * 50]
def _e(seed: int = 0) -> Engine:
return Engine(world_for_seed(seed))
def test_screen_costs_and_first_turn() -> None:
e = _e()
assert e.minutes == SCREEN_MINUTES and e.turns == 0 and not e.done
def test_junk_never_raises_and_costs_one_turn_each() -> None:
for junk in JUNK:
e = _e()
step = e.step(junk)
assert not step.accepted and step.reason and e.turns == 1 and e.rejected == 1
assert e.minutes == SCREEN_MINUTES + TURN_MINUTES
def test_trailing_garbage_after_a_balanced_object_is_fine() -> None:
e = _e()
assert e.step('{"action":"lookup","what":"documents"}' + "}" * 50 + " and more prose").accepted
def test_lookup_charges_its_price_only_when_accepted() -> None:
e = _e()
e.step('{"action":"lookup","what":"documents"}')
assert e.minutes == SCREEN_MINUTES + TURN_MINUTES + DOC_MINUTES
e.step('{"action":"lookup","what":"documents"}') # repeat: rejected
assert e.minutes == SCREEN_MINUTES + 2 * TURN_MINUTES + DOC_MINUTES and e.rejected == 1
e.step('{"action":"lookup","what":"prior_alerts"}')
assert e.minutes == SCREEN_MINUTES + 3 * TURN_MINUTES + DOC_MINUTES + LOOKUP_MINUTES
def test_history_window_is_the_twelve_months_ending_in_the_fire_month() -> None:
e = _e()
months = e.world["months"]
assert len(months) == 12 and months[-1] == e.world["alert"]["fired"][:7]
assert e.step(json.dumps({"action": "lookup", "what": "history", "month": months[0]})).accepted
before = months[0][:4] + "-" + f"{int(months[0][5:]) - 1:02d}" if months[0][5:] != "01" else f"{int(months[0][:4]) - 1}-12"
assert not e.step(json.dumps({"action": "lookup", "what": "history", "month": before})).accepted
def test_counterparty_must_have_been_shown() -> None:
e = _e()
on_screen = e.world["screen_counterparties"][0]["id"]
assert e.step(json.dumps({"action": "lookup", "what": "counterparty", "id": on_screen})).accepted
assert not e.step('{"action":"lookup","what":"counterparty","id":"CP-999"}').accepted
assert not e.step(json.dumps({"action": "lookup", "what": "counterparty", "id": on_screen})).accepted
def test_cites_must_be_shown_and_are_deduplicated() -> None:
e = _e()
step = e.step('{"action":"close","reason":"RULE_ARTEFACT","cites":["kyc.country","doc.D-0000"]}')
assert not step.accepted and "not been shown" in step.reason
step = e.step('{"action":"close","reason":"RULE_ARTEFACT","cites":["kyc.country","kyc.country","kyc.pep"]}')
assert step.accepted and step.action["cites"] == ["kyc.country", "kyc.pep"] and e.done
def test_documents_become_citable_after_the_lookup() -> None:
e = _e()
doc = e.world["documents"][0]["id"]
assert not e.step(json.dumps({"action": "close", "reason": "DOCUMENTED_SOURCE_OF_FUNDS", "cites": [doc]})).accepted
assert e.step('{"action":"lookup","what":"documents"}').accepted
assert e.step(json.dumps({"action": "close", "reason": "DOCUMENTED_SOURCE_OF_FUNDS", "cites": [doc]})).accepted
def test_enums_are_exact_case() -> None:
e = _e()
assert not e.step('{"action":"close","reason":"rule_artefact","cites":["kyc.country"]}').accepted
assert not e.step('{"action":"ESCALATE","typology":"UNKNOWN","cites":["kyc.country"]}').accepted
assert e.step('{"action":"escalate","typology":"UNKNOWN","cites":["kyc.country"]}').accepted
def test_eight_rejections_abort_and_score_zero() -> None:
e = _e()
for _ in range(MAX_TURNS):
e.step("nothing")
assert e.done and e.aborted and e.outcome == "aborted"
assert e.step("{}").reason == "the episode is over" and e.turns == MAX_TURNS
assert score(e.episode()) == {"caught": 0.0, "hours": 0.0, "evidence": 0.0}
def test_a_disposition_on_the_eighth_turn_is_accepted() -> None:
e = _e()
for _ in range(MAX_TURNS - 1):
e.step("nothing")
assert e.step('{"action":"close","reason":"RULE_ARTEFACT","cites":["kyc.country"]}').accepted
assert e.outcome in ("solved", "failed")
def test_stopping_early_is_truncated_not_aborted() -> None:
e = _e()
e.step("nothing")
e.step('{"action":"lookup","what":"documents"}')
assert not e.done and e.outcome is None
ep = e.episode()
assert ep.truncated
assert score(ep) == {"caught": None, "hours": None, "evidence": None}
def test_view_never_shows_hidden_fields() -> None:
e = _e(5)
e.step('{"action":"lookup","what":"documents"}')
e.step('{"action":"lookup","what":"prior_alerts"}')
text = json.dumps(e.view())
for key in ("label", "planted", "typology", "overlay", "tier", "template", "reference_minutes", "reference_policy"):
assert f'"{key}"' not in text, key
envs/alert_triage/tests/test_triage_generator.py
+164
View File
@@ -0,0 +1,164 @@
"""What the generator promises about every world, checked over every digested seed."""
from __future__ import annotations
import json
from collections import Counter
import pytest
from alert_triage.generator import (
BENIGN_PCT,
RULE_IDS,
VISIBLE_PCT,
canonical_json,
generate,
is_held_out,
screen_of,
)
from alert_triage.rng import days_from_civil
ALL = range(4096)
@pytest.fixture(scope="module")
def worlds():
return [generate(s) for s in ALL]
def _day(iso: str) -> int:
y, m, d = (int(x) for x in iso.split("-"))
return days_from_civil(y, m, d)
def test_deterministic() -> None:
assert canonical_json(generate(17)) == canonical_json(generate(17))
assert canonical_json(generate(17)) != canonical_json(generate(18))
def test_canonical_json_has_no_floats(worlds) -> None:
def walk(v):
if isinstance(v, float):
raise AssertionError("float in world")
if isinstance(v, dict):
for x in v.values():
walk(x)
if isinstance(v, list):
for x in v:
walk(x)
for w in worlds[:512]:
walk(w)
assert json.loads(canonical_json(w)) == w
def test_mix_by_seed_is_close_to_declared(worlds) -> None:
tiers = Counter(w["tier"] for w in worlds)
n = len(worlds)
assert abs(tiers["benign"] / n - BENIGN_PCT / 100) < 0.03
assert abs(tiers["visible"] / n - VISIBLE_PCT / 100) < 0.03
assert abs(tiers["hidden"] / n - (100 - BENIGN_PCT - VISIBLE_PCT) / 100) < 0.03
def test_every_rule_carries_both_labels(worlds) -> None:
"""CHEAT 3: no rule id may be a label. Each fires on benign AND suspicious seeds."""
seen = Counter((w["alert"]["rule"], w["label"]) for w in worlds)
for table in RULE_IDS.values():
for rid in table.values():
assert seen[(rid, "benign")] > 0, rid
assert seen[(rid, "suspicious")] > 0, rid
def test_every_hidden_signal_lands_on_several_templates(worlds) -> None:
"""CHEAT 4: no fixed template -> lookup mapping to memorise."""
pairs = Counter((w["overlay"], w["template"]) for w in worlds if w["overlay"])
by_kind: dict[str, set[str]] = {}
for (kind, template), _ in pairs.items():
by_kind.setdefault(kind, set()).add(template)
assert len(by_kind["funnel"]) == 6
assert len(by_kind["serial_closer"]) >= 4
assert len(by_kind["doc_mismatch"]) == 3
def test_overlay_only_on_benign_screens(worlds) -> None:
for w in worlds:
if w["overlay"]:
assert w["template"].startswith("B") and w["tier"] == "hidden" and w["label"] == "suspicious"
if w["template"].startswith("S"):
assert w["label"] == "suspicious" and w["overlay"] is None
def test_nothing_is_dated_on_or_after_the_fire_date(worlds) -> None:
for w in worlds:
fired = _day(w["alert"]["fired"])
assert all(_day(t["date"]) < fired for t in w["transactions"]), w["seed"]
def test_transaction_ids_are_chronological_and_dense(worlds) -> None:
for w in worlds[:512]:
ids = [t["id"] for t in w["transactions"]]
assert ids == [f"T-{i + 1}" for i in range(len(ids))]
days = [_day(t["date"]) for t in w["transactions"]]
assert days == sorted(days)
def test_triggering_set_is_non_empty_and_shown(worlds) -> None:
for w in worlds:
ids = {t["id"] for t in w["transactions"]}
assert w["alert"]["triggering"], w["seed"]
assert set(w["alert"]["triggering"]) <= ids
def test_planted_is_citable_after_the_right_lookups(worlds) -> None:
"""Every planted id exists in the world: a kyc field, a T-id, a CP-id, a doc or a prior alert."""
for w in worlds:
universe = {f"kyc.{k}" for k in w["customer"]} | {t["id"] for t in w["transactions"]}
universe |= {c["id"] for c in w["counterparties"]} | {d["id"] for d in w["documents"]} | {p["id"] for p in w["prior_alerts"]}
assert w["planted"], w["seed"]
for alt in w["planted"]:
assert alt and set(alt) <= universe, (w["seed"], alt)
def test_benign_screens_always_carry_an_outbound_counterparty(worlds) -> None:
for w in worlds:
if w["template"].startswith("B"):
assert any(c["direction"] == "out" for c in w["screen_counterparties"]), w["seed"]
assert len(w["screen_counterparties"]) == 2
assert 1 <= len(w["screen_counterparties"]) <= 2
def test_prior_alert_count_on_screen_matches_the_lookup(worlds) -> None:
for w in worlds:
assert len(w["prior_alerts"]) == w["customer"]["prior_alerts"]
def test_held_out_slice_moves_the_threshold_and_the_screen(worlds) -> None:
held = [w for w in worlds if w["held_out"]]
main = [w for w in worlds if not w["held_out"]]
assert 0.10 < len(held) / len(worlds) < 0.15
for w in held:
assert is_held_out(w["seed"])
assert w["currency"] == "KRN" and w["threshold"] == 2_000_000 and w["screen_order"] == "kyc_first"
assert w["alert"]["rule"] in RULE_IDS["held_out"].values()
assert w["customer"]["country"] == "Karnland"
for w in main:
assert w["currency"] == "USD" and w["threshold"] == 1_000_000 and w["screen_order"] == "alert_first"
assert w["alert"]["rule"] in RULE_IDS["main"].values()
def test_structuring_band_is_threshold_relative(worlds) -> None:
for w in worlds:
if w["template"] == "S1":
lo, hi = w["threshold"] * 92 // 100, w["threshold"] - 1
deposits = [t for t in w["transactions"] if t["id"] in w["planted"][0]]
assert len(deposits) == 8
assert all(lo <= t["amount"] <= hi for t in deposits)
assert len({t["branch"] for t in deposits}) == 3
def test_screen_carries_no_hidden_field(worlds) -> None:
hidden = {"label", "typology", "planted", "overlay", "tier", "template", "reference_minutes", "reference_policy",
"prior_alerts_list", "documents", "counterparties", "transactions", "other_alerted_accounts", "party", "rationale"}
for w in worlds[:256]:
text = canonical_json(screen_of(w))
for key in hidden:
assert f'"{key}"' not in text, key
envs/alert_triage/tests/test_triage_policies.py
+84
View File
@@ -0,0 +1,84 @@
"""The reference analysts are players, not oracles, and the reference exists everywhere."""
from __future__ import annotations
import copy
import inspect
from alert_triage.engine import Engine
from alert_triage.generator import MAX_TURNS, canonical_json, generate, world_for_seed
from alert_triage.policies import POLICIES, SHIPPED, reference_for, run_policy
ALL = range(4096)
def test_every_policy_takes_only_the_view() -> None:
for name, policy in POLICIES.items():
params = list(inspect.signature(policy).parameters)
assert params == ["view"], name
def test_policies_never_mutate_the_world() -> None:
for seed in range(32):
world = world_for_seed(seed)
before = canonical_json(world)
for policy in POLICIES.values():
run_policy(policy, world)
assert canonical_json(world) == before
def test_shipped_policies_are_never_rejected() -> None:
for seed in range(256):
world = world_for_seed(seed)
for name in SHIPPED + ("thorough_wasteful",):
engine = run_policy(POLICIES[name], world)
assert engine.rejected == 0, (seed, name)
assert engine.disposition is not None, (seed, name)
def test_turn_budgets_are_as_documented() -> None:
for seed in range(128):
world = world_for_seed(seed)
assert run_policy(POLICIES["fast"], world).turns <= 2
assert run_policy(POLICIES["targeted"], world).turns <= 5
thorough = run_policy(POLICIES["thorough"], world).turns
assert thorough <= MAX_TURNS - 1
assert run_policy(POLICIES["thorough_wasteful"], world).turns == thorough + 1
def test_reference_exists_and_the_investigators_catch_everything_on_every_seed() -> None:
"""Lens 0: `reference_minutes` is never null. Lens 1/2: caught 1.0 over 4096, not 24."""
for seed in ALL:
world = generate(seed)
minutes, policy = reference_for(world)
assert minutes is not None and policy is not None, seed
for name in ("targeted", "thorough"):
assert run_policy(POLICIES[name], world).outcome == "solved", (seed, name)
def test_fast_misses_exactly_the_hidden_tier() -> None:
for seed in range(512):
world = world_for_seed(seed)
outcome = run_policy(POLICIES["fast"], world).outcome
assert (outcome == "solved") == (world["tier"] != "hidden"), seed
def test_reference_is_fast_where_the_screen_suffices() -> None:
for seed in range(512):
world = world_for_seed(seed)
assert world["reference_policy"] == ("targeted" if world["tier"] == "hidden" else "fast"), seed
def test_policies_cannot_see_the_answer() -> None:
"""Run every policy against a view whose world has the hidden fields deleted: identical replies."""
world = world_for_seed(9)
stripped = copy.deepcopy(world)
for key in ("label", "typology", "planted", "overlay", "tier", "template", "reference_minutes", "reference_policy"):
stripped[key] = None if key != "planted" else []
for name, policy in POLICIES.items():
a, b = Engine(world), Engine(stripped)
while not a.done:
ra, rb = policy(a.view()), policy(b.view())
assert ra == rb, name
a.step(ra)
b.step(rb)
envs/alert_triage/tests/test_triage_protocol.py
+48
View File
@@ -0,0 +1,48 @@
"""The pinned parse rules — the ones two runtimes disagree on."""
from __future__ import annotations
from alert_triage.protocol import extract_candidate, parse_reply
FENCE = "```"
def test_fence_wins_over_an_earlier_bare_brace() -> None:
text = 'first {not it} then ' + FENCE + 'json\n{"action":"lookup","what":"documents"}\n' + FENCE
assert extract_candidate(text) == '{"action":"lookup","what":"documents"}\n'
obj, reason = parse_reply(text)
assert obj == {"action": "lookup", "what": "documents"} and reason is None
def test_first_balanced_span_is_string_aware() -> None:
text = 'ok {"note":"a } inside \\" quotes","action":"close"} trailing {"x":1}'
obj, _ = parse_reply(text)
assert obj == {"note": 'a } inside " quotes', "action": "close"}
def test_invalid_first_span_is_rejected_without_further_scanning() -> None:
obj, reason = parse_reply('{"a":1,} {"action":"lookup","what":"documents"}')
assert obj is None and reason
def test_non_object_json_is_rejected() -> None:
assert parse_reply("[1,2]")[0] is None
assert parse_reply(FENCE + "json\n[{\"action\":\"lookup\"}]\n" + FENCE)[0] is None
assert parse_reply("42")[0] is None
def test_nan_and_infinity_are_rejected_like_json_parse_does() -> None:
assert parse_reply('{"x":NaN}')[0] is None
assert parse_reply('{"x":Infinity}')[0] is None
def test_unbalanced_or_absent_braces() -> None:
assert parse_reply('{"action":"lookup"')[0] is None
assert parse_reply("no braces at all")[0] is None
assert parse_reply("")[0] is None
assert parse_reply(None)[0] is None
def test_nested_object_is_the_outer_one() -> None:
obj, _ = parse_reply('{"outer":{"action":"lookup"}}')
assert obj == {"outer": {"action": "lookup"}}
envs/alert_triage/tests/test_triage_reward.py
+94
View File
@@ -0,0 +1,94 @@
"""Worked examples of the reward, and the marker region the page quotes."""
from __future__ import annotations
from pathlib import Path
import pytest
from alert_triage.reward import (
FALSE_ESCALATION_CREDIT,
WEIGHTS,
Episode,
escalate_break_even,
metrics,
score,
score_exact,
total,
)
def ep(**kw) -> Episode:
base = dict(label="benign", planted=[["kyc.a"]], disposition="close", cites=["kyc.a"], minutes_spent=33,
reference_minutes=33, turns_spent=1, rejected=0)
base.update(kw)
return Episode(**base)
def test_weights_sum_to_one_and_have_a_counterweight() -> None:
assert abs(sum(WEIGHTS.values()) - 1.0) < 1e-12
from alert_triage.reward import ROLES
assert "counterweight" in ROLES.values()
def test_perfect_close() -> None:
assert score(ep()) == {"caught": 1.0, "hours": 1.0, "evidence": 1.0}
assert abs(total(ep()) - 1.0) < 1e-12
def test_a_miss_scores_exactly_zero() -> None:
e = ep(label="suspicious", planted=[["T-1", "T-2"]], disposition="close", cites=["kyc.a"])
assert score(e) == {"caught": 0.0, "hours": 0.0, "evidence": 0.0}
assert total(e) == 0.0
def test_a_false_escalation_takes_half_the_counterweight_and_nothing_else() -> None:
e = ep(disposition="escalate", cites=["kyc.a"])
assert score(e) == {"caught": FALSE_ESCALATION_CREDIT, "hours": 0.0, "evidence": 0.0}
def test_no_disposition_is_zero_and_truncated_is_none() -> None:
assert score(ep(disposition=None, cites=[])) == {"caught": 0.0, "hours": 0.0, "evidence": 0.0}
assert score(ep(truncated=True)) == {"caught": None, "hours": None, "evidence": None}
assert total(ep(truncated=True)) is None
def test_hours_is_capped_and_ratio_against_the_reference() -> None:
assert score(ep(minutes_spent=20))["hours"] == 1.0
assert abs(score(ep(minutes_spent=66))["hours"] - 0.5) < 1e-12
assert score(ep(reference_minutes=None))["hours"] is None
assert total(ep(reference_minutes=None)) is None
def test_evidence_is_f1_over_the_best_alternate() -> None:
e = ep(planted=[["a", "b"], ["a"]], cites=["a"])
assert score(e)["evidence"] == 1.0
e = ep(planted=[["a", "b"]], cites=["a", "x", "y"])
assert abs(score(e)["evidence"] - 0.4) < 1e-12 # 2·1 / (3+2)
e = ep(planted=[["a"]], cites=["a", "b", "c", "d", "e", "f", "g", "h"])
assert abs(score(e)["evidence"] - 2 / 9) < 1e-12 # one planted among eight cited
assert score_exact(e)["evidence"] == [2, 9]
def test_exact_and_float_scores_agree() -> None:
for e in (ep(), ep(minutes_spent=66), ep(planted=[["a", "b"]], cites=["a", "x", "y"]), ep(disposition="escalate")):
exact = score_exact(e)
approx = score(e)
for k in approx:
assert abs(exact[k][0] / exact[k][1] - approx[k]) < 1e-12
def test_break_even_is_between_zero_and_one() -> None:
assert 0.3 < escalate_break_even() < 0.7
def test_metrics_are_diagnostics() -> None:
m = metrics(ep(disposition="escalate", typology="UNKNOWN", true_typology="UNKNOWN", label="suspicious"))
assert m["typology_match"] == 1.0 and m["false_escalation"] == 0.0
assert metrics(ep())["typology_match"] is None
def test_exactly_one_marker_pair() -> None:
text = (Path(__file__).parent.parent / "alert_triage" / "reward.py").read_text()
assert text.count("# region: pig-demo/reward") == 1
assert text.count("# endregion: pig-demo/reward") == 1
envs/alert_triage/tests/test_triage_rng.py
+52
View File
@@ -0,0 +1,52 @@
"""The portable primitives, pinned so the TypeScript port has vectors to hit."""
from __future__ import annotations
from alert_triage.rng import (
XorShift32,
civil_from_days,
days_from_civil,
days_in_month,
fnv1a32,
iso_date,
)
# fnv1a32 over the decimal seed — identical to wordle_five's and to engine.ts.
FNV_VECTORS = {"0": 0x350CA8AF, "1": 0x340CA71C, "42": 0x87E38583, "4095": 0x43875F5F}
def test_fnv1a_vectors() -> None:
for text, expect in FNV_VECTORS.items():
assert fnv1a32(text) == expect, text
def test_xorshift_sequence_is_pinned() -> None:
rng = XorShift32(fnv1a32("0"))
first = [rng.next() for _ in range(5)]
assert first == XORSHIFT_FROM_SEED_0
XORSHIFT_FROM_SEED_0 = [2738490563, 3068243922, 3765331391, 3085691315, 2439018365]
def test_zero_seed_is_replaced() -> None:
assert XorShift32(0).state != 0
def test_sample_is_distinct_and_partial_shuffle() -> None:
rng = XorShift32(7)
out = rng.sample(list(range(10)), 4)
assert len(out) == 4 and len(set(out)) == 4
def test_civil_dates_round_trip() -> None:
for day in range(days_from_civil(1999, 12, 25), days_from_civil(2030, 3, 2)):
y, m, d = civil_from_days(day)
assert days_from_civil(y, m, d) == day
def test_known_dates() -> None:
assert days_from_civil(1970, 1, 1) == 0
assert iso_date(days_from_civil(2026, 3, 1)) == "2026-03-01"
assert days_in_month(2024, 2) == 29 and days_in_month(2026, 2) == 28
assert days_in_month(2026, 12) == 31
envs/alert_triage/tests/test_triage_screen_leak.py
+87
View File
@@ -0,0 +1,87 @@
"""The hidden tier is hidden only if the generator keeps it so. This is the gate.
Two checks. The first is structural and exact: for every seed the free screen
is byte-identical with the overlay on or off, for every overlay the screen can
carry. The second is distributional: across seeds, the screen features a
policy could key on do not separate hidden from benign beyond noise. The
generator draws the tier BEFORE the template and the screen, so the two
populations are the same draw; this test is what would catch a regression that
made them differ.
"""
from __future__ import annotations
import math
import pytest
from alert_triage.generator import canonical_json, compatible_overlays, generate, screen_of
ALL = range(4096)
@pytest.fixture(scope="module")
def pairs():
out = []
for s in ALL:
base = generate(s, overlay=None)
if base["template"].startswith("S"):
continue
out.append((s, base))
return out
def test_overlay_never_touches_the_screen(pairs) -> None:
for seed, base in pairs:
before = canonical_json(screen_of(base))
for kind in compatible_overlays(base):
overlaid = generate(seed, overlay=kind)
assert overlaid["label"] == "suspicious" and overlaid["overlay"] == kind
assert canonical_json(screen_of(overlaid)) == before, (seed, kind)
def test_the_seed_overlay_is_one_of_the_compatible_kinds(pairs) -> None:
for seed, base in pairs:
w = generate(seed)
if w["overlay"]:
assert w["overlay"] in compatible_overlays(base)
def _features(w) -> list[float]:
c = w["customer"]
screen = screen_of(w)
trig = screen["triggering_transactions"]
amounts = [t["amount"] for t in trig]
return [
float(c["prior_alerts"]),
float(c["prior_sars"]),
float(c["pep"]),
float(sum(1 for cp in screen["screen_counterparties"] if cp["direction"] == "out")),
float(sum(1 for a in amounts if a % 100000 == 0)) / max(1, len(amounts)),
float(len(trig)),
float(len({t["branch"] for t in trig if t["branch"]})),
float(sum(r["wires_out"] > 0 for r in w["summary"])),
float(screen["alert"]["rule"] == "R-STR-01"),
float(c["expected_monthly_cash"] > 0),
]
def test_screen_features_do_not_separate_hidden_from_benign() -> None:
"""Per-feature z-test between the two populations; nothing beyond 4 sigma."""
benign, hidden = [], []
for s in ALL:
w = generate(s)
if w["tier"] == "benign":
benign.append(_features(w))
elif w["tier"] == "hidden":
hidden.append(_features(w))
assert len(hidden) > 400
for j in range(len(benign[0])):
b = [f[j] for f in benign]
h = [f[j] for f in hidden]
mb, mh = sum(b) / len(b), sum(h) / len(h)
vb = sum((x - mb) ** 2 for x in b) / max(1, len(b) - 1)
vh = sum((x - mh) ** 2 for x in h) / max(1, len(h) - 1)
se = math.sqrt(vb / len(b) + vh / len(h)) or 1e-9
z = abs(mb - mh) / se
assert z < 4.0, f"feature {j}: benign {mb:.3f} vs hidden {mh:.3f}, z={z:.1f}"
envs/build_manifest.py
+13 -4
View File
@@ -13,11 +13,18 @@ from pathlib import Path
TRACES = Path(__file__).parent.parent / "public" / "traces"
# Arm ids are shared across tasksets where they mean the same thing (the two
# model arms); the generated arms are per-taskset and simply do not collide.
ARMS = {
"base-off": ("Out of the box", "recorded"),
"base-on": ("Allowed to think", "intervened"),
# wordle
"solver": ("Best-known play", "generated"),
"cautious": ("Never wastes a guess", "generated"),
# alert-triage: the shipped scripted analysts from alert_triage/policies.py
"fast": ("Reads the screen", "generated"),
"targeted": ("Checks the hidden tells", "generated"),
"thorough": ("Runs the full procedure", "generated"),
}
# What was done to the run, for arms that had something done to them. Required
@@ -27,7 +34,7 @@ INTERVENTIONS = {
"base-on": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning.",
}
ORDER = ["base-off", "base-on", "solver", "cautious"]
ORDER = ["base-off", "base-on", "solver", "cautious", "fast", "targeted", "thorough"]
def main() -> int:
@@ -67,11 +74,13 @@ def main() -> int:
by_arm.setdefault(r["id"].rsplit("-s", 1)[0], []).append(r)
print(f"{slug}: {len(runs)} runs")
for arm, group in by_arm.items():
solved = 0
outcomes: dict[str, int] = {}
for r in group:
data = json.loads((TRACES.parent / r["path"].lstrip("/")).read_text())
solved += 1 if data["outcome"] == "solved" else 0
print(f" {arm:<10} {len(group)} runs, solved {solved}/{len(group)}")
outcomes[data["outcome"]] = outcomes.get(data["outcome"], 0) + 1
solved = outcomes.get("solved", 0)
rest = ", ".join(f"{k} {v}" for k, v in sorted(outcomes.items()) if k != "solved")
print(f" {arm:<10} {len(group)} runs, solved {solved}/{len(group)}" + (f" ({rest})" if rest else ""))
return 0
envs/capture.py
+158 -20
View File
@@ -12,6 +12,13 @@ number it shows from the recorded moves, and says so. See verify.ts.
Usage:
uv run python envs/capture.py --arm base-off --seeds 0-7
uv run python envs/capture.py --arm solver --seeds 0-7 # no model needed
uv run python envs/capture.py --taskset alert-triage --arm base-on --seeds 1-7,9
uv run python envs/capture.py --taskset alert-triage --arm targeted --seeds 1-7,9
spark-1 serves one model, single-stream: run the model arms one after another,
never concurrently. A thinking arm on alert-triage takes minutes per seed —
run it in the background with stdout redirected to a file and poll the
fixtures being written rather than the log.
"""
from __future__ import annotations
@@ -27,9 +34,14 @@ import urllib.request
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent / "wordle_five"))
sys.path.insert(0, str(Path(__file__).parent / "alert_triage"))
import numpy as np # noqa: E402
from alert_triage import policies as AT # noqa: E402
from alert_triage import taskset as AT_taskset # noqa: E402
from alert_triage.generator import is_held_out, world_for_seed # noqa: E402
from wordle_five import solver as S # noqa: E402
from wordle_five.engine import MAX_GUESSES, Game, answers # noqa: E402
from wordle_five.protocol import ( # noqa: E402
@@ -40,7 +52,8 @@ from wordle_five.protocol import ( # noqa: E402
)
from wordle_five.reward import Episode, metrics, score # noqa: E402
OUT = Path(__file__).parent.parent / "public" / "traces" / "wordle"
TRACES = Path(__file__).parent.parent / "public" / "traces"
OUT = TRACES / "wordle"
ENDPOINT = os.environ.get("PIG_DEMO_INFERENCE", "http://100.127.247.67:8001/v1/chat/completions")
MODEL = os.environ.get("PIG_DEMO_MODEL", "brain-qwen38-dspark")
@@ -52,7 +65,7 @@ ARMS = {
}
def call_model(messages: list[dict], thinking: bool) -> dict:
def call_model(messages: list[dict], thinking: bool, max_tokens: int | None = None, timeout: int = 300) -> dict:
"""One completion. Returns reply, reasoning and the real call metrics.
Never raises on an upstream failure — a dropped call becomes a turn with a
@@ -64,7 +77,7 @@ def call_model(messages: list[dict], thinking: bool) -> dict:
"model": MODEL,
"messages": messages,
"temperature": 0.7,
"max_tokens": 2048 if thinking else 512,
"max_tokens": max_tokens or (2048 if thinking else 512),
"chat_template_kwargs": {"enable_thinking": bool(thinking)},
}
request = urllib.request.Request(
@@ -74,7 +87,7 @@ def call_model(messages: list[dict], thinking: bool) -> dict:
)
started = time.time()
try:
with urllib.request.urlopen(request, timeout=300) as response:
with urllib.request.urlopen(request, timeout=timeout) as response:
payload = json.loads(response.read())
except (urllib.error.URLError, TimeoutError, OSError) as exc:
return {
@@ -245,33 +258,158 @@ def capture(arm: str, seed: int) -> dict:
}
# ------------------------------------------------------------ alert-triage --
#
# The same fixture shape as wordle — runId, seed, model, capturedAt, rewards,
# metrics, truncated, outcome, turns[{reply, reasoning, call, info}] — driven
# through `alert_triage.taskset.play_episode`, the loop the probe and the
# tests share. The browser regenerates the world from the seed and replays the
# reply strings; nothing else in the fixture is trusted by the page.
AT_ARMS = {
"base-off": {"label": "Out of the box", "thinking": False, "max_tokens": 1024, "timeout": 300},
"base-on": {"label": "Allowed to think", "thinking": True, "max_tokens": 4096, "timeout": 900},
"fast": {"label": "Reads the screen", "thinking": None, "policy": "fast"},
"targeted": {"label": "Checks the hidden tells", "thinking": None, "policy": "targeted"},
"thorough": {"label": "Runs the full procedure", "thinking": None, "policy": "thorough"},
}
AT_MODEL_NAME = {"fast": "fast-analyst", "targeted": "targeted-analyst", "thorough": "thorough-analyst"}
# A thinking budget the model exhausts is recorded as finishReason "length"
# with whatever content survived (usually none, which the engine rejects).
# That is a thing the model did under the budget it was given, not a capture
# error, and the fixture says so rather than retrying until it looks better.
def _messages_from(prompt: str, transcript: list[dict]) -> list[dict]:
"""The chat a model sees: the system prompt, the screen, then each reply and what it got back."""
messages = [{"role": "system", "content": prompt}, {"role": "user", "content": transcript[0]["observation"]}]
for entry in transcript[1:]:
messages.append({"role": "assistant", "content": entry["reply"] or ""})
messages.append({"role": "user", "content": entry["observation"]})
return messages
def _generated_call() -> dict:
return {"promptTokens": None, "completionTokens": None, "reasoningTokens": None, "durationMs": None, "finishReason": "generated"}
def capture_alert_triage(arm: str, seed: int) -> dict:
config = AT_ARMS[arm]
calls: list[dict] = []
if config["thinking"] is None:
policy = AT.POLICIES[config["policy"]]
def respond(prompt: str, transcript: list[dict], view: dict) -> str | None:
reply = policy(view)
calls.append({"reply": reply, "reasoning": None, "call": _generated_call()})
return reply
else:
def respond(prompt: str, transcript: list[dict], view: dict) -> str | None:
result = call_model(_messages_from(prompt, transcript), bool(config["thinking"]),
max_tokens=config["max_tokens"], timeout=config["timeout"])
calls.append(result)
return result["reply"]
world = world_for_seed(seed)
played = AT_taskset.play_episode(seed, respond, world)
steps = played["transcript"][1:]
assert len(steps) == len(calls), "one model call per engine step"
turns = [
{
**call,
"info": {"action": step["action"], "rejection": step["rejection"], "observation": step["observation"]},
}
for call, step in zip(calls, steps)
]
return {
"runId": f"{arm}-s{seed}",
"seed": seed,
"model": AT_MODEL_NAME.get(arm, MODEL),
"capturedAt": time.strftime("%Y-%m-%d"),
"rewards": played["rewards"],
"metrics": played["metrics"],
"truncated": played["truncated"],
"outcome": played["outcome"],
"info": {**played["info"], "tier": world["tier"], "screen": played["transcript"][0]["observation"]},
"turns": turns,
}
def _at_summary(episode: dict) -> str:
r = episode["rewards"]
fmt = lambda v: " -- " if v is None else f"{v:.2f}" # noqa: E731
actions = []
for t in episode["turns"]:
a = t["info"]["action"]
if a is None:
actions.append("REJ")
elif a["action"] == "lookup":
actions.append(a.get("month") or a.get("id") or a["what"])
else:
actions.append(a["action"].upper())
return (
f"{episode['info']['tier']:<8}{episode['info']['template']:<3} {episode['outcome']:<8}"
f" caught={fmt(r['caught'])} hours={fmt(r['hours'])} evid={fmt(r['evidence'])}"
f" {episode['metrics']['hours_spent']:.2f}h {actions}"
)
TASKSETS = {
"wordle-five": {"arms": ARMS, "out": TRACES / "wordle", "capture": capture},
"alert-triage": {"arms": AT_ARMS, "out": TRACES / "alert-triage", "capture": capture_alert_triage},
}
def parse_seeds(spec: str) -> list[int]:
if "-" in spec:
lo, hi = spec.split("-")
return list(range(int(lo), int(hi) + 1))
return [int(s) for s in spec.split(",")]
""""0-7", "1,3", or a mix: "1-7,9"."""
seeds: list[int] = []
for part in spec.split(","):
if "-" in part:
lo, hi = part.split("-")
seeds.extend(range(int(lo), int(hi) + 1))
else:
seeds.append(int(part))
return seeds
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--arm", required=True, choices=sorted(ARMS))
parser.add_argument("--taskset", default="wordle-five", choices=sorted(TASKSETS))
parser.add_argument("--arm", required=True)
parser.add_argument("--seeds", default="0-7")
args = parser.parse_args()
OUT.mkdir(parents=True, exist_ok=True)
taskset = TASKSETS[args.taskset]
if args.arm not in taskset["arms"]:
parser.error(f"--arm must be one of {sorted(taskset['arms'])} for {args.taskset}")
out: Path = taskset["out"]
out.mkdir(parents=True, exist_ok=True)
for seed in parse_seeds(args.seeds):
if args.taskset == "alert-triage" and is_held_out(seed):
# The held-out bucket is never captured, probed or trained on. A
# fixture for one would put a held-out alert behind a permalink.
print(f"{args.arm}-s{seed}: seed {seed} is held out — skipped", flush=True)
continue
started = time.time()
episode = capture(args.arm, seed)
path = OUT / f"{episode['runId']}.json"
episode = taskset["capture"](args.arm, seed)
path = out / f"{episode['runId']}.json"
path.write_text(json.dumps(episode, indent=2) + "\n")
guesses = [t["info"]["guess"] for t in episode["turns"]]
print(
f"{episode['runId']:>16} {episode['answer']} {episode['outcome']:<7}"
f" solved={episode['rewards']['solved']:.0f}"
f" econ={episode['rewards']['economy']:.2f}"
f" cons={episode['rewards']['consistency']:.2f}"
f" {time.time()-started:5.1f}s {guesses}"
)
if args.taskset == "wordle-five":
guesses = [t["info"]["guess"] for t in episode["turns"]]
print(
f"{episode['runId']:>16} {episode['answer']} {episode['outcome']:<7}"
f" solved={episode['rewards']['solved']:.0f}"
f" econ={episode['rewards']['economy']:.2f}"
f" cons={episode['rewards']['consistency']:.2f}"
f" {time.time()-started:5.1f}s {guesses}",
flush=True,
)
else:
print(f"{episode['runId']:>16} {_at_summary(episode)} {time.time()-started:6.1f}s", flush=True)
return 0
envs/probe.py
+31 -4
View File
@@ -10,8 +10,9 @@ A blended float hides the failure this is built to catch: a component that is
flat across every policy is measuring nothing, and it will still move the total
because the other components move.
Run: uv run python envs/probe.py
Exits non-zero on any violated assertion.
Run: uv run python envs/probe.py [--taskset wordle-five|alert-triage]
Runs every registered taskset by default. Exits non-zero on any violated
assertion.
"""
from __future__ import annotations
@@ -20,6 +21,7 @@ import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent / "wordle_five"))
sys.path.insert(0, str(Path(__file__).parent / "alert_triage"))
from wordle_five import solver as S # noqa: E402
from wordle_five.engine import ( # noqa: E402
@@ -32,6 +34,8 @@ from wordle_five.engine import ( # noqa: E402
from wordle_five.protocol import parse_guess # noqa: E402
from wordle_five.reward import ROLES, WEIGHTS, Episode, metrics, score, total # noqa: E402
from alert_triage import ladder as alert_triage_ladder # noqa: E402
SEEDS = list(range(16))
COMPONENTS = ("solved", "economy", "consistency")
@@ -162,7 +166,7 @@ def play(policy, seed: int) -> Episode:
)
def run() -> int:
def run_wordle() -> int:
print(f"Probing wordle-five over {len(SEEDS)} seeds\n")
results: dict[str, dict[str, float]] = {}
extra: dict[str, dict[str, float]] = {}
@@ -243,5 +247,28 @@ def run() -> int:
return 0
# Each taskset registers its ladder here. The alert-triage rungs live in
# envs/alert_triage/alert_triage/ladder.py beside the policies they probe.
TASKSETS = {
"wordle-five": run_wordle,
"alert-triage": alert_triage_ladder.run,
}
def main(argv: list[str]) -> int:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--taskset", choices=sorted(TASKSETS), action="append")
args = parser.parse_args(argv)
chosen = args.taskset or list(TASKSETS)
status = 0
for i, name in enumerate(chosen):
if i:
print("\n" + "=" * 78 + "\n")
status |= TASKSETS[name]()
return status
if __name__ == "__main__":
raise SystemExit(run())
raise SystemExit(main(sys.argv[1:]))
envs/verify_fixtures.py
+45 -8
View File
@@ -16,17 +16,20 @@ import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent / "wordle_five"))
sys.path.insert(0, str(Path(__file__).parent / "alert_triage"))
from alert_triage import taskset as AT # noqa: E402
from alert_triage.generator import is_held_out # noqa: E402
from wordle_five import solver as S # noqa: E402
from wordle_five.engine import Game # noqa: E402
from wordle_five.protocol import parse_guess # noqa: E402
from wordle_five.reward import Episode, score # noqa: E402
TRACES = Path(__file__).parent.parent / "public" / "traces" / "wordle"
TRACES = Path(__file__).parent.parent / "public" / "traces"
TOLERANCE = 1e-9
def replay(fixture: dict) -> dict[str, float]:
def replay_wordle(fixture: dict) -> dict[str, float]:
game = Game(seed=fixture["seed"])
if game.answer != fixture["answer"]:
raise AssertionError(
@@ -49,27 +52,61 @@ def replay(fixture: dict) -> dict[str, float]:
return score(episode)
def replay_alert_triage(fixture: dict) -> dict[str, float | None]:
"""The world comes from the seed, the score from the reply strings. Nothing
else in the fixture is trusted — the same rule the browser's verify uses."""
seed = fixture["seed"]
if is_held_out(seed):
raise AssertionError(f"seed {seed} is in the held-out bucket and must never be captured")
played = AT.replay(seed, [turn["reply"] for turn in fixture["turns"]])
if played["outcome"] != fixture["outcome"]:
raise AssertionError(f"replay gives outcome {played['outcome']!r}, fixture claims {fixture['outcome']!r}")
return played["rewards"]
REPLAY = {"wordle": replay_wordle, "alert-triage": replay_alert_triage}
def _delta(recomputed, recorded) -> float:
"""None is "not scored". It matches only None; against a number it is a mismatch, not a zero."""
if recomputed is None or recorded is None:
return 0.0 if recomputed is recorded else float("inf")
return abs(recomputed - recorded)
def main() -> int:
fixtures = sorted(TRACES.glob("*.json"))
fixtures = [f for f in fixtures if f.name != "manifest.json"]
fixtures = [
f for demo in sorted(p for p in TRACES.iterdir() if p.is_dir())
for f in sorted(demo.glob("*.json"))
]
if not fixtures:
print("no fixtures found — nothing to verify")
return 0
failures = 0
for path in fixtures:
demo = path.parent.name
if demo not in REPLAY:
failures += 1
print(f"{path.parent.name}/{path.name:<20} NO REPLAYER for demo '{demo}'")
continue
data = json.loads(path.read_text())
recomputed = replay(data)
try:
recomputed = REPLAY[demo](data)
except AssertionError as exc:
failures += 1
print(f"{path.parent.name}/{path.name:<20} MISMATCH {exc}")
continue
recorded = data["rewards"]
deltas = {k: abs(recomputed[k] - recorded[k]) for k in recorded}
deltas = {k: _delta(recomputed[k], recorded[k]) for k in recorded}
worst = max(deltas.values())
status = "ok" if worst <= TOLERANCE else "MISMATCH"
if worst > TOLERANCE:
failures += 1
culprit = max(deltas, key=deltas.get)
print(f"{path.name:>20} {status} worst delta {worst:.3g} on '{culprit}'")
print(f"{path.parent.name}/{path.name:<20} {status} worst delta {worst:.3g} on '{culprit}'")
else:
print(f"{path.name:>20} {status} delta {worst:.1e}")
print(f"{path.parent.name}/{path.name:<20} {status} delta {worst:.1e}")
print(f"\n{len(fixtures)} fixtures, {failures} mismatched")
return 1 if failures else 0
public/og/alert-triage.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 145 KiB

public/traces/alert-triage/base-off-s1.json
+92
View File
@@ -0,0 +1,92 @@
{
"runId": "base-off-s1",
"seed": 1,
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 0.55,
"evidence": 0.2857142857142857
},
"metrics": {
"hours_spent": 1.0,
"reference_hours": 0.55,
"lookups": 1.0,
"turns_used": 2.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": null
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "benign",
"template": "B4",
"overlay": null,
"typology": null,
"planted": [
[
"kyc.expected_monthly_cash"
],
[
"kyc.expected_monthly_cash",
"kyc.business_type"
]
],
"reference_minutes": 33,
"reference_policy": "fast",
"held_out": false,
"tier": "benign",
"screen": "ALERT A-8971 rule R-STR-01 \u2014 Three or more cash deposits just under the reporting threshold within ten days\n fired 2026-04-26\n triggering transactions:\n T-213 2026-04-16 cash_in 9,440.00 USD branch BR-052\n T-217 2026-04-21 cash_in 8,950.00 USD branch BR-052\n T-219 2026-04-22 cash_in 8,880.00 USD branch BR-052\nKYC FILE C-9647 Oakhill Kitchen Group\n kyc.account_opened = 2019-07-11\n kyc.business_type = pizzeria\n kyc.country = United States\n kyc.customer_type = business\n kyc.expected_activity = Cash-intensive food service; daily takings deposited most trading days.\n kyc.expected_monthly_cash = 86,000.00 USD\n kyc.kyc_refreshed = 2026-03-13\n kyc.pep = False\n kyc.prior_alerts = 4\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 492,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 42,400 0 0 0 0 13,935 18\n 2025-06 38,160 0 0 0 0 12,582 19\n 2025-07 44,260 0 0 0 0 15,928 18\n 2025-08 41,680 0 0 0 0 13,940 18\n 2025-09 44,950 0 0 0 0 15,498 17\n 2025-10 35,860 0 0 0 0 13,284 19\n 2025-11 42,370 0 0 0 0 12,792 19\n 2025-12 39,280 0 0 0 0 13,776 19\n 2026-01 39,860 0 0 0 0 13,940 19\n 2026-02 39,280 0 0 0 0 12,718 18\n 2026-03 36,510 0 0 0 0 11,693 19\n 2026-04 71,720 0 0 0 0 14,218 18\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 out 48 txns 103,976.00 USD\n CP-2 out 12 txns 60,331.50 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "{\"action\":\"lookup\",\"what\":\"prior_alerts\"}",
"reasoning": null,
"call": {
"promptTokens": 1531,
"completionTokens": 12,
"reasoningTokens": null,
"durationMs": 1886,
"finishReason": "stop"
},
"info": {
"action": {
"action": "lookup",
"what": "prior_alerts"
},
"rejection": null,
"observation": "PRIOR ALERTS:\n A-3217 R-WIRE-04 fired 2024-07-08 closed after 7 day(s) by Beatriz Grimaldi\n Closed: destination and amount match the remittance declared at onboarding; within stated income.\n A-4028 R-DORM-05 fired 2024-10-12 closed after 9 day(s) by Mateo Jensen\n Closed: reactivation explained by a documented source of funds; document reviewed and retained.\n A-2622 R-VEL-02 fired 2025-01-17 closed after 5 day(s) by Samir Pereira\n Closed: pass-through matches the settlement terms in the agreement on file; counterparties verified.\n A-1389 R-STR-01 fired 2026-03-06 closed after 3 day(s) by Marta Eskildsen\n Closed: deposits are documented daily takings; amounts vary with trading days; within kyc.expected_monthly_cash.\n\nHours spent so far: 0.95. You have 7 replies left."
}
},
{
"reply": "{\"action\":\"close\",\"reason\":\"PREVIOUSLY_REVIEWED\",\"cites\":[\"A-1389\",\"T-213\",\"T-217\",\"T-219\",\"kyc.expected_activity\",\"kyc.expected_monthly_cash\"],\"note\":\"Same rule R-STR-01 was fired on 2026-03-06 (A-1389) and closed by Marta Eskildsen after confirming the deposits were documented daily takings varying with trading days, within expected monthly cash. The current three deposits (T-213, T-217, T-219) follow the identical pattern: cash-intensive pizzeria, amounts just under the 10k threshold consistent with splitting daily takings, all at the same branch, and April's total cash-in of 71,720 is well within the 86,000 expected monthly cash. No new counterparty, no change in activity profile. This is a repeat rule artefact on a known, documented cash-deposit pattern.\"}",
"reasoning": null,
"call": {
"promptTokens": 1827,
"completionTokens": 226,
"reasoningTokens": null,
"durationMs": 8850,
"finishReason": "stop"
},
"info": {
"action": {
"action": "close",
"reason": "PREVIOUSLY_REVIEWED",
"cites": [
"A-1389",
"T-213",
"T-217",
"T-219",
"kyc.expected_activity",
"kyc.expected_monthly_cash"
],
"note": "Same rule R-STR-01 was fired on 2026-03-06 (A-1389) and closed by Marta Eskildsen after confirming the deposits were documented daily takings varying with trading days, within expected monthly cash. The current three deposits (T-213, T-217, T-219) follow the identical pattern: cash-intensive pizzeria, amounts just under the 10k threshold consistent with splitting daily takings, all at the same branch, and April's total cash-in of 71,720 is well within the 86,000 expected monthly cash. No new counterparty, no change in activity profile. This is a repeat rule artefact on a known, documented cash-deposit pattern."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/base-off-s2.json
+145
View File
@@ -0,0 +1,145 @@
{
"runId": "base-off-s2",
"seed": 2,
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 0.34146341463414637,
"evidence": 0.4
},
"metrics": {
"hours_spent": 2.05,
"reference_hours": 0.7,
"lookups": 4.0,
"turns_used": 5.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": null
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "benign",
"template": "B6",
"overlay": null,
"typology": null,
"planted": [
[
"doc.D-aded"
]
],
"reference_minutes": 42,
"reference_policy": "fast",
"held_out": false,
"tier": "benign",
"screen": "ALERT A-9760 rule R-DORM-05 \u2014 Large credit to an account with no material activity for nine months\n fired 2026-04-23\n triggering transactions:\n T-4 2026-04-18 wire_in 356,000.00 USD CP-1\n T-5 2026-04-19 wire_out 291,900.00 USD CP-2\nKYC FILE C-7059 Gabriela Kowalczyk\n kyc.account_opened = 2012-03-09\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Retired; low activity expected.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2025-12-12\n kyc.occupation = retired\n kyc.pep = False\n kyc.prior_alerts = 2\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 31,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 0 0 0 0 0 0 0\n 2025-06 0 0 0 0 0 0 0\n 2025-07 0 170 0 0 0 0 1\n 2025-08 0 0 0 0 0 0 0\n 2025-09 0 0 0 0 0 0 0\n 2025-10 0 0 0 0 0 0 0\n 2025-11 0 0 0 0 0 0 0\n 2025-12 0 120 0 0 0 0 1\n 2026-01 0 170 0 0 0 0 1\n 2026-02 0 0 0 0 0 0 0\n 2026-03 0 0 0 0 0 0 0\n 2026-04 0 0 356,000 291,900 0 0 2\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 in 1 txns 356,000.00 USD\n CP-2 out 1 txns 291,900.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "```json\n{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-1\"}\n```",
"reasoning": null,
"call": {
"promptTokens": 1358,
"completionTokens": 22,
"reasoningTokens": null,
"durationMs": 1702,
"finishReason": "stop"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-1"
},
"rejection": null,
"observation": "COUNTERPARTY CP-1 Eskildsen & Laurent LLP, client trust account\n type law_firm_trust; jurisdiction United States; first seen 2026-04-18\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-4\n\nHours spent so far: 0.95. You have 7 replies left."
}
},
{
"reply": "```json\n{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-2\"}\n```",
"reasoning": null,
"call": {
"promptTokens": 1478,
"completionTokens": 22,
"reasoningTokens": null,
"durationMs": 1266,
"finishReason": "stop"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-2"
},
"rejection": null,
"observation": "COUNTERPARTY CP-2 Brightwater Securities\n type brokerage; jurisdiction United States; first seen 2026-04-19\n other alerted accounts sharing this counterparty in twelve months: 1\n transactions: T-5\n\nHours spent so far: 1.40. You have 6 replies left."
}
},
{
"reply": "```json\n{\"action\":\"lookup\",\"what\":\"prior_alerts\"}\n```",
"reasoning": null,
"call": {
"promptTokens": 1587,
"completionTokens": 17,
"reasoningTokens": null,
"durationMs": 509,
"finishReason": "stop"
},
"info": {
"action": {
"action": "lookup",
"what": "prior_alerts"
},
"rejection": null,
"observation": "PRIOR ALERTS:\n A-4278 R-DORM-05 fired 2024-12-23 closed after 9 day(s) by Amara Tanaka\n Closed: reactivation explained by a documented source of funds; document reviewed and retained.\n A-4841 R-WIRE-04 fired 2026-02-02 closed after 5 day(s) by Alicia Haddad\n Closed: destination and amount match the remittance declared at onboarding; within stated income.\n\nHours spent so far: 1.85. You have 5 replies left."
}
},
{
"reply": "```json\n{\"action\":\"lookup\",\"what\":\"documents\"}\n```",
"reasoning": null,
"call": {
"promptTokens": 1762,
"completionTokens": 15,
"reasoningTokens": null,
"durationMs": 553,
"finishReason": "stop"
},
"info": {
"action": {
"action": "lookup",
"what": "documents"
},
"rejection": null,
"observation": "DOCUMENTS ON FILE:\n doc.D-3211 [identity] Government-issued identity document, verified at onboarding\n doc.D-aded [executor_letter] Letter from Alicia Haddad, executor, via Eskildsen & Laurent LLP, client trust account: distribution from the estate of Aaron Eskildsen to the named beneficiary.\n\nHours spent so far: 2.00. You have 4 replies left."
}
},
{
"reply": "```json\n{\"action\":\"close\",\"reason\":\"DOCUMENTED_SOURCE_OF_FUNDS\",\"cites\":[\"doc.D-aded\",\"T-4\",\"CP-1\",\"CP-2\"],\"note\":\"The $356,000 wire-in from CP-1 (Eskildsen & Laurent LLP client trust account) is an estate distribution to a named beneficiary, documented by doc.D-aded (executor letter). The $291,900 wire-out to CP-2 (Brightwater Securities) is consistent with moving estate proceeds into an investment account. Amounts are consistent with an estate distribution and not with the customer's $31,000 stated income, but the source is documented and legitimate. Prior alert A-4278 was similarly closed for a documented source of funds.\"}\n```",
"reasoning": null,
"call": {
"promptTokens": 1885,
"completionTokens": 170,
"reasoningTokens": null,
"durationMs": 6879,
"finishReason": "stop"
},
"info": {
"action": {
"action": "close",
"reason": "DOCUMENTED_SOURCE_OF_FUNDS",
"cites": [
"doc.D-aded",
"T-4",
"CP-1",
"CP-2"
],
"note": "The $356,000 wire-in from CP-1 (Eskildsen & Laurent LLP client trust account) is an estate distribution to a named beneficiary, documented by doc.D-aded (executor letter). The $291,900 wire-out to CP-2 (Brightwater Securities) is consistent with moving estate proceeds into an investment account. Amounts are consistent with an estate distribution and not with the customer's $31,000 stated income, but the source is documented and legitimate. Prior alert A-4278 was similarly closed for a documented source of funds."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/base-off-s3.json
+91
View File
@@ -0,0 +1,91 @@
{
"runId": "base-off-s3",
"seed": 3,
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 0.55,
"evidence": 0.3333333333333333
},
"metrics": {
"hours_spent": 1.0,
"reference_hours": 0.55,
"lookups": 1.0,
"turns_used": 2.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": null
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "benign",
"template": "B4",
"overlay": null,
"typology": null,
"planted": [
[
"kyc.expected_monthly_cash"
],
[
"kyc.expected_monthly_cash",
"kyc.business_type"
]
],
"reference_minutes": 33,
"reference_policy": "fast",
"held_out": false,
"tier": "benign",
"screen": "ALERT A-9453 rule R-STR-01 \u2014 Three or more cash deposits just under the reporting threshold within ten days\n fired 2026-04-23\n triggering transactions:\n T-200 2026-04-14 cash_in 9,770.00 USD branch BR-089\n T-201 2026-04-15 cash_in 9,430.00 USD branch BR-014\n T-206 2026-04-22 cash_in 8,640.00 USD branch BR-089\nKYC FILE C-4718 Lakeside Kitchen LLC\n kyc.account_opened = 2016-10-03\n kyc.business_type = pizzeria\n kyc.country = United States\n kyc.customer_type = business\n kyc.expected_activity = Cash-intensive food service; daily takings deposited most trading days.\n kyc.expected_monthly_cash = 97,000.00 USD\n kyc.kyc_refreshed = 2025-07-13\n kyc.pep = False\n kyc.prior_alerts = 0\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 588,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 52,750 0 0 0 0 16,816 17\n 2025-06 45,240 0 0 0 0 16,317 17\n 2025-07 49,740 0 0 0 0 16,493 17\n 2025-08 40,470 0 0 0 0 16,944 17\n 2025-09 49,720 0 0 0 0 18,522 19\n 2025-10 44,630 0 0 0 0 17,199 16\n 2025-11 49,060 0 0 0 0 18,992 16\n 2025-12 52,570 0 0 0 0 17,992 19\n 2026-01 50,320 0 0 0 0 15,038 17\n 2026-02 51,710 0 0 0 0 17,826 16\n 2026-03 52,240 0 0 0 0 16,934 17\n 2026-04 78,100 0 0 0 0 18,673 18\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 out 48 txns 132,956.60 USD\n CP-2 out 12 txns 74,793.60 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "{\"action\":\"lookup\",\"what\":\"prior_alerts\"}",
"reasoning": null,
"call": {
"promptTokens": 1531,
"completionTokens": 12,
"reasoningTokens": null,
"durationMs": 2064,
"finishReason": "stop"
},
"info": {
"action": {
"action": "lookup",
"what": "prior_alerts"
},
"rejection": null,
"observation": "PRIOR ALERTS: none on file.\n\nHours spent so far: 0.95. You have 7 replies left."
}
},
{
"reply": "{\"action\":\"close\",\"reason\":\"CONSISTENT_WITH_PROFILE\",\"cites\":[\"T-200\",\"T-201\",\"T-206\",\"kyc.expected_activity\",\"kyc.expected_monthly_cash\"],\"note\":\"Lakeside Kitchen LLC is a cash-intensive pizzeria with a documented expectation of ~$97k/month in cash deposits. The three triggering deposits ($9,770, $9,430, $8,640) are each just under the $10,000 reporting threshold but are consistent with the customer's normal pattern of daily takings deposited across multiple trading days. April's total cash-in of $78,100 is within the expected range and the account has no prior alerts, no SARs, and no PEP flags. The rule fired because three sub-threshold deposits occurred within ten days, but this is a natural consequence of a high-volume cash business depositing daily takings in amounts that individually stay below the CTR threshold. No structuring indicators (e.g., sudden spike, new counterparty, unusual timing) are present.\"}",
"reasoning": null,
"call": {
"promptTokens": 1584,
"completionTokens": 233,
"reasoningTokens": null,
"durationMs": 9906,
"finishReason": "stop"
},
"info": {
"action": {
"action": "close",
"reason": "CONSISTENT_WITH_PROFILE",
"cites": [
"T-200",
"T-201",
"T-206",
"kyc.expected_activity",
"kyc.expected_monthly_cash"
],
"note": "Lakeside Kitchen LLC is a cash-intensive pizzeria with a documented expectation of ~$97k/month in cash deposits. The three triggering deposits ($9,770, $9,430, $8,640) are each just under the $10,000 reporting threshold but are consistent with the customer's normal pattern of daily takings deposited across multiple trading days. April's total cash-in of $78,100 is within the expected range and the account has no prior alerts, no SARs, and no PEP flags. The rule fired because three sub-threshold deposits occurred within ten days, but this is a natural consequence of a high-volume cash business depositing daily takings in amounts that individually stay below the CTR threshold. No structuring indicators (e.g., sudden spike, new counterparty, unusual timing) are present."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/base-off-s4.json
+177
View File
@@ -0,0 +1,177 @@
{
"runId": "base-off-s4",
"seed": 4,
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 0.22,
"evidence": 0.7058823529411765
},
"metrics": {
"hours_spent": 2.5,
"reference_hours": 0.55,
"lookups": 5.0,
"turns_used": 6.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": 1.0
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "suspicious",
"template": "S1",
"overlay": null,
"typology": "STRUCTURING",
"planted": [
[
"T-46",
"T-47",
"T-49",
"T-50",
"T-52",
"T-53",
"T-54",
"T-56"
]
],
"reference_minutes": 33,
"reference_policy": "fast",
"held_out": false,
"tier": "visible",
"screen": "ALERT A-8199 rule R-STR-01 \u2014 Three or more cash deposits just under the reporting threshold within ten days\n fired 2026-06-14\n triggering transactions:\n T-49 2026-06-05 cash_in 9,320.00 USD branch BR-027\n T-50 2026-06-06 cash_in 9,820.00 USD branch BR-031\n T-52 2026-06-07 cash_in 9,880.00 USD branch BR-068\n T-53 2026-06-09 cash_in 9,730.00 USD branch BR-027\n T-54 2026-06-10 cash_in 9,680.00 USD branch BR-031\n T-56 2026-06-11 cash_in 9,850.00 USD branch BR-068\nKYC FILE C-5480 Bilal Quintero\n kyc.account_opened = 2017-03-29\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Salaried individual; payroll deposits and routine spending.\n kyc.expected_monthly_cash = 100.00 USD\n kyc.kyc_refreshed = 2026-05-04\n kyc.occupation = delivery driver\n kyc.pep = False\n kyc.prior_alerts = 3\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 45,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-07 0 220 0 0 3,410 0 3\n 2025-08 0 480 0 0 3,382 0 3\n 2025-09 0 570 0 0 3,444 0 4\n 2025-10 0 570 0 0 3,454 0 5\n 2025-11 0 660 0 0 3,460 0 4\n 2025-12 0 550 0 0 3,403 0 3\n 2026-01 0 680 0 0 3,351 0 5\n 2026-02 0 490 0 0 3,445 0 4\n 2026-03 0 540 0 0 3,433 0 4\n 2026-04 0 430 0 0 3,387 0 4\n 2026-05 0 660 0 0 3,411 0 5\n 2026-06 77,210 420 0 66,400 3,348 0 13\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-2 out 1 txns 66,400.00 USD\n CP-1 in 12 txns 40,928.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "{\"action\":\"lookup\",\"what\":\"history\",\"month\":\"2026-06\"}",
"reasoning": null,
"call": {
"promptTokens": 1588,
"completionTokens": 20,
"reasoningTokens": null,
"durationMs": 1398,
"finishReason": "stop"
},
"info": {
"action": {
"action": "lookup",
"what": "history",
"month": "2026-06"
},
"rejection": null,
"observation": "HISTORY 2026-06: 13 transactions\n T-45 2026-06-02 ach_in 3,348.00 USD CP-1\n T-46 2026-06-02 cash_in 9,400.00 USD branch BR-031\n T-47 2026-06-03 cash_in 9,530.00 USD branch BR-068\n T-48 2026-06-04 cash_out 110.00 USD branch BR-027\n T-49 2026-06-05 cash_in 9,320.00 USD branch BR-027\n T-50 2026-06-06 cash_in 9,820.00 USD branch BR-031\n T-51 2026-06-07 cash_out 150.00 USD branch BR-027\n T-52 2026-06-07 cash_in 9,880.00 USD branch BR-068\n T-53 2026-06-09 cash_in 9,730.00 USD branch BR-027\n T-54 2026-06-10 cash_in 9,680.00 USD branch BR-031\n T-55 2026-06-11 cash_out 160.00 USD branch BR-068\n T-56 2026-06-11 cash_in 9,850.00 USD branch BR-068\n T-57 2026-06-13 wire_out 66,400.00 USD CP-2\n\nHours spent so far: 0.95. You have 7 replies left."
}
},
{
"reply": "{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-2\"}",
"reasoning": null,
"call": {
"promptTokens": 2151,
"completionTokens": 17,
"reasoningTokens": null,
"durationMs": 1282,
"finishReason": "stop"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-2"
},
"rejection": null,
"observation": "COUNTERPARTY CP-2 Redfern Holdings Co.\n type business; jurisdiction United States; first seen 2026-06-13\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-57\n\nHours spent so far: 1.40. You have 6 replies left."
}
},
{
"reply": "{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-1\"}",
"reasoning": null,
"call": {
"promptTokens": 2258,
"completionTokens": 17,
"reasoningTokens": null,
"durationMs": 677,
"finishReason": "stop"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-1"
},
"rejection": null,
"observation": "COUNTERPARTY CP-1 Stonebridge Diner Group\n type business; jurisdiction United States; first seen 2025-07-03\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-1, T-5, T-7, T-11, T-16, T-21, T-23, T-28, T-32, T-36, T-40, T-45\n\nHours spent so far: 1.85. You have 5 replies left."
}
},
{
"reply": "{\"action\":\"lookup\",\"what\":\"prior_alerts\"}",
"reasoning": null,
"call": {
"promptTokens": 2417,
"completionTokens": 12,
"reasoningTokens": null,
"durationMs": 563,
"finishReason": "stop"
},
"info": {
"action": {
"action": "lookup",
"what": "prior_alerts"
},
"rejection": null,
"observation": "PRIOR ALERTS:\n A-1684 R-STR-01 fired 2025-01-17 closed after 5 day(s) by Andre Abara\n Closed: deposits are documented daily takings; amounts vary with trading days; within kyc.expected_monthly_cash.\n A-3624 R-STR-01 fired 2025-03-06 closed after 8 day(s) by Bilal Laurent\n Closed: deposits are documented daily takings; amounts vary with trading days; within kyc.expected_monthly_cash.\n A-2017 R-CASH-03 fired 2025-05-09 closed after 7 day(s) by Sofia Grimaldi\n Closed: seasonal peak consistent with kyc.expected_activity; volume within stated expectation.\n\nHours spent so far: 2.30. You have 4 replies left."
}
},
{
"reply": "{\"action\":\"lookup\",\"what\":\"documents\"}",
"reasoning": null,
"call": {
"promptTokens": 2655,
"completionTokens": 10,
"reasoningTokens": null,
"durationMs": 640,
"finishReason": "stop"
},
"info": {
"action": {
"action": "lookup",
"what": "documents"
},
"rejection": null,
"observation": "DOCUMENTS ON FILE:\n doc.D-c608 [identity] Government-issued identity document, verified at onboarding\n\nHours spent so far: 2.45. You have 3 replies left."
}
},
{
"reply": "{\"action\":\"escalate\",\"typology\":\"STRUCTURING\",\"cites\":[\"T-49\",\"T-50\",\"T-52\",\"T-53\",\"T-54\",\"T-56\",\"kyc.expected_monthly_cash\",\"CP-2\",\"T-57\"],\"narrative\":\"Six cash deposits of 9,320\u20139,880 USD each over 06-05 to 06-11, all just under the 10,000 reporting threshold, at three rotating branches (BR-027, BR-031, BR-068). This is a stark departure from the customer's profile: a salaried delivery driver with kyc.expected_monthly_cash of 100 USD and a 12-month history of zero cash-in until June 2026, when 77,210 USD in cash appeared. On 06-13 the customer wired the full 66,400 USD to CP-2 Redfern Holdings Co., a business first seen on that same date \u2014 a rapid movement of the aggregated cash. The pattern of just-under-threshold deposits across multiple branches followed by a single outflow to a newly-seen counterparty is classic structuring/layering, not documented daily takings. Prior R-STR-01 closures (A-1684, A-3624) referenced amounts within expected monthly cash, which does not apply here given the 770x jump and the immediate wire-out. Escalating for investigation.\"}",
"reasoning": null,
"call": {
"promptTokens": 2725,
"completionTokens": 338,
"reasoningTokens": null,
"durationMs": 15080,
"finishReason": "stop"
},
"info": {
"action": {
"action": "escalate",
"typology": "STRUCTURING",
"cites": [
"T-49",
"T-50",
"T-52",
"T-53",
"T-54",
"T-56",
"kyc.expected_monthly_cash",
"CP-2",
"T-57"
],
"narrative": "Six cash deposits of 9,320\u20139,880 USD each over 06-05 to 06-11, all just under the 10,000 reporting threshold, at three rotating branches (BR-027, BR-031, BR-068). This is a stark departure from the customer's profile: a salaried delivery driver with kyc.expected_monthly_cash of 100 USD and a 12-month history of zero cash-in until June 2026, when 77,210 USD in cash appeared. On 06-13 the customer wired the full 66,400 USD to CP-2 Redfern Holdings Co., a business first seen on that same date \u2014 a rapid movement of the aggregated cash. The pattern of just-under-threshold deposits across multiple branches followed by a single outflow to a newly-seen counterparty is classic structuring/layering, not documented daily takings. Prior R-STR-01 closures (A-1684, A-3624) referenced amounts within expected monthly cash, which does not apply here given the 770x jump and the immediate wire-out. Escalating for investigation."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/base-off-s5.json
+150
View File
@@ -0,0 +1,150 @@
{
"runId": "base-off-s5",
"seed": 5,
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 0.34146341463414637,
"evidence": 0.3333333333333333
},
"metrics": {
"hours_spent": 2.05,
"reference_hours": 0.7,
"lookups": 4.0,
"turns_used": 5.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": null
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "benign",
"template": "B3",
"overlay": null,
"typology": null,
"planted": [
[
"doc.D-8578"
],
[
"doc.D-8578",
"kyc.occupation"
]
],
"reference_minutes": 42,
"reference_policy": "fast",
"held_out": false,
"tier": "benign",
"screen": "ALERT A-6081 rule R-VEL-02 \u2014 Funds received and moved out again within 48 hours\n fired 2026-03-22\n triggering transactions:\n T-52 2026-03-18 wire_in 252,000.00 USD CP-2\n T-54 2026-03-19 wire_out 219,200.00 USD CP-3\nKYC FILE C-7077 Andre Nakamura\n kyc.account_opened = 2022-01-15\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Salaried individual; payroll deposits and routine spending.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2025-04-22\n kyc.occupation = teacher\n kyc.pep = False\n kyc.prior_alerts = 3\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 106,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-04 0 590 0 0 8,866 0 4\n 2025-05 0 820 0 0 8,921 0 4\n 2025-06 0 920 0 0 8,952 0 5\n 2025-07 0 680 0 0 8,870 0 3\n 2025-08 0 1,140 0 0 8,906 0 5\n 2025-09 0 390 0 0 8,870 0 4\n 2025-10 0 850 0 0 8,922 0 4\n 2025-11 0 1,020 0 0 8,916 0 5\n 2025-12 0 950 0 0 8,901 0 5\n 2026-01 0 880 0 0 8,854 0 4\n 2026-02 0 1,170 0 0 8,916 0 5\n 2026-03 0 720 252,000 219,200 8,956 0 6\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-2 in 1 txns 252,000.00 USD\n CP-3 out 1 txns 219,200.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-2\"}",
"reasoning": null,
"call": {
"promptTokens": 1433,
"completionTokens": 17,
"reasoningTokens": null,
"durationMs": 1207,
"finishReason": "stop"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-2"
},
"rejection": null,
"observation": "COUNTERPARTY CP-2 Iwata Title & Escrow\n type title_company; jurisdiction United States; first seen 2026-03-18\n other alerted accounts sharing this counterparty in twelve months: 1\n transactions: T-52\n\nHours spent so far: 0.95. You have 7 replies left."
}
},
{
"reply": "{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-3\"}",
"reasoning": null,
"call": {
"promptTokens": 1543,
"completionTokens": 17,
"reasoningTokens": null,
"durationMs": 552,
"finishReason": "stop"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-3"
},
"rejection": null,
"observation": "COUNTERPARTY CP-3 Harbor Securities\n type brokerage; jurisdiction United States; first seen 2026-03-19\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-54\n\nHours spent so far: 1.40. You have 6 replies left."
}
},
{
"reply": "{\"action\":\"lookup\",\"what\":\"prior_alerts\"}",
"reasoning": null,
"call": {
"promptTokens": 1647,
"completionTokens": 12,
"reasoningTokens": null,
"durationMs": 540,
"finishReason": "stop"
},
"info": {
"action": {
"action": "lookup",
"what": "prior_alerts"
},
"rejection": null,
"observation": "PRIOR ALERTS:\n A-4824 R-VEL-02 fired 2025-03-19 closed after 4 day(s) by Jamal Whitlock\n Closed: pass-through matches the settlement terms in the agreement on file; counterparties verified.\n A-1410 R-WIRE-04 fired 2025-04-24 closed after 2 day(s) by Lars Farrow\n Closed: destination and amount match the remittance declared at onboarding; within stated income.\n A-2317 R-CASH-03 fired 2025-09-02 closed after 6 day(s) by Chidi Zapata\n Closed: seasonal peak consistent with kyc.expected_activity; volume within stated expectation.\n\nHours spent so far: 1.85. You have 5 replies left."
}
},
{
"reply": "{\"action\":\"lookup\",\"what\":\"documents\"}",
"reasoning": null,
"call": {
"promptTokens": 1875,
"completionTokens": 10,
"reasoningTokens": null,
"durationMs": 636,
"finishReason": "stop"
},
"info": {
"action": {
"action": "lookup",
"what": "documents"
},
"rejection": null,
"observation": "DOCUMENTS ON FILE:\n doc.D-5045 [identity] Government-issued identity document, verified at onboarding\n doc.D-8578 [closing_statement] Closing statement \u2014 sale of 266 Birch Avenue; net proceeds to seller disbursed by Iwata Title & Escrow.\n\nHours spent so far: 2.00. You have 4 replies left."
}
},
{
"reply": "{\"action\":\"close\",\"reason\":\"DOCUMENTED_SOURCE_OF_FUNDS\",\"cites\":[\"doc.D-8578\",\"CP-2\",\"CP-3\",\"T-52\",\"T-54\"],\"note\":\"Wire-in from Iwata Title & Escrow (CP-2) matches the closing statement on file (doc.D-8578) for the sale of 266 Birch Avenue; next-day wire-out to Harbor Securities (CP-3, brokerage) is consistent with investment of real-estate proceeds. Source of funds documented; amounts within the property sale context.\"}",
"reasoning": null,
"call": {
"promptTokens": 1988,
"completionTokens": 124,
"reasoningTokens": null,
"durationMs": 5392,
"finishReason": "stop"
},
"info": {
"action": {
"action": "close",
"reason": "DOCUMENTED_SOURCE_OF_FUNDS",
"cites": [
"doc.D-8578",
"CP-2",
"CP-3",
"T-52",
"T-54"
],
"note": "Wire-in from Iwata Title & Escrow (CP-2) matches the closing statement on file (doc.D-8578) for the sale of 266 Birch Avenue; next-day wire-out to Harbor Securities (CP-3, brokerage) is consistent with investment of real-estate proceeds. Source of funds documented; amounts within the property sale context."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/base-off-s6.json
+90
View File
@@ -0,0 +1,90 @@
{
"runId": "base-off-s6",
"seed": 6,
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 0.55,
"evidence": 0.6666666666666666
},
"metrics": {
"hours_spent": 1.0,
"reference_hours": 0.55,
"lookups": 1.0,
"turns_used": 2.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": null
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "benign",
"template": "B1",
"overlay": null,
"typology": null,
"planted": [
[
"kyc.expected_activity",
"kyc.expected_monthly_cash"
],
[
"kyc.expected_monthly_cash"
]
],
"reference_minutes": 33,
"reference_policy": "fast",
"held_out": false,
"tier": "benign",
"screen": "ALERT A-7014 rule R-CASH-03 \u2014 Monthly cash deposits materially above the trailing six-month average\n fired 2026-05-15\n triggering transactions:\n T-83 2026-05-02 cash_in 2,140.00 USD branch BR-102\n T-84 2026-05-03 cash_in 2,930.00 USD branch BR-102\n T-85 2026-05-04 cash_in 2,610.00 USD branch BR-031\n T-86 2026-05-05 cash_in 2,290.00 USD branch BR-031\n T-87 2026-05-06 cash_in 2,930.00 USD branch BR-031\n T-88 2026-05-08 cash_in 2,190.00 USD branch BR-031\n T-89 2026-05-09 cash_in 2,270.00 USD branch BR-031\n T-90 2026-05-10 cash_in 1,970.00 USD branch BR-102\n T-91 2026-05-11 cash_in 2,540.00 USD branch BR-031\n T-92 2026-05-12 cash_in 2,540.00 USD branch BR-031\n T-93 2026-05-13 cash_in 1,970.00 USD branch BR-102\n T-94 2026-05-14 cash_in 2,930.00 USD branch BR-031\nKYC FILE C-3047 Lakeside Landscaping Inc.\n kyc.account_opened = 2013-08-05\n kyc.business_type = landscaping services\n kyc.country = United States\n kyc.customer_type = business\n kyc.expected_activity = Seasonal business; cash receipts peak April to September.\n kyc.expected_monthly_cash = 38,000.00 USD\n kyc.kyc_refreshed = 2025-04-30\n kyc.pep = False\n kyc.prior_alerts = 2\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 266,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-06 28,130 0 0 2,900 0 7,716 10\n 2025-07 24,190 0 0 0 0 8,652 11\n 2025-08 27,600 0 0 0 0 7,538 10\n 2025-09 22,400 0 0 6,300 0 6,691 13\n 2025-10 5,480 0 0 0 0 1,686 4\n 2025-11 7,530 0 0 0 0 1,673 4\n 2025-12 6,520 0 0 3,900 0 2,119 5\n 2026-01 8,260 0 0 0 0 1,811 5\n 2026-02 5,740 0 0 0 0 1,989 4\n 2026-03 6,100 0 0 3,300 0 2,094 6\n 2026-04 24,540 0 0 0 0 5,537 10\n 2026-05 29,310 0 0 0 0 9,777 13\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 out 12 txns 57,289.16 USD\n CP-2 out 4 txns 16,400.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "Looking at this alert, the key question is whether the May cash deposits are genuinely out of line with the customer's profile.\n\nLet me note the facts I've already been shown:\n- Business: landscaping, seasonal cash peaks April\u2013September (kyc.expected_activity)\n- Expected monthly cash: 38,000 USD (kyc.expected_monthly_cash)\n- May 2026 cash in: 29,310 \u2014 below the stated expectation\n- The rule R-CASH-03 fired because May is \"materially above the trailing six-month average\" \u2014 but the trailing six months (Nov\u2013Apr) include the dead winter season, so a low base mechanically trips the rule at the start of season.\n- April 2026 already showed the seasonal uptick (24,540).\n- No wires in, all outflows to CP-1/CP-2 are ordinary business payables; no PEP, no prior SARs.\n\nThis looks like a rule artefact (seasonal peak vs. low winter base), not structuring. The individual deposits are all well under the 10,000 reporting threshold and there's no rapid movement or funneling pattern.\n\nLet me confirm there's no prior alert that already covered this seasonal pattern, which would strengthen a clean close.\n\n{\"action\":\"lookup\",\"what\":\"prior_alerts\"}",
"reasoning": null,
"call": {
"promptTokens": 1867,
"completionTokens": 295,
"reasoningTokens": null,
"durationMs": 13229,
"finishReason": "stop"
},
"info": {
"action": {
"action": "lookup",
"what": "prior_alerts"
},
"rejection": null,
"observation": "PRIOR ALERTS:\n A-4237 R-CASH-03 fired 2024-08-10 closed after 7 day(s) by Bilal Rasmussen\n Closed: seasonal peak consistent with kyc.expected_activity; volume within stated expectation.\n A-2077 R-WIRE-04 fired 2025-04-23 closed after 3 day(s) by Dana Pereira\n Closed: destination and amount match the remittance declared at onboarding; within stated income.\n\nHours spent so far: 0.95. You have 7 replies left."
}
},
{
"reply": "The prior alert A-4237 is decisive. It is the same rule (R-CASH-03), fired in the same seasonal window (August), and was closed by a prior analyst as \"seasonal peak consistent with kyc.expected_activity; volume within stated expectation.\" This exact pattern has already been reviewed and dispositioned.\n\nThe current May volume (29,310) is below the stated monthly expectation (38,000), the trigger is purely mechanical (seasonal peak vs. low winter base), there are no PEP flags, no prior SARs, and the outflows are ordinary business payables. This is a rule artefact that has been previously reviewed.\n\n{\"action\":\"close\",\"reason\":\"PREVIOUSLY_REVIEWED\",\"cites\":[\"A-4237\",\"kyc.expected_activity\",\"kyc.expected_monthly_cash\",\"T-84\"],\"note\":\"Same rule R-CASH-03, same seasonal window; prior alert A-4237 already closed as seasonal peak consistent with profile. May cash (29,310) is below stated monthly expectation (38,000). Trigger is mechanical (low winter base). No PEP, no prior SARs, outflows are ordinary business payables.\"}",
"reasoning": null,
"call": {
"promptTokens": 2318,
"completionTokens": 269,
"reasoningTokens": null,
"durationMs": 20616,
"finishReason": "stop"
},
"info": {
"action": {
"action": "close",
"reason": "PREVIOUSLY_REVIEWED",
"cites": [
"A-4237",
"kyc.expected_activity",
"kyc.expected_monthly_cash",
"T-84"
],
"note": "Same rule R-CASH-03, same seasonal window; prior alert A-4237 already closed as seasonal peak consistent with profile. May cash (29,310) is below stated monthly expectation (38,000). Trigger is mechanical (low winter base). No PEP, no prior SARs, outflows are ordinary business payables."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/base-off-s7.json
+144
View File
@@ -0,0 +1,144 @@
{
"runId": "base-off-s7",
"seed": 7,
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 0.34146341463414637,
"evidence": 0.5
},
"metrics": {
"hours_spent": 2.05,
"reference_hours": 0.7,
"lookups": 4.0,
"turns_used": 5.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": null
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "benign",
"template": "B6",
"overlay": null,
"typology": null,
"planted": [
[
"doc.D-b931"
]
],
"reference_minutes": 42,
"reference_policy": "fast",
"held_out": false,
"tier": "benign",
"screen": "ALERT A-5644 rule R-DORM-05 \u2014 Large credit to an account with no material activity for nine months\n fired 2026-06-14\n triggering transactions:\n T-4 2026-06-07 wire_in 249,000.00 USD CP-1\n T-5 2026-06-10 wire_out 134,400.00 USD CP-2\nKYC FILE C-1843 Leila Bergstrom\n kyc.account_opened = 2017-03-20\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Retired; low activity expected.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2026-01-13\n kyc.occupation = retired\n kyc.pep = False\n kyc.prior_alerts = 0\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 33,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-07 0 0 0 0 0 0 0\n 2025-08 0 0 0 0 0 0 0\n 2025-09 0 60 0 0 0 0 1\n 2025-10 0 0 0 0 0 0 0\n 2025-11 0 150 0 0 0 0 1\n 2025-12 0 0 0 0 0 0 0\n 2026-01 0 0 0 0 0 0 0\n 2026-02 0 0 0 0 0 0 0\n 2026-03 0 50 0 0 0 0 1\n 2026-04 0 0 0 0 0 0 0\n 2026-05 0 0 0 0 0 0 0\n 2026-06 0 0 249,000 134,400 0 0 2\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 in 1 txns 249,000.00 USD\n CP-2 out 1 txns 134,400.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-1\"}",
"reasoning": null,
"call": {
"promptTokens": 1354,
"completionTokens": 17,
"reasoningTokens": null,
"durationMs": 1503,
"finishReason": "stop"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-1"
},
"rejection": null,
"observation": "COUNTERPARTY CP-1 Xiang & Farrow LLP, client trust account\n type law_firm_trust; jurisdiction United States; first seen 2026-06-07\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-4\n\nHours spent so far: 0.95. You have 7 replies left."
}
},
{
"reply": "{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-2\"}",
"reasoning": null,
"call": {
"promptTokens": 1469,
"completionTokens": 17,
"reasoningTokens": null,
"durationMs": 685,
"finishReason": "stop"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-2"
},
"rejection": null,
"observation": "COUNTERPARTY CP-2 Oakhill Securities\n type brokerage; jurisdiction United States; first seen 2026-06-10\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-5\n\nHours spent so far: 1.40. You have 6 replies left."
}
},
{
"reply": "{\"action\":\"lookup\",\"what\":\"prior_alerts\"}",
"reasoning": null,
"call": {
"promptTokens": 1573,
"completionTokens": 12,
"reasoningTokens": null,
"durationMs": 5002,
"finishReason": "stop"
},
"info": {
"action": {
"action": "lookup",
"what": "prior_alerts"
},
"rejection": null,
"observation": "PRIOR ALERTS: none on file.\n\nHours spent so far: 1.85. You have 5 replies left."
}
},
{
"reply": "{\"action\":\"lookup\",\"what\":\"documents\"}",
"reasoning": null,
"call": {
"promptTokens": 1626,
"completionTokens": 10,
"reasoningTokens": null,
"durationMs": 737,
"finishReason": "stop"
},
"info": {
"action": {
"action": "lookup",
"what": "documents"
},
"rejection": null,
"observation": "DOCUMENTS ON FILE:\n doc.D-1dc2 [identity] Government-issued identity document, verified at onboarding\n doc.D-b931 [executor_letter] Letter from Hana Sandoval, executor, via Xiang & Farrow LLP, client trust account: distribution from the estate of Elena Okafor to the named beneficiary.\n\nHours spent so far: 2.00. You have 4 replies left."
}
},
{
"reply": "{\"action\":\"close\",\"reason\":\"DOCUMENTED_SOURCE_OF_FUNDS\",\"cites\":[\"doc.D-b931\",\"T-4\",\"CP-1\"],\"note\":\"The $249,000 wire-in from Xiang & Farrow LLP client trust account (CP-1) is documented as a distribution from the estate of Elena Okafor to the named beneficiary, Leila Bergstrom, per executor letter doc.D-b931. The subsequent wire-out of $134,400 to Oakhill Securities (CP-2, a US brokerage) is consistent with a retired individual moving funds into investment accounts. No PEP status, no prior alerts or SARs, and the customer profile (retired, low activity) aligns with a one-time estate distribution. Both counterparties are US-based, legitimate financial/legal entities with no shared alert history. Closing as documented source of funds.\"}",
"reasoning": null,
"call": {
"promptTokens": 1746,
"completionTokens": 189,
"reasoningTokens": null,
"durationMs": 7295,
"finishReason": "stop"
},
"info": {
"action": {
"action": "close",
"reason": "DOCUMENTED_SOURCE_OF_FUNDS",
"cites": [
"doc.D-b931",
"T-4",
"CP-1"
],
"note": "The $249,000 wire-in from Xiang & Farrow LLP client trust account (CP-1) is documented as a distribution from the estate of Elena Okafor to the named beneficiary, Leila Bergstrom, per executor letter doc.D-b931. The subsequent wire-out of $134,400 to Oakhill Securities (CP-2, a US brokerage) is consistent with a retired individual moving funds into investment accounts. No PEP status, no prior alerts or SARs, and the customer profile (retired, low activity) aligns with a one-time estate distribution. Both counterparties are US-based, legitimate financial/legal entities with no shared alert history. Closing as documented source of funds."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/base-off-s9.json
+171
View File
@@ -0,0 +1,171 @@
{
"runId": "base-off-s9",
"seed": 9,
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 0.0,
"hours": 0.0,
"evidence": 0.0
},
"metrics": {
"hours_spent": 2.5,
"reference_hours": 2.05,
"lookups": 5.0,
"turns_used": 6.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 1.0,
"typology_match": null
},
"truncated": false,
"outcome": "failed",
"info": {
"label": "suspicious",
"template": "B3",
"overlay": "funnel",
"typology": "FUNNEL_ACCOUNT",
"planted": [
[
"CP-3",
"T-52"
],
[
"T-52"
]
],
"reference_minutes": 123,
"reference_policy": "targeted",
"held_out": false,
"tier": "hidden",
"screen": "ALERT A-9537 rule R-VEL-02 \u2014 Funds received and moved out again within 48 hours\n fired 2026-04-19\n triggering transactions:\n T-51 2026-04-16 wire_in 462,000.00 USD CP-2\n T-52 2026-04-17 wire_out 448,100.00 USD CP-3\nKYC FILE C-9673 Samir Tanaka\n kyc.account_opened = 2019-12-21\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Salaried individual; payroll deposits and routine spending.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2025-04-04\n kyc.occupation = electrician\n kyc.pep = False\n kyc.prior_alerts = 4\n kyc.prior_sars = 1\n kyc.remittance_declared = none declared\n kyc.stated_income = 84,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 0 1,090 0 0 6,958 0 5\n 2025-06 0 630 0 0 7,009 0 5\n 2025-07 0 780 0 0 6,957 0 4\n 2025-08 0 470 0 0 6,989 0 3\n 2025-09 0 1,040 0 0 6,959 0 5\n 2025-10 0 740 0 0 6,996 0 4\n 2025-11 0 860 0 0 6,947 0 5\n 2025-12 0 900 0 0 6,940 0 5\n 2026-01 0 650 0 0 7,025 0 3\n 2026-02 0 460 0 0 7,026 0 3\n 2026-03 0 620 0 0 6,962 0 3\n 2026-04 0 960 462,000 448,100 6,944 0 7\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-2 in 1 txns 462,000.00 USD\n CP-3 out 1 txns 448,100.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "I need to investigate this alert. The customer is a salaried electrician earning $84,000/year (~$6,992/month) with a very consistent pattern of ACH deposits around $7,000/month. Suddenly in April 2026, a $462,000 wire came in and $448,100 went out within a day. This is dramatically inconsistent with their profile. Let me gather more information.\n\n{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-2\"}",
"reasoning": null,
"call": {
"promptTokens": 1432,
"completionTokens": 120,
"reasoningTokens": null,
"durationMs": 4843,
"finishReason": "stop"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-2"
},
"rejection": null,
"observation": "COUNTERPARTY CP-2 Okafor Title & Escrow\n type title_company; jurisdiction United States; first seen 2026-04-16\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-51\n\nHours spent so far: 0.95. You have 7 replies left."
}
},
{
"reply": "{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-3\"}",
"reasoning": null,
"call": {
"promptTokens": 1645,
"completionTokens": 17,
"reasoningTokens": null,
"durationMs": 504,
"finishReason": "stop"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-3"
},
"rejection": null,
"observation": "COUNTERPARTY CP-3 Summit Securities\n type brokerage; jurisdiction United States; first seen 2026-04-17\n other alerted accounts sharing this counterparty in twelve months: 8\n transactions: T-52\n\nHours spent so far: 1.40. You have 6 replies left."
}
},
{
"reply": "{\"action\":\"lookup\",\"what\":\"prior_alerts\"}",
"reasoning": null,
"call": {
"promptTokens": 1749,
"completionTokens": 12,
"reasoningTokens": null,
"durationMs": 514,
"finishReason": "stop"
},
"info": {
"action": {
"action": "lookup",
"what": "prior_alerts"
},
"rejection": null,
"observation": "PRIOR ALERTS:\n A-2906 R-VEL-02 fired 2025-01-09 closed after 4 day(s) by Hana Xiang\n Closed: pass-through matches the settlement terms in the agreement on file; counterparties verified.\n A-2547 R-DORM-05 fired 2025-07-01 escalated \u2014 report filed after 2 day(s) by Jamal Ferreira\n Escalated: pattern not explained by the file at the time.\n A-2855 R-DORM-05 fired 2025-07-17 closed after 8 day(s) by Carlos Zapata\n Closed: reactivation explained by a documented source of funds; document reviewed and retained.\n A-2904 R-VEL-02 fired 2025-12-05 closed after 5 day(s) by Dana Grimaldi\n Closed: pass-through matches the settlement terms in the agreement on file; counterparties verified.\n\nHours spent so far: 1.85. You have 5 replies left."
}
},
{
"reply": "{\"action\":\"lookup\",\"what\":\"documents\"}",
"reasoning": null,
"call": {
"promptTokens": 2037,
"completionTokens": 10,
"reasoningTokens": null,
"durationMs": 589,
"finishReason": "stop"
},
"info": {
"action": {
"action": "lookup",
"what": "documents"
},
"rejection": null,
"observation": "DOCUMENTS ON FILE:\n doc.D-8b75 [identity] Government-issued identity document, verified at onboarding\n doc.D-838c [closing_statement] Closing statement \u2014 sale of 82 Alder Street; net proceeds to seller disbursed by Okafor Title & Escrow.\n\nHours spent so far: 2.00. You have 4 replies left."
}
},
{
"reply": "{\"action\":\"lookup\",\"what\":\"history\",\"month\":\"2026-04\"}",
"reasoning": null,
"call": {
"promptTokens": 2150,
"completionTokens": 20,
"reasoningTokens": null,
"durationMs": 634,
"finishReason": "stop"
},
"info": {
"action": {
"action": "lookup",
"what": "history",
"month": "2026-04"
},
"rejection": null,
"observation": "HISTORY 2026-04: 7 transactions\n T-46 2026-04-04 ach_in 6,944.00 USD CP-1\n T-47 2026-04-04 cash_out 220.00 USD branch BR-068\n T-48 2026-04-06 cash_out 280.00 USD branch BR-068\n T-49 2026-04-10 cash_out 340.00 USD branch BR-068\n T-50 2026-04-12 cash_out 120.00 USD branch BR-068\n T-51 2026-04-16 wire_in 462,000.00 USD CP-2\n T-52 2026-04-17 wire_out 448,100.00 USD CP-3\n\nHours spent so far: 2.45. You have 3 replies left."
}
},
{
"reply": "Now I have a complete picture. Let me assess:\n\n**The case:**\n- Customer is a salaried electrician earning ~$7,000/month, with a 12-month pattern of consistent ACH payroll deposits and small cash withdrawals.\n- A $462,000 wire arrived from Okafor Title & Escrow (CP-2), a US title company, first seen 2026-04-16.\n- $448,100 went out the next day to Summit Securities (CP-3), a US brokerage, first seen 2026-04-17.\n- A closing statement on file (doc.D-838c) documents the sale of 82 Alder Street, with net proceeds disbursed by Okafor Title & Escrow.\n- Prior alerts A-2906 and A-2904 (both R-VEL-02, same rule) were closed as pass-throughs matching settlement terms with verified counterparties.\n- No other alerted accounts share CP-2; CP-3 (brokerage) is shared with 8 other accounts, which is typical for a major brokerage.\n\n**Assessment:** This is a documented real estate sale. The customer sold a property, received the net proceeds via a title company (standard for US real estate closings), and wired the funds to a brokerage account (likely to invest the proceeds). The source of funds is documented on file. The counterparty (title company) is a legitimate US entity with no other alerted accounts sharing it. The outflow to a brokerage is a routine investment activity.\n\nThis is consistent with a documented source of funds.\n\n{\"action\":\"close\",\"reason\":\"DOCUMENTED_SOURCE_OF_FUNDS\",\"cites\":[\"doc.D-838c\",\"T-51\",\"CP-2\",\"CP-3\",\"kyc.occupation\",\"kyc.stated_income\"],\"note\":\"The $462,000 wire-in from Okafor Title & Escrow (CP-2) is the net proceeds from the sale of 82 Alder Street, as documented in the closing statement on file (doc.D-838c). The $448,100 wire-out to Summit Securities (CP-3), a US brokerage, is consistent with investing real estate sale proceeds. The customer's profile as a salaried electrician ($84,000/yr) does not account for this amount, but the documented property sale does. CP-2 is a title company with no other alerted accounts sharing it; CP-3 is a major brokerage shared by 8 accounts, typical for a large financial institution. Prior R-VEL-02 alerts on this customer (A-2906, A-2904) were similarly closed as settlement pass-throughs. No structuring, funnel, mule, or rapid-movement typology is indicated.\"}",
"reasoning": null,
"call": {
"promptTokens": 2476,
"completionTokens": 626,
"reasoningTokens": null,
"durationMs": 38579,
"finishReason": "stop"
},
"info": {
"action": {
"action": "close",
"reason": "DOCUMENTED_SOURCE_OF_FUNDS",
"cites": [
"doc.D-838c",
"T-51",
"CP-2",
"CP-3",
"kyc.occupation",
"kyc.stated_income"
],
"note": "The $462,000 wire-in from Okafor Title & Escrow (CP-2) is the net proceeds from the sale of 82 Alder Street, as documented in the closing statement on file (doc.D-838c). The $448,100 wire-out to Summit Securities (CP-3), a US brokerage, is consistent with investing real estate sale proceeds. The customer's profile as a salaried electrician ($84,000/yr) does not account for this amount, but the documented property sale does. CP-2 is a title company with no other alerted accounts sharing it; CP-3 is a major brokerage shared by 8 accounts, typical for a large financial institution. Prior R-VEL-02 alerts on this customer (A-2906, A-2904) were similarly closed as settlement pass-throughs. No structuring, funnel, mule, or rapid-movement typology is indicated."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/base-on-s1.json
+107
View File
File diff suppressed because one or more lines are too long
public/traces/alert-triage/base-on-s2.json
+126
View File
@@ -0,0 +1,126 @@
{
"runId": "base-on-s2",
"seed": 2,
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 0.4375,
"evidence": 0.4
},
"metrics": {
"hours_spent": 1.6,
"reference_hours": 0.7,
"lookups": 3.0,
"turns_used": 4.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": null
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "benign",
"template": "B6",
"overlay": null,
"typology": null,
"planted": [
[
"doc.D-aded"
]
],
"reference_minutes": 42,
"reference_policy": "fast",
"held_out": false,
"tier": "benign",
"screen": "ALERT A-9760 rule R-DORM-05 \u2014 Large credit to an account with no material activity for nine months\n fired 2026-04-23\n triggering transactions:\n T-4 2026-04-18 wire_in 356,000.00 USD CP-1\n T-5 2026-04-19 wire_out 291,900.00 USD CP-2\nKYC FILE C-7059 Gabriela Kowalczyk\n kyc.account_opened = 2012-03-09\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Retired; low activity expected.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2025-12-12\n kyc.occupation = retired\n kyc.pep = False\n kyc.prior_alerts = 2\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 31,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 0 0 0 0 0 0 0\n 2025-06 0 0 0 0 0 0 0\n 2025-07 0 170 0 0 0 0 1\n 2025-08 0 0 0 0 0 0 0\n 2025-09 0 0 0 0 0 0 0\n 2025-10 0 0 0 0 0 0 0\n 2025-11 0 0 0 0 0 0 0\n 2025-12 0 120 0 0 0 0 1\n 2026-01 0 170 0 0 0 0 1\n 2026-02 0 0 0 0 0 0 0\n 2026-03 0 0 0 0 0 0 0\n 2026-04 0 0 356,000 291,900 0 0 2\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 in 1 txns 356,000.00 USD\n CP-2 out 1 txns 291,900.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "\n\n{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-1\"}",
"reasoning": "Let me analyze this alert carefully.\n\nAlert A-9760: R-DORM-05 \u2014 Large credit to an account with no material activity for nine months\nFired 2026-04-23\nTriggering transactions:\n- T-4: 2026-04-18 wire_in 356,000.00 USD from CP-1\n- T-5: 2026-04-19 wire_out 291,900.00 USD to CP-2\n\nKYC:\n- Gabriela Kowalczyk, individual, US\n- Account opened 2012-03-09\n- Retired, low activity expected\n- Expected monthly cash 0\n- KYC refreshed 2025-12-12\n- Stated income 31,000 USD\n- PEP false\n- Prior alerts: 2\n- Prior SARs: 0\n- Remittance declared: none\n\nTwelve-month summary:\n- Account essentially dormant (only small cash out of 170, 120, 170 in Jul, Dec, Jan)\n- April 2026: wire in 356,000, wire out 291,900\n\nSo a retired person with stated income of 31k receives a 356k wire and the next day wires out 291,900 to another party. This is a classic rapid movement / layering pattern. The account was dormant for ~9 months, then a large credit comes in and most of it moves out next day. This looks like a possible mule/funnel account or rapid movement.\n\nKey considerations:\n- Large credit inconsistent with profile (retired, low income)\n- Rapid movement: 291,900 out next day (82% of inflow)\n- CP-1 and CP-2 unknown - need to look up\n- Prior alerts = 2 - need to check if previously reviewed\n\nLet me think about what to do. I have 8 replies. Each lookup costs 0.40h except documents 0.10h.\n\nThe most decisive facts:\n1. The rapid movement (in then out next day, 82%)\n2. The counterparties - are they shared with other alerted accounts? (funnel/mule indicator)\n3. Prior alerts - were they escalated or closed?\n\nLet me look up the counterparties first. CP-1 and CP-2. The counterparty lookup tells me \"how many OTHER alerted accounts share it\" which is a strong mule/funnel indicator.\n\nActually, let me be strategic. The pattern here strongly suggests escalation (RAPID_MOVEMENT or MONEY_MULE or FUNNEL_ACCOUNT). But I should confirm with lookups to cite proper facts.\n\nLet me think about which disposition:\n- CONSISTENT_WITH_PROFILE: No, a 356k wire to a retired person with 31k income and dormant account is NOT consistent.\n- DOCUMENTED_SOURCE_OF_FUNDS: Would need documents.\n- PREVIOUSLY_REVIEWED: If prior alerts cover this.\n- RULE_ARTEFACT: If the rule fired in error.\n- Escalate: RAPID_MOVEMENT, MONEY_MULE, FUNNEL_ACCOUNT, etc.\n\nThe strongest signal: rapid movement of funds (in 356k, out 291.9k next day) to an account inconsistent with profile. This is a strong escalation candidate.\n\nBut I should check:\n1. Counterparty lookups to see if these are known/shared (mule/funnel)\n2. Prior alerts to see if this was already reviewed\n\nLet me start with counterparty lookups. Actually, let me look up both CP-1 and CP-2, and prior alerts. That's 3 lookups = 1.2h. Plus I've spent 0.5h. Total 1.7h. I have plenty of budget (8 replies * 0.05 = 0.4h base cost, but the \"hours\" seem to be a separate resource - actually the cost is in analyst-hours and each reply costs 0.05h, lookups cost 0.40h).\n\nWait, let me re-read. \"You have 8 replies. Each reply must be exactly ONE JSON object... and every reply costs 0.05 hours whether or not it is accepted.\" And lookups cost +0.40h or +0.10h.\n\nSo the constraint is 8 replies, not hours. The hours are just a cost metric. Let me count replies: I have 8. Each lookup is a reply. So I can do up to 8 lookups/actions.\n\nLet me be efficient. I'll do:\n1. Lookup counterparty CP-1\n2. Lookup counterparty CP-2\n3. Lookup prior_alerts\n\nThen decide.\n\nActually, the most decisive single fact might be the counterparty sharing. Let me look up CP-1 and CP-2 to see \"how many OTHER alerted accounts share it.\" If CP-1 or CP-2 is shared with many other alerted accounts, that's a strong mule/funnel indicator.\n\nLet me do the lookups. I'll batch my thinking but each reply is one JSON.\n\nLet me start with CP-1.\n\nActually, let me reconsider. The pattern of \"large credit to dormant account, then rapid outflow\" is the textbook RAPID_MOVEMENT typology. The counterparty info will tell me if it's a mule/funnel network.\n\nLet me look up both counterparties and prior alerts. I'll do them one at a time.\n\nReply 1: lookup counterparty CP-1\nReply 2: lookup counterparty CP-2\nReply 3: lookup prior_alerts\n\nThen I'll have enough to decide.\n\nLet me start.\n",
"call": {
"promptTokens": 1394,
"completionTokens": 1306,
"reasoningTokens": null,
"durationMs": 56324,
"finishReason": "stop"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-1"
},
"rejection": null,
"observation": "COUNTERPARTY CP-1 Eskildsen & Laurent LLP, client trust account\n type law_firm_trust; jurisdiction United States; first seen 2026-04-18\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-4\n\nHours spent so far: 0.95. You have 7 replies left."
}
},
{
"reply": "\n\n{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-2\"}",
"reasoning": "Let me analyze what I have so far.\n\nThe alert is R-DORM-05 \u2014 Large credit to an account with no material activity for nine months.\n\nKey facts:\n- Customer: Gabriela Kowalczyk, retired, low activity expected\n- Account opened 2012-03-09\n- Country: US\n- KYC refreshed 2025-12-12\n- Stated income: 31,000 USD\n- Prior alerts: 2, prior SARs: 0\n- Remittance declared: none declared\n\nTriggering transactions:\n- T-4: 2026-04-18 wire_in 356,000 USD from CP-1\n- T-5: 2026-04-19 wire_out 291,900 USD to CP-2\n\nTwelve-month summary shows the account was essentially dormant (only tiny cash out of 170, 120, 170 in a few months) until April 2026 when the big wires hit.\n\nCP-1 = Eskildsen & Laurent LLP, client trust account (law_firm_trust), US jurisdiction, first seen 2026-04-18, 0 other alerted accounts sharing it.\n\nThis is interesting. A law firm trust account sending 356,000 USD to a retired individual. This could be a legitimate settlement or inheritance/lawyer distribution. But then 291,900 went out to CP-2 the next day.\n\nLet me think about the typology. The account was dormant for 9+ months, then a large credit arrives, and the next day most of it (291,900 out of 356,000, ~82%) is wired out to another party. This is a classic rapid movement / funnel pattern.\n\nThe question is whether this is:\n1. A legitimate source of funds (law firm trust = settlement/inheritance) \u2192 could be DOCUMENTED_SOURCE_OF_FUNDS or CONSISTENT_WITH_PROFILE\n2. A mule/funnel account \u2192 RAPID_MOVEMENT, MONEY_MULE, or FUNNEL_ACCOUNT\n\nLet me consider. The customer is retired with stated income of 31,000. A 356,000 wire from a law firm trust account is a large amount but could be a settlement or estate distribution. However, the rapid outflow the next day to a different counterparty (CP-2) is a red flag for layering / rapid movement / mule activity.\n\nI need to look at CP-2 to understand where the money went. Let me also consider prior alerts (2 prior alerts on this customer).\n\nLet me look up CP-2 and prior alerts. Let me be strategic with my 7 remaining replies.\n\nActually, let me think about what decides the case. The key tension:\n- Law firm trust account as source = potentially documented/legitimate\n- But rapid movement out to CP-2 next day = layering\n\nI should look at CP-2 to see who receives the funds. If CP-2 is another law firm or a related entity, it might be legitimate. If CP-2 is a high-risk jurisdiction or unknown, it's a red flag.\n\nLet me look up CP-2.\n",
"call": {
"promptTokens": 1509,
"completionTokens": 746,
"reasoningTokens": null,
"durationMs": 23849,
"finishReason": "stop"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-2"
},
"rejection": null,
"observation": "COUNTERPARTY CP-2 Brightwater Securities\n type brokerage; jurisdiction United States; first seen 2026-04-19\n other alerted accounts sharing this counterparty in twelve months: 1\n transactions: T-5\n\nHours spent so far: 1.40. You have 6 replies left."
}
},
{
"reply": "\n\n{\"action\":\"lookup\",\"what\":\"documents\"}",
"reasoning": "Let me analyze this alert carefully.\n\n**Alert A-9760**: Rule R-DORM-05 \u2014 Large credit to an account with no material activity for nine months\n- T-4: 2026-04-18 wire_in 356,000 USD from CP-1\n- T-5: 2026-04-19 wire_out 291,900 USD to CP-2\n\n**Customer**: Gabriela Kowalczyk, C-7059\n- Account opened 2012-03-09\n- US individual, retired\n- Expected activity: low\n- Stated income: 31,000 USD\n- KYC refreshed 2025-12-12\n- 2 prior alerts, 0 prior SARs\n\n**Twelve-month summary**: Essentially dormant. Only tiny cash-outs (170, 120, 170) in 2025-07, 2025-12, 2026-01. Then in 2026-04: 356,000 wires in, 291,900 wires out.\n\n**Counterparties**:\n- CP-1: Eskildsen & Laurent LLP, client trust account, law_firm_trust, US, first seen 2026-04-18, 0 other alerted accounts share it. This is a law firm trust account \u2014 consistent with a legal settlement/inheritance/claim payout to a retired individual.\n- CP-2: Brightwater Securities, brokerage, US, first seen 2026-04-19, 1 other alerted account shares it. This is a brokerage \u2014 consistent with depositing funds into an investment account.\n\n**Analysis**:\n- The customer is a long-standing (2012) US retiree with low activity, consistent with profile.\n- The large credit of 356,000 comes from a law firm trust account (client trust) \u2014 a very common and legitimate source for large one-off funds (legal settlement, inheritance, class action payout, etc.).\n- The next day, 291,900 (about 82% of it) is wired out to a brokerage (Brightwater Securities) \u2014 consistent with a retiree moving money into investments.\n- The remaining ~64,100 stays in the account.\n- The counterparty CP-1 is a law firm trust with 0 other alerted accounts \u2014 not a shared/suspicious entity.\n- CP-2 is a brokerage with 1 other alerted account \u2014 that's mild, but a brokerage is a normal destination.\n\nThis looks like a legitimate pattern: a retired customer receives a legal payout (settlement/inheritance from a law firm trust) and immediately moves most of it into a brokerage account. This is CONSISTENT_WITH_PROFILE or DOCUMENTED_SOURCE_OF_FUNDS.\n\nThe key question: is this a rule artefact or consistent with profile? The rule fired because the account was dormant for 9 months and then received a large credit. But the source is a law firm trust account (legitimate), and the outflow is to a brokerage (legitimate). \n\nLet me think about whether I need more info. The prior alerts (2) and documents could help. Let me consider:\n- Prior alerts: knowing the disposition of the 2 prior alerts would tell me if this customer has been reviewed before (PREVIOUSLY_REVIEWED option) and what the rationale was.\n- Documents: could show a legal settlement letter or source of funds documentation (DOCUMENTED_SOURCE_OF_FUNDS).\n\nThe strongest close here would be either:\n1. CONSISTENT_WITH_PROFILE \u2014 the pattern (law firm trust \u2192 brokerage) is consistent with a retiree receiving a legal payout and investing it.\n2. DOCUMENTED_SOURCE_OF_FUNDS \u2014 if documents show the source.\n3. PREVIOUSLY_REVIEWED \u2014 if prior alerts were already reviewed and cleared.\n\nLet me check prior alerts and documents to strengthen my case. Actually, let me be strategic. I have 6 replies left. Let me look at prior alerts and documents.\n\nActually, the most decisive evidence for a clean close is documents showing the source of funds (e.g., a legal settlement agreement or inheritance document). Let me check documents first, and prior alerts.\n\nLet me look at documents (0.10) and prior_alerts (0.40).\n\nActually, let me think about which is more decisive. The transaction pattern already strongly suggests legitimacy:\n- In from law firm trust (CP-1, 0 shared alerts)\n- Out to brokerage (CP-2)\n\nThis is a classic \"legal settlement \u2192 investment\" pattern for a retiree. The KYC says retired, low activity. A large one-off legal payout is entirely consistent with a retired person's profile (e.g., they may have had a personal injury claim, inheritance, or class action).\n\nI think the best close reason is CONSISTENT_WITH_PROFILE, citing the KYC (retired, low activity), T-4 (from law firm trust), T-5 (to brokerage), and the counterparties.\n\nBut let me check documents to see if there's a documented source, which would make it a stronger DOCUMENTED_SOURCE_OF_FUNDS close. And prior alerts to see if it's PREVIOUSLY_REVIEWED.\n\nLet me do both lookups to be thorough, then decide. I have enough replies.\n\nLet me look at documents first (cheapest, 0.10) and prior alerts.\n\nActually, I'll do both in sequence since I need to see results. Let me start with documents.\n",
"call": {
"promptTokens": 1613,
"completionTokens": 1246,
"reasoningTokens": null,
"durationMs": 42157,
"finishReason": "stop"
},
"info": {
"action": {
"action": "lookup",
"what": "documents"
},
"rejection": null,
"observation": "DOCUMENTS ON FILE:\n doc.D-3211 [identity] Government-issued identity document, verified at onboarding\n doc.D-aded [executor_letter] Letter from Alicia Haddad, executor, via Eskildsen & Laurent LLP, client trust account: distribution from the estate of Aaron Eskildsen to the named beneficiary.\n\nHours spent so far: 1.55. You have 5 replies left."
}
},
{
"reply": "\n\n{\"action\":\"close\",\"reason\":\"DOCUMENTED_SOURCE_OF_FUNDS\",\"cites\":[\"doc.D-aded\",\"T-4\",\"T-5\",\"kyc.expected_activity\"],\"note\":\"Wire-in of 356K from Eskildsen & Laurent LLP client trust account is documented by executor letter (doc.D-aded) as an estate distribution to the named beneficiary. Next-day wire-out of 291.9K to Brightwater Securities (brokerage) is a normal disposition of estate funds. Customer is retired with low activity expected; the dormant period is consistent with profile. CP-1 (trust account) has zero other alerted accounts. No PEP, no prior SARs, KYC refreshed 2025-12-12.\"}",
"reasoning": "Let me analyze what I have:\n\n- Alert A-9760: Large credit to an account with no material activity for nine months\n- Customer: Gabriela Kowalczyk, retired, low activity expected\n- Triggering transactions:\n - T-4: 2026-04-18 wire_in 356,000 USD from CP-1 (Eskildsen & Laurent LLP, client trust account)\n - T-5: 2026-04-19 wire_out 291,900 USD to CP-2 (Brightwater Securities, brokerage)\n\n- CP-1: Law firm trust account, US jurisdiction, first seen 2026-04-18, 0 other alerted accounts sharing it\n- CP-2: Brokerage, US jurisdiction, first seen 2026-04-19, 1 other alerted account sharing it\n- Documents:\n - doc.D-3211: Identity document\n - doc.D-aded: Executor letter from Alicia Haddad, via Eskildsen & Laurent LLP, stating distribution from the estate of Aaron Eskildsen to the named beneficiary\n\nSo the picture is:\n- Retired customer with low activity (consistent with profile)\n- Large wire IN from a law firm trust account (estate distribution)\n- Next day, wire OUT to a brokerage account\n- There's documentation: an executor letter explaining the source of funds (estate distribution)\n- CP-1 has 0 other alerted accounts sharing it\n- CP-2 has 1 other alerted account sharing it (slightly concerning but not definitive)\n\nThe key question: Is this a legitimate estate distribution being moved to a brokerage account, or is this suspicious?\n\nArguments for closing (DOCUMENTED_SOURCE_OF_FUNDS):\n- The customer is retired with low activity expected\n- The source is a law firm trust account (estate distribution)\n- There's an executor letter documenting the source of funds\n- The outflow is to a brokerage (Brightwater Securities), which is a normal place to put estate money\n- CP-1 has no other alerted accounts\n- The amounts are consistent (356K in, 291.9K out - the difference could be fees or partial distribution)\n- KYC was refreshed recently (2025-12-12)\n- No PEP status\n- No prior SARs\n\nArguments for escalating:\n- Rapid movement (next day outflow)\n- Large amount relative to stated income (31K)\n- CP-2 has 1 other alerted account\n- Account was dormant for 9 months\n\nHowever, the rapid movement to a brokerage is actually quite normal for estate distributions - people often deposit estate money into investment accounts. The documentation clearly explains the source. The law firm trust account is a legitimate source. The executor letter is a strong document.\n\nI think this is a DOCUMENTED_SOURCE_OF_FUNDS case. The executor letter from the law firm explains the source of the 356K wire. The outflow to a brokerage is a normal disposition of estate funds. The customer profile (retired, low activity) is consistent with receiving a large lump sum after being dormant.\n\nLet me cite:\n- doc.D-aded (executor letter documenting source)\n- T-4 (the wire in from trust account)\n- T-5 (the wire out to brokerage)\n- kyc.expected_activity (retired, low activity expected)\n- kyc.occupation (retired)\n\nI'll close with DOCUMENTED_SOURCE_OF_FUNDS.\n",
"call": {
"promptTokens": 1731,
"completionTokens": 930,
"reasoningTokens": null,
"durationMs": 48857,
"finishReason": "stop"
},
"info": {
"action": {
"action": "close",
"reason": "DOCUMENTED_SOURCE_OF_FUNDS",
"cites": [
"doc.D-aded",
"T-4",
"T-5",
"kyc.expected_activity"
],
"note": "Wire-in of 356K from Eskildsen & Laurent LLP client trust account is documented by executor letter (doc.D-aded) as an estate distribution to the named beneficiary. Next-day wire-out of 291.9K to Brightwater Securities (brokerage) is a normal disposition of estate funds. Customer is retired with low activity expected; the dormant period is consistent with profile. CP-1 (trust account) has zero other alerted accounts. No PEP, no prior SARs, KYC refreshed 2025-12-12."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/fast-s1.json
+68
View File
@@ -0,0 +1,68 @@
{
"runId": "fast-s1",
"seed": 1,
"model": "fast-analyst",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 1.0,
"evidence": 1.0
},
"metrics": {
"hours_spent": 0.55,
"reference_hours": 0.55,
"lookups": 0.0,
"turns_used": 1.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": null
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "benign",
"template": "B4",
"overlay": null,
"typology": null,
"planted": [
[
"kyc.expected_monthly_cash"
],
[
"kyc.expected_monthly_cash",
"kyc.business_type"
]
],
"reference_minutes": 33,
"reference_policy": "fast",
"held_out": false,
"tier": "benign",
"screen": "ALERT A-8971 rule R-STR-01 \u2014 Three or more cash deposits just under the reporting threshold within ten days\n fired 2026-04-26\n triggering transactions:\n T-213 2026-04-16 cash_in 9,440.00 USD branch BR-052\n T-217 2026-04-21 cash_in 8,950.00 USD branch BR-052\n T-219 2026-04-22 cash_in 8,880.00 USD branch BR-052\nKYC FILE C-9647 Oakhill Kitchen Group\n kyc.account_opened = 2019-07-11\n kyc.business_type = pizzeria\n kyc.country = United States\n kyc.customer_type = business\n kyc.expected_activity = Cash-intensive food service; daily takings deposited most trading days.\n kyc.expected_monthly_cash = 86,000.00 USD\n kyc.kyc_refreshed = 2026-03-13\n kyc.pep = False\n kyc.prior_alerts = 4\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 492,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 42,400 0 0 0 0 13,935 18\n 2025-06 38,160 0 0 0 0 12,582 19\n 2025-07 44,260 0 0 0 0 15,928 18\n 2025-08 41,680 0 0 0 0 13,940 18\n 2025-09 44,950 0 0 0 0 15,498 17\n 2025-10 35,860 0 0 0 0 13,284 19\n 2025-11 42,370 0 0 0 0 12,792 19\n 2025-12 39,280 0 0 0 0 13,776 19\n 2026-01 39,860 0 0 0 0 13,940 19\n 2026-02 39,280 0 0 0 0 12,718 18\n 2026-03 36,510 0 0 0 0 11,693 19\n 2026-04 71,720 0 0 0 0 14,218 18\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 out 48 txns 103,976.00 USD\n CP-2 out 12 txns 60,331.50 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "{\"action\": \"close\", \"reason\": \"CONSISTENT_WITH_PROFILE\", \"cites\": [\"kyc.expected_monthly_cash\"], \"note\": \"Cash volume within the expectation on file; band deposits are ordinary takings.\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "close",
"reason": "CONSISTENT_WITH_PROFILE",
"cites": [
"kyc.expected_monthly_cash"
],
"note": "Cash volume within the expectation on file; band deposits are ordinary takings."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/fast-s2.json
+83
View File
@@ -0,0 +1,83 @@
{
"runId": "fast-s2",
"seed": 2,
"model": "fast-analyst",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 1.0,
"evidence": 1.0
},
"metrics": {
"hours_spent": 0.7,
"reference_hours": 0.7,
"lookups": 1.0,
"turns_used": 2.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": null
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "benign",
"template": "B6",
"overlay": null,
"typology": null,
"planted": [
[
"doc.D-aded"
]
],
"reference_minutes": 42,
"reference_policy": "fast",
"held_out": false,
"tier": "benign",
"screen": "ALERT A-9760 rule R-DORM-05 \u2014 Large credit to an account with no material activity for nine months\n fired 2026-04-23\n triggering transactions:\n T-4 2026-04-18 wire_in 356,000.00 USD CP-1\n T-5 2026-04-19 wire_out 291,900.00 USD CP-2\nKYC FILE C-7059 Gabriela Kowalczyk\n kyc.account_opened = 2012-03-09\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Retired; low activity expected.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2025-12-12\n kyc.occupation = retired\n kyc.pep = False\n kyc.prior_alerts = 2\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 31,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 0 0 0 0 0 0 0\n 2025-06 0 0 0 0 0 0 0\n 2025-07 0 170 0 0 0 0 1\n 2025-08 0 0 0 0 0 0 0\n 2025-09 0 0 0 0 0 0 0\n 2025-10 0 0 0 0 0 0 0\n 2025-11 0 0 0 0 0 0 0\n 2025-12 0 120 0 0 0 0 1\n 2026-01 0 170 0 0 0 0 1\n 2026-02 0 0 0 0 0 0 0\n 2026-03 0 0 0 0 0 0 0\n 2026-04 0 0 356,000 291,900 0 0 2\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 in 1 txns 356,000.00 USD\n CP-2 out 1 txns 291,900.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "{\"action\": \"lookup\", \"what\": \"documents\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "documents"
},
"rejection": null,
"observation": "DOCUMENTS ON FILE:\n doc.D-3211 [identity] Government-issued identity document, verified at onboarding\n doc.D-aded [executor_letter] Letter from Alicia Haddad, executor, via Eskildsen & Laurent LLP, client trust account: distribution from the estate of Aaron Eskildsen to the named beneficiary.\n\nHours spent so far: 0.65. You have 7 replies left."
}
},
{
"reply": "{\"action\": \"close\", \"reason\": \"DOCUMENTED_SOURCE_OF_FUNDS\", \"cites\": [\"doc.D-aded\"], \"note\": \"Source of funds documented: executor_letter.\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "close",
"reason": "DOCUMENTED_SOURCE_OF_FUNDS",
"cites": [
"doc.D-aded"
],
"note": "Source of funds documented: executor_letter."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/fast-s3.json
+68
View File
@@ -0,0 +1,68 @@
{
"runId": "fast-s3",
"seed": 3,
"model": "fast-analyst",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 1.0,
"evidence": 1.0
},
"metrics": {
"hours_spent": 0.55,
"reference_hours": 0.55,
"lookups": 0.0,
"turns_used": 1.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": null
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "benign",
"template": "B4",
"overlay": null,
"typology": null,
"planted": [
[
"kyc.expected_monthly_cash"
],
[
"kyc.expected_monthly_cash",
"kyc.business_type"
]
],
"reference_minutes": 33,
"reference_policy": "fast",
"held_out": false,
"tier": "benign",
"screen": "ALERT A-9453 rule R-STR-01 \u2014 Three or more cash deposits just under the reporting threshold within ten days\n fired 2026-04-23\n triggering transactions:\n T-200 2026-04-14 cash_in 9,770.00 USD branch BR-089\n T-201 2026-04-15 cash_in 9,430.00 USD branch BR-014\n T-206 2026-04-22 cash_in 8,640.00 USD branch BR-089\nKYC FILE C-4718 Lakeside Kitchen LLC\n kyc.account_opened = 2016-10-03\n kyc.business_type = pizzeria\n kyc.country = United States\n kyc.customer_type = business\n kyc.expected_activity = Cash-intensive food service; daily takings deposited most trading days.\n kyc.expected_monthly_cash = 97,000.00 USD\n kyc.kyc_refreshed = 2025-07-13\n kyc.pep = False\n kyc.prior_alerts = 0\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 588,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 52,750 0 0 0 0 16,816 17\n 2025-06 45,240 0 0 0 0 16,317 17\n 2025-07 49,740 0 0 0 0 16,493 17\n 2025-08 40,470 0 0 0 0 16,944 17\n 2025-09 49,720 0 0 0 0 18,522 19\n 2025-10 44,630 0 0 0 0 17,199 16\n 2025-11 49,060 0 0 0 0 18,992 16\n 2025-12 52,570 0 0 0 0 17,992 19\n 2026-01 50,320 0 0 0 0 15,038 17\n 2026-02 51,710 0 0 0 0 17,826 16\n 2026-03 52,240 0 0 0 0 16,934 17\n 2026-04 78,100 0 0 0 0 18,673 18\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 out 48 txns 132,956.60 USD\n CP-2 out 12 txns 74,793.60 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "{\"action\": \"close\", \"reason\": \"CONSISTENT_WITH_PROFILE\", \"cites\": [\"kyc.expected_monthly_cash\"], \"note\": \"Cash volume within the expectation on file; band deposits are ordinary takings.\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "close",
"reason": "CONSISTENT_WITH_PROFILE",
"cites": [
"kyc.expected_monthly_cash"
],
"note": "Cash volume within the expectation on file; band deposits are ordinary takings."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/fast-s4.json
+76
View File
@@ -0,0 +1,76 @@
{
"runId": "fast-s4",
"seed": 4,
"model": "fast-analyst",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 1.0,
"evidence": 0.8571428571428571
},
"metrics": {
"hours_spent": 0.55,
"reference_hours": 0.55,
"lookups": 0.0,
"turns_used": 1.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": 1.0
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "suspicious",
"template": "S1",
"overlay": null,
"typology": "STRUCTURING",
"planted": [
[
"T-46",
"T-47",
"T-49",
"T-50",
"T-52",
"T-53",
"T-54",
"T-56"
]
],
"reference_minutes": 33,
"reference_policy": "fast",
"held_out": false,
"tier": "visible",
"screen": "ALERT A-8199 rule R-STR-01 \u2014 Three or more cash deposits just under the reporting threshold within ten days\n fired 2026-06-14\n triggering transactions:\n T-49 2026-06-05 cash_in 9,320.00 USD branch BR-027\n T-50 2026-06-06 cash_in 9,820.00 USD branch BR-031\n T-52 2026-06-07 cash_in 9,880.00 USD branch BR-068\n T-53 2026-06-09 cash_in 9,730.00 USD branch BR-027\n T-54 2026-06-10 cash_in 9,680.00 USD branch BR-031\n T-56 2026-06-11 cash_in 9,850.00 USD branch BR-068\nKYC FILE C-5480 Bilal Quintero\n kyc.account_opened = 2017-03-29\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Salaried individual; payroll deposits and routine spending.\n kyc.expected_monthly_cash = 100.00 USD\n kyc.kyc_refreshed = 2026-05-04\n kyc.occupation = delivery driver\n kyc.pep = False\n kyc.prior_alerts = 3\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 45,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-07 0 220 0 0 3,410 0 3\n 2025-08 0 480 0 0 3,382 0 3\n 2025-09 0 570 0 0 3,444 0 4\n 2025-10 0 570 0 0 3,454 0 5\n 2025-11 0 660 0 0 3,460 0 4\n 2025-12 0 550 0 0 3,403 0 3\n 2026-01 0 680 0 0 3,351 0 5\n 2026-02 0 490 0 0 3,445 0 4\n 2026-03 0 540 0 0 3,433 0 4\n 2026-04 0 430 0 0 3,387 0 4\n 2026-05 0 660 0 0 3,411 0 5\n 2026-06 77,210 420 0 66,400 3,348 0 13\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-2 out 1 txns 66,400.00 USD\n CP-1 in 12 txns 40,928.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "{\"action\": \"escalate\", \"typology\": \"STRUCTURING\", \"cites\": [\"T-49\", \"T-50\", \"T-52\", \"T-53\", \"T-54\", \"T-56\"], \"narrative\": \"Repeated cash deposits just under the threshold across branches.\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "escalate",
"typology": "STRUCTURING",
"cites": [
"T-49",
"T-50",
"T-52",
"T-53",
"T-54",
"T-56"
],
"narrative": "Repeated cash deposits just under the threshold across branches."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/fast-s5.json
+87
View File
@@ -0,0 +1,87 @@
{
"runId": "fast-s5",
"seed": 5,
"model": "fast-analyst",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 1.0,
"evidence": 1.0
},
"metrics": {
"hours_spent": 0.7,
"reference_hours": 0.7,
"lookups": 1.0,
"turns_used": 2.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": null
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "benign",
"template": "B3",
"overlay": null,
"typology": null,
"planted": [
[
"doc.D-8578"
],
[
"doc.D-8578",
"kyc.occupation"
]
],
"reference_minutes": 42,
"reference_policy": "fast",
"held_out": false,
"tier": "benign",
"screen": "ALERT A-6081 rule R-VEL-02 \u2014 Funds received and moved out again within 48 hours\n fired 2026-03-22\n triggering transactions:\n T-52 2026-03-18 wire_in 252,000.00 USD CP-2\n T-54 2026-03-19 wire_out 219,200.00 USD CP-3\nKYC FILE C-7077 Andre Nakamura\n kyc.account_opened = 2022-01-15\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Salaried individual; payroll deposits and routine spending.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2025-04-22\n kyc.occupation = teacher\n kyc.pep = False\n kyc.prior_alerts = 3\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 106,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-04 0 590 0 0 8,866 0 4\n 2025-05 0 820 0 0 8,921 0 4\n 2025-06 0 920 0 0 8,952 0 5\n 2025-07 0 680 0 0 8,870 0 3\n 2025-08 0 1,140 0 0 8,906 0 5\n 2025-09 0 390 0 0 8,870 0 4\n 2025-10 0 850 0 0 8,922 0 4\n 2025-11 0 1,020 0 0 8,916 0 5\n 2025-12 0 950 0 0 8,901 0 5\n 2026-01 0 880 0 0 8,854 0 4\n 2026-02 0 1,170 0 0 8,916 0 5\n 2026-03 0 720 252,000 219,200 8,956 0 6\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-2 in 1 txns 252,000.00 USD\n CP-3 out 1 txns 219,200.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "{\"action\": \"lookup\", \"what\": \"documents\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "documents"
},
"rejection": null,
"observation": "DOCUMENTS ON FILE:\n doc.D-5045 [identity] Government-issued identity document, verified at onboarding\n doc.D-8578 [closing_statement] Closing statement \u2014 sale of 266 Birch Avenue; net proceeds to seller disbursed by Iwata Title & Escrow.\n\nHours spent so far: 0.65. You have 7 replies left."
}
},
{
"reply": "{\"action\": \"close\", \"reason\": \"DOCUMENTED_SOURCE_OF_FUNDS\", \"cites\": [\"doc.D-8578\"], \"note\": \"Source of funds documented: closing_statement.\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "close",
"reason": "DOCUMENTED_SOURCE_OF_FUNDS",
"cites": [
"doc.D-8578"
],
"note": "Source of funds documented: closing_statement."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/fast-s6.json
+69
View File
@@ -0,0 +1,69 @@
{
"runId": "fast-s6",
"seed": 6,
"model": "fast-analyst",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 1.0,
"evidence": 1.0
},
"metrics": {
"hours_spent": 0.55,
"reference_hours": 0.55,
"lookups": 0.0,
"turns_used": 1.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": null
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "benign",
"template": "B1",
"overlay": null,
"typology": null,
"planted": [
[
"kyc.expected_activity",
"kyc.expected_monthly_cash"
],
[
"kyc.expected_monthly_cash"
]
],
"reference_minutes": 33,
"reference_policy": "fast",
"held_out": false,
"tier": "benign",
"screen": "ALERT A-7014 rule R-CASH-03 \u2014 Monthly cash deposits materially above the trailing six-month average\n fired 2026-05-15\n triggering transactions:\n T-83 2026-05-02 cash_in 2,140.00 USD branch BR-102\n T-84 2026-05-03 cash_in 2,930.00 USD branch BR-102\n T-85 2026-05-04 cash_in 2,610.00 USD branch BR-031\n T-86 2026-05-05 cash_in 2,290.00 USD branch BR-031\n T-87 2026-05-06 cash_in 2,930.00 USD branch BR-031\n T-88 2026-05-08 cash_in 2,190.00 USD branch BR-031\n T-89 2026-05-09 cash_in 2,270.00 USD branch BR-031\n T-90 2026-05-10 cash_in 1,970.00 USD branch BR-102\n T-91 2026-05-11 cash_in 2,540.00 USD branch BR-031\n T-92 2026-05-12 cash_in 2,540.00 USD branch BR-031\n T-93 2026-05-13 cash_in 1,970.00 USD branch BR-102\n T-94 2026-05-14 cash_in 2,930.00 USD branch BR-031\nKYC FILE C-3047 Lakeside Landscaping Inc.\n kyc.account_opened = 2013-08-05\n kyc.business_type = landscaping services\n kyc.country = United States\n kyc.customer_type = business\n kyc.expected_activity = Seasonal business; cash receipts peak April to September.\n kyc.expected_monthly_cash = 38,000.00 USD\n kyc.kyc_refreshed = 2025-04-30\n kyc.pep = False\n kyc.prior_alerts = 2\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 266,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-06 28,130 0 0 2,900 0 7,716 10\n 2025-07 24,190 0 0 0 0 8,652 11\n 2025-08 27,600 0 0 0 0 7,538 10\n 2025-09 22,400 0 0 6,300 0 6,691 13\n 2025-10 5,480 0 0 0 0 1,686 4\n 2025-11 7,530 0 0 0 0 1,673 4\n 2025-12 6,520 0 0 3,900 0 2,119 5\n 2026-01 8,260 0 0 0 0 1,811 5\n 2026-02 5,740 0 0 0 0 1,989 4\n 2026-03 6,100 0 0 3,300 0 2,094 6\n 2026-04 24,540 0 0 0 0 5,537 10\n 2026-05 29,310 0 0 0 0 9,777 13\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 out 12 txns 57,289.16 USD\n CP-2 out 4 txns 16,400.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "{\"action\": \"close\", \"reason\": \"CONSISTENT_WITH_PROFILE\", \"cites\": [\"kyc.expected_activity\", \"kyc.expected_monthly_cash\"], \"note\": \"Seasonal peak the file anticipated; within expected monthly cash.\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "close",
"reason": "CONSISTENT_WITH_PROFILE",
"cites": [
"kyc.expected_activity",
"kyc.expected_monthly_cash"
],
"note": "Seasonal peak the file anticipated; within expected monthly cash."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/fast-s7.json
+83
View File
@@ -0,0 +1,83 @@
{
"runId": "fast-s7",
"seed": 7,
"model": "fast-analyst",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 1.0,
"evidence": 1.0
},
"metrics": {
"hours_spent": 0.7,
"reference_hours": 0.7,
"lookups": 1.0,
"turns_used": 2.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": null
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "benign",
"template": "B6",
"overlay": null,
"typology": null,
"planted": [
[
"doc.D-b931"
]
],
"reference_minutes": 42,
"reference_policy": "fast",
"held_out": false,
"tier": "benign",
"screen": "ALERT A-5644 rule R-DORM-05 \u2014 Large credit to an account with no material activity for nine months\n fired 2026-06-14\n triggering transactions:\n T-4 2026-06-07 wire_in 249,000.00 USD CP-1\n T-5 2026-06-10 wire_out 134,400.00 USD CP-2\nKYC FILE C-1843 Leila Bergstrom\n kyc.account_opened = 2017-03-20\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Retired; low activity expected.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2026-01-13\n kyc.occupation = retired\n kyc.pep = False\n kyc.prior_alerts = 0\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 33,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-07 0 0 0 0 0 0 0\n 2025-08 0 0 0 0 0 0 0\n 2025-09 0 60 0 0 0 0 1\n 2025-10 0 0 0 0 0 0 0\n 2025-11 0 150 0 0 0 0 1\n 2025-12 0 0 0 0 0 0 0\n 2026-01 0 0 0 0 0 0 0\n 2026-02 0 0 0 0 0 0 0\n 2026-03 0 50 0 0 0 0 1\n 2026-04 0 0 0 0 0 0 0\n 2026-05 0 0 0 0 0 0 0\n 2026-06 0 0 249,000 134,400 0 0 2\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 in 1 txns 249,000.00 USD\n CP-2 out 1 txns 134,400.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "{\"action\": \"lookup\", \"what\": \"documents\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "documents"
},
"rejection": null,
"observation": "DOCUMENTS ON FILE:\n doc.D-1dc2 [identity] Government-issued identity document, verified at onboarding\n doc.D-b931 [executor_letter] Letter from Hana Sandoval, executor, via Xiang & Farrow LLP, client trust account: distribution from the estate of Elena Okafor to the named beneficiary.\n\nHours spent so far: 0.65. You have 7 replies left."
}
},
{
"reply": "{\"action\": \"close\", \"reason\": \"DOCUMENTED_SOURCE_OF_FUNDS\", \"cites\": [\"doc.D-b931\"], \"note\": \"Source of funds documented: executor_letter.\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "close",
"reason": "DOCUMENTED_SOURCE_OF_FUNDS",
"cites": [
"doc.D-b931"
],
"note": "Source of funds documented: executor_letter."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/fast-s9.json
+87
View File
@@ -0,0 +1,87 @@
{
"runId": "fast-s9",
"seed": 9,
"model": "fast-analyst",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 0.0,
"hours": 0.0,
"evidence": 0.0
},
"metrics": {
"hours_spent": 0.7,
"reference_hours": 2.05,
"lookups": 1.0,
"turns_used": 2.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 1.0,
"typology_match": null
},
"truncated": false,
"outcome": "failed",
"info": {
"label": "suspicious",
"template": "B3",
"overlay": "funnel",
"typology": "FUNNEL_ACCOUNT",
"planted": [
[
"CP-3",
"T-52"
],
[
"T-52"
]
],
"reference_minutes": 123,
"reference_policy": "targeted",
"held_out": false,
"tier": "hidden",
"screen": "ALERT A-9537 rule R-VEL-02 \u2014 Funds received and moved out again within 48 hours\n fired 2026-04-19\n triggering transactions:\n T-51 2026-04-16 wire_in 462,000.00 USD CP-2\n T-52 2026-04-17 wire_out 448,100.00 USD CP-3\nKYC FILE C-9673 Samir Tanaka\n kyc.account_opened = 2019-12-21\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Salaried individual; payroll deposits and routine spending.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2025-04-04\n kyc.occupation = electrician\n kyc.pep = False\n kyc.prior_alerts = 4\n kyc.prior_sars = 1\n kyc.remittance_declared = none declared\n kyc.stated_income = 84,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 0 1,090 0 0 6,958 0 5\n 2025-06 0 630 0 0 7,009 0 5\n 2025-07 0 780 0 0 6,957 0 4\n 2025-08 0 470 0 0 6,989 0 3\n 2025-09 0 1,040 0 0 6,959 0 5\n 2025-10 0 740 0 0 6,996 0 4\n 2025-11 0 860 0 0 6,947 0 5\n 2025-12 0 900 0 0 6,940 0 5\n 2026-01 0 650 0 0 7,025 0 3\n 2026-02 0 460 0 0 7,026 0 3\n 2026-03 0 620 0 0 6,962 0 3\n 2026-04 0 960 462,000 448,100 6,944 0 7\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-2 in 1 txns 462,000.00 USD\n CP-3 out 1 txns 448,100.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "{\"action\": \"lookup\", \"what\": \"documents\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "documents"
},
"rejection": null,
"observation": "DOCUMENTS ON FILE:\n doc.D-8b75 [identity] Government-issued identity document, verified at onboarding\n doc.D-838c [closing_statement] Closing statement \u2014 sale of 82 Alder Street; net proceeds to seller disbursed by Okafor Title & Escrow.\n\nHours spent so far: 0.65. You have 7 replies left."
}
},
{
"reply": "{\"action\": \"close\", \"reason\": \"DOCUMENTED_SOURCE_OF_FUNDS\", \"cites\": [\"doc.D-838c\"], \"note\": \"Source of funds documented: closing_statement.\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "close",
"reason": "DOCUMENTED_SOURCE_OF_FUNDS",
"cites": [
"doc.D-838c"
],
"note": "Source of funds documented: closing_statement."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/targeted-s1.json
+146
View File
@@ -0,0 +1,146 @@
{
"runId": "targeted-s1",
"seed": 1,
"model": "targeted-analyst",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 0.2682926829268293,
"evidence": 1.0
},
"metrics": {
"hours_spent": 2.05,
"reference_hours": 0.55,
"lookups": 4.0,
"turns_used": 5.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": null
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "benign",
"template": "B4",
"overlay": null,
"typology": null,
"planted": [
[
"kyc.expected_monthly_cash"
],
[
"kyc.expected_monthly_cash",
"kyc.business_type"
]
],
"reference_minutes": 33,
"reference_policy": "fast",
"held_out": false,
"tier": "benign",
"screen": "ALERT A-8971 rule R-STR-01 \u2014 Three or more cash deposits just under the reporting threshold within ten days\n fired 2026-04-26\n triggering transactions:\n T-213 2026-04-16 cash_in 9,440.00 USD branch BR-052\n T-217 2026-04-21 cash_in 8,950.00 USD branch BR-052\n T-219 2026-04-22 cash_in 8,880.00 USD branch BR-052\nKYC FILE C-9647 Oakhill Kitchen Group\n kyc.account_opened = 2019-07-11\n kyc.business_type = pizzeria\n kyc.country = United States\n kyc.customer_type = business\n kyc.expected_activity = Cash-intensive food service; daily takings deposited most trading days.\n kyc.expected_monthly_cash = 86,000.00 USD\n kyc.kyc_refreshed = 2026-03-13\n kyc.pep = False\n kyc.prior_alerts = 4\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 492,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 42,400 0 0 0 0 13,935 18\n 2025-06 38,160 0 0 0 0 12,582 19\n 2025-07 44,260 0 0 0 0 15,928 18\n 2025-08 41,680 0 0 0 0 13,940 18\n 2025-09 44,950 0 0 0 0 15,498 17\n 2025-10 35,860 0 0 0 0 13,284 19\n 2025-11 42,370 0 0 0 0 12,792 19\n 2025-12 39,280 0 0 0 0 13,776 19\n 2026-01 39,860 0 0 0 0 13,940 19\n 2026-02 39,280 0 0 0 0 12,718 18\n 2026-03 36,510 0 0 0 0 11,693 19\n 2026-04 71,720 0 0 0 0 14,218 18\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 out 48 txns 103,976.00 USD\n CP-2 out 12 txns 60,331.50 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "{\"action\": \"lookup\", \"what\": \"documents\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "documents"
},
"rejection": null,
"observation": "DOCUMENTS ON FILE:\n doc.D-e0ed [registration] Business registration \u2014 Oakhill Kitchen Group\n doc.D-000c [licence] Food service licence, renewed annually\n\nHours spent so far: 0.65. You have 7 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "prior_alerts"
},
"rejection": null,
"observation": "PRIOR ALERTS:\n A-3217 R-WIRE-04 fired 2024-07-08 closed after 7 day(s) by Beatriz Grimaldi\n Closed: destination and amount match the remittance declared at onboarding; within stated income.\n A-4028 R-DORM-05 fired 2024-10-12 closed after 9 day(s) by Mateo Jensen\n Closed: reactivation explained by a documented source of funds; document reviewed and retained.\n A-2622 R-VEL-02 fired 2025-01-17 closed after 5 day(s) by Samir Pereira\n Closed: pass-through matches the settlement terms in the agreement on file; counterparties verified.\n A-1389 R-STR-01 fired 2026-03-06 closed after 3 day(s) by Marta Eskildsen\n Closed: deposits are documented daily takings; amounts vary with trading days; within kyc.expected_monthly_cash.\n\nHours spent so far: 1.10. You have 6 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-1\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-1"
},
"rejection": null,
"observation": "COUNTERPARTY CP-1 Highland Produce Co.\n type business; jurisdiction United States; first seen 2025-05-06\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-5, T-7, T-11, T-12, T-20, T-22, T-29, T-33, T-42, T-50, T-51, T-53, T-64, T-66, T-69, T-71, T-75, T-81, T-85, T-90, T-94, T-95, T-97, T-101, T-114, T-116, T-120, T-121, T-130, T-133, T-135, T-141, T-150, T-152, T-157, T-163, T-168, T-176, T-178, T-182, T-192, T-195, T-198, T-201, T-206, T-211, T-212, T-216\n\nHours spent so far: 1.55. You have 5 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-2"
},
"rejection": null,
"observation": "COUNTERPARTY CP-2 Highland Foods Co.\n type business; jurisdiction United States; first seen 2025-05-03\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-3, T-25, T-38, T-61, T-76, T-92, T-113, T-131, T-153, T-171, T-189, T-208\n\nHours spent so far: 2.00. You have 4 replies left."
}
},
{
"reply": "{\"action\": \"close\", \"reason\": \"CONSISTENT_WITH_PROFILE\", \"cites\": [\"kyc.expected_monthly_cash\"], \"note\": \"Cash volume within the expectation on file; band deposits are ordinary takings.\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "close",
"reason": "CONSISTENT_WITH_PROFILE",
"cites": [
"kyc.expected_monthly_cash"
],
"note": "Cash volume within the expectation on file; band deposits are ordinary takings."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/targeted-s2.json
+142
View File
@@ -0,0 +1,142 @@
{
"runId": "targeted-s2",
"seed": 2,
"model": "targeted-analyst",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 0.34146341463414637,
"evidence": 1.0
},
"metrics": {
"hours_spent": 2.05,
"reference_hours": 0.7,
"lookups": 4.0,
"turns_used": 5.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": null
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "benign",
"template": "B6",
"overlay": null,
"typology": null,
"planted": [
[
"doc.D-aded"
]
],
"reference_minutes": 42,
"reference_policy": "fast",
"held_out": false,
"tier": "benign",
"screen": "ALERT A-9760 rule R-DORM-05 \u2014 Large credit to an account with no material activity for nine months\n fired 2026-04-23\n triggering transactions:\n T-4 2026-04-18 wire_in 356,000.00 USD CP-1\n T-5 2026-04-19 wire_out 291,900.00 USD CP-2\nKYC FILE C-7059 Gabriela Kowalczyk\n kyc.account_opened = 2012-03-09\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Retired; low activity expected.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2025-12-12\n kyc.occupation = retired\n kyc.pep = False\n kyc.prior_alerts = 2\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 31,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 0 0 0 0 0 0 0\n 2025-06 0 0 0 0 0 0 0\n 2025-07 0 170 0 0 0 0 1\n 2025-08 0 0 0 0 0 0 0\n 2025-09 0 0 0 0 0 0 0\n 2025-10 0 0 0 0 0 0 0\n 2025-11 0 0 0 0 0 0 0\n 2025-12 0 120 0 0 0 0 1\n 2026-01 0 170 0 0 0 0 1\n 2026-02 0 0 0 0 0 0 0\n 2026-03 0 0 0 0 0 0 0\n 2026-04 0 0 356,000 291,900 0 0 2\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 in 1 txns 356,000.00 USD\n CP-2 out 1 txns 291,900.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "{\"action\": \"lookup\", \"what\": \"documents\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "documents"
},
"rejection": null,
"observation": "DOCUMENTS ON FILE:\n doc.D-3211 [identity] Government-issued identity document, verified at onboarding\n doc.D-aded [executor_letter] Letter from Alicia Haddad, executor, via Eskildsen & Laurent LLP, client trust account: distribution from the estate of Aaron Eskildsen to the named beneficiary.\n\nHours spent so far: 0.65. You have 7 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "prior_alerts"
},
"rejection": null,
"observation": "PRIOR ALERTS:\n A-4278 R-DORM-05 fired 2024-12-23 closed after 9 day(s) by Amara Tanaka\n Closed: reactivation explained by a documented source of funds; document reviewed and retained.\n A-4841 R-WIRE-04 fired 2026-02-02 closed after 5 day(s) by Alicia Haddad\n Closed: destination and amount match the remittance declared at onboarding; within stated income.\n\nHours spent so far: 1.10. You have 6 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-1\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-1"
},
"rejection": null,
"observation": "COUNTERPARTY CP-1 Eskildsen & Laurent LLP, client trust account\n type law_firm_trust; jurisdiction United States; first seen 2026-04-18\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-4\n\nHours spent so far: 1.55. You have 5 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-2"
},
"rejection": null,
"observation": "COUNTERPARTY CP-2 Brightwater Securities\n type brokerage; jurisdiction United States; first seen 2026-04-19\n other alerted accounts sharing this counterparty in twelve months: 1\n transactions: T-5\n\nHours spent so far: 2.00. You have 4 replies left."
}
},
{
"reply": "{\"action\": \"close\", \"reason\": \"DOCUMENTED_SOURCE_OF_FUNDS\", \"cites\": [\"doc.D-aded\"], \"note\": \"Source of funds documented: executor_letter.\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "close",
"reason": "DOCUMENTED_SOURCE_OF_FUNDS",
"cites": [
"doc.D-aded"
],
"note": "Source of funds documented: executor_letter."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/targeted-s3.json
+146
View File
@@ -0,0 +1,146 @@
{
"runId": "targeted-s3",
"seed": 3,
"model": "targeted-analyst",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 0.2682926829268293,
"evidence": 1.0
},
"metrics": {
"hours_spent": 2.05,
"reference_hours": 0.55,
"lookups": 4.0,
"turns_used": 5.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": null
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "benign",
"template": "B4",
"overlay": null,
"typology": null,
"planted": [
[
"kyc.expected_monthly_cash"
],
[
"kyc.expected_monthly_cash",
"kyc.business_type"
]
],
"reference_minutes": 33,
"reference_policy": "fast",
"held_out": false,
"tier": "benign",
"screen": "ALERT A-9453 rule R-STR-01 \u2014 Three or more cash deposits just under the reporting threshold within ten days\n fired 2026-04-23\n triggering transactions:\n T-200 2026-04-14 cash_in 9,770.00 USD branch BR-089\n T-201 2026-04-15 cash_in 9,430.00 USD branch BR-014\n T-206 2026-04-22 cash_in 8,640.00 USD branch BR-089\nKYC FILE C-4718 Lakeside Kitchen LLC\n kyc.account_opened = 2016-10-03\n kyc.business_type = pizzeria\n kyc.country = United States\n kyc.customer_type = business\n kyc.expected_activity = Cash-intensive food service; daily takings deposited most trading days.\n kyc.expected_monthly_cash = 97,000.00 USD\n kyc.kyc_refreshed = 2025-07-13\n kyc.pep = False\n kyc.prior_alerts = 0\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 588,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 52,750 0 0 0 0 16,816 17\n 2025-06 45,240 0 0 0 0 16,317 17\n 2025-07 49,740 0 0 0 0 16,493 17\n 2025-08 40,470 0 0 0 0 16,944 17\n 2025-09 49,720 0 0 0 0 18,522 19\n 2025-10 44,630 0 0 0 0 17,199 16\n 2025-11 49,060 0 0 0 0 18,992 16\n 2025-12 52,570 0 0 0 0 17,992 19\n 2026-01 50,320 0 0 0 0 15,038 17\n 2026-02 51,710 0 0 0 0 17,826 16\n 2026-03 52,240 0 0 0 0 16,934 17\n 2026-04 78,100 0 0 0 0 18,673 18\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 out 48 txns 132,956.60 USD\n CP-2 out 12 txns 74,793.60 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "{\"action\": \"lookup\", \"what\": \"documents\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "documents"
},
"rejection": null,
"observation": "DOCUMENTS ON FILE:\n doc.D-f244 [registration] Business registration \u2014 Lakeside Kitchen LLC\n doc.D-ee54 [licence] Food service licence, renewed annually\n\nHours spent so far: 0.65. You have 7 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "prior_alerts"
},
"rejection": null,
"observation": "PRIOR ALERTS: none on file.\n\nHours spent so far: 1.10. You have 6 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-1\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-1"
},
"rejection": null,
"observation": "COUNTERPARTY CP-1 Blue Ridge Produce LLC\n type business; jurisdiction United States; first seen 2025-05-08\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-4, T-9, T-10, T-13, T-23, T-25, T-28, T-30, T-35, T-41, T-48, T-51, T-55, T-57, T-63, T-66, T-70, T-74, T-75, T-79, T-92, T-95, T-99, T-101, T-104, T-106, T-111, T-113, T-125, T-127, T-128, T-133, T-145, T-146, T-150, T-153, T-160, T-165, T-168, T-171, T-174, T-176, T-179, T-184, T-194, T-196, T-199, T-204\n\nHours spent so far: 1.55. You have 5 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-2"
},
"rejection": null,
"observation": "COUNTERPARTY CP-2 Copperfield Foods Group\n type business; jurisdiction United States; first seen 2025-05-03\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-2, T-21, T-39, T-56, T-76, T-90, T-108, T-122, T-139, T-157, T-177, T-190\n\nHours spent so far: 2.00. You have 4 replies left."
}
},
{
"reply": "{\"action\": \"close\", \"reason\": \"CONSISTENT_WITH_PROFILE\", \"cites\": [\"kyc.expected_monthly_cash\"], \"note\": \"Cash volume within the expectation on file; band deposits are ordinary takings.\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "close",
"reason": "CONSISTENT_WITH_PROFILE",
"cites": [
"kyc.expected_monthly_cash"
],
"note": "Cash volume within the expectation on file; band deposits are ordinary takings."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/targeted-s4.json
+76
View File
@@ -0,0 +1,76 @@
{
"runId": "targeted-s4",
"seed": 4,
"model": "targeted-analyst",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 1.0,
"evidence": 0.8571428571428571
},
"metrics": {
"hours_spent": 0.55,
"reference_hours": 0.55,
"lookups": 0.0,
"turns_used": 1.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": 1.0
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "suspicious",
"template": "S1",
"overlay": null,
"typology": "STRUCTURING",
"planted": [
[
"T-46",
"T-47",
"T-49",
"T-50",
"T-52",
"T-53",
"T-54",
"T-56"
]
],
"reference_minutes": 33,
"reference_policy": "fast",
"held_out": false,
"tier": "visible",
"screen": "ALERT A-8199 rule R-STR-01 \u2014 Three or more cash deposits just under the reporting threshold within ten days\n fired 2026-06-14\n triggering transactions:\n T-49 2026-06-05 cash_in 9,320.00 USD branch BR-027\n T-50 2026-06-06 cash_in 9,820.00 USD branch BR-031\n T-52 2026-06-07 cash_in 9,880.00 USD branch BR-068\n T-53 2026-06-09 cash_in 9,730.00 USD branch BR-027\n T-54 2026-06-10 cash_in 9,680.00 USD branch BR-031\n T-56 2026-06-11 cash_in 9,850.00 USD branch BR-068\nKYC FILE C-5480 Bilal Quintero\n kyc.account_opened = 2017-03-29\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Salaried individual; payroll deposits and routine spending.\n kyc.expected_monthly_cash = 100.00 USD\n kyc.kyc_refreshed = 2026-05-04\n kyc.occupation = delivery driver\n kyc.pep = False\n kyc.prior_alerts = 3\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 45,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-07 0 220 0 0 3,410 0 3\n 2025-08 0 480 0 0 3,382 0 3\n 2025-09 0 570 0 0 3,444 0 4\n 2025-10 0 570 0 0 3,454 0 5\n 2025-11 0 660 0 0 3,460 0 4\n 2025-12 0 550 0 0 3,403 0 3\n 2026-01 0 680 0 0 3,351 0 5\n 2026-02 0 490 0 0 3,445 0 4\n 2026-03 0 540 0 0 3,433 0 4\n 2026-04 0 430 0 0 3,387 0 4\n 2026-05 0 660 0 0 3,411 0 5\n 2026-06 77,210 420 0 66,400 3,348 0 13\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-2 out 1 txns 66,400.00 USD\n CP-1 in 12 txns 40,928.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "{\"action\": \"escalate\", \"typology\": \"STRUCTURING\", \"cites\": [\"T-49\", \"T-50\", \"T-52\", \"T-53\", \"T-54\", \"T-56\"], \"narrative\": \"Repeated cash deposits just under the threshold across branches.\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "escalate",
"typology": "STRUCTURING",
"cites": [
"T-49",
"T-50",
"T-52",
"T-53",
"T-54",
"T-56"
],
"narrative": "Repeated cash deposits just under the threshold across branches."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/targeted-s5.json
+146
View File
@@ -0,0 +1,146 @@
{
"runId": "targeted-s5",
"seed": 5,
"model": "targeted-analyst",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 0.34146341463414637,
"evidence": 1.0
},
"metrics": {
"hours_spent": 2.05,
"reference_hours": 0.7,
"lookups": 4.0,
"turns_used": 5.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": null
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "benign",
"template": "B3",
"overlay": null,
"typology": null,
"planted": [
[
"doc.D-8578"
],
[
"doc.D-8578",
"kyc.occupation"
]
],
"reference_minutes": 42,
"reference_policy": "fast",
"held_out": false,
"tier": "benign",
"screen": "ALERT A-6081 rule R-VEL-02 \u2014 Funds received and moved out again within 48 hours\n fired 2026-03-22\n triggering transactions:\n T-52 2026-03-18 wire_in 252,000.00 USD CP-2\n T-54 2026-03-19 wire_out 219,200.00 USD CP-3\nKYC FILE C-7077 Andre Nakamura\n kyc.account_opened = 2022-01-15\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Salaried individual; payroll deposits and routine spending.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2025-04-22\n kyc.occupation = teacher\n kyc.pep = False\n kyc.prior_alerts = 3\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 106,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-04 0 590 0 0 8,866 0 4\n 2025-05 0 820 0 0 8,921 0 4\n 2025-06 0 920 0 0 8,952 0 5\n 2025-07 0 680 0 0 8,870 0 3\n 2025-08 0 1,140 0 0 8,906 0 5\n 2025-09 0 390 0 0 8,870 0 4\n 2025-10 0 850 0 0 8,922 0 4\n 2025-11 0 1,020 0 0 8,916 0 5\n 2025-12 0 950 0 0 8,901 0 5\n 2026-01 0 880 0 0 8,854 0 4\n 2026-02 0 1,170 0 0 8,916 0 5\n 2026-03 0 720 252,000 219,200 8,956 0 6\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-2 in 1 txns 252,000.00 USD\n CP-3 out 1 txns 219,200.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "{\"action\": \"lookup\", \"what\": \"documents\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "documents"
},
"rejection": null,
"observation": "DOCUMENTS ON FILE:\n doc.D-5045 [identity] Government-issued identity document, verified at onboarding\n doc.D-8578 [closing_statement] Closing statement \u2014 sale of 266 Birch Avenue; net proceeds to seller disbursed by Iwata Title & Escrow.\n\nHours spent so far: 0.65. You have 7 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "prior_alerts"
},
"rejection": null,
"observation": "PRIOR ALERTS:\n A-4824 R-VEL-02 fired 2025-03-19 closed after 4 day(s) by Jamal Whitlock\n Closed: pass-through matches the settlement terms in the agreement on file; counterparties verified.\n A-1410 R-WIRE-04 fired 2025-04-24 closed after 2 day(s) by Lars Farrow\n Closed: destination and amount match the remittance declared at onboarding; within stated income.\n A-2317 R-CASH-03 fired 2025-09-02 closed after 6 day(s) by Chidi Zapata\n Closed: seasonal peak consistent with kyc.expected_activity; volume within stated expectation.\n\nHours spent so far: 1.10. You have 6 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-2"
},
"rejection": null,
"observation": "COUNTERPARTY CP-2 Iwata Title & Escrow\n type title_company; jurisdiction United States; first seen 2026-03-18\n other alerted accounts sharing this counterparty in twelve months: 1\n transactions: T-52\n\nHours spent so far: 1.55. You have 5 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-3\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-3"
},
"rejection": null,
"observation": "COUNTERPARTY CP-3 Harbor Securities\n type brokerage; jurisdiction United States; first seen 2026-03-19\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-54\n\nHours spent so far: 2.00. You have 4 replies left."
}
},
{
"reply": "{\"action\": \"close\", \"reason\": \"DOCUMENTED_SOURCE_OF_FUNDS\", \"cites\": [\"doc.D-8578\"], \"note\": \"Source of funds documented: closing_statement.\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "close",
"reason": "DOCUMENTED_SOURCE_OF_FUNDS",
"cites": [
"doc.D-8578"
],
"note": "Source of funds documented: closing_statement."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/targeted-s6.json
+147
View File
@@ -0,0 +1,147 @@
{
"runId": "targeted-s6",
"seed": 6,
"model": "targeted-analyst",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 0.2682926829268293,
"evidence": 1.0
},
"metrics": {
"hours_spent": 2.05,
"reference_hours": 0.55,
"lookups": 4.0,
"turns_used": 5.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": null
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "benign",
"template": "B1",
"overlay": null,
"typology": null,
"planted": [
[
"kyc.expected_activity",
"kyc.expected_monthly_cash"
],
[
"kyc.expected_monthly_cash"
]
],
"reference_minutes": 33,
"reference_policy": "fast",
"held_out": false,
"tier": "benign",
"screen": "ALERT A-7014 rule R-CASH-03 \u2014 Monthly cash deposits materially above the trailing six-month average\n fired 2026-05-15\n triggering transactions:\n T-83 2026-05-02 cash_in 2,140.00 USD branch BR-102\n T-84 2026-05-03 cash_in 2,930.00 USD branch BR-102\n T-85 2026-05-04 cash_in 2,610.00 USD branch BR-031\n T-86 2026-05-05 cash_in 2,290.00 USD branch BR-031\n T-87 2026-05-06 cash_in 2,930.00 USD branch BR-031\n T-88 2026-05-08 cash_in 2,190.00 USD branch BR-031\n T-89 2026-05-09 cash_in 2,270.00 USD branch BR-031\n T-90 2026-05-10 cash_in 1,970.00 USD branch BR-102\n T-91 2026-05-11 cash_in 2,540.00 USD branch BR-031\n T-92 2026-05-12 cash_in 2,540.00 USD branch BR-031\n T-93 2026-05-13 cash_in 1,970.00 USD branch BR-102\n T-94 2026-05-14 cash_in 2,930.00 USD branch BR-031\nKYC FILE C-3047 Lakeside Landscaping Inc.\n kyc.account_opened = 2013-08-05\n kyc.business_type = landscaping services\n kyc.country = United States\n kyc.customer_type = business\n kyc.expected_activity = Seasonal business; cash receipts peak April to September.\n kyc.expected_monthly_cash = 38,000.00 USD\n kyc.kyc_refreshed = 2025-04-30\n kyc.pep = False\n kyc.prior_alerts = 2\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 266,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-06 28,130 0 0 2,900 0 7,716 10\n 2025-07 24,190 0 0 0 0 8,652 11\n 2025-08 27,600 0 0 0 0 7,538 10\n 2025-09 22,400 0 0 6,300 0 6,691 13\n 2025-10 5,480 0 0 0 0 1,686 4\n 2025-11 7,530 0 0 0 0 1,673 4\n 2025-12 6,520 0 0 3,900 0 2,119 5\n 2026-01 8,260 0 0 0 0 1,811 5\n 2026-02 5,740 0 0 0 0 1,989 4\n 2026-03 6,100 0 0 3,300 0 2,094 6\n 2026-04 24,540 0 0 0 0 5,537 10\n 2026-05 29,310 0 0 0 0 9,777 13\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 out 12 txns 57,289.16 USD\n CP-2 out 4 txns 16,400.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "{\"action\": \"lookup\", \"what\": \"documents\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "documents"
},
"rejection": null,
"observation": "DOCUMENTS ON FILE:\n doc.D-ab7d [registration] Business registration \u2014 Lakeside Landscaping Inc.\n doc.D-f811 [licence] Trade licence, landscaping services, renewed annually\n\nHours spent so far: 0.65. You have 7 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "prior_alerts"
},
"rejection": null,
"observation": "PRIOR ALERTS:\n A-4237 R-CASH-03 fired 2024-08-10 closed after 7 day(s) by Bilal Rasmussen\n Closed: seasonal peak consistent with kyc.expected_activity; volume within stated expectation.\n A-2077 R-WIRE-04 fired 2025-04-23 closed after 3 day(s) by Dana Pereira\n Closed: destination and amount match the remittance declared at onboarding; within stated income.\n\nHours spent so far: 1.10. You have 6 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-1\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-1"
},
"rejection": null,
"observation": "COUNTERPARTY CP-1 Lakeside Supply LLC\n type business; jurisdiction United States; first seen 2025-06-25\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-9, T-17, T-28, T-40, T-48, T-52, T-55, T-62, T-64, T-70, T-80, T-95\n\nHours spent so far: 1.55. You have 5 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-2"
},
"rejection": null,
"observation": "COUNTERPARTY CP-2 Harbor Equipment Group\n type business; jurisdiction United States; first seen 2025-06-06\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-3, T-33, T-53, T-67\n\nHours spent so far: 2.00. You have 4 replies left."
}
},
{
"reply": "{\"action\": \"close\", \"reason\": \"CONSISTENT_WITH_PROFILE\", \"cites\": [\"kyc.expected_activity\", \"kyc.expected_monthly_cash\"], \"note\": \"Seasonal peak the file anticipated; within expected monthly cash.\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "close",
"reason": "CONSISTENT_WITH_PROFILE",
"cites": [
"kyc.expected_activity",
"kyc.expected_monthly_cash"
],
"note": "Seasonal peak the file anticipated; within expected monthly cash."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/targeted-s7.json
+142
View File
@@ -0,0 +1,142 @@
{
"runId": "targeted-s7",
"seed": 7,
"model": "targeted-analyst",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 0.34146341463414637,
"evidence": 1.0
},
"metrics": {
"hours_spent": 2.05,
"reference_hours": 0.7,
"lookups": 4.0,
"turns_used": 5.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": null
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "benign",
"template": "B6",
"overlay": null,
"typology": null,
"planted": [
[
"doc.D-b931"
]
],
"reference_minutes": 42,
"reference_policy": "fast",
"held_out": false,
"tier": "benign",
"screen": "ALERT A-5644 rule R-DORM-05 \u2014 Large credit to an account with no material activity for nine months\n fired 2026-06-14\n triggering transactions:\n T-4 2026-06-07 wire_in 249,000.00 USD CP-1\n T-5 2026-06-10 wire_out 134,400.00 USD CP-2\nKYC FILE C-1843 Leila Bergstrom\n kyc.account_opened = 2017-03-20\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Retired; low activity expected.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2026-01-13\n kyc.occupation = retired\n kyc.pep = False\n kyc.prior_alerts = 0\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 33,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-07 0 0 0 0 0 0 0\n 2025-08 0 0 0 0 0 0 0\n 2025-09 0 60 0 0 0 0 1\n 2025-10 0 0 0 0 0 0 0\n 2025-11 0 150 0 0 0 0 1\n 2025-12 0 0 0 0 0 0 0\n 2026-01 0 0 0 0 0 0 0\n 2026-02 0 0 0 0 0 0 0\n 2026-03 0 50 0 0 0 0 1\n 2026-04 0 0 0 0 0 0 0\n 2026-05 0 0 0 0 0 0 0\n 2026-06 0 0 249,000 134,400 0 0 2\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 in 1 txns 249,000.00 USD\n CP-2 out 1 txns 134,400.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "{\"action\": \"lookup\", \"what\": \"documents\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "documents"
},
"rejection": null,
"observation": "DOCUMENTS ON FILE:\n doc.D-1dc2 [identity] Government-issued identity document, verified at onboarding\n doc.D-b931 [executor_letter] Letter from Hana Sandoval, executor, via Xiang & Farrow LLP, client trust account: distribution from the estate of Elena Okafor to the named beneficiary.\n\nHours spent so far: 0.65. You have 7 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "prior_alerts"
},
"rejection": null,
"observation": "PRIOR ALERTS: none on file.\n\nHours spent so far: 1.10. You have 6 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-1\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-1"
},
"rejection": null,
"observation": "COUNTERPARTY CP-1 Xiang & Farrow LLP, client trust account\n type law_firm_trust; jurisdiction United States; first seen 2026-06-07\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-4\n\nHours spent so far: 1.55. You have 5 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-2"
},
"rejection": null,
"observation": "COUNTERPARTY CP-2 Oakhill Securities\n type brokerage; jurisdiction United States; first seen 2026-06-10\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-5\n\nHours spent so far: 2.00. You have 4 replies left."
}
},
{
"reply": "{\"action\": \"close\", \"reason\": \"DOCUMENTED_SOURCE_OF_FUNDS\", \"cites\": [\"doc.D-b931\"], \"note\": \"Source of funds documented: executor_letter.\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "close",
"reason": "DOCUMENTED_SOURCE_OF_FUNDS",
"cites": [
"doc.D-b931"
],
"note": "Source of funds documented: executor_letter."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/targeted-s9.json
+147
View File
@@ -0,0 +1,147 @@
{
"runId": "targeted-s9",
"seed": 9,
"model": "targeted-analyst",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 1.0,
"evidence": 1.0
},
"metrics": {
"hours_spent": 2.05,
"reference_hours": 2.05,
"lookups": 4.0,
"turns_used": 5.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": 1.0
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "suspicious",
"template": "B3",
"overlay": "funnel",
"typology": "FUNNEL_ACCOUNT",
"planted": [
[
"CP-3",
"T-52"
],
[
"T-52"
]
],
"reference_minutes": 123,
"reference_policy": "targeted",
"held_out": false,
"tier": "hidden",
"screen": "ALERT A-9537 rule R-VEL-02 \u2014 Funds received and moved out again within 48 hours\n fired 2026-04-19\n triggering transactions:\n T-51 2026-04-16 wire_in 462,000.00 USD CP-2\n T-52 2026-04-17 wire_out 448,100.00 USD CP-3\nKYC FILE C-9673 Samir Tanaka\n kyc.account_opened = 2019-12-21\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Salaried individual; payroll deposits and routine spending.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2025-04-04\n kyc.occupation = electrician\n kyc.pep = False\n kyc.prior_alerts = 4\n kyc.prior_sars = 1\n kyc.remittance_declared = none declared\n kyc.stated_income = 84,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 0 1,090 0 0 6,958 0 5\n 2025-06 0 630 0 0 7,009 0 5\n 2025-07 0 780 0 0 6,957 0 4\n 2025-08 0 470 0 0 6,989 0 3\n 2025-09 0 1,040 0 0 6,959 0 5\n 2025-10 0 740 0 0 6,996 0 4\n 2025-11 0 860 0 0 6,947 0 5\n 2025-12 0 900 0 0 6,940 0 5\n 2026-01 0 650 0 0 7,025 0 3\n 2026-02 0 460 0 0 7,026 0 3\n 2026-03 0 620 0 0 6,962 0 3\n 2026-04 0 960 462,000 448,100 6,944 0 7\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-2 in 1 txns 462,000.00 USD\n CP-3 out 1 txns 448,100.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "{\"action\": \"lookup\", \"what\": \"documents\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "documents"
},
"rejection": null,
"observation": "DOCUMENTS ON FILE:\n doc.D-8b75 [identity] Government-issued identity document, verified at onboarding\n doc.D-838c [closing_statement] Closing statement \u2014 sale of 82 Alder Street; net proceeds to seller disbursed by Okafor Title & Escrow.\n\nHours spent so far: 0.65. You have 7 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "prior_alerts"
},
"rejection": null,
"observation": "PRIOR ALERTS:\n A-2906 R-VEL-02 fired 2025-01-09 closed after 4 day(s) by Hana Xiang\n Closed: pass-through matches the settlement terms in the agreement on file; counterparties verified.\n A-2547 R-DORM-05 fired 2025-07-01 escalated \u2014 report filed after 2 day(s) by Jamal Ferreira\n Escalated: pattern not explained by the file at the time.\n A-2855 R-DORM-05 fired 2025-07-17 closed after 8 day(s) by Carlos Zapata\n Closed: reactivation explained by a documented source of funds; document reviewed and retained.\n A-2904 R-VEL-02 fired 2025-12-05 closed after 5 day(s) by Dana Grimaldi\n Closed: pass-through matches the settlement terms in the agreement on file; counterparties verified.\n\nHours spent so far: 1.10. You have 6 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-2"
},
"rejection": null,
"observation": "COUNTERPARTY CP-2 Okafor Title & Escrow\n type title_company; jurisdiction United States; first seen 2026-04-16\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-51\n\nHours spent so far: 1.55. You have 5 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-3\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-3"
},
"rejection": null,
"observation": "COUNTERPARTY CP-3 Summit Securities\n type brokerage; jurisdiction United States; first seen 2026-04-17\n other alerted accounts sharing this counterparty in twelve months: 8\n transactions: T-52\n\nHours spent so far: 2.00. You have 4 replies left."
}
},
{
"reply": "{\"action\": \"escalate\", \"typology\": \"FUNNEL_ACCOUNT\", \"cites\": [\"CP-3\", \"T-52\"], \"narrative\": \"CP-3 is shared with 8 other alerted accounts; funds converge on it.\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "escalate",
"typology": "FUNNEL_ACCOUNT",
"cites": [
"CP-3",
"T-52"
],
"narrative": "CP-3 is shared with 8 other alerted accounts; funds converge on it."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/thorough-s1.json
+186
View File
@@ -0,0 +1,186 @@
{
"runId": "thorough-s1",
"seed": 1,
"model": "thorough-analyst",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 0.1864406779661017,
"evidence": 1.0
},
"metrics": {
"hours_spent": 2.95,
"reference_hours": 0.55,
"lookups": 6.0,
"turns_used": 7.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": null
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "benign",
"template": "B4",
"overlay": null,
"typology": null,
"planted": [
[
"kyc.expected_monthly_cash"
],
[
"kyc.expected_monthly_cash",
"kyc.business_type"
]
],
"reference_minutes": 33,
"reference_policy": "fast",
"held_out": false,
"tier": "benign",
"screen": "ALERT A-8971 rule R-STR-01 \u2014 Three or more cash deposits just under the reporting threshold within ten days\n fired 2026-04-26\n triggering transactions:\n T-213 2026-04-16 cash_in 9,440.00 USD branch BR-052\n T-217 2026-04-21 cash_in 8,950.00 USD branch BR-052\n T-219 2026-04-22 cash_in 8,880.00 USD branch BR-052\nKYC FILE C-9647 Oakhill Kitchen Group\n kyc.account_opened = 2019-07-11\n kyc.business_type = pizzeria\n kyc.country = United States\n kyc.customer_type = business\n kyc.expected_activity = Cash-intensive food service; daily takings deposited most trading days.\n kyc.expected_monthly_cash = 86,000.00 USD\n kyc.kyc_refreshed = 2026-03-13\n kyc.pep = False\n kyc.prior_alerts = 4\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 492,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 42,400 0 0 0 0 13,935 18\n 2025-06 38,160 0 0 0 0 12,582 19\n 2025-07 44,260 0 0 0 0 15,928 18\n 2025-08 41,680 0 0 0 0 13,940 18\n 2025-09 44,950 0 0 0 0 15,498 17\n 2025-10 35,860 0 0 0 0 13,284 19\n 2025-11 42,370 0 0 0 0 12,792 19\n 2025-12 39,280 0 0 0 0 13,776 19\n 2026-01 39,860 0 0 0 0 13,940 19\n 2026-02 39,280 0 0 0 0 12,718 18\n 2026-03 36,510 0 0 0 0 11,693 19\n 2026-04 71,720 0 0 0 0 14,218 18\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 out 48 txns 103,976.00 USD\n CP-2 out 12 txns 60,331.50 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "{\"action\": \"lookup\", \"what\": \"documents\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "documents"
},
"rejection": null,
"observation": "DOCUMENTS ON FILE:\n doc.D-e0ed [registration] Business registration \u2014 Oakhill Kitchen Group\n doc.D-000c [licence] Food service licence, renewed annually\n\nHours spent so far: 0.65. You have 7 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "prior_alerts"
},
"rejection": null,
"observation": "PRIOR ALERTS:\n A-3217 R-WIRE-04 fired 2024-07-08 closed after 7 day(s) by Beatriz Grimaldi\n Closed: destination and amount match the remittance declared at onboarding; within stated income.\n A-4028 R-DORM-05 fired 2024-10-12 closed after 9 day(s) by Mateo Jensen\n Closed: reactivation explained by a documented source of funds; document reviewed and retained.\n A-2622 R-VEL-02 fired 2025-01-17 closed after 5 day(s) by Samir Pereira\n Closed: pass-through matches the settlement terms in the agreement on file; counterparties verified.\n A-1389 R-STR-01 fired 2026-03-06 closed after 3 day(s) by Marta Eskildsen\n Closed: deposits are documented daily takings; amounts vary with trading days; within kyc.expected_monthly_cash.\n\nHours spent so far: 1.10. You have 6 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-1\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-1"
},
"rejection": null,
"observation": "COUNTERPARTY CP-1 Highland Produce Co.\n type business; jurisdiction United States; first seen 2025-05-06\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-5, T-7, T-11, T-12, T-20, T-22, T-29, T-33, T-42, T-50, T-51, T-53, T-64, T-66, T-69, T-71, T-75, T-81, T-85, T-90, T-94, T-95, T-97, T-101, T-114, T-116, T-120, T-121, T-130, T-133, T-135, T-141, T-150, T-152, T-157, T-163, T-168, T-176, T-178, T-182, T-192, T-195, T-198, T-201, T-206, T-211, T-212, T-216\n\nHours spent so far: 1.55. You have 5 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-2"
},
"rejection": null,
"observation": "COUNTERPARTY CP-2 Highland Foods Co.\n type business; jurisdiction United States; first seen 2025-05-03\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-3, T-25, T-38, T-61, T-76, T-92, T-113, T-131, T-153, T-171, T-189, T-208\n\nHours spent so far: 2.00. You have 4 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-04\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "history",
"month": "2026-04"
},
"rejection": null,
"observation": "HISTORY 2026-04: 18 transactions\n T-204 2026-04-01 cash_in 3,380.00 USD branch BR-052\n T-205 2026-04-04 cash_in 3,420.00 USD branch BR-052\n T-206 2026-04-07 ach_out 2,091.00 USD CP-1\n T-207 2026-04-08 cash_in 4,850.00 USD branch BR-052\n T-208 2026-04-10 ach_out 5,436.60 USD CP-2\n T-209 2026-04-11 cash_in 5,220.00 USD branch BR-052\n T-210 2026-04-13 cash_in 4,850.00 USD branch BR-052\n T-211 2026-04-13 ach_out 2,509.20 USD CP-1\n T-212 2026-04-14 ach_out 1,672.80 USD CP-1\n T-213 2026-04-16 cash_in 9,440.00 USD branch BR-052\n T-214 2026-04-16 cash_in 5,010.00 USD branch BR-052\n T-215 2026-04-19 cash_in 3,970.00 USD branch BR-052\n T-216 2026-04-19 ach_out 2,509.20 USD CP-1\n T-217 2026-04-21 cash_in 8,950.00 USD branch BR-052\n T-218 2026-04-21 cash_in 4,640.00 USD branch BR-052\n T-219 2026-04-22 cash_in 8,880.00 USD branch BR-052\n T-220 2026-04-24 cash_in 4,720.00 USD branch BR-052\n T-221 2026-04-25 cash_in 4,390.00 USD branch BR-014\n\nHours spent so far: 2.45. You have 3 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-03\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "history",
"month": "2026-03"
},
"rejection": null,
"observation": "HISTORY 2026-03: 19 transactions\n T-185 2026-03-01 cash_in 2,020.00 USD branch BR-052\n T-186 2026-03-02 cash_in 2,850.00 USD branch BR-014\n T-187 2026-03-03 cash_in 2,990.00 USD branch BR-014\n T-188 2026-03-04 cash_in 2,500.00 USD branch BR-014\n T-189 2026-03-04 ach_out 4,526.40 USD CP-2\n T-190 2026-03-08 cash_in 2,900.00 USD branch BR-014\n T-191 2026-03-11 cash_in 2,610.00 USD branch BR-052\n T-192 2026-03-11 ach_out 1,886.00 USD CP-1\n T-193 2026-03-12 cash_in 2,150.00 USD branch BR-052\n T-194 2026-03-13 cash_in 2,230.00 USD branch BR-014\n T-195 2026-03-15 ach_out 1,886.00 USD CP-1\n T-196 2026-03-17 cash_in 2,310.00 USD branch BR-014\n T-197 2026-03-20 cash_in 2,690.00 USD branch BR-014\n T-198 2026-03-20 ach_out 1,886.00 USD CP-1\n T-199 2026-03-21 cash_in 3,170.00 USD branch BR-052\n T-200 2026-03-22 cash_in 2,900.00 USD branch BR-014\n T-201 2026-03-23 ach_out 1,508.80 USD CP-1\n T-202 2026-03-29 cash_in 2,740.00 USD branch BR-052\n T-203 2026-03-31 cash_in 2,450.00 USD branch BR-014\n\nHours spent so far: 2.90. You have 2 replies left."
}
},
{
"reply": "{\"action\": \"close\", \"reason\": \"CONSISTENT_WITH_PROFILE\", \"cites\": [\"kyc.expected_monthly_cash\"], \"note\": \"Cash volume within the expectation on file; band deposits are ordinary takings.\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "close",
"reason": "CONSISTENT_WITH_PROFILE",
"cites": [
"kyc.expected_monthly_cash"
],
"note": "Cash volume within the expectation on file; band deposits are ordinary takings."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/thorough-s2.json
+182
View File
@@ -0,0 +1,182 @@
{
"runId": "thorough-s2",
"seed": 2,
"model": "thorough-analyst",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 0.23728813559322035,
"evidence": 1.0
},
"metrics": {
"hours_spent": 2.95,
"reference_hours": 0.7,
"lookups": 6.0,
"turns_used": 7.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": null
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "benign",
"template": "B6",
"overlay": null,
"typology": null,
"planted": [
[
"doc.D-aded"
]
],
"reference_minutes": 42,
"reference_policy": "fast",
"held_out": false,
"tier": "benign",
"screen": "ALERT A-9760 rule R-DORM-05 \u2014 Large credit to an account with no material activity for nine months\n fired 2026-04-23\n triggering transactions:\n T-4 2026-04-18 wire_in 356,000.00 USD CP-1\n T-5 2026-04-19 wire_out 291,900.00 USD CP-2\nKYC FILE C-7059 Gabriela Kowalczyk\n kyc.account_opened = 2012-03-09\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Retired; low activity expected.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2025-12-12\n kyc.occupation = retired\n kyc.pep = False\n kyc.prior_alerts = 2\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 31,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 0 0 0 0 0 0 0\n 2025-06 0 0 0 0 0 0 0\n 2025-07 0 170 0 0 0 0 1\n 2025-08 0 0 0 0 0 0 0\n 2025-09 0 0 0 0 0 0 0\n 2025-10 0 0 0 0 0 0 0\n 2025-11 0 0 0 0 0 0 0\n 2025-12 0 120 0 0 0 0 1\n 2026-01 0 170 0 0 0 0 1\n 2026-02 0 0 0 0 0 0 0\n 2026-03 0 0 0 0 0 0 0\n 2026-04 0 0 356,000 291,900 0 0 2\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 in 1 txns 356,000.00 USD\n CP-2 out 1 txns 291,900.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "{\"action\": \"lookup\", \"what\": \"documents\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "documents"
},
"rejection": null,
"observation": "DOCUMENTS ON FILE:\n doc.D-3211 [identity] Government-issued identity document, verified at onboarding\n doc.D-aded [executor_letter] Letter from Alicia Haddad, executor, via Eskildsen & Laurent LLP, client trust account: distribution from the estate of Aaron Eskildsen to the named beneficiary.\n\nHours spent so far: 0.65. You have 7 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "prior_alerts"
},
"rejection": null,
"observation": "PRIOR ALERTS:\n A-4278 R-DORM-05 fired 2024-12-23 closed after 9 day(s) by Amara Tanaka\n Closed: reactivation explained by a documented source of funds; document reviewed and retained.\n A-4841 R-WIRE-04 fired 2026-02-02 closed after 5 day(s) by Alicia Haddad\n Closed: destination and amount match the remittance declared at onboarding; within stated income.\n\nHours spent so far: 1.10. You have 6 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-1\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-1"
},
"rejection": null,
"observation": "COUNTERPARTY CP-1 Eskildsen & Laurent LLP, client trust account\n type law_firm_trust; jurisdiction United States; first seen 2026-04-18\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-4\n\nHours spent so far: 1.55. You have 5 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-2"
},
"rejection": null,
"observation": "COUNTERPARTY CP-2 Brightwater Securities\n type brokerage; jurisdiction United States; first seen 2026-04-19\n other alerted accounts sharing this counterparty in twelve months: 1\n transactions: T-5\n\nHours spent so far: 2.00. You have 4 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-04\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "history",
"month": "2026-04"
},
"rejection": null,
"observation": "HISTORY 2026-04: 2 transactions\n T-4 2026-04-18 wire_in 356,000.00 USD CP-1\n T-5 2026-04-19 wire_out 291,900.00 USD CP-2\n\nHours spent so far: 2.45. You have 3 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-03\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "history",
"month": "2026-03"
},
"rejection": null,
"observation": "HISTORY 2026-03: 0 transactions\n\nHours spent so far: 2.90. You have 2 replies left."
}
},
{
"reply": "{\"action\": \"close\", \"reason\": \"DOCUMENTED_SOURCE_OF_FUNDS\", \"cites\": [\"doc.D-aded\"], \"note\": \"Source of funds documented: executor_letter.\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "close",
"reason": "DOCUMENTED_SOURCE_OF_FUNDS",
"cites": [
"doc.D-aded"
],
"note": "Source of funds documented: executor_letter."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/thorough-s3.json
+186
View File
@@ -0,0 +1,186 @@
{
"runId": "thorough-s3",
"seed": 3,
"model": "thorough-analyst",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 0.1864406779661017,
"evidence": 1.0
},
"metrics": {
"hours_spent": 2.95,
"reference_hours": 0.55,
"lookups": 6.0,
"turns_used": 7.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": null
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "benign",
"template": "B4",
"overlay": null,
"typology": null,
"planted": [
[
"kyc.expected_monthly_cash"
],
[
"kyc.expected_monthly_cash",
"kyc.business_type"
]
],
"reference_minutes": 33,
"reference_policy": "fast",
"held_out": false,
"tier": "benign",
"screen": "ALERT A-9453 rule R-STR-01 \u2014 Three or more cash deposits just under the reporting threshold within ten days\n fired 2026-04-23\n triggering transactions:\n T-200 2026-04-14 cash_in 9,770.00 USD branch BR-089\n T-201 2026-04-15 cash_in 9,430.00 USD branch BR-014\n T-206 2026-04-22 cash_in 8,640.00 USD branch BR-089\nKYC FILE C-4718 Lakeside Kitchen LLC\n kyc.account_opened = 2016-10-03\n kyc.business_type = pizzeria\n kyc.country = United States\n kyc.customer_type = business\n kyc.expected_activity = Cash-intensive food service; daily takings deposited most trading days.\n kyc.expected_monthly_cash = 97,000.00 USD\n kyc.kyc_refreshed = 2025-07-13\n kyc.pep = False\n kyc.prior_alerts = 0\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 588,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 52,750 0 0 0 0 16,816 17\n 2025-06 45,240 0 0 0 0 16,317 17\n 2025-07 49,740 0 0 0 0 16,493 17\n 2025-08 40,470 0 0 0 0 16,944 17\n 2025-09 49,720 0 0 0 0 18,522 19\n 2025-10 44,630 0 0 0 0 17,199 16\n 2025-11 49,060 0 0 0 0 18,992 16\n 2025-12 52,570 0 0 0 0 17,992 19\n 2026-01 50,320 0 0 0 0 15,038 17\n 2026-02 51,710 0 0 0 0 17,826 16\n 2026-03 52,240 0 0 0 0 16,934 17\n 2026-04 78,100 0 0 0 0 18,673 18\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 out 48 txns 132,956.60 USD\n CP-2 out 12 txns 74,793.60 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "{\"action\": \"lookup\", \"what\": \"documents\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "documents"
},
"rejection": null,
"observation": "DOCUMENTS ON FILE:\n doc.D-f244 [registration] Business registration \u2014 Lakeside Kitchen LLC\n doc.D-ee54 [licence] Food service licence, renewed annually\n\nHours spent so far: 0.65. You have 7 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "prior_alerts"
},
"rejection": null,
"observation": "PRIOR ALERTS: none on file.\n\nHours spent so far: 1.10. You have 6 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-1\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-1"
},
"rejection": null,
"observation": "COUNTERPARTY CP-1 Blue Ridge Produce LLC\n type business; jurisdiction United States; first seen 2025-05-08\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-4, T-9, T-10, T-13, T-23, T-25, T-28, T-30, T-35, T-41, T-48, T-51, T-55, T-57, T-63, T-66, T-70, T-74, T-75, T-79, T-92, T-95, T-99, T-101, T-104, T-106, T-111, T-113, T-125, T-127, T-128, T-133, T-145, T-146, T-150, T-153, T-160, T-165, T-168, T-171, T-174, T-176, T-179, T-184, T-194, T-196, T-199, T-204\n\nHours spent so far: 1.55. You have 5 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-2"
},
"rejection": null,
"observation": "COUNTERPARTY CP-2 Copperfield Foods Group\n type business; jurisdiction United States; first seen 2025-05-03\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-2, T-21, T-39, T-56, T-76, T-90, T-108, T-122, T-139, T-157, T-177, T-190\n\nHours spent so far: 2.00. You have 4 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-04\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "history",
"month": "2026-04"
},
"rejection": null,
"observation": "HISTORY 2026-04: 18 transactions\n T-189 2026-04-01 cash_in 4,840.00 USD branch BR-014\n T-190 2026-04-03 ach_out 7,065.80 USD CP-2\n T-191 2026-04-04 cash_in 5,650.00 USD branch BR-014\n T-192 2026-04-05 cash_in 6,050.00 USD branch BR-014\n T-193 2026-04-07 cash_in 4,790.00 USD branch BR-089\n T-194 2026-04-07 ach_out 2,523.50 USD CP-1\n T-195 2026-04-10 cash_in 3,880.00 USD branch BR-089\n T-196 2026-04-10 ach_out 3,028.20 USD CP-1\n T-197 2026-04-11 cash_in 5,340.00 USD branch BR-089\n T-198 2026-04-13 cash_in 5,240.00 USD branch BR-089\n T-199 2026-04-13 ach_out 3,028.20 USD CP-1\n T-200 2026-04-14 cash_in 9,770.00 USD branch BR-089\n T-201 2026-04-15 cash_in 9,430.00 USD branch BR-014\n T-202 2026-04-16 cash_in 3,930.00 USD branch BR-089\n T-203 2026-04-17 cash_in 4,390.00 USD branch BR-014\n T-204 2026-04-20 ach_out 3,028.20 USD CP-1\n T-205 2026-04-21 cash_in 6,150.00 USD branch BR-014\n T-206 2026-04-22 cash_in 8,640.00 USD branch BR-089\n\nHours spent so far: 2.45. You have 3 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-03\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "history",
"month": "2026-03"
},
"rejection": null,
"observation": "HISTORY 2026-03: 17 transactions\n T-172 2026-03-01 cash_in 5,420.00 USD branch BR-089\n T-173 2026-03-02 cash_in 5,330.00 USD branch BR-089\n T-174 2026-03-04 ach_out 3,175.20 USD CP-1\n T-175 2026-03-05 cash_in 3,740.00 USD branch BR-089\n T-176 2026-03-06 ach_out 3,704.40 USD CP-1\n T-177 2026-03-06 ach_out 5,292.00 USD CP-2\n T-178 2026-03-08 cash_in 4,490.00 USD branch BR-014\n T-179 2026-03-08 ach_out 2,646.00 USD CP-1\n T-180 2026-03-11 cash_in 4,180.00 USD branch BR-089\n T-181 2026-03-12 cash_in 3,660.00 USD branch BR-089\n T-182 2026-03-13 cash_in 3,520.00 USD branch BR-089\n T-183 2026-03-15 cash_in 4,540.00 USD branch BR-089\n T-184 2026-03-15 ach_out 2,116.80 USD CP-1\n T-185 2026-03-21 cash_in 4,320.00 USD branch BR-089\n T-186 2026-03-22 cash_in 5,380.00 USD branch BR-089\n T-187 2026-03-24 cash_in 3,830.00 USD branch BR-089\n T-188 2026-03-26 cash_in 3,830.00 USD branch BR-089\n\nHours spent so far: 2.90. You have 2 replies left."
}
},
{
"reply": "{\"action\": \"close\", \"reason\": \"CONSISTENT_WITH_PROFILE\", \"cites\": [\"kyc.expected_monthly_cash\"], \"note\": \"Cash volume within the expectation on file; band deposits are ordinary takings.\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "close",
"reason": "CONSISTENT_WITH_PROFILE",
"cites": [
"kyc.expected_monthly_cash"
],
"note": "Cash volume within the expectation on file; band deposits are ordinary takings."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/thorough-s4.json
+196
View File
@@ -0,0 +1,196 @@
{
"runId": "thorough-s4",
"seed": 4,
"model": "thorough-analyst",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 0.1864406779661017,
"evidence": 1.0
},
"metrics": {
"hours_spent": 2.95,
"reference_hours": 0.55,
"lookups": 6.0,
"turns_used": 7.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": 1.0
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "suspicious",
"template": "S1",
"overlay": null,
"typology": "STRUCTURING",
"planted": [
[
"T-46",
"T-47",
"T-49",
"T-50",
"T-52",
"T-53",
"T-54",
"T-56"
]
],
"reference_minutes": 33,
"reference_policy": "fast",
"held_out": false,
"tier": "visible",
"screen": "ALERT A-8199 rule R-STR-01 \u2014 Three or more cash deposits just under the reporting threshold within ten days\n fired 2026-06-14\n triggering transactions:\n T-49 2026-06-05 cash_in 9,320.00 USD branch BR-027\n T-50 2026-06-06 cash_in 9,820.00 USD branch BR-031\n T-52 2026-06-07 cash_in 9,880.00 USD branch BR-068\n T-53 2026-06-09 cash_in 9,730.00 USD branch BR-027\n T-54 2026-06-10 cash_in 9,680.00 USD branch BR-031\n T-56 2026-06-11 cash_in 9,850.00 USD branch BR-068\nKYC FILE C-5480 Bilal Quintero\n kyc.account_opened = 2017-03-29\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Salaried individual; payroll deposits and routine spending.\n kyc.expected_monthly_cash = 100.00 USD\n kyc.kyc_refreshed = 2026-05-04\n kyc.occupation = delivery driver\n kyc.pep = False\n kyc.prior_alerts = 3\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 45,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-07 0 220 0 0 3,410 0 3\n 2025-08 0 480 0 0 3,382 0 3\n 2025-09 0 570 0 0 3,444 0 4\n 2025-10 0 570 0 0 3,454 0 5\n 2025-11 0 660 0 0 3,460 0 4\n 2025-12 0 550 0 0 3,403 0 3\n 2026-01 0 680 0 0 3,351 0 5\n 2026-02 0 490 0 0 3,445 0 4\n 2026-03 0 540 0 0 3,433 0 4\n 2026-04 0 430 0 0 3,387 0 4\n 2026-05 0 660 0 0 3,411 0 5\n 2026-06 77,210 420 0 66,400 3,348 0 13\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-2 out 1 txns 66,400.00 USD\n CP-1 in 12 txns 40,928.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "{\"action\": \"lookup\", \"what\": \"documents\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "documents"
},
"rejection": null,
"observation": "DOCUMENTS ON FILE:\n doc.D-c608 [identity] Government-issued identity document, verified at onboarding\n\nHours spent so far: 0.65. You have 7 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "prior_alerts"
},
"rejection": null,
"observation": "PRIOR ALERTS:\n A-1684 R-STR-01 fired 2025-01-17 closed after 5 day(s) by Andre Abara\n Closed: deposits are documented daily takings; amounts vary with trading days; within kyc.expected_monthly_cash.\n A-3624 R-STR-01 fired 2025-03-06 closed after 8 day(s) by Bilal Laurent\n Closed: deposits are documented daily takings; amounts vary with trading days; within kyc.expected_monthly_cash.\n A-2017 R-CASH-03 fired 2025-05-09 closed after 7 day(s) by Sofia Grimaldi\n Closed: seasonal peak consistent with kyc.expected_activity; volume within stated expectation.\n\nHours spent so far: 1.10. You have 6 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-2"
},
"rejection": null,
"observation": "COUNTERPARTY CP-2 Redfern Holdings Co.\n type business; jurisdiction United States; first seen 2026-06-13\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-57\n\nHours spent so far: 1.55. You have 5 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-1\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-1"
},
"rejection": null,
"observation": "COUNTERPARTY CP-1 Stonebridge Diner Group\n type business; jurisdiction United States; first seen 2025-07-03\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-1, T-5, T-7, T-11, T-16, T-21, T-23, T-28, T-32, T-36, T-40, T-45\n\nHours spent so far: 2.00. You have 4 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-06\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "history",
"month": "2026-06"
},
"rejection": null,
"observation": "HISTORY 2026-06: 13 transactions\n T-45 2026-06-02 ach_in 3,348.00 USD CP-1\n T-46 2026-06-02 cash_in 9,400.00 USD branch BR-031\n T-47 2026-06-03 cash_in 9,530.00 USD branch BR-068\n T-48 2026-06-04 cash_out 110.00 USD branch BR-027\n T-49 2026-06-05 cash_in 9,320.00 USD branch BR-027\n T-50 2026-06-06 cash_in 9,820.00 USD branch BR-031\n T-51 2026-06-07 cash_out 150.00 USD branch BR-027\n T-52 2026-06-07 cash_in 9,880.00 USD branch BR-068\n T-53 2026-06-09 cash_in 9,730.00 USD branch BR-027\n T-54 2026-06-10 cash_in 9,680.00 USD branch BR-031\n T-55 2026-06-11 cash_out 160.00 USD branch BR-068\n T-56 2026-06-11 cash_in 9,850.00 USD branch BR-068\n T-57 2026-06-13 wire_out 66,400.00 USD CP-2\n\nHours spent so far: 2.45. You have 3 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-05\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "history",
"month": "2026-05"
},
"rejection": null,
"observation": "HISTORY 2026-05: 5 transactions\n T-40 2026-05-02 ach_in 3,411.00 USD CP-1\n T-41 2026-05-02 cash_out 190.00 USD branch BR-068\n T-42 2026-05-15 cash_out 120.00 USD branch BR-027\n T-43 2026-05-18 cash_out 150.00 USD branch BR-068\n T-44 2026-05-29 cash_out 200.00 USD branch BR-027\n\nHours spent so far: 2.90. You have 2 replies left."
}
},
{
"reply": "{\"action\": \"escalate\", \"typology\": \"STRUCTURING\", \"cites\": [\"T-49\", \"T-50\", \"T-52\", \"T-53\", \"T-54\", \"T-56\", \"T-46\", \"T-47\"], \"narrative\": \"Repeated cash deposits just under the threshold across branches.\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "escalate",
"typology": "STRUCTURING",
"cites": [
"T-49",
"T-50",
"T-52",
"T-53",
"T-54",
"T-56",
"T-46",
"T-47"
],
"narrative": "Repeated cash deposits just under the threshold across branches."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/thorough-s5.json
+186
View File
@@ -0,0 +1,186 @@
{
"runId": "thorough-s5",
"seed": 5,
"model": "thorough-analyst",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 0.23728813559322035,
"evidence": 1.0
},
"metrics": {
"hours_spent": 2.95,
"reference_hours": 0.7,
"lookups": 6.0,
"turns_used": 7.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": null
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "benign",
"template": "B3",
"overlay": null,
"typology": null,
"planted": [
[
"doc.D-8578"
],
[
"doc.D-8578",
"kyc.occupation"
]
],
"reference_minutes": 42,
"reference_policy": "fast",
"held_out": false,
"tier": "benign",
"screen": "ALERT A-6081 rule R-VEL-02 \u2014 Funds received and moved out again within 48 hours\n fired 2026-03-22\n triggering transactions:\n T-52 2026-03-18 wire_in 252,000.00 USD CP-2\n T-54 2026-03-19 wire_out 219,200.00 USD CP-3\nKYC FILE C-7077 Andre Nakamura\n kyc.account_opened = 2022-01-15\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Salaried individual; payroll deposits and routine spending.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2025-04-22\n kyc.occupation = teacher\n kyc.pep = False\n kyc.prior_alerts = 3\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 106,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-04 0 590 0 0 8,866 0 4\n 2025-05 0 820 0 0 8,921 0 4\n 2025-06 0 920 0 0 8,952 0 5\n 2025-07 0 680 0 0 8,870 0 3\n 2025-08 0 1,140 0 0 8,906 0 5\n 2025-09 0 390 0 0 8,870 0 4\n 2025-10 0 850 0 0 8,922 0 4\n 2025-11 0 1,020 0 0 8,916 0 5\n 2025-12 0 950 0 0 8,901 0 5\n 2026-01 0 880 0 0 8,854 0 4\n 2026-02 0 1,170 0 0 8,916 0 5\n 2026-03 0 720 252,000 219,200 8,956 0 6\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-2 in 1 txns 252,000.00 USD\n CP-3 out 1 txns 219,200.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "{\"action\": \"lookup\", \"what\": \"documents\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "documents"
},
"rejection": null,
"observation": "DOCUMENTS ON FILE:\n doc.D-5045 [identity] Government-issued identity document, verified at onboarding\n doc.D-8578 [closing_statement] Closing statement \u2014 sale of 266 Birch Avenue; net proceeds to seller disbursed by Iwata Title & Escrow.\n\nHours spent so far: 0.65. You have 7 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "prior_alerts"
},
"rejection": null,
"observation": "PRIOR ALERTS:\n A-4824 R-VEL-02 fired 2025-03-19 closed after 4 day(s) by Jamal Whitlock\n Closed: pass-through matches the settlement terms in the agreement on file; counterparties verified.\n A-1410 R-WIRE-04 fired 2025-04-24 closed after 2 day(s) by Lars Farrow\n Closed: destination and amount match the remittance declared at onboarding; within stated income.\n A-2317 R-CASH-03 fired 2025-09-02 closed after 6 day(s) by Chidi Zapata\n Closed: seasonal peak consistent with kyc.expected_activity; volume within stated expectation.\n\nHours spent so far: 1.10. You have 6 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-2"
},
"rejection": null,
"observation": "COUNTERPARTY CP-2 Iwata Title & Escrow\n type title_company; jurisdiction United States; first seen 2026-03-18\n other alerted accounts sharing this counterparty in twelve months: 1\n transactions: T-52\n\nHours spent so far: 1.55. You have 5 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-3\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-3"
},
"rejection": null,
"observation": "COUNTERPARTY CP-3 Harbor Securities\n type brokerage; jurisdiction United States; first seen 2026-03-19\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-54\n\nHours spent so far: 2.00. You have 4 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-03\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "history",
"month": "2026-03"
},
"rejection": null,
"observation": "HISTORY 2026-03: 6 transactions\n T-49 2026-03-02 cash_out 160.00 USD branch BR-052\n T-50 2026-03-04 cash_out 320.00 USD branch BR-052\n T-51 2026-03-05 ach_in 8,956.00 USD CP-1\n T-52 2026-03-18 wire_in 252,000.00 USD CP-2\n T-53 2026-03-19 cash_out 240.00 USD branch BR-089\n T-54 2026-03-19 wire_out 219,200.00 USD CP-3\n\nHours spent so far: 2.45. You have 3 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-02\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "history",
"month": "2026-02"
},
"rejection": null,
"observation": "HISTORY 2026-02: 5 transactions\n T-44 2026-02-01 ach_in 8,916.00 USD CP-1\n T-45 2026-02-16 cash_out 150.00 USD branch BR-089\n T-46 2026-02-18 cash_out 390.00 USD branch BR-052\n T-47 2026-02-21 cash_out 380.00 USD branch BR-052\n T-48 2026-02-26 cash_out 250.00 USD branch BR-089\n\nHours spent so far: 2.90. You have 2 replies left."
}
},
{
"reply": "{\"action\": \"close\", \"reason\": \"DOCUMENTED_SOURCE_OF_FUNDS\", \"cites\": [\"doc.D-8578\"], \"note\": \"Source of funds documented: closing_statement.\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "close",
"reason": "DOCUMENTED_SOURCE_OF_FUNDS",
"cites": [
"doc.D-8578"
],
"note": "Source of funds documented: closing_statement."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/thorough-s6.json
+187
View File
@@ -0,0 +1,187 @@
{
"runId": "thorough-s6",
"seed": 6,
"model": "thorough-analyst",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 0.1864406779661017,
"evidence": 1.0
},
"metrics": {
"hours_spent": 2.95,
"reference_hours": 0.55,
"lookups": 6.0,
"turns_used": 7.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": null
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "benign",
"template": "B1",
"overlay": null,
"typology": null,
"planted": [
[
"kyc.expected_activity",
"kyc.expected_monthly_cash"
],
[
"kyc.expected_monthly_cash"
]
],
"reference_minutes": 33,
"reference_policy": "fast",
"held_out": false,
"tier": "benign",
"screen": "ALERT A-7014 rule R-CASH-03 \u2014 Monthly cash deposits materially above the trailing six-month average\n fired 2026-05-15\n triggering transactions:\n T-83 2026-05-02 cash_in 2,140.00 USD branch BR-102\n T-84 2026-05-03 cash_in 2,930.00 USD branch BR-102\n T-85 2026-05-04 cash_in 2,610.00 USD branch BR-031\n T-86 2026-05-05 cash_in 2,290.00 USD branch BR-031\n T-87 2026-05-06 cash_in 2,930.00 USD branch BR-031\n T-88 2026-05-08 cash_in 2,190.00 USD branch BR-031\n T-89 2026-05-09 cash_in 2,270.00 USD branch BR-031\n T-90 2026-05-10 cash_in 1,970.00 USD branch BR-102\n T-91 2026-05-11 cash_in 2,540.00 USD branch BR-031\n T-92 2026-05-12 cash_in 2,540.00 USD branch BR-031\n T-93 2026-05-13 cash_in 1,970.00 USD branch BR-102\n T-94 2026-05-14 cash_in 2,930.00 USD branch BR-031\nKYC FILE C-3047 Lakeside Landscaping Inc.\n kyc.account_opened = 2013-08-05\n kyc.business_type = landscaping services\n kyc.country = United States\n kyc.customer_type = business\n kyc.expected_activity = Seasonal business; cash receipts peak April to September.\n kyc.expected_monthly_cash = 38,000.00 USD\n kyc.kyc_refreshed = 2025-04-30\n kyc.pep = False\n kyc.prior_alerts = 2\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 266,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-06 28,130 0 0 2,900 0 7,716 10\n 2025-07 24,190 0 0 0 0 8,652 11\n 2025-08 27,600 0 0 0 0 7,538 10\n 2025-09 22,400 0 0 6,300 0 6,691 13\n 2025-10 5,480 0 0 0 0 1,686 4\n 2025-11 7,530 0 0 0 0 1,673 4\n 2025-12 6,520 0 0 3,900 0 2,119 5\n 2026-01 8,260 0 0 0 0 1,811 5\n 2026-02 5,740 0 0 0 0 1,989 4\n 2026-03 6,100 0 0 3,300 0 2,094 6\n 2026-04 24,540 0 0 0 0 5,537 10\n 2026-05 29,310 0 0 0 0 9,777 13\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 out 12 txns 57,289.16 USD\n CP-2 out 4 txns 16,400.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "{\"action\": \"lookup\", \"what\": \"documents\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "documents"
},
"rejection": null,
"observation": "DOCUMENTS ON FILE:\n doc.D-ab7d [registration] Business registration \u2014 Lakeside Landscaping Inc.\n doc.D-f811 [licence] Trade licence, landscaping services, renewed annually\n\nHours spent so far: 0.65. You have 7 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "prior_alerts"
},
"rejection": null,
"observation": "PRIOR ALERTS:\n A-4237 R-CASH-03 fired 2024-08-10 closed after 7 day(s) by Bilal Rasmussen\n Closed: seasonal peak consistent with kyc.expected_activity; volume within stated expectation.\n A-2077 R-WIRE-04 fired 2025-04-23 closed after 3 day(s) by Dana Pereira\n Closed: destination and amount match the remittance declared at onboarding; within stated income.\n\nHours spent so far: 1.10. You have 6 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-1\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-1"
},
"rejection": null,
"observation": "COUNTERPARTY CP-1 Lakeside Supply LLC\n type business; jurisdiction United States; first seen 2025-06-25\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-9, T-17, T-28, T-40, T-48, T-52, T-55, T-62, T-64, T-70, T-80, T-95\n\nHours spent so far: 1.55. You have 5 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-2"
},
"rejection": null,
"observation": "COUNTERPARTY CP-2 Harbor Equipment Group\n type business; jurisdiction United States; first seen 2025-06-06\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-3, T-33, T-53, T-67\n\nHours spent so far: 2.00. You have 4 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-05\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "history",
"month": "2026-05"
},
"rejection": null,
"observation": "HISTORY 2026-05: 13 transactions\n T-83 2026-05-02 cash_in 2,140.00 USD branch BR-102\n T-84 2026-05-03 cash_in 2,930.00 USD branch BR-102\n T-85 2026-05-04 cash_in 2,610.00 USD branch BR-031\n T-86 2026-05-05 cash_in 2,290.00 USD branch BR-031\n T-87 2026-05-06 cash_in 2,930.00 USD branch BR-031\n T-88 2026-05-08 cash_in 2,190.00 USD branch BR-031\n T-89 2026-05-09 cash_in 2,270.00 USD branch BR-031\n T-90 2026-05-10 cash_in 1,970.00 USD branch BR-102\n T-91 2026-05-11 cash_in 2,540.00 USD branch BR-031\n T-92 2026-05-12 cash_in 2,540.00 USD branch BR-031\n T-93 2026-05-13 cash_in 1,970.00 USD branch BR-102\n T-94 2026-05-14 cash_in 2,930.00 USD branch BR-031\n T-95 2026-05-14 ach_out 9,777.43 USD CP-1\n\nHours spent so far: 2.45. You have 3 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-04\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "history",
"month": "2026-04"
},
"rejection": null,
"observation": "HISTORY 2026-04: 10 transactions\n T-73 2026-04-04 cash_in 3,270.00 USD branch BR-102\n T-74 2026-04-09 cash_in 2,740.00 USD branch BR-031\n T-75 2026-04-12 cash_in 3,180.00 USD branch BR-102\n T-76 2026-04-13 cash_in 2,990.00 USD branch BR-102\n T-77 2026-04-16 cash_in 2,430.00 USD branch BR-102\n T-78 2026-04-18 cash_in 2,570.00 USD branch BR-102\n T-79 2026-04-24 cash_in 2,370.00 USD branch BR-031\n T-80 2026-04-24 ach_out 5,537.66 USD CP-1\n T-81 2026-04-27 cash_in 2,370.00 USD branch BR-031\n T-82 2026-04-30 cash_in 2,620.00 USD branch BR-102\n\nHours spent so far: 2.90. You have 2 replies left."
}
},
{
"reply": "{\"action\": \"close\", \"reason\": \"CONSISTENT_WITH_PROFILE\", \"cites\": [\"kyc.expected_activity\", \"kyc.expected_monthly_cash\"], \"note\": \"Seasonal peak the file anticipated; within expected monthly cash.\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "close",
"reason": "CONSISTENT_WITH_PROFILE",
"cites": [
"kyc.expected_activity",
"kyc.expected_monthly_cash"
],
"note": "Seasonal peak the file anticipated; within expected monthly cash."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/thorough-s7.json
+182
View File
@@ -0,0 +1,182 @@
{
"runId": "thorough-s7",
"seed": 7,
"model": "thorough-analyst",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 0.23728813559322035,
"evidence": 1.0
},
"metrics": {
"hours_spent": 2.95,
"reference_hours": 0.7,
"lookups": 6.0,
"turns_used": 7.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": null
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "benign",
"template": "B6",
"overlay": null,
"typology": null,
"planted": [
[
"doc.D-b931"
]
],
"reference_minutes": 42,
"reference_policy": "fast",
"held_out": false,
"tier": "benign",
"screen": "ALERT A-5644 rule R-DORM-05 \u2014 Large credit to an account with no material activity for nine months\n fired 2026-06-14\n triggering transactions:\n T-4 2026-06-07 wire_in 249,000.00 USD CP-1\n T-5 2026-06-10 wire_out 134,400.00 USD CP-2\nKYC FILE C-1843 Leila Bergstrom\n kyc.account_opened = 2017-03-20\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Retired; low activity expected.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2026-01-13\n kyc.occupation = retired\n kyc.pep = False\n kyc.prior_alerts = 0\n kyc.prior_sars = 0\n kyc.remittance_declared = none declared\n kyc.stated_income = 33,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-07 0 0 0 0 0 0 0\n 2025-08 0 0 0 0 0 0 0\n 2025-09 0 60 0 0 0 0 1\n 2025-10 0 0 0 0 0 0 0\n 2025-11 0 150 0 0 0 0 1\n 2025-12 0 0 0 0 0 0 0\n 2026-01 0 0 0 0 0 0 0\n 2026-02 0 0 0 0 0 0 0\n 2026-03 0 50 0 0 0 0 1\n 2026-04 0 0 0 0 0 0 0\n 2026-05 0 0 0 0 0 0 0\n 2026-06 0 0 249,000 134,400 0 0 2\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-1 in 1 txns 249,000.00 USD\n CP-2 out 1 txns 134,400.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "{\"action\": \"lookup\", \"what\": \"documents\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "documents"
},
"rejection": null,
"observation": "DOCUMENTS ON FILE:\n doc.D-1dc2 [identity] Government-issued identity document, verified at onboarding\n doc.D-b931 [executor_letter] Letter from Hana Sandoval, executor, via Xiang & Farrow LLP, client trust account: distribution from the estate of Elena Okafor to the named beneficiary.\n\nHours spent so far: 0.65. You have 7 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "prior_alerts"
},
"rejection": null,
"observation": "PRIOR ALERTS: none on file.\n\nHours spent so far: 1.10. You have 6 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-1\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-1"
},
"rejection": null,
"observation": "COUNTERPARTY CP-1 Xiang & Farrow LLP, client trust account\n type law_firm_trust; jurisdiction United States; first seen 2026-06-07\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-4\n\nHours spent so far: 1.55. You have 5 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-2"
},
"rejection": null,
"observation": "COUNTERPARTY CP-2 Oakhill Securities\n type brokerage; jurisdiction United States; first seen 2026-06-10\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-5\n\nHours spent so far: 2.00. You have 4 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-06\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "history",
"month": "2026-06"
},
"rejection": null,
"observation": "HISTORY 2026-06: 2 transactions\n T-4 2026-06-07 wire_in 249,000.00 USD CP-1\n T-5 2026-06-10 wire_out 134,400.00 USD CP-2\n\nHours spent so far: 2.45. You have 3 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-05\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "history",
"month": "2026-05"
},
"rejection": null,
"observation": "HISTORY 2026-05: 0 transactions\n\nHours spent so far: 2.90. You have 2 replies left."
}
},
{
"reply": "{\"action\": \"close\", \"reason\": \"DOCUMENTED_SOURCE_OF_FUNDS\", \"cites\": [\"doc.D-b931\"], \"note\": \"Source of funds documented: executor_letter.\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "close",
"reason": "DOCUMENTED_SOURCE_OF_FUNDS",
"cites": [
"doc.D-b931"
],
"note": "Source of funds documented: executor_letter."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/alert-triage/thorough-s9.json
+187
View File
@@ -0,0 +1,187 @@
{
"runId": "thorough-s9",
"seed": 9,
"model": "thorough-analyst",
"capturedAt": "2026-08-28",
"rewards": {
"caught": 1.0,
"hours": 0.6949152542372882,
"evidence": 1.0
},
"metrics": {
"hours_spent": 2.95,
"reference_hours": 2.05,
"lookups": 6.0,
"turns_used": 7.0,
"rejected_replies": 0.0,
"false_escalation": 0.0,
"missed": 0.0,
"typology_match": 1.0
},
"truncated": false,
"outcome": "solved",
"info": {
"label": "suspicious",
"template": "B3",
"overlay": "funnel",
"typology": "FUNNEL_ACCOUNT",
"planted": [
[
"CP-3",
"T-52"
],
[
"T-52"
]
],
"reference_minutes": 123,
"reference_policy": "targeted",
"held_out": false,
"tier": "hidden",
"screen": "ALERT A-9537 rule R-VEL-02 \u2014 Funds received and moved out again within 48 hours\n fired 2026-04-19\n triggering transactions:\n T-51 2026-04-16 wire_in 462,000.00 USD CP-2\n T-52 2026-04-17 wire_out 448,100.00 USD CP-3\nKYC FILE C-9673 Samir Tanaka\n kyc.account_opened = 2019-12-21\n kyc.country = United States\n kyc.customer_type = individual\n kyc.expected_activity = Salaried individual; payroll deposits and routine spending.\n kyc.expected_monthly_cash = 0.00 USD\n kyc.kyc_refreshed = 2025-04-04\n kyc.occupation = electrician\n kyc.pep = False\n kyc.prior_alerts = 4\n kyc.prior_sars = 1\n kyc.remittance_declared = none declared\n kyc.stated_income = 84,000.00 USD\nTWELVE-MONTH SUMMARY (USD)\n month cash in cash out wires in wires out ach in ach out txns\n 2025-05 0 1,090 0 0 6,958 0 5\n 2025-06 0 630 0 0 7,009 0 5\n 2025-07 0 780 0 0 6,957 0 4\n 2025-08 0 470 0 0 6,989 0 3\n 2025-09 0 1,040 0 0 6,959 0 5\n 2025-10 0 740 0 0 6,996 0 4\n 2025-11 0 860 0 0 6,947 0 5\n 2025-12 0 900 0 0 6,940 0 5\n 2026-01 0 650 0 0 7,025 0 3\n 2026-02 0 460 0 0 7,026 0 3\n 2026-03 0 620 0 0 6,962 0 3\n 2026-04 0 960 462,000 448,100 6,944 0 7\nCOUNTERPARTIES on screen (top by twelve-month volume):\n CP-2 in 1 txns 462,000.00 USD\n CP-3 out 1 txns 448,100.00 USD\nReporting threshold in this jurisdiction: 10,000.00 USD\nSYNTHETIC DATA \u2014 no real customer, account or transaction.\n\nHours spent so far: 0.50. You have 8 replies left."
},
"turns": [
{
"reply": "{\"action\": \"lookup\", \"what\": \"documents\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "documents"
},
"rejection": null,
"observation": "DOCUMENTS ON FILE:\n doc.D-8b75 [identity] Government-issued identity document, verified at onboarding\n doc.D-838c [closing_statement] Closing statement \u2014 sale of 82 Alder Street; net proceeds to seller disbursed by Okafor Title & Escrow.\n\nHours spent so far: 0.65. You have 7 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"prior_alerts\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "prior_alerts"
},
"rejection": null,
"observation": "PRIOR ALERTS:\n A-2906 R-VEL-02 fired 2025-01-09 closed after 4 day(s) by Hana Xiang\n Closed: pass-through matches the settlement terms in the agreement on file; counterparties verified.\n A-2547 R-DORM-05 fired 2025-07-01 escalated \u2014 report filed after 2 day(s) by Jamal Ferreira\n Escalated: pattern not explained by the file at the time.\n A-2855 R-DORM-05 fired 2025-07-17 closed after 8 day(s) by Carlos Zapata\n Closed: reactivation explained by a documented source of funds; document reviewed and retained.\n A-2904 R-VEL-02 fired 2025-12-05 closed after 5 day(s) by Dana Grimaldi\n Closed: pass-through matches the settlement terms in the agreement on file; counterparties verified.\n\nHours spent so far: 1.10. You have 6 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-2\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-2"
},
"rejection": null,
"observation": "COUNTERPARTY CP-2 Okafor Title & Escrow\n type title_company; jurisdiction United States; first seen 2026-04-16\n other alerted accounts sharing this counterparty in twelve months: 0\n transactions: T-51\n\nHours spent so far: 1.55. You have 5 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"counterparty\", \"id\": \"CP-3\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "counterparty",
"id": "CP-3"
},
"rejection": null,
"observation": "COUNTERPARTY CP-3 Summit Securities\n type brokerage; jurisdiction United States; first seen 2026-04-17\n other alerted accounts sharing this counterparty in twelve months: 8\n transactions: T-52\n\nHours spent so far: 2.00. You have 4 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-04\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "history",
"month": "2026-04"
},
"rejection": null,
"observation": "HISTORY 2026-04: 7 transactions\n T-46 2026-04-04 ach_in 6,944.00 USD CP-1\n T-47 2026-04-04 cash_out 220.00 USD branch BR-068\n T-48 2026-04-06 cash_out 280.00 USD branch BR-068\n T-49 2026-04-10 cash_out 340.00 USD branch BR-068\n T-50 2026-04-12 cash_out 120.00 USD branch BR-068\n T-51 2026-04-16 wire_in 462,000.00 USD CP-2\n T-52 2026-04-17 wire_out 448,100.00 USD CP-3\n\nHours spent so far: 2.45. You have 3 replies left."
}
},
{
"reply": "{\"action\": \"lookup\", \"what\": \"history\", \"month\": \"2026-03\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "lookup",
"what": "history",
"month": "2026-03"
},
"rejection": null,
"observation": "HISTORY 2026-03: 3 transactions\n T-43 2026-03-02 ach_in 6,962.00 USD CP-1\n T-44 2026-03-13 cash_out 370.00 USD branch BR-027\n T-45 2026-03-18 cash_out 250.00 USD branch BR-027\n\nHours spent so far: 2.90. You have 2 replies left."
}
},
{
"reply": "{\"action\": \"escalate\", \"typology\": \"FUNNEL_ACCOUNT\", \"cites\": [\"CP-3\", \"T-52\"], \"narrative\": \"CP-3 is shared with 8 other alerted accounts; funds converge on it.\"}",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"action": {
"action": "escalate",
"typology": "FUNNEL_ACCOUNT",
"cites": [
"CP-3",
"T-52"
],
"narrative": "CP-3 is shared with 8 other alerted accounts; funds converge on it."
},
"rejection": null,
"observation": "Disposition recorded. The episode is over."
}
}
]
}
public/traces/manifest.json
+310
View File
@@ -1,4 +1,314 @@
{
"alert-triage": [
{
"id": "base-off-s1",
"label": "Out of the box",
"path": "/traces/alert-triage/base-off-s1.json",
"kind": "recorded",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 1
},
{
"id": "base-off-s2",
"label": "Out of the box",
"path": "/traces/alert-triage/base-off-s2.json",
"kind": "recorded",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 2
},
{
"id": "base-off-s3",
"label": "Out of the box",
"path": "/traces/alert-triage/base-off-s3.json",
"kind": "recorded",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 3
},
{
"id": "base-off-s4",
"label": "Out of the box",
"path": "/traces/alert-triage/base-off-s4.json",
"kind": "recorded",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 4
},
{
"id": "base-off-s5",
"label": "Out of the box",
"path": "/traces/alert-triage/base-off-s5.json",
"kind": "recorded",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 5
},
{
"id": "base-off-s6",
"label": "Out of the box",
"path": "/traces/alert-triage/base-off-s6.json",
"kind": "recorded",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 6
},
{
"id": "base-off-s7",
"label": "Out of the box",
"path": "/traces/alert-triage/base-off-s7.json",
"kind": "recorded",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 7
},
{
"id": "base-off-s9",
"label": "Out of the box",
"path": "/traces/alert-triage/base-off-s9.json",
"kind": "recorded",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 9
},
{
"id": "base-on-s1",
"label": "Allowed to think",
"path": "/traces/alert-triage/base-on-s1.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 1,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "base-on-s2",
"label": "Allowed to think",
"path": "/traces/alert-triage/base-on-s2.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 2,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "fast-s1",
"label": "Reads the screen",
"path": "/traces/alert-triage/fast-s1.json",
"kind": "generated",
"model": "fast-analyst",
"capturedAt": "2026-08-28",
"seed": 1
},
{
"id": "fast-s2",
"label": "Reads the screen",
"path": "/traces/alert-triage/fast-s2.json",
"kind": "generated",
"model": "fast-analyst",
"capturedAt": "2026-08-28",
"seed": 2
},
{
"id": "fast-s3",
"label": "Reads the screen",
"path": "/traces/alert-triage/fast-s3.json",
"kind": "generated",
"model": "fast-analyst",
"capturedAt": "2026-08-28",
"seed": 3
},
{
"id": "fast-s4",
"label": "Reads the screen",
"path": "/traces/alert-triage/fast-s4.json",
"kind": "generated",
"model": "fast-analyst",
"capturedAt": "2026-08-28",
"seed": 4
},
{
"id": "fast-s5",
"label": "Reads the screen",
"path": "/traces/alert-triage/fast-s5.json",
"kind": "generated",
"model": "fast-analyst",
"capturedAt": "2026-08-28",
"seed": 5
},
{
"id": "fast-s6",
"label": "Reads the screen",
"path": "/traces/alert-triage/fast-s6.json",
"kind": "generated",
"model": "fast-analyst",
"capturedAt": "2026-08-28",
"seed": 6
},
{
"id": "fast-s7",
"label": "Reads the screen",
"path": "/traces/alert-triage/fast-s7.json",
"kind": "generated",
"model": "fast-analyst",
"capturedAt": "2026-08-28",
"seed": 7
},
{
"id": "fast-s9",
"label": "Reads the screen",
"path": "/traces/alert-triage/fast-s9.json",
"kind": "generated",
"model": "fast-analyst",
"capturedAt": "2026-08-28",
"seed": 9
},
{
"id": "targeted-s1",
"label": "Checks the hidden tells",
"path": "/traces/alert-triage/targeted-s1.json",
"kind": "generated",
"model": "targeted-analyst",
"capturedAt": "2026-08-28",
"seed": 1
},
{
"id": "targeted-s2",
"label": "Checks the hidden tells",
"path": "/traces/alert-triage/targeted-s2.json",
"kind": "generated",
"model": "targeted-analyst",
"capturedAt": "2026-08-28",
"seed": 2
},
{
"id": "targeted-s3",
"label": "Checks the hidden tells",
"path": "/traces/alert-triage/targeted-s3.json",
"kind": "generated",
"model": "targeted-analyst",
"capturedAt": "2026-08-28",
"seed": 3
},
{
"id": "targeted-s4",
"label": "Checks the hidden tells",
"path": "/traces/alert-triage/targeted-s4.json",
"kind": "generated",
"model": "targeted-analyst",
"capturedAt": "2026-08-28",
"seed": 4
},
{
"id": "targeted-s5",
"label": "Checks the hidden tells",
"path": "/traces/alert-triage/targeted-s5.json",
"kind": "generated",
"model": "targeted-analyst",
"capturedAt": "2026-08-28",
"seed": 5
},
{
"id": "targeted-s6",
"label": "Checks the hidden tells",
"path": "/traces/alert-triage/targeted-s6.json",
"kind": "generated",
"model": "targeted-analyst",
"capturedAt": "2026-08-28",
"seed": 6
},
{
"id": "targeted-s7",
"label": "Checks the hidden tells",
"path": "/traces/alert-triage/targeted-s7.json",
"kind": "generated",
"model": "targeted-analyst",
"capturedAt": "2026-08-28",
"seed": 7
},
{
"id": "targeted-s9",
"label": "Checks the hidden tells",
"path": "/traces/alert-triage/targeted-s9.json",
"kind": "generated",
"model": "targeted-analyst",
"capturedAt": "2026-08-28",
"seed": 9
},
{
"id": "thorough-s1",
"label": "Runs the full procedure",
"path": "/traces/alert-triage/thorough-s1.json",
"kind": "generated",
"model": "thorough-analyst",
"capturedAt": "2026-08-28",
"seed": 1
},
{
"id": "thorough-s2",
"label": "Runs the full procedure",
"path": "/traces/alert-triage/thorough-s2.json",
"kind": "generated",
"model": "thorough-analyst",
"capturedAt": "2026-08-28",
"seed": 2
},
{
"id": "thorough-s3",
"label": "Runs the full procedure",
"path": "/traces/alert-triage/thorough-s3.json",
"kind": "generated",
"model": "thorough-analyst",
"capturedAt": "2026-08-28",
"seed": 3
},
{
"id": "thorough-s4",
"label": "Runs the full procedure",
"path": "/traces/alert-triage/thorough-s4.json",
"kind": "generated",
"model": "thorough-analyst",
"capturedAt": "2026-08-28",
"seed": 4
},
{
"id": "thorough-s5",
"label": "Runs the full procedure",
"path": "/traces/alert-triage/thorough-s5.json",
"kind": "generated",
"model": "thorough-analyst",
"capturedAt": "2026-08-28",
"seed": 5
},
{
"id": "thorough-s6",
"label": "Runs the full procedure",
"path": "/traces/alert-triage/thorough-s6.json",
"kind": "generated",
"model": "thorough-analyst",
"capturedAt": "2026-08-28",
"seed": 6
},
{
"id": "thorough-s7",
"label": "Runs the full procedure",
"path": "/traces/alert-triage/thorough-s7.json",
"kind": "generated",
"model": "thorough-analyst",
"capturedAt": "2026-08-28",
"seed": 7
},
{
"id": "thorough-s9",
"label": "Runs the full procedure",
"path": "/traces/alert-triage/thorough-s9.json",
"kind": "generated",
"model": "thorough-analyst",
"capturedAt": "2026-08-28",
"seed": 9
}
],
"wordle": [
{
"id": "base-off-s0",
scripts/conformance.mjs
+308 -145
View File
@@ -2,31 +2,47 @@
/**
* The cross-language conformance gate.
*
* `envs/wordle_five/wordle_five/engine.py` is the reference implementation and
* `src/demos/wordle/engine.ts` is a port of it. The site's entire argument is
* that the board you play in the browser is the same environment the model was
* evaluated in, so the two implementations agreeing is not a nice-to-have — it
* is the claim.
* Every environment under `envs/` is the reference implementation, and the
* matching `src/demos/<slug>/engine.ts` is a port of it. The site's entire
* argument is that the board you play in the browser is the same environment
* the model was evaluated in, so the two implementations agreeing is not a
* nice-to-have — it is the claim. Each environment commits the digest(s) its
* Python computes to `envs/<pkg>/CONFORMANCE.txt`; this script recomputes them
* from the TypeScript and fails if any differ.
*
* ── wordle ──────────────────────────────────────────────────────────────────
* A hand-written vector file would only ever catch the cases somebody thought
* of, and the case nobody thinks of is always the same one: repeated letters,
* where a green must claim its letter before any yellow is assigned. So the
* gate is exhaustive instead. For every answer A in `words/answers.json`, in
* list order, concatenate `score(G, A)` for every guess G in that same list, in
* that same order, and stream the whole thing through SHA-256. That is the
* digest `engine.py`'s `conformance_digest()` computes, and it is the digest
* this script computes from the TypeScript.
* gate is exhaustive. For every answer A in `words/answers.json`, in list
* order, concatenate `score(G, A)` for every guess G in that same list, in that
* same order, and stream the whole thing through SHA-256. Both the list order
* and the two nested loop orders are part of the definition.
*
* Both the answer list order and the two nested loop orders are part of the
* definition. Sorting the list, deduplicating it, or swapping the loops all
* produce a different, equally valid, completely useless number.
* ── alert-triage ────────────────────────────────────────────────────────────
* Three digests, because three things can drift:
* world canonical JSON of the generated world for seeds 0–4095, with the
* reference minutes the shipped policies compute. One RNG draw out
* of order anywhere in the generator changes every seed after it.
* scorer `scoreExact` over a fixed grid of synthetic episodes, as exact
* fractions so no float formatting is hashed.
* protocol parse + engine over a committed corpus of reply strings —
* accepted actions, rejection reasons and the minute meter after
* every turn. JSON-in-prose with fences and a brace scan is where
* two runtimes disagree, and the rejection reasons quote the
* offending value with Python's `repr`, so this is the digest that
* catches a port that "mostly works".
*
* Usage:
* node scripts/conformance.mjs full digest, compared to CONFORMANCE.txt
* node scripts/conformance.mjs --limit 50 first 50 answers only, no comparison
* node scripts/conformance.mjs every environment, compared to its CONFORMANCE.txt
* node scripts/conformance.mjs --only wordle one environment
* node scripts/conformance.mjs --limit 50 wordle: first 50 answers; alert-triage: first 50 seeds
* of the world digest. No comparison.
*
* Exits 0 with a note (not an error) when `engine.ts` does not exist yet — the
* TypeScript port and this script are allowed to arrive in either order.
* An environment whose `engine.ts` does not exist yet is noted and skipped
* (exit 0) — the TypeScript port and this gate are allowed to arrive in
* either order. An environment whose CONFORMANCE.txt does not exist yet has
* its digests printed with the command that writes the file.
*/
import fs from 'node:fs';
@@ -36,142 +52,224 @@ import { spawnSync } from 'node:child_process';
import { abs, cyan, die, dim, exists, green, read, red, rel } from './_lib.mjs';
const ANSWERS = abs('envs', 'wordle_five', 'words', 'answers.json');
const EXPECTED_FILE = abs('envs', 'wordle_five', 'CONFORMANCE.txt');
/* ------------------------------------------------------------------ flags */
const limitFlag = process.argv.indexOf('--limit');
const limit = limitFlag === -1 ? 0 : Number(process.argv[limitFlag + 1] ?? 0);
const argv = process.argv.slice(2);
const limitFlag = argv.indexOf('--limit');
const limit = limitFlag === -1 ? 0 : Number(argv[limitFlag + 1] ?? 0);
if (limitFlag !== -1 && (!Number.isInteger(limit) || limit <= 0)) {
die('--limit takes a positive integer, e.g. `--limit 50`.');
}
const onlyFlag = argv.indexOf('--only');
const only = onlyFlag === -1 ? null : argv[onlyFlag + 1];
/* ------------------------------------------------------ find the TS engine */
function findEngine() {
const preferred = abs('src', 'demos', 'wordle', 'engine.ts');
if (exists(preferred)) return preferred;
const demosDir = abs('src', 'demos');
if (!exists(demosDir)) return null;
const found = fs
.readdirSync(demosDir, { withFileTypes: true })
.filter((e) => e.isDirectory() && !e.name.startsWith('_'))
.map((e) => path.join(demosDir, e.name, 'engine.ts'))
.filter(exists);
// More than one engine.ts and no `wordle` directory means there is nothing to
// guess between; say so rather than silently conforming the wrong one.
return found.length === 1 ? found[0] : null;
}
const engine = findEngine();
if (!engine) {
console.log(dim('conformance: engine not present yet — no src/demos/wordle/engine.ts to check.'));
console.log(dim('The TypeScript port and this gate may land in either order; re-run once it exists.'));
process.exit(0);
}
if (!exists(ANSWERS)) die(`${rel(ANSWERS)} does not exist. It defines both the pairs and their order.`);
if (!exists(abs('node_modules', '.bin', 'tsx'))) die('tsx is not installed, and the engine is TypeScript. Run `pnpm install`.');
/* ------------------------------------------------------------------- run it */
const probeDir = fs.mkdtempSync(path.join(os.tmpdir(), 'pig-conformance-'));
const probe = path.join(probeDir, 'digest.mts');
fs.writeFileSync(probe, probeSource(), 'utf8');
const started = Date.now();
console.log(dim(`conformance: hashing every (guess, answer) pair through ${rel(engine)}...`));
// The tsx bin is a POSIX shell shim, not a JS file, so it is the executable
// here rather than an argument to node.
const run = spawnSync(abs('node_modules', '.bin', 'tsx'), [probe, engine, ANSWERS, String(limit)], {
cwd: abs('.'),
encoding: 'utf8',
maxBuffer: 16 * 1024 * 1024,
timeout: 15 * 60_000,
// The probe lives in a temp directory, where tsx would never find the
// repo's tsconfig — and without it the `@/` alias in engine.ts does not
// resolve and the import fails for a reason that looks nothing like the cause.
env: { ...process.env, TSX_TSCONFIG_PATH: abs('tsconfig.json') },
stdio: ['ignore', 'pipe', 'inherit'],
});
fs.rmSync(probeDir, { recursive: true, force: true });
if (run.error) die(`could not run the digest probe: ${run.error.message}`);
if (run.status !== 0) die(`the digest probe exited ${run.status}. See the error above.`);
let result;
try {
result = JSON.parse(String(run.stdout).trim().split('\n').pop());
} catch {
die(`the digest probe produced no parseable result. Output was:\n${run.stdout}`);
}
if (result.error) {
die(
`${rel(engine)}: ${result.error}\n` +
' The gate needs a scoring function it can recognise. Export one named `scoreGuess` (or `score`), ' +
'taking (guess, answer) and returning the pattern — either a "GYXXG" string or an array of ' +
"'exact' | 'present' | 'absent' tiles. Exporting `conformanceDigest()` directly also works.",
);
}
const seconds = ((Date.now() - started) / 1000).toFixed(1);
console.log(` scored ${result.pairs.toLocaleString('en-US')} pairs from ${result.answers.toLocaleString('en-US')} answers in ${seconds}s`);
console.log(` via ${cyan(result.via)}`);
console.log(` digest ${cyan(result.digest)}`);
if (limit) {
console.log('');
console.log(dim(`Partial run (--limit ${limit}): not compared to ${rel(EXPECTED_FILE)}. Run without --limit for the real gate.`));
process.exit(0);
}
/* ----------------------------------------------------------- compare it */
if (!exists(EXPECTED_FILE)) {
console.log('');
console.log(green('conformance: computed, but there is nothing to compare against yet.'));
console.log(`Write the digest to ${rel(EXPECTED_FILE)} once the Python agrees:`);
console.log(dim(` python -c "from wordle_five.engine import conformance_digest; print(conformance_digest())"`));
process.exit(0);
}
const expected = read(EXPECTED_FILE).match(/\b[0-9a-f]{64}\b/)?.[0];
if (!expected) {
die(`${rel(EXPECTED_FILE)} contains no 64-character hex digest.`);
}
if (expected !== result.digest) {
console.error('');
console.error(`${red('FAIL')} the TypeScript engine and ${rel(EXPECTED_FILE)} disagree.`);
console.error(` expected ${expected}`);
console.error(` got ${result.digest}`);
console.error('');
console.error(
' One of the two implementations is wrong, and the board on the site is therefore not the\n' +
' environment the model was evaluated in. The usual culprit is the two-pass scoring: every\n' +
' green must claim its letter out of the pool BEFORE any yellow is assigned, or a guess like\n' +
' SASSY against BASIS marks an S yellow that the greens have already spent.\n' +
' If the Python changed on purpose, re-run its own `conformance_digest()` and update the file.',
);
process.exit(1);
}
console.log('');
console.log(green(`conformance: the TypeScript engine matches ${rel(EXPECTED_FILE)} exactly.`));
/* ------------------------------------------------------------------ probe */
/* ----------------------------------------------------------- environments */
/**
* Runs under tsx so it can import the TypeScript engine directly.
* @typedef {{
* name: string,
* engine: string,
* expected: string,
* args: string[],
* probe: () => string,
* writeHint: string,
* diagnose: string,
* describe: (result: any, seconds: string) => string[],
* }} Environment
*/
/** @type {Environment[]} */
const ENVIRONMENTS = [
{
name: 'wordle',
engine: abs('src', 'demos', 'wordle', 'engine.ts'),
expected: abs('envs', 'wordle_five', 'CONFORMANCE.txt'),
args: [abs('envs', 'wordle_five', 'words', 'answers.json')],
probe: wordleProbeSource,
writeHint: 'python -c "from wordle_five.engine import conformance_digest; print(conformance_digest())"',
diagnose:
' The usual culprit is the two-pass scoring: every green must claim its letter out of the pool\n' +
' BEFORE any yellow is assigned, or a guess like SASSY against BASIS marks an S yellow that the\n' +
' greens have already spent.',
describe: (result, seconds) => [
` scored ${result.pairs.toLocaleString('en-US')} pairs from ${result.answers.toLocaleString('en-US')} answers in ${seconds}s`,
` via ${cyan(result.via)}`,
],
},
{
name: 'alert-triage',
engine: abs('src', 'demos', 'alert-triage', 'engine.ts'),
expected: abs('envs', 'alert_triage', 'CONFORMANCE.txt'),
args: [abs('envs', 'alert_triage', 'conformance', 'replies.json')],
probe: triageProbeSource,
writeHint: 'uv run python -m alert_triage.conformance --write (from envs/alert_triage)',
diagnose:
' world: an RNG draw made out of order, or a `//` that became a `/`. Bisect by seed: dump\n' +
' canonicalJson(worldForSeed(s)) on both sides and diff the first seed that differs.\n' +
' scorer: scoreExact() — check the fraction forms and the best-alternate tie-break (first wins).\n' +
' protocol: the parse rules (fence wins; first balanced span; no rescanning) or a rejection\n' +
' reason whose repr() of the offending value differs from Python.',
describe: (result, seconds) => [
` hashed ${result.seeds.toLocaleString('en-US')} worlds, ${result.grid.toLocaleString('en-US')} grid episodes and ${result.cases} corpus cases in ${seconds}s`,
],
},
];
/* ------------------------------------------------------------------ run it */
if (!exists(abs('node_modules', '.bin', 'tsx'))) die('tsx is not installed, and the engines are TypeScript. Run `pnpm install`.');
const selected = ENVIRONMENTS.filter((env) => only === null || env.name === only);
if (only !== null && selected.length === 0) {
die(`--only ${only}: no such environment. Known: ${ENVIRONMENTS.map((e) => e.name).join(', ')}.`);
}
let failed = false;
for (const env of selected) {
if (!runEnvironment(env)) failed = true;
console.log('');
}
process.exit(failed ? 1 : 0);
/** @param {Environment} env */
function runEnvironment(env) {
if (!exists(env.engine)) {
console.log(dim(`conformance[${env.name}]: engine not present yet — no ${rel(env.engine)} to check.`));
console.log(dim('The TypeScript port and this gate may land in either order; re-run once it exists.'));
return true;
}
for (const arg of env.args) {
if (!exists(arg)) die(`${rel(arg)} does not exist. It is part of the definition of the ${env.name} digest.`);
}
const probeDir = fs.mkdtempSync(path.join(os.tmpdir(), 'pig-conformance-'));
const probe = path.join(probeDir, 'digest.mts');
fs.writeFileSync(probe, env.probe(), 'utf8');
const started = Date.now();
console.log(dim(`conformance[${env.name}]: hashing through ${rel(env.engine)}...`));
// The tsx bin is a POSIX shell shim, not a JS file, so it is the executable
// here rather than an argument to node.
const run = spawnSync(abs('node_modules', '.bin', 'tsx'), [probe, env.engine, ...env.args, String(limit)], {
cwd: abs('.'),
encoding: 'utf8',
maxBuffer: 16 * 1024 * 1024,
timeout: 15 * 60_000,
// The probe lives in a temp directory, where tsx would never find the
// repo's tsconfig — and without it the `@/` alias in an engine does not
// resolve and the import fails for a reason that looks nothing like the cause.
env: { ...process.env, TSX_TSCONFIG_PATH: abs('tsconfig.json') },
stdio: ['ignore', 'pipe', 'inherit'],
});
fs.rmSync(probeDir, { recursive: true, force: true });
if (run.error) die(`could not run the ${env.name} digest probe: ${run.error.message}`);
if (run.status !== 0) die(`the ${env.name} digest probe exited ${run.status}. See the error above.`);
let result;
try {
result = JSON.parse(String(run.stdout).trim().split('\n').pop());
} catch {
die(`the ${env.name} digest probe produced no parseable result. Output was:\n${run.stdout}`);
}
if (result.error) {
die(`${rel(env.engine)}: ${result.error}`);
}
const seconds = ((Date.now() - started) / 1000).toFixed(1);
for (const line of env.describe(result, seconds)) console.log(line);
/** @type {Record<string, string>} */
const digests = result.digests;
const names = Object.keys(digests);
for (const name of names) console.log(` ${names.length > 1 ? name.padEnd(9) : 'digest'} ${cyan(digests[name])}`);
if (limit) {
console.log('');
console.log(dim(`Partial run (--limit ${limit}): not compared to ${rel(env.expected)}. Run without --limit for the real gate.`));
return true;
}
/* ----------------------------------------------------------- compare */
if (!exists(env.expected)) {
console.log('');
console.log(green(`conformance[${env.name}]: computed, but there is nothing to compare against yet.`));
console.log(`Write the digest(s) to ${rel(env.expected)} once the Python agrees:`);
console.log(dim(` ${env.writeHint}`));
return true;
}
const expected = parseExpected(read(env.expected), names);
if (!expected) {
die(`${rel(env.expected)} contains no 64-character hex digest${names.length > 1 ? ` for each of: ${names.join(', ')}` : ''}.`);
}
const mismatched = names.filter((name) => expected[name] !== digests[name]);
if (mismatched.length) {
console.error('');
console.error(`${red('FAIL')} the TypeScript engine and ${rel(env.expected)} disagree${names.length > 1 ? ` on: ${mismatched.join(', ')}` : ''}.`);
for (const name of mismatched) {
const label = names.length > 1 ? ` (${name})` : '';
console.error(` expected${label} ${expected[name]}`);
console.error(` got${label} ${digests[name]}`);
}
console.error('');
console.error(
' One of the two implementations is wrong, and the board on the site is therefore not the\n' +
' environment the model was evaluated in.\n' +
env.diagnose +
'\n If the Python changed on purpose, re-run its own digest and update the file:\n' +
` ${env.writeHint}`,
);
return false;
}
console.log('');
console.log(green(`conformance[${env.name}]: the TypeScript engine matches ${rel(env.expected)} exactly${names.length > 1 ? ` (${names.join(', ')})` : ''}.`));
return true;
}
/**
* Reads the committed digests. A single-digest file is one hex string on a
* line; a multi-digest file is `hex name description` per line, and every
* name the probe produced must be present.
*
* Kept here as a string rather than as a checked-in `.mts`: it is an
* implementation detail of this script, and a stray TypeScript file in
* @param {string} text
* @param {string[]} names
* @returns {Record<string, string> | null}
*/
function parseExpected(text, names) {
/** @type {Record<string, string>} */
const out = {};
if (names.length === 1) {
const hex = text.match(/\b[0-9a-f]{64}\b/)?.[0];
if (!hex) return null;
out[names[0]] = hex;
return out;
}
for (const line of text.split('\n')) {
const m = line.match(/^\s*([0-9a-f]{64})\s+(\S+)/);
if (m) out[m[2]] = m[1];
}
return names.every((n) => n in out) ? out : null;
}
/* ------------------------------------------------------------------ probes */
/**
* Each probe runs under tsx so it can import the TypeScript engine directly,
* and prints one JSON line: `{ digests: { <name>: hex }, ...counts }` or
* `{ error }`.
*
* Kept here as strings rather than as checked-in `.mts` files: they are
* implementation details of this script, and a stray TypeScript file in
* `scripts/` would be swept into a typecheck it is deliberately outside of.
*
* A function declaration, not a `const`, so it is hoisted above the code that
* writes it to disk near the top of this file.
* Function declarations, not `const`s, so they are hoisted above the table
* that references them near the top of this file.
*/
function probeSource() {
function wordleProbeSource() {
return `
import { createHash } from 'node:crypto';
import { readFileSync } from 'node:fs';
@@ -195,7 +293,7 @@ const pool = limit > 0 ? answers.slice(0, limit) : answers;
// here — it is the implementation's own statement of the definition.
if (typeof mod.conformanceDigest === 'function' && limit === 0) {
const digest = await mod.conformanceDigest();
say({ digest, via: 'engine.conformanceDigest()', pairs: pool.length * pool.length, answers: pool.length });
say({ digests: { wordle: digest }, via: 'engine.conformanceDigest()', pairs: pool.length * pool.length, answers: pool.length });
process.exit(0);
}
@@ -209,7 +307,13 @@ const name =
(typeof mod.default === 'function' ? 'default' : null);
if (!name) {
say({ error: 'no scoring function is exported. Found: ' + Object.keys(mod).join(', ') });
say({
error:
'no scoring function is exported. Found: ' + Object.keys(mod).join(', ') +
'. The gate needs a scoring function it can recognise. Export one named scoreGuess (or score), ' +
'taking (guess, answer) and returning the pattern — either a "GYXXG" string or an array of ' +
"'exact' | 'present' | 'absent' tiles. Exporting conformanceDigest() directly also works.",
});
process.exit(0);
}
const score = mod[name];
@@ -271,6 +375,65 @@ for (let i = 0; i < pool.length; i += 1) {
}
if (process.stderr.isTTY) process.stderr.write(' '.repeat(40) + '\\r');
say({ digest: hash.digest('hex'), via: 'engine.' + name + '(guess, answer)', pairs: pool.length * pool.length, answers: pool.length });
say({ digests: { wordle: hash.digest('hex') }, via: 'engine.' + name + '(guess, answer)', pairs: pool.length * pool.length, answers: pool.length });
`;
}
/**
* The alert-triage probe. Three digests, each SHA-256 over newline-terminated
* canonical JSON lines, exactly as \`alert_triage/conformance.py\` defines
* them. The engine exports the lines; the probe only hashes them, so the
* engine stays free of node:crypto and browser-clean.
*/
function triageProbeSource() {
return `
import { createHash } from 'node:crypto';
import { readFileSync } from 'node:fs';
const [enginePath, corpusPath, limitRaw] = process.argv.slice(2);
const limit = Number(limitRaw ?? 0);
const say = (value: unknown) => console.log(JSON.stringify(value));
let mod: any;
try {
mod = await import(enginePath);
} catch (error: any) {
say({ error: 'could not import it: ' + String(error?.message ?? error).split('\\n')[0] });
process.exit(0);
}
for (const name of ['worldLines', 'scorerLines', 'protocolLines']) {
if (typeof mod[name] !== 'function') {
say({ error: 'does not export ' + name + '(). Found: ' + Object.keys(mod).join(', ') });
process.exit(0);
}
}
function digestOf(lines: Iterable<string>, label: string, total: number): [string, number] {
const hash = createHash('sha256');
let count = 0;
for (const line of lines) {
hash.update(line, 'utf8');
hash.update('\\n');
count += 1;
if (count % 256 === 0 && process.stderr.isTTY) process.stderr.write(' ' + label + ' ' + count + '/' + total + '\\r');
}
if (process.stderr.isTTY) process.stderr.write(' '.repeat(40) + '\\r');
return [hash.digest('hex'), count];
}
const SEEDS = 4096;
const seedCount = limit > 0 ? Math.min(limit, SEEDS) : SEEDS;
const seeds = Array.from({ length: seedCount }, (_, i) => i);
const corpus = JSON.parse(readFileSync(corpusPath, 'utf8'));
try {
const [world] = digestOf(mod.worldLines(seeds), 'world', seedCount);
const [scorer, grid] = digestOf(mod.scorerLines(), 'scorer', 0);
const [protocol, cases] = digestOf(mod.protocolLines(corpus), 'protocol', corpus.length);
say({ digests: { world, scorer, protocol }, seeds: seedCount, grid, cases });
} catch (error: any) {
say({ error: String(error?.stack ?? error?.message ?? error).split('\\n').slice(0, 3).join(' | ') });
}
`;
}
src/components/demo/DemoShell.tsx
+1 -1
View File
@@ -744,7 +744,7 @@ function ShellSkeleton() {
* The run a visitor sees before choosing one.
*
* `runs[0]` is the manifest's first entry — the weakest agent on seed 0, which
* for wordle is a failed game with thinking off. So the Watch tab opened on a
* for the first demo shipped was a failed game with thinking off. So the Watch tab opened on a
* loss with an empty reasoning panel and the Reward tab on a row of zeros:
* the model's least interesting attempt, chosen by accident of sort order.
*
src/demos/alert-triage/__tests__/engine.test.ts
+275
View File
@@ -0,0 +1,275 @@
import assert from 'node:assert/strict';
import { readFileSync } from 'node:fs';
import { test } from 'node:test';
import {
Engine,
XorShift32,
canonicalJson,
civilFromDays,
daysFromCivil,
daysInMonth,
extractCandidate,
fnv1a32,
generate,
isHeldOut,
isoDate,
parseReply,
protocolTrace,
pyJsonDumps,
pyJsonLoads,
pyRepr,
referenceFor,
runPolicy,
screenOf,
scoreExact,
worldForSeed,
POLICIES,
} from '../engine';
import type { CorpusCase } from '../engine';
/*
* The cross-language digest in envs/alert_triage/CONFORMANCE.txt is the real
* gate (`pnpm conformance`). These vectors exist so that a break is LOCATED,
* not merely detected: each one is pinned identically in
* envs/alert_triage/tests/, or was produced by the Python on the day the port
* landed, and names the primitive that drifted.
*/
/* ---------------------------------------------------------------- rng.py */
test('fnv1a32 vectors match test_triage_rng.py', () => {
assert.equal(fnv1a32('0'), 0x350ca8af);
assert.equal(fnv1a32('1'), 0x340ca71c);
assert.equal(fnv1a32('42'), 0x87e38583);
assert.equal(fnv1a32('4095'), 0x43875f5f);
});
test('xorshift32 from seed 0 is pinned', () => {
const rng = new XorShift32(fnv1a32('0'));
assert.deepEqual(Array.from({ length: 5 }, () => rng.next()), [2738490563, 3068243922, 3765331391, 3085691315, 2439018365]);
assert.notEqual(new XorShift32(0).state, 0);
});
test('sample is a distinct partial shuffle', () => {
const out = new XorShift32(7).sample([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], 4);
assert.equal(out.length, 4);
assert.equal(new Set(out).size, 4);
});
test('civil dates round-trip and hit the known ones', () => {
for (let day = daysFromCivil(1999, 12, 25); day < daysFromCivil(2030, 3, 2); day += 1) {
const [y, m, d] = civilFromDays(day);
assert.equal(daysFromCivil(y, m, d), day);
}
assert.equal(daysFromCivil(1970, 1, 1), 0);
assert.equal(daysFromCivil(2026, 3, 1), 20513);
assert.equal(daysFromCivil(2000, 2, 29), 11016);
assert.equal(isoDate(20000), '2024-10-04');
assert.equal(isoDate(0), '1970-01-01');
assert.equal(isoDate(-1), '1969-12-31');
assert.equal(daysInMonth(2024, 2), 29);
assert.equal(daysInMonth(2026, 2), 28);
assert.equal(daysInMonth(2026, 12), 31);
});
/* ---------------------------------------------------------- generator.py */
test('the held-out slice is fnv1a(seed) % 8 == 7', () => {
const heldOut = Array.from({ length: 64 }, (_, s) => s).filter(isHeldOut);
assert.deepEqual(heldOut, [0, 8, 11, 19, 20, 28, 37, 46, 55]);
});
// Produced by `world_for_seed` in Python the day the port landed. A different
// template or alert id here means an RNG draw moved; a different reference
// means a policy diverged.
const WORLD_VECTORS: [number, string, string, string | null, string, string, string, number, string, string][] = [
[0, 'B6', 'hidden', 'funnel', 'suspicious', 'A-9571', '2026-04-18', 123, 'targeted', 'Emeka Zapata'],
[1, 'B4', 'benign', null, 'benign', 'A-8971', '2026-04-26', 33, 'fast', 'Oakhill Kitchen Group'],
[2, 'B6', 'benign', null, 'benign', 'A-9760', '2026-04-23', 42, 'fast', 'Gabriela Kowalczyk'],
[3, 'B4', 'benign', null, 'benign', 'A-9453', '2026-04-23', 33, 'fast', 'Lakeside Kitchen LLC'],
[4, 'S1', 'visible', null, 'suspicious', 'A-8199', '2026-06-14', 33, 'fast', 'Bilal Quintero'],
[5, 'B3', 'benign', null, 'benign', 'A-6081', '2026-03-22', 42, 'fast', 'Andre Nakamura'],
[6, 'B1', 'benign', null, 'benign', 'A-7014', '2026-05-15', 33, 'fast', 'Lakeside Landscaping Inc.'],
[7, 'B6', 'benign', null, 'benign', 'A-5644', '2026-06-14', 42, 'fast', 'Leila Bergstrom'],
[8, 'S3', 'visible', null, 'suspicious', 'A-5828', '2026-04-24', 33, 'fast', 'Dana Yilmaz'],
[9, 'B3', 'hidden', 'funnel', 'suspicious', 'A-9537', '2026-04-19', 123, 'targeted', 'Samir Tanaka'],
[10, 'S1', 'visible', null, 'suspicious', 'A-5846', '2026-06-26', 33, 'fast', 'Malik Marchetti'],
[11, 'B3', 'benign', null, 'benign', 'A-9882', '2026-05-16', 42, 'fast', 'Naomi Abara'],
];
test('world vectors match the Python generator', () => {
for (const [seed, template, tier, overlay, label, alertId, fired, minutes, policy, name] of WORLD_VECTORS) {
const w = worldForSeed(seed);
assert.deepEqual(
[w.template, w.tier, w.overlay, w.label, w.alert.id, w.alert.fired, w.reference_minutes, w.reference_policy, w.customer.name],
[template, tier, overlay, label, alertId, fired, minutes, policy, name],
`seed ${seed}`,
);
}
});
test('the screen is byte-identical with a hidden overlay on or off', () => {
for (let seed = 0; seed < 64; seed += 1) {
const auto = generate(seed, 'auto');
if (auto.tier === 'visible') continue;
const twin = generate(seed, null);
assert.equal(canonicalJson(screenOf(auto)), canonicalJson(screenOf(twin)), `seed ${seed}`);
}
});
test('canonical JSON is sorted, compact and drops undefined', () => {
assert.equal(canonicalJson({ b: 1, a: [null, true, 'x'], c: undefined }), '{"a":[null,true,"x"],"b":1}');
assert.equal(canonicalJson('dash — “quotes”'), '"dash — “quotes”"');
assert.throws(() => canonicalJson(1.5));
});
/* ----------------------------------------------------------- protocol.py */
test('python repr vectors', () => {
const strings: [string, string][] = [
['a', "'a'"],
["it's", '"it\'s"'],
['say "hi"', "'say \"hi\"'"],
['it\'s "both"', "'it\\'s \"both\"'"],
['tab\there', "'tab\\there'"],
['nl\n', "'nl\\n'"],
['\x01\x7f\x80\xa0', "'\\x01\\x7f\\x80\\xa0'"],
['café 
 ​', "'café \\u2028 \\u200b'"],
['😀', "'😀'"],
['back\\slash', "'back\\\\slash'"],
];
for (const [input, expected] of strings) assert.equal(pyRepr(input), expected, JSON.stringify(input));
const numbers: [string, string][] = [
['1', '1'],
['-0', '0'],
['1.0', '1.0'],
['1.5', '1.5'],
['1e5', '100000.0'],
['1E16', '1e+16'],
['1e-5', '1e-05'],
['0.0001', '0.0001'],
['12345678901234567890', '12345678901234567890'],
['-2.5e-7', '-2.5e-07'],
['1e22', '1e+22'],
['123456789012345.6', '123456789012345.6'],
['1e400', 'inf'],
['0.1', '0.1'],
['100.0', '100.0'],
];
for (const [raw, expected] of numbers) assert.equal(pyRepr(pyJsonLoads(raw)), expected, raw);
assert.equal(pyRepr(null), 'None');
assert.equal(pyRepr(undefined), 'None');
assert.equal(pyRepr(true), 'True');
assert.equal(pyRepr(pyJsonLoads('[1, "a", null]')), "[1, 'a', None]");
assert.equal(pyRepr(pyJsonLoads('{"k": 2.0, "z": [true]}')), "{'k': 2.0, 'z': [True]}");
});
test('the JSON reader accepts what json.loads accepts and nothing more', () => {
assert.throws(() => pyJsonLoads('{"a":1,}'));
assert.throws(() => pyJsonLoads('{"a":NaN}'));
assert.throws(() => pyJsonLoads('{"a":Infinity}'));
assert.throws(() => pyJsonLoads('{"a":"x\ny"}'));
assert.throws(() => pyJsonLoads('{"a":1} {"b":2}'));
assert.throws(() => pyJsonLoads("{'a':1}"));
assert.equal(pyJsonLoads(' \t"\\ud83d\\ude00\\u00e9" \n'), '😀é');
assert.equal(pyJsonLoads('"\\ud800"'), '\ud800');
const obj = pyJsonLoads('{"__proto__": 1, "a": {"b": [1e2]}}');
assert.ok(obj instanceof Map);
assert.equal(pyRepr(obj), "{'__proto__': 1, 'a': {'b': [100.0]}}");
});
test('the pinned candidate rules', () => {
assert.equal(extractCandidate('{"a":1} ```json\n{"b":2}\n```'), '{"b":2}\n');
assert.equal(extractCandidate('text {"a":"}"} {"b":2}'), '{"a":"}"}');
assert.equal(extractCandidate('text {"a":"\\"}"} x'), '{"a":"\\"}"}');
assert.equal(extractCandidate('{"a":1'), null);
assert.equal(extractCandidate('no braces'), null);
assert.equal(extractCandidate('[{"a":1}]'), '{"a":1}');
assert.deepEqual(parseReply(''), [null, 'no JSON object found in the reply']);
assert.deepEqual(parseReply(null), [null, 'no JSON object found in the reply']);
assert.deepEqual(parseReply('{"a":1,} {"a":1}'), [null, 'the JSON object could not be parsed']);
assert.deepEqual(parseReply('```json\n[1]\n```'), [null, 'the reply parsed but is not a JSON object']);
});
test('pyJsonDumps writes what json.dumps writes', () => {
assert.equal(pyJsonDumps({ action: 'close', cites: ['a', 'b'], note: 'é\x7f"' }), '{"action": "close", "cites": ["a", "b"], "note": "\\u00e9\\u007f\\""}');
});
/* ------------------------------------------------------------- engine.py */
test('the rejection reasons quote Python repr', () => {
const engine = new Engine(worldForSeed(1));
assert.equal(engine.step('{"action":"Lookup"}').reason, "unknown action 'Lookup'; use lookup, close or escalate");
assert.equal(engine.step('{"what":"documents"}').reason, 'unknown action None; use lookup, close or escalate');
assert.equal(engine.step('{"action":"lookup","what":3.0}').reason, "unknown lookup 3.0; use history, counterparty, prior_alerts or documents");
assert.equal(engine.step('{"action":"lookup","what":"history","month":["x"]}').reason, "month ['x'] is not in the twelve-month window");
assert.equal(engine.step('{"action":"close","reason":"CONSISTENT_WITH_PROFILE","cites":["doc.x"]}').reason, 'cites an id you have not been shown: doc.x');
assert.equal(engine.turns, 5);
assert.equal(engine.rejected, 5);
assert.equal(engine.minutes, 30 + 5 * 3);
});
test('aborted is exactly eight turns; fewer is truncated and unscored', () => {
const aborted = new Engine(worldForSeed(3));
for (let i = 0; i < 8; i += 1) aborted.step('thinking');
assert.equal(aborted.outcome, 'aborted');
assert.deepEqual(scoreExact(aborted.episode()), { caught: [0, 1], hours: [0, 1], evidence: [0, 1] });
const truncated = new Engine(worldForSeed(3));
truncated.step('thinking');
assert.equal(truncated.outcome, null);
assert.deepEqual(scoreExact(truncated.episode()), { caught: null, hours: null, evidence: null });
});
test('the reference is the cheapest correct shipped policy and never null', () => {
for (let seed = 0; seed < 96; seed += 1) {
const world = generate(seed);
const [minutes, policy] = referenceFor(world);
assert.notEqual(minutes, null, `seed ${seed}`);
assert.equal(runPolicy(POLICIES[policy!]!, world).minutes, minutes);
if (world.tier === 'hidden') {
assert.notEqual(runPolicy(POLICIES.fast!, world).outcome, 'solved', `fast cannot see the hidden signal on seed ${seed}`);
assert.equal(runPolicy(POLICIES.targeted!, world).outcome, 'solved', `targeted catches seed ${seed}`);
}
}
});
test('the committed corpus replays to the same rejection reasons the Python recorded', () => {
const corpus: CorpusCase[] = JSON.parse(
readFileSync(new URL('../../../../envs/alert_triage/conformance/replies.json', import.meta.url), 'utf8'),
);
const reasons = protocolTrace(corpus)
.flatMap((entry) => entry.turns as { reason: string | null }[])
.map((t) => t.reason)
.filter((r): r is string => r !== null);
assert.deepEqual(reasons, [
"month 'BADMONTH' is not in the twelve-month window",
'the JSON object could not be parsed',
"unknown close reason 'consistent_with_profile'",
"unknown action 'Lookup'; use lookup, close or escalate",
'cites must be a non-empty list of ids',
'cites must be a non-empty list of ids',
"unknown typology 'structuring'",
'cites must be a non-empty list of ids',
'prior_alerts has already been looked up',
"counterparty 'CP-99' has not appeared on your screen or in a fetched month",
"counterparty 'cp-1' has not appeared on your screen or in a fetched month",
"month '2024-01' is not in the twelve-month window",
"month '2026-1' is not in the twelve-month window",
'cites an id you have not been shown: doc.probate_letter',
'no JSON object found in the reply',
'no JSON object found in the reply',
'no JSON object found in the reply',
'the JSON object could not be parsed',
'the reply parsed but is not a JSON object',
'unknown action None; use lookup, close or escalate',
'no JSON object found in the reply',
'unknown action None; use lookup, close or escalate',
"counterparty 'CP-3' has not appeared on your screen or in a fetched month",
'no JSON object found in the reply',
"unknown lookup 'emails'; use history, counterparty, prior_alerts or documents",
]);
});
src/demos/alert-triage/__tests__/game.test.ts
+112
View File
@@ -0,0 +1,112 @@
import assert from 'node:assert/strict';
import { test } from 'node:test';
import { describe } from '../adapter';
import { Engine, isHeldOut, worldForSeed } from '../engine';
import { initState, play, replay } from '../game';
import { thorough, closeBlind, inaction } from '../policies';
/**
* The seam, exercised the way the shell exercises it: a fresh board per
* seed, one reply at a time through `play()`, and a recording through
* `replay()`. The engine's own conformance is the port's business; these
* check that the board never shows what it must not and always agrees with
* the engine it wraps.
*/
const SEEDS = Array.from({ length: 48 }, (_, i) => i);
test('a fresh board shows the screen, the free-screen hours, and no answer', () => {
for (const seed of SEEDS) {
const state = initState(seed);
assert.equal(state.seed, seed);
assert.equal(state.reveal, null, `seed ${seed} leaked the label before a disposition`);
assert.equal(state.status, 'playing');
assert.equal(state.turnsSpent, 0);
assert.equal(state.minutesSpent, 30);
assert.equal(state.lookups.length, 0);
assert.equal(state.screen.summary.length, 12);
assert.ok(state.shown.includes(state.screen.alert.id));
assert.equal(state.screen.screen_order, isHeldOut(seed) ? 'kyc_first' : 'alert_first');
}
});
test('the thorough analyst, played through play(), reaches the right call on every seed', () => {
for (const seed of SEEDS) {
let state = initState(seed);
let guard = 0;
while (state.status === 'playing' && guard < 12) {
// The policy reads the engine's view; rebuild the same view off the board.
const view = {
screen: state.screen,
lookups: state.lookups,
turns_spent: state.turnsSpent,
turns_left: 8 - state.turnsSpent,
rejected: state.rejected,
minutes_spent: state.minutesSpent,
last_rejection: state.lastRejection,
};
state = play(state, thorough(view));
guard += 1;
}
assert.equal(state.status === 'closed' || state.status === 'escalated', true, `seed ${seed} never decided`);
assert.equal(state.outcome, 'solved', `seed ${seed}: thorough got it wrong`);
assert.ok(state.reveal, `seed ${seed} has no reveal after the episode ended`);
assert.equal(state.reveal?.label, worldForSeed(seed).label);
assert.equal(state.reveal?.referenceMinutes, worldForSeed(seed).reference_minutes);
assert.equal(state.rejected, 0);
// Playing on after the end is a no-op, never a new board.
assert.equal(play(state, '{"action":"lookup","what":"documents"}'), state);
}
});
test('replay() agrees with the engine step for step and one snapshot per reply', () => {
for (const seed of SEEDS.slice(0, 16)) {
const engine = new Engine(worldForSeed(seed));
const replies: string[] = [];
while (!engine.done) {
const reply = thorough(engine.view());
replies.push(reply);
engine.step(reply);
}
const snapshots = replay(seed, replies);
assert.equal(snapshots.length, replies.length);
const final = snapshots[snapshots.length - 1]!;
assert.equal(final.minutesSpent, engine.minutes);
assert.equal(final.turnsSpent, engine.turns);
assert.deepEqual(final.disposition, engine.disposition);
assert.deepEqual(final.shown.sort(), [...engine.shown].sort());
// A fixture's own reference wins over the world's for the reveal.
assert.equal(replay(seed, replies, { referenceMinutes: 99 }).at(-1)?.reveal?.referenceMinutes, 99);
}
});
test('refusals cost a turn, are announced, and eight of them abort', () => {
let state = initState(3);
state = play(state, inaction());
assert.equal(state.turnsSpent, 1);
assert.equal(state.rejected, 1);
assert.equal(state.minutesSpent, 33);
assert.ok(state.lastRejection);
assert.equal(state.reveal, null);
assert.match(describe(state, initState(3), 0).announce, /refused/);
for (let i = 1; i < 8; i += 1) state = play(state, inaction());
assert.equal(state.status, 'aborted');
assert.equal(state.outcome, 'aborted');
assert.ok(state.reveal, 'an aborted episode still reveals the answer');
assert.match(describe(state, null, 7).announce, /stays in the queue/);
});
test('citing an id never shown is refused, and a blind close is scored against the label', () => {
const state = play(initState(5), '{"action":"close","reason":"DOCUMENTED_SOURCE_OF_FUNDS","cites":["doc.probate_letter"],"note":""}');
assert.equal(state.rejected, 1);
assert.match(state.lastRejection ?? '', /not been shown/);
const blind = play(initState(5), closeBlind());
assert.equal(blind.status, 'closed');
assert.equal(blind.outcome, worldForSeed(5).label === 'benign' ? 'solved' : 'failed');
const step = describe(blind, initState(5), 0);
assert.equal(step.caption, 'closed');
assert.match(step.announce, /citing 1 id/);
});
src/demos/alert-triage/__tests__/scorer.test.ts
+73
View File
@@ -0,0 +1,73 @@
import assert from 'node:assert/strict';
import { test } from 'node:test';
import { bestAlternate } from '../grade';
import { evidence, f1Exact, score, type Episode } from '../score';
/**
* Vectors the page's copy relies on, checked against the browser scorer. The
* cross-language digest proves the port matches the Python; these prove the
* sentences on the Reward tab match the port.
*/
const base: Episode = {
label: 'benign',
planted: [['kyc.expected_monthly_cash'], ['kyc.expected_monthly_cash', 'kyc.business_type']],
disposition: 'close',
cites: ['kyc.expected_monthly_cash'],
minutes_spent: 33,
reference_minutes: 33,
turns_spent: 1,
rejected: 0,
truncated: false,
};
test('a correct, well-cited close at the reference scores 1.0 everywhere', () => {
assert.deepEqual(score(base), { caught: 1, hours: 1, evidence: 1 });
});
test('a missed escalation scores exactly 0.000 on every component', () => {
assert.deepEqual(score({ ...base, label: 'suspicious', planted: [['T-1', 'T-2']] }), {
caught: 0,
hours: 0,
evidence: 0,
});
});
test('a false escalation earns the credit on caught and nothing else', () => {
assert.deepEqual(score({ ...base, disposition: 'escalate', cites: ['T-1'] }), {
caught: 0.5,
hours: 0,
evidence: 0,
});
});
test('no disposition is zero, not null; a truncated run is null, not zero', () => {
assert.deepEqual(score({ ...base, disposition: null, cites: [] }), { caught: 0, hours: 0, evidence: 0 });
assert.deepEqual(score({ ...base, truncated: true }), { caught: null, hours: null, evidence: null });
});
test('hours is capped at one and null without a reference', () => {
assert.equal(score({ ...base, minutes_spent: 20 }).hours, 1);
assert.equal(score({ ...base, minutes_spent: 66 }).hours, 0.5);
assert.equal(score({ ...base, reference_minutes: null }).hours, null);
});
test('precision bites: one planted id among eight cited is F1 0.22', () => {
const cites = ['kyc.expected_monthly_cash', 'T-1', 'T-2', 'T-3', 'T-4', 'T-5', 'T-6', 'T-7'];
assert.deepEqual(f1Exact(cites, ['kyc.expected_monthly_cash']), [2, 9]);
assert.equal(evidence({ ...base, cites }), 2 / 9);
});
test('the best alternate is scored, and the board can recover which one', () => {
const cites = ['kyc.expected_monthly_cash', 'kyc.business_type'];
assert.deepEqual(bestAlternate(cites, base.planted), base.planted[1]);
assert.equal(evidence({ ...base, cites }), 1);
// A tie goes to the first alternate, as the scorer's strict `>` does.
assert.deepEqual(bestAlternate(['T-9'], base.planted), base.planted[0]);
});
test('planted ids are matched exact-case and an unshown id simply misses', () => {
assert.equal(evidence({ ...base, cites: ['KYC.EXPECTED_MONTHLY_CASH'] }), 0);
assert.equal(evidence({ ...base, cites: ['doc.probate_letter', 'kyc.expected_monthly_cash'] }), 2 / 3);
});
src/demos/alert-triage/adapter.ts
+138
View File
@@ -0,0 +1,138 @@
import type { DemoEpisode, DemoStep } from '@/lib/demo-kit';
import { hours } from './format';
import { replay } from './game';
import type { TriageState } from './types';
/**
* A recorded episode becomes a list of board snapshots.
*
* Nothing is read off the fixture but the seed and the reply strings. The
* world is regenerated from the seed and every reply is stepped through the
* same engine the environment ran, so the board you scrub through is what the
* engine did with what the model said — not what the capture script wrote
* down about it. A fixture whose `info` disagrees with its own replies cannot
* fool this adapter, because the adapter never looks.
*
* `reference_hours` is the one metric read, and only for the reveal: it comes
* from running the shipped analysts, which the browser has no business doing.
* Its absence leaves the reveal without a reference, never with a zero.
*
* Snapshots, not deltas: the scrubber jumps to any step.
*/
export function adapt(episode: DemoEpisode): DemoStep<TriageState>[] {
const referenceHours = episode.metrics?.['reference_hours'];
const referenceMinutes = typeof referenceHours === 'number' && Number.isFinite(referenceHours)
? Math.round(referenceHours * 60)
: null;
const snapshots = replay(
episode.seed,
episode.turns.map((turn) => turn.reply),
{ referenceMinutes },
);
let previous: TriageState | null = null;
return episode.turns.map((turn, index) => {
const state = snapshots[index] ?? snapshots[snapshots.length - 1];
if (!state) {
// Cannot happen: replay returns one snapshot per reply. Typed defensively
// so a broken port degrades to a readable step rather than a crash.
throw new Error(`replay produced no snapshot for turn ${index + 1}`);
}
const { announce, caption } = describe(state, previous, index);
previous = state;
return {
index,
state,
reply: turn.reply,
reasoning: turn.reasoning,
call: turn.call,
announce,
caption,
};
});
}
/**
* What a screen reader hears when a step lands, and the six words under the
* chip. Derived from the change between snapshots, so it says what the engine
* did rather than what the reply claimed.
*/
export function describe(
state: TriageState,
previous: TriageState | null,
index: number,
): { announce: string; caption: string } {
const turn = `Turn ${index + 1}`;
const meter = `${hours(state.minutesSpent)} hours spent.`;
const wasDone = previous !== null && previous.status !== 'playing';
if (wasDone) {
return { announce: `${turn}: the episode was already over.`, caption: 'after the end' };
}
if (state.lastRejection !== null && state.turnsSpent === (previous?.turnsSpent ?? 0) + 1 && state.lookups.length === (previous?.lookups.length ?? 0) && state.disposition === null) {
const why = state.lastRejection;
return {
announce: `${turn} refused: ${why}. It still cost a turn. ${meter}`,
caption: `refused: ${shorten(why)}`,
};
}
const d = state.disposition;
if (d !== null) {
const what = d.action === 'close' ? `closed, ${d.reason.toLowerCase().replace(/_/g, ' ')}` : `escalated as ${d.typology.toLowerCase().replace(/_/g, ' ')}`;
const cites = `citing ${d.cites.length} ${d.cites.length === 1 ? 'id' : 'ids'}`;
const verdict =
state.outcome === 'solved'
? 'Right call.'
: state.reveal?.label === 'suspicious'
? 'Missed: the alert was suspicious.'
: 'A clean customer was escalated.';
return {
announce: `${turn}: ${what}, ${cites}. ${verdict} ${meter}`,
caption: d.action === 'close' ? 'closed' : 'escalated',
};
}
if (state.status === 'aborted') {
return {
announce: `${turn}: no disposition in eight replies. The alert stays in the queue. ${meter}`,
caption: 'aborted',
};
}
const lookup = state.lookups[state.lookups.length - 1];
if (lookup && state.lookups.length > (previous?.lookups.length ?? 0)) {
switch (lookup.what) {
case 'history':
return {
announce: `${turn}: looked up ${lookup.key}, ${lookup.result.transactions.length} transactions. ${meter}`,
caption: `history ${lookup.key}`,
};
case 'counterparty':
return {
announce: `${turn}: looked up ${lookup.key}, ${lookup.result.name}, shared with ${lookup.result.other_alerted_accounts} other alerted accounts. ${meter}`,
caption: `counterparty ${lookup.key}`,
};
case 'prior_alerts':
return {
announce: `${turn}: looked up prior alerts, ${lookup.result.alerts.length} on file. ${meter}`,
caption: 'prior alerts',
};
case 'documents':
return {
announce: `${turn}: looked up documents, ${lookup.result.documents.length} on file. ${meter}`,
caption: 'documents',
};
}
}
return { announce: `${turn}: nothing changed. ${meter}`, caption: 'no change' };
}
function shorten(text: string): string {
return text.length > 40 ? `${text.slice(0, 37)}…` : text;
}
src/demos/alert-triage/controls.tsx
+354
View File
@@ -0,0 +1,354 @@
import { useId, useMemo, useState } from 'react';
import { Button } from '@/components/ui/button';
import { cn } from '@/lib/utils';
import { enumLabel, fieldLabel, hours } from './format';
import { MAX_TURNS, PRICES, play } from './game';
import { DOC_MINUTES, LOOKUP_MINUTES, TURN_MINUTES } from './score';
import { CLOSE_REASONS, TYPOLOGIES, type CloseReason, type TriageState, type Typology } from './types';
/**
* The analyst's controls: four lookups with their prices on the button, and
* a disposition with a citation checklist.
*
* Every control produces exactly the JSON object a model would have to
* produce, and hands it to the same engine. The visitor never sees a control
* the model does not have, and the reply the engine received is shown under
* the controls so the action set is learned by using it. The controls own no
* state the board does not: the next board comes back from `play()` and goes
* to `onChange`, and the only local state here is what is half-typed.
*/
type Panel = 'none' | 'close' | 'escalate';
const price = (minutes: number) => `+${hours(minutes)} h`;
function Section({ title, children }: { title: string; children: React.ReactNode }) {
return (
<fieldset className="min-w-0 rounded-lg border border-border p-3">
<legend className="px-1 text-xs font-semibold uppercase tracking-wide text-muted">{title}</legend>
{children}
</fieldset>
);
}
const SELECT =
'tap min-w-0 rounded-md border border-border bg-surface px-2 text-sm text-fg focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-brand disabled:opacity-50';
/** The citable ids, grouped the way the screen groups them. */
function groupShown(shown: readonly string[]) {
const groups: { label: string; ids: string[] }[] = [
{ label: 'KYC file', ids: [] },
{ label: 'Transactions', ids: [] },
{ label: 'Counterparties', ids: [] },
{ label: 'Prior alerts', ids: [] },
{ label: 'Documents', ids: [] },
{ label: 'Other', ids: [] },
];
const byId = (a: string, b: string) => {
const na = Number(a.replace(/^\D+/, ''));
const nb = Number(b.replace(/^\D+/, ''));
return Number.isFinite(na) && Number.isFinite(nb) && na !== nb ? na - nb : a.localeCompare(b);
};
for (const id of shown) {
const index = id.startsWith('kyc.')
? 0
: id.startsWith('T-')
? 1
: id.startsWith('CP-')
? 2
: id.startsWith('A-')
? 3
: id.startsWith('doc.')
? 4
: 5;
groups[index]!.ids.push(id);
}
for (const group of groups) group.ids.sort(byId);
return groups.filter((group) => group.ids.length > 0);
}
export function Controls({
state,
onChange,
}: {
state: TriageState;
onChange: (next: TriageState) => void;
}) {
const ids = useId();
const [month, setMonth] = useState('');
const [cp, setCp] = useState('');
const [panel, setPanel] = useState<Panel>('none');
const [reason, setReason] = useState<CloseReason>('CONSISTENT_WITH_PROFILE');
const [typology, setTypology] = useState<Typology>('STRUCTURING');
const [cites, setCites] = useState<string[]>([]);
const [prose, setProse] = useState('');
const [lastReply, setLastReply] = useState<string | null>(null);
const done = state.status !== 'playing';
const left = MAX_TURNS - state.turnsSpent;
const send = (object: Record<string, unknown>) => {
const reply = JSON.stringify(object);
setLastReply(reply);
onChange(play(state, reply));
};
const monthsLeft = useMemo(
() => state.screen.months.filter((m) => !state.lookups.some((l) => l.what === 'history' && l.key === m)),
[state.screen.months, state.lookups],
);
const cpsLeft = useMemo(
() =>
state.shown
.filter((id) => id.startsWith('CP-'))
.filter((id) => !state.lookups.some((l) => l.what === 'counterparty' && l.key === id))
.sort((a, b) => Number(a.slice(3)) - Number(b.slice(3))),
[state.shown, state.lookups],
);
const priorsMade = state.lookups.some((l) => l.what === 'prior_alerts');
const docsMade = state.lookups.some((l) => l.what === 'documents');
const groups = useMemo(() => groupShown(state.shown), [state.shown]);
const toggleCite = (id: string) =>
setCites((current) => (current.includes(id) ? current.filter((c) => c !== id) : [...current, id]));
const submitDisposition = () => {
if (cites.length === 0 || done) return;
if (panel === 'close') send({ action: 'close', reason, cites, note: prose });
else if (panel === 'escalate') send({ action: 'escalate', typology, cites, narrative: prose });
setPanel('none');
setCites([]);
setProse('');
};
const status = done
? state.status === 'aborted'
? `No disposition in ${MAX_TURNS} replies. The alert stays in the queue.`
: `Disposition recorded after ${hours(state.minutesSpent)} hours. The board shows what the generator planted.`
: state.lastRejection
? `That reply was not accepted: ${state.lastRejection}. It still cost a turn. ${left} ${left === 1 ? 'reply' : 'replies'} left.`
: `Hours spent so far: ${hours(state.minutesSpent)}. ${left} ${left === 1 ? 'reply' : 'replies'} left.`;
return (
<div className="flex flex-col gap-3" style={{ paddingBottom: 'max(0px, var(--safe-bottom))' }}>
{/* The board is a group of tables; the outcome of every click is said
here in words, where it announces itself. */}
<p aria-live="polite" className="text-sm text-muted">
{status}
</p>
<Section title={`Lookups — each reply also costs ${price(TURN_MINUTES)}`}>
<div className="grid grid-cols-1 gap-2">
<div className="flex min-w-0 gap-2">
<label className="sr-only" htmlFor={`${ids}-month`}>
Month to itemise
</label>
<select
id={`${ids}-month`}
className={cn(SELECT, 'min-w-[7rem] flex-1')}
value={month}
disabled={done || monthsLeft.length === 0}
onChange={(event) => setMonth(event.target.value)}
>
<option value="">{monthsLeft.length === 0 ? 'every month itemised' : 'month…'}</option>
{monthsLeft.map((m) => (
<option key={m} value={m}>
{m}
</option>
))}
</select>
<Button
type="button"
variant="outline"
size="touch"
disabled={done || month === ''}
onClick={() => {
send({ action: 'lookup', what: 'history', month });
setMonth('');
}}
>
History <span className="nums text-muted">{price(PRICES.history)}</span>
</Button>
</div>
<div className="flex min-w-0 gap-2">
<label className="sr-only" htmlFor={`${ids}-cp`}>
Counterparty to look up
</label>
<select
id={`${ids}-cp`}
className={cn(SELECT, 'min-w-[7rem] flex-1')}
value={cp}
disabled={done || cpsLeft.length === 0}
onChange={(event) => setCp(event.target.value)}
>
<option value="">{cpsLeft.length === 0 ? 'no counterparty left' : 'counterparty…'}</option>
{cpsLeft.map((id) => (
<option key={id} value={id}>
{id}
</option>
))}
</select>
<Button
type="button"
variant="outline"
size="touch"
disabled={done || cp === ''}
onClick={() => {
send({ action: 'lookup', what: 'counterparty', id: cp });
setCp('');
}}
>
Counterparty <span className="nums text-muted">{price(PRICES.counterparty)}</span>
</Button>
</div>
<div className="flex flex-wrap gap-2">
<Button
type="button"
variant="outline"
size="touch"
disabled={done || priorsMade}
onClick={() => send({ action: 'lookup', what: 'prior_alerts' })}
>
Prior alerts <span className="nums text-muted">{priorsMade ? 'done' : price(LOOKUP_MINUTES)}</span>
</Button>
<Button
type="button"
variant="outline"
size="touch"
disabled={done || docsMade}
onClick={() => send({ action: 'lookup', what: 'documents' })}
>
Documents <span className="nums text-muted">{docsMade ? 'done' : price(DOC_MINUTES)}</span>
</Button>
</div>
</div>
</Section>
<Section title="Disposition — ends the episode">
<div className="flex flex-wrap gap-2" role="group" aria-label="Choose a disposition">
<Button
type="button"
size="touch"
variant={panel === 'close' ? 'default' : 'outline'}
aria-pressed={panel === 'close'}
disabled={done}
onClick={() => setPanel(panel === 'close' ? 'none' : 'close')}
>
Close
</Button>
<Button
type="button"
size="touch"
variant={panel === 'escalate' ? 'default' : 'outline'}
aria-pressed={panel === 'escalate'}
disabled={done}
onClick={() => setPanel(panel === 'escalate' ? 'none' : 'escalate')}
>
Escalate
</Button>
</div>
{panel !== 'none' && !done ? (
<form
className="mt-3 flex flex-col gap-3"
onSubmit={(event) => {
event.preventDefault();
submitDisposition();
}}
>
<label className="flex flex-col gap-1 text-xs text-muted">
{panel === 'close' ? 'Reason' : 'Typology'}
{panel === 'close' ? (
<select className={SELECT} value={reason} onChange={(event) => setReason(event.target.value as CloseReason)}>
{CLOSE_REASONS.map((r) => (
<option key={r} value={r}>
{enumLabel(r)}
</option>
))}
</select>
) : (
<select className={SELECT} value={typology} onChange={(event) => setTypology(event.target.value as Typology)}>
{TYPOLOGIES.map((t) => (
<option key={t} value={t}>
{enumLabel(t)}
</option>
))}
</select>
)}
</label>
<fieldset className="min-w-0">
<legend className="mb-1 text-xs text-muted">
Cite the facts that decide it — only ids you have been shown
</legend>
<div className="flex max-h-56 flex-col gap-2 overflow-y-auto rounded-md border border-border p-2">
{groups.map((group) => (
<div key={group.label}>
<p className="mb-1 text-[0.65rem] font-semibold uppercase tracking-wide text-muted">{group.label}</p>
<div className="flex flex-wrap gap-1">
{group.ids.map((id) => {
const on = cites.includes(id);
return (
<label
key={id}
className={cn(
'inline-flex cursor-pointer select-none items-center gap-1 rounded-md border px-1.5 py-1 font-mono text-[0.7rem] transition-colors duration-1',
on ? 'border-brand bg-accent-subtle text-fg' : 'border-border text-muted hover:bg-surface-2',
)}
title={id.startsWith('kyc.') ? fieldLabel(id) : id}
>
<input type="checkbox" className="sr-only" checked={on} onChange={() => toggleCite(id)} />
{id}
</label>
);
})}
</div>
</div>
))}
</div>
</fieldset>
<label className="flex flex-col gap-1 text-xs text-muted">
<span>
{panel === 'close' ? 'Note' : 'Narrative'}{' '}
<span className="rounded border border-border px-1 py-px text-[0.65rem] uppercase tracking-wide">
displayed, never scored
</span>
</span>
<textarea
className="min-h-16 rounded-md border border-border bg-surface px-2 py-1 text-sm text-fg focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-brand"
value={prose}
onChange={(event) => setProse(event.target.value)}
placeholder={panel === 'close' ? 'Why this is consistent with the file…' : 'What the pattern is and why it is reportable…'}
/>
</label>
<div className="flex flex-wrap items-center gap-2">
<Button type="submit" size="touch" disabled={cites.length === 0}>
{panel === 'close' ? 'Close the alert' : 'Escalate for filing'}
</Button>
<span className="nums text-xs text-muted">
{cites.length} cited · {price(TURN_MINUTES)}
</span>
</div>
</form>
) : null}
</Section>
{lastReply ? (
<details className="text-xs text-muted">
<summary className="cursor-pointer">The reply the engine received</summary>
<pre className="mt-1 overflow-x-auto rounded-md bg-surface-2 p-2 font-mono text-[0.7rem] text-fg [contain:inline-size]">
{lastReply}
</pre>
</details>
) : null}
</div>
);
}
export default Controls;
src/demos/alert-triage/demo.tsx
+92
View File
@@ -0,0 +1,92 @@
import { defineDemo, type DemoEpisode, type RewardValues } from '@/lib/demo-kit';
import { adapt } from './adapter';
import Controls from './controls';
import { initState, replay } from './game';
import meta from './meta';
import { narrative } from './narrative';
import { reward } from './reward';
import { score } from './score';
import Board from './surface';
import type { TriageState } from './types';
export default defineDemo<TriageState>({
meta,
narrative,
reward,
anatomy: {
task:
'Triage one transaction-monitoring alert on a synthetic customer: read the screen, pay analyst-hours for any lookups, then close it with a reason or escalate it with a typology, citing the facts.',
actions:
'One JSON object per reply — a lookup (a month’s history, a counterparty, prior alerts, documents) or a disposition (close with a reason, escalate with a typology), citing only ids you were shown. Anything else is refused and still costs a turn.',
grader:
'Regenerates the alert from the seed, compares the disposition and the cited ids to what the generator planted, and charges the hours. Deterministic Python: no model, no human, no prose read anywhere.',
score:
'Over half for nothing slipping through, a quarter for clearing it within the reference hours, a fifth for citing the facts that decide it. A missed escalation zeroes all three.',
},
provenance: {
envPackage: 'alert_triage',
tasksetId: 'alert-triage',
verifiersVersion: '0.3.2.dev12',
command: 'uv run python envs/probe.py --taskset alert-triage',
credits: [
{
label: 'FinCEN — guidance on preparing a complete and sufficient SAR narrative (the templates are written from public typology descriptions, not from any case)',
href: 'https://www.fincen.gov/sites/default/files/shared/sarnarrcompletguidfinal_112003.pdf',
},
{
label: 'FATF — methods and trends (typology reports)',
href: 'https://www.fatf-gafi.org/en/topics/methods-and-trends.html',
},
{
label: 'verifiers — the environment API this mirrors',
href: 'https://github.com/PrimeIntellect-ai/verifiers',
},
{
label: 'Our synthetic name and word lists, and where they came from',
href: 'https://git.karti.ai/PIG/PIG-Demo/src/branch/main/envs/alert_triage/names/PROVENANCE.md',
},
],
},
adapt,
Surface: Board,
interactive: {
init: initState,
Controls: Controls,
},
/**
* Re-derive the score from the replies alone.
*
* The world is regenerated from the seed and the replies are stepped through
* the engine; the label, the planted set and the hours all come out of that,
* never off the fixture. The one thing read from the recording is
* `reference_hours` — it comes from running the shipped analysts, which the
* browser has no business doing — and a missing one makes the run
* *unverifiable* rather than wrong.
*/
verify: (episode: DemoEpisode): RewardValues | null => {
const referenceHours = episode.metrics?.['reference_hours'];
const referenceMinutes =
typeof referenceHours === 'number' && Number.isFinite(referenceHours) ? Math.round(referenceHours * 60) : null;
const snapshots = replay(episode.seed, episode.turns.map((turn) => turn.reply), { referenceMinutes });
const final = snapshots[snapshots.length - 1];
// A run with no replies, or one that stopped before a disposition and
// before the eighth turn, never reached a terminal state: not scored.
if (!final || final.reveal === null) return null;
const d = final.disposition;
if (final.status === 'playing') return null;
return score({
label: final.reveal.label,
planted: final.reveal.planted,
disposition: d === null ? null : d.action,
cites: d === null ? [] : d.cites,
minutes_spent: final.minutesSpent,
reference_minutes: referenceMinutes,
turns_spent: final.turnsSpent,
rejected: final.rejected,
truncated: false,
});
},
});
src/demos/alert-triage/engine.ts
+406
View File
@@ -0,0 +1,406 @@
/**
* One episode, in the browser: the free screen, up to eight replies, and the
* hour meter.
*
* This is the entry point of the port of `envs/alert_triage/alert_triage/` —
* the generator, the protocol, the reward and the scripted analysts live in
* the sibling modules and are re-exported here. "Port" is meant strictly:
* `scripts/conformance.mjs` computes the three digests in
* `envs/alert_triage/CONFORMANCE.txt` from this module and fails the build
* if any differs from what the Python computed. That gate is what lets the
* page claim it verified a recorded run rather than merely replayed it.
*
* Two discipline rules, enforced here rather than in the grader:
*
* - **Shown ids.** A disposition may cite only ids the engine has actually
* rendered — KYC fields and triggering transactions from the screen, plus
* whatever lookups returned.
* - **Abort versus truncate.** `aborted` is exactly MAX_TURNS turns spent
* with no accepted disposition, and scores 0.000. Anything that stops
* earlier with no disposition is `truncated` and is not scored at all.
*
* Keep this module pure and dependency-free. It runs on the main thread and
* under `node --test`; hashing is the conformance script's business.
*/
import { MAX_TURNS, canonicalJson, generate, screenOf } from './generator';
import type { Counterparty, Document, Label, PriorAlert, Screen, Txn, World } from './generator';
import { ACTIONS, CLOSE_REASONS, LOOKUPS, TYPOLOGIES, isWindowMonth, parseReply, pyRepr } from './protocol';
import type { JsonValue, Lookup, LookupResult } from './protocol';
import { POLICIES, SHIPPED } from './policies';
import type { LookupRecord, Policy, View } from './policies';
import { DOC_MINUTES, LOOKUP_MINUTES, SCREEN_MINUTES, TURN_MINUTES, scoreExact } from './score';
import type { Episode } from './score';
export * from './rng';
export * from './generator';
export * from './protocol';
export * from './score';
export * from './policies';
const LOOKUP_PRICE: Record<Lookup, number> = {
history: LOOKUP_MINUTES,
counterparty: LOOKUP_MINUTES,
prior_alerts: LOOKUP_MINUTES,
documents: DOC_MINUTES,
};
/** The canonical form of an accepted action. */
export type EngineAction =
| { action: 'lookup'; what: 'history'; month: string }
| { action: 'lookup'; what: 'counterparty'; id: string }
| { action: 'lookup'; what: 'prior_alerts' | 'documents' }
| { action: 'close'; reason: string; cites: string[]; note: string }
| { action: 'escalate'; typology: string; cites: string[]; narrative: string };
export type Disposition = Extract<EngineAction, { action: 'close' | 'escalate' }>;
export interface Step {
accepted: boolean;
action: EngineAction | null;
/** Why a rejected reply was rejected. */
reason: string | null;
/** What an accepted lookup returned. */
result: LookupResult | null;
minutes_spent: number;
turns_spent: number;
done: boolean;
}
export type Outcome = 'solved' | 'failed' | 'aborted';
export class Engine {
readonly world: World;
turns = 0;
rejected = 0;
minutes = SCREEN_MINUTES;
actions: EngineAction[] = [];
lookups: LookupRecord[] = [];
disposition: Disposition | null = null;
shown = new Set<string>();
lastRejection: string | null = null;
private readonly screenCache: Screen;
constructor(world: World) {
this.world = world;
this.screenCache = this.buildScreen();
const screen = this.screenCache;
for (const k of Object.keys(screen.customer)) {
if (k !== 'id' && k !== 'name' && (screen.customer as unknown as Record<string, unknown>)[k] !== undefined) this.shown.add(`kyc.${k}`);
}
this.shown.add(screen.alert.id);
for (const t of screen.triggering_transactions) {
this.shown.add(t.id);
if (t.cp) this.shown.add(t.cp);
}
for (const cp of screen.screen_counterparties) this.shown.add(cp.id);
}
/* ------------------------------------------------------------- views */
private buildScreen(): Screen {
const screen = screenOf(this.world);
if (screen.screen_order === 'kyc_first') screen.summary = [...screen.summary].reverse();
return screen;
}
screen(): Screen {
return this.screenCache;
}
/**
* Everything the model has been shown, and nothing it has not. The scripted
* policies take this and only this: no label, typology, planted set,
* overlay or reference is in it.
*/
view(): View {
return {
screen: this.screenCache,
lookups: [...this.lookups],
turns_spent: this.turns,
turns_left: MAX_TURNS - this.turns,
rejected: this.rejected,
minutes_spent: this.minutes,
last_rejection: this.lastRejection,
};
}
get done(): boolean {
return this.disposition !== null || this.turns >= MAX_TURNS;
}
get aborted(): boolean {
return this.disposition === null && this.turns >= MAX_TURNS;
}
/* -------------------------------------------------------------- step */
step(reply: string | null | undefined): Step {
if (this.done) {
return { accepted: false, action: null, reason: 'the episode is over', result: null, minutes_spent: this.minutes, turns_spent: this.turns, done: true };
}
const [obj, parseReason] = parseReply(reply);
let action: EngineAction | null = null;
let reason = parseReason;
if (obj !== null) [action, reason] = this.validate(obj);
this.turns += 1;
this.minutes += TURN_MINUTES;
if (action === null) {
this.rejected += 1;
this.lastRejection = reason;
return { accepted: false, action: null, reason, result: null, minutes_spent: this.minutes, turns_spent: this.turns, done: this.done };
}
this.lastRejection = null;
this.actions.push(action);
let result: LookupResult | null = null;
if (action.action === 'lookup') {
this.minutes += LOOKUP_PRICE[action.what];
result = this.lookup(action);
const key = action.what === 'history' ? action.month : action.what === 'counterparty' ? action.id : null;
this.lookups.push({ what: action.what, key, result });
} else {
this.disposition = action;
}
return { accepted: true, action, reason: null, result, minutes_spent: this.minutes, turns_spent: this.turns, done: this.done };
}
private validate(obj: Map<string, JsonValue>): [EngineAction | null, string | null] {
// A missing key is Python's `.get()` None; `pyRepr` prints both as None.
const get = (k: string): JsonValue | undefined => obj.get(k);
const action = get('action');
if (!(ACTIONS as readonly unknown[]).includes(action)) {
return [null, `unknown action ${pyRepr(action)}; use lookup, close or escalate`];
}
if (action === 'lookup') {
const what = get('what');
if (!(LOOKUPS as readonly unknown[]).includes(what)) {
return [null, `unknown lookup ${pyRepr(what)}; use history, counterparty, prior_alerts or documents`];
}
const lookup = what as Lookup;
if (lookup === 'history') {
const month = get('month');
if (!isWindowMonth(month, this.world.months)) {
return [null, `month ${pyRepr(month)} is not in the twelve-month window`];
}
if (this.lookups.some((l) => l.what === 'history' && l.key === month)) {
return [null, `history for ${month} has already been looked up`];
}
return [{ action: 'lookup', what: 'history', month }, null];
}
if (lookup === 'counterparty') {
const cid = get('id');
if (typeof cid !== 'string' || !this.shown.has(cid) || !cid.startsWith('CP-')) {
return [null, `counterparty ${pyRepr(cid)} has not appeared on your screen or in a fetched month`];
}
if (this.lookups.some((l) => l.what === 'counterparty' && l.key === cid)) {
return [null, `counterparty ${cid} has already been looked up`];
}
return [{ action: 'lookup', what: 'counterparty', id: cid }, null];
}
if (this.lookups.some((l) => l.what === lookup)) {
return [null, `${lookup} has already been looked up`];
}
return [{ action: 'lookup', what: lookup }, null];
}
const cites = get('cites');
if (!Array.isArray(cites) || cites.length === 0) return [null, 'cites must be a non-empty list of ids'];
const seen: string[] = [];
for (const c of cites) {
if (typeof c !== 'string') return [null, 'cites must be a non-empty list of ids'];
if (!seen.includes(c)) seen.push(c);
}
for (const c of seen) {
if (!this.shown.has(c)) return [null, `cites an id you have not been shown: ${c}`];
}
if (action === 'close') {
const reason = get('reason');
if (!(CLOSE_REASONS as readonly unknown[]).includes(reason)) return [null, `unknown close reason ${pyRepr(reason)}`];
const note = get('note');
return [{ action: 'close', reason: reason as string, cites: seen, note: typeof note === 'string' ? note : '' }, null];
}
const typology = get('typology');
if (!(TYPOLOGIES as readonly unknown[]).includes(typology)) return [null, `unknown typology ${pyRepr(typology)}`];
const narrative = get('narrative');
return [{ action: 'escalate', typology: typology as string, cites: seen, narrative: typeof narrative === 'string' ? narrative : '' }, null];
}
private lookup(action: Extract<EngineAction, { action: 'lookup' }>): LookupResult {
const w = this.world;
if (action.what === 'history') {
const rows: Txn[] = w.transactions.filter((t) => t.month === action.month);
for (const t of rows) {
this.shown.add(t.id);
if (t.cp) this.shown.add(t.cp);
}
return { month: action.month, transactions: rows };
}
if (action.what === 'counterparty') {
const cp = w.counterparties.find((c) => c.id === action.id)!;
for (const id of cp.transactions) this.shown.add(id);
const copy: Counterparty = { ...cp };
return copy;
}
if (action.what === 'prior_alerts') {
for (const p of w.prior_alerts) this.shown.add(p.id);
const alerts: PriorAlert[] = [...w.prior_alerts];
return { alerts };
}
for (const d of w.documents) this.shown.add(d.id);
const documents: Document[] = [...w.documents];
return { documents };
}
/* ----------------------------------------------------------- outcome */
/** solved | failed | aborted, or null while running / if truncated. */
get outcome(): Outcome | null {
if (this.disposition !== null) {
const label: Label = this.world.label;
const right = (label === 'benign') === (this.disposition.action === 'close');
return right ? 'solved' : 'failed';
}
return this.aborted ? 'aborted' : null;
}
episode(truncated: boolean | null = null): Episode {
const d = this.disposition;
return {
label: this.world.label,
planted: this.world.planted,
disposition: d === null ? null : d.action,
cites: d === null ? [] : [...d.cites],
minutes_spent: this.minutes,
reference_minutes: this.world.reference_minutes ?? null,
turns_spent: this.turns,
rejected: this.rejected,
truncated: truncated === null ? !this.done : truncated,
typology: d === null ? null : d.action === 'escalate' ? d.typology : null,
true_typology: this.world.typology ?? null,
lookups: this.lookups.length,
};
}
}
/* ------------------------------------------------------------ reference */
/** Play a world to the end under one policy. Returns the finished Engine. */
export function runPolicy(policy: Policy, world: World): Engine {
const engine = new Engine(world);
while (!engine.done) engine.step(policy(engine.view()));
return engine;
}
/**
* The cheapest shipped policy that reaches the correct disposition, in
* minutes. `[null, null]` if none does — which the Python's tests assert
* never happens for seeds 0–4095.
*/
export function referenceFor(world: World): [number | null, string | null] {
let best: [number, string] | null = null;
for (const name of SHIPPED) {
const engine = runPolicy(POLICIES[name]!, world);
if (engine.outcome === 'solved' && (best === null || engine.minutes < best[0])) best = [engine.minutes, name];
}
return best ?? [null, null];
}
/**
* The world with `reference_minutes` attached — part of the digest.
*
* The policies read only the engine's view, never the world, so attaching
* their result here is not circular; it is the same move as wordle's
* `reference_depth`, made part of the world so the browser recomputes it
* instead of trusting the recorded number.
*/
export function worldForSeed(seed: number): World {
const world = generate(seed);
const [minutes, policy] = referenceFor(world);
world.reference_minutes = minutes;
world.reference_policy = policy;
return world;
}
/** Re-score a recorded run from its reply strings alone. What `verify` does. */
export function replay(seed: number, replies: readonly (string | null)[], world: World | null = null): { engine: Engine; episode: Episode } {
const engine = new Engine(world ?? worldForSeed(seed));
for (const reply of replies) {
if (engine.done) break;
engine.step(reply);
}
return { engine, episode: engine.episode() };
}
/* ---------------------------------------------------------- conformance */
// The three digests, as the lines that get hashed. `conformance.py` is the
// definition; these produce the same byte strings for SHA-256 to consume. No
// hashing here so the module stays browser-clean.
const GRID_LABELS = ['benign', 'suspicious'] as const;
const GRID_DISPOSITIONS = ['close', 'escalate', 'none', 'truncated'] as const;
const GRID_CITES: string[][] = [['a'], ['a', 'b'], ['a', 'b', 'c', 'd'], ['x'], ['a', 'x'], ['a', 'b', 'x', 'y', 'z'], ['a', 'a', 'b'], ['kyc.z']];
const GRID_PLANTED: string[][][] = [[['a']], [['a', 'b']], [['a', 'b'], ['a']], [['c'], ['a', 'b', 'c', 'd']]];
const GRID_MINUTES = [33, 42, 60, 96, 150, 204];
const GRID_REFERENCE: (number | null)[] = [33, 42, 96, null];
/** Digest (a): one canonical world per seed, `reference_minutes` included. */
export function* worldLines(seeds: Iterable<number>): Generator<string> {
for (const seed of seeds) yield canonicalJson(worldForSeed(seed));
}
/** Digest (b): `scoreExact` over the fixed grid, in `conformance.py`'s loop order. */
export function* scorerLines(): Generator<string> {
for (const label of GRID_LABELS) {
for (const disposition of GRID_DISPOSITIONS) {
for (const cites of GRID_CITES) {
for (const planted of GRID_PLANTED) {
for (const minutes of GRID_MINUTES) {
for (const reference of GRID_REFERENCE) {
const none = disposition === 'none' || disposition === 'truncated';
const ep: Episode = {
label,
planted,
disposition: none ? null : disposition,
cites: none ? [] : cites,
minutes_spent: minutes,
reference_minutes: reference,
turns_spent: 1,
rejected: 0,
truncated: disposition === 'truncated',
};
yield canonicalJson(scoreExact(ep));
}
}
}
}
}
}
}
export interface CorpusCase {
seed: number;
note?: string;
replies: (string | null)[];
}
/** Digest (c): parse + engine over the reply corpus on fixed worlds. */
export function protocolTrace(corpus: readonly CorpusCase[]): Record<string, unknown>[] {
const out: Record<string, unknown>[] = [];
for (const c of corpus) {
const engine = new Engine(worldForSeed(c.seed));
const turns: Record<string, unknown>[] = [];
for (const reply of c.replies) {
if (engine.done) break;
const step = engine.step(reply);
turns.push({ accepted: step.accepted, action: step.action, reason: step.reason, minutes: step.minutes_spent, turns: step.turns_spent });
}
const ep = engine.episode();
out.push({ seed: c.seed, turns, outcome: engine.outcome, truncated: ep.truncated, score: scoreExact(ep) });
}
return out;
}
export function* protocolLines(corpus: readonly CorpusCase[]): Generator<string> {
for (const entry of protocolTrace(corpus)) yield canonicalJson(entry);
}
src/demos/alert-triage/format.ts
+58
View File
@@ -0,0 +1,58 @@
/**
* Rendering helpers shared by the board and the controls. Pure, no React.
*
* `money` mirrors `protocol.money()` in the Python so the board shows an
* amount the way the model was shown it — the same digits, the same currency
* code after them. The currency is fictional on held-out seeds (KRN), which is
* exactly why it is a code and never a symbol.
*/
export function money(cents: number, currency: string): string {
const sign = cents < 0 ? '-' : '';
const abs = Math.abs(cents);
const whole = Math.trunc(abs / 100).toLocaleString('en-US');
const minor = String(abs % 100).padStart(2, '0');
return `${sign}${whole}.${minor} ${currency}`;
}
/** Whole units, no minor part, for the summary table. */
export function wholeUnits(cents: number): string {
return Math.trunc(cents / 100).toLocaleString('en-US');
}
export function hours(minutes: number): string {
return (minutes / 60).toFixed(2);
}
const CHANNEL_LABEL: Record<string, string> = {
cash_in: 'cash in',
cash_out: 'cash out',
wire_in: 'wire in',
wire_out: 'wire out',
ach_in: 'ACH in',
ach_out: 'ACH out',
p2p_in: 'P2P in',
p2p_out: 'P2P out',
};
export function channelLabel(channel: string): string {
return CHANNEL_LABEL[channel] ?? channel.replace('_', ' ');
}
/** `kyc.expected_monthly_cash` -> `expected monthly cash`. */
export function fieldLabel(key: string): string {
return key.replace(/^kyc\./, '').replace(/_/g, ' ');
}
/** `CONSISTENT_WITH_PROFILE` -> `consistent with profile`. */
export function enumLabel(value: string): string {
return value.toLowerCase().replace(/_/g, ' ');
}
/** The KYC keys the surface renders, in the order the Python renders them (sorted). */
export function kycEntries(customer: Record<string, unknown>): [string, unknown][] {
return Object.keys(customer)
.filter((key) => key !== 'id' && key !== 'name')
.sort()
.map((key) => [key, customer[key]]);
}
src/demos/alert-triage/game.ts
+137
View File
@@ -0,0 +1,137 @@
/**
* The seam between the board and the engine.
*
* Everything the demo knows about `./engine` passes through here: the adapter
* replays a recording, the controls take the visitor's next reply, `verify`
* re-scores, and all three get a `TriageState` snapshot back. The engine is a
* mutable object with a `step()`; the shell wants immutable snapshots it can
* scrub through. So a snapshot carries the replies that produced it, and the
* next step is "rebuild from the seed, replay them all, step once more". That
* is a few hundred RNG draws and at most eight steps — cheap enough to do on
* every click, and it keeps `play()` pure, which is what makes the Play tab
* and the replay agree by construction rather than by care.
*
* `worldForSeed` runs the three shipped analysts to find the reference, so a
* world is memoised per seed. The engine never mutates its world.
*/
import { Engine, MAX_TURNS, worldForSeed, type World } from './engine';
import { DOC_MINUTES, LOOKUP_MINUTES } from './score';
import type { Disposition, Lookup, LookupKind, Reveal, TriageState } from './types';
export { MAX_TURNS };
/** What each lookup adds to the hour meter, in minutes. The generator's config. */
export const PRICES: Record<LookupKind, number> = {
history: LOOKUP_MINUTES,
counterparty: LOOKUP_MINUTES,
prior_alerts: LOOKUP_MINUTES,
documents: DOC_MINUTES,
};
const worlds = new Map<number, World>();
function worldFor(seed: number): World {
let world = worlds.get(seed);
if (!world) {
world = worldForSeed(seed);
worlds.set(seed, world);
}
return world;
}
/**
* The engine's lookup records, narrowed onto the board's discriminated union.
* The engine validated `what`, so the result shape follows from it.
*/
function lookupsOf(engine: Engine): Lookup[] {
return engine.lookups.map((record): Lookup => {
switch (record.what) {
case 'history':
return { what: 'history', key: String(record.key), result: record.result as Extract<Lookup, { what: 'history' }>['result'] };
case 'counterparty':
return { what: 'counterparty', key: String(record.key), result: record.result as Extract<Lookup, { what: 'counterparty' }>['result'] };
case 'prior_alerts':
return { what: 'prior_alerts', key: null, result: record.result as Extract<Lookup, { what: 'prior_alerts' }>['result'] };
case 'documents':
return { what: 'documents', key: null, result: record.result as Extract<Lookup, { what: 'documents' }>['result'] };
}
});
}
function snapshot(
engine: Engine,
seed: number,
replies: (string | null)[],
referenceMinutes: number | null | undefined,
): TriageState {
const d = engine.disposition;
const status: TriageState['status'] =
d !== null ? (d.action === 'close' ? 'closed' : 'escalated') : engine.aborted ? 'aborted' : 'playing';
// The answer key stays off the board until the episode is over.
const reveal: Reveal | null = engine.done
? {
label: engine.world.label,
typology: engine.world.typology ?? null,
planted: engine.world.planted,
template: engine.world.template,
overlay: engine.world.overlay,
tier: engine.world.tier ?? null,
// A recording's own metric wins over the world's, so the reveal shows
// the number the fixture was scored against; a fresh board uses what
// `worldForSeed` computed.
referenceMinutes: referenceMinutes !== undefined ? referenceMinutes : engine.world.reference_minutes ?? null,
}
: null;
return {
seed,
screen: engine.screen(),
lookups: lookupsOf(engine),
turnsSpent: engine.turns,
rejected: engine.rejected,
minutesSpent: engine.minutes,
lastRejection: engine.lastRejection,
// The engine checked `reason`/`typology` against the enums, so the
// widening from `string` is a fact about the engine, not a hope.
disposition: d as Disposition | null,
status,
outcome: engine.outcome,
shown: [...engine.shown],
replies,
reveal,
};
}
/**
* One snapshot per reply, in order. A reply after the episode is over
* produces the same board again — the engine ignores it, as the Python does.
*/
export function replay(
seed: number,
replies: readonly (string | null)[],
options: { referenceMinutes?: number | null } = {},
): TriageState[] {
const engine = new Engine(worldFor(seed));
const played: (string | null)[] = [];
const out: TriageState[] = [];
for (const reply of replies) {
played.push(reply);
if (!engine.done) engine.step(reply);
out.push(snapshot(engine, seed, [...played], options.referenceMinutes));
}
return out;
}
/** A fresh board for a seed. Deterministic in the seed — the shell relies on it. */
export function initState(seed: number): TriageState {
return snapshot(new Engine(worldFor(seed)), seed, [], undefined);
}
/** Play one reply, returning the next board. Pure — never mutates its input. */
export function play(state: TriageState, reply: string | null): TriageState {
if (state.status !== 'playing') return state;
const snapshots = replay(state.seed, [...state.replies, reply]);
return snapshots[snapshots.length - 1] ?? state;
}
src/demos/alert-triage/generator.ts
+1113
View File
File diff suppressed because it is too large Load Diff
src/demos/alert-triage/grade.ts
+26
View File
@@ -0,0 +1,26 @@
/**
* One helper the scorer does not need but the board does.
*
* `evidence()` in `./score` takes the best alternate's F1 and discards which
* alternate it was. The reveal on the board wants to mark each cited id as
* planted or not, and "planted" is only well-defined against the alternate the
* grader actually scored — so this recovers it, with the same tie-break the
* scorer uses (first alternate wins a tie). Pure, no `?raw`, testable.
*/
import { f1Exact } from './score';
export function bestAlternate(cites: readonly string[], planted: readonly string[][]): string[] {
let best: string[] = planted[0] ?? [];
let bestNum = 0;
let bestDen = 1;
for (const alt of planted) {
const [num, den] = f1Exact(cites, alt);
if (num * bestDen > bestNum * den) {
bestNum = num;
bestDen = den;
best = alt;
}
}
return best;
}
src/demos/alert-triage/meta.ts
+23
View File
@@ -0,0 +1,23 @@
import { defineMeta } from '@/lib/demo-kit';
/**
* Eager, serialisable, React-free: this is imported for every demo on every
* page. Everything expensive is behind `demo.tsx`.
*
* `status` starts as `spec`: `check-demos` rule 7 refuses `live` until the
* capture stage has put recorded runs for this slug in the traces manifest.
* The ship stage flips it. Nothing else here changes at that point.
*/
export default defineMeta({
slug: 'alert-triage',
title: 'Alert Triage',
tagline:
'Work one transaction-monitoring alert against the customer’s file and twelve months of history, then close it or escalate it for a filing.',
vertical: 'financial-crime',
status: 'spec',
order: 0,
icon: 'Siren',
persona: 'BSA / AML Officer',
rewardLine: 'Miss nothing, then clear it fast',
ogImage: '/og/alert-triage.png',
});
src/demos/alert-triage/names.ts
+28
View File
@@ -0,0 +1,28 @@
/**
* The synthetic name and word lists, imported from the environment so the
* two sides can never drift: one file, read by both. Provenance is in
* `envs/alert_triage/names/PROVENANCE.md`. All four are tiny (under 2 kB
* together) and are inlined into this demo's lazy chunk.
*/
import firstNamesJson from '../../../envs/alert_triage/names/first_names.json';
import surnamesJson from '../../../envs/alert_triage/names/surnames.json';
import wordsJson from '../../../envs/alert_triage/names/words.json';
import jurisdictionsJson from '../../../envs/alert_triage/names/jurisdictions.json';
export const FIRST_NAMES: readonly string[] = firstNamesJson;
export const SURNAMES: readonly string[] = surnamesJson;
export const WORDS: {
readonly adjectives: readonly string[];
readonly nouns: readonly string[];
readonly suffixes: readonly string[];
readonly branches: readonly string[];
readonly streets: readonly string[];
} = wordsJson;
export const JURISDICTIONS: {
readonly home: Readonly<Record<string, string>>;
readonly monitored: readonly string[];
readonly other: readonly string[];
} = jurisdictionsJson;
src/demos/alert-triage/narrative.ts
+88
View File
@@ -0,0 +1,88 @@
import type { Narrative } from '@/lib/demo-kit';
/**
* What this environment argues, tab by tab, addressed to the BSA officer.
*
* Every claim below is one a reader can check against the surface under it.
* The two that matter most are the reward claim — which says what the
* recorded seeds show AND what the full ladder shows, because on eight seeds
* they disagree — and the limits, which say where this is easier than the
* job it stands in for.
*
* The numbers quoted in `claims.reward` were measured on 2026-08-28 from
* `uv run python envs/probe.py --taskset alert-triage` (894 main-slice seeds)
* and from the 24 fixtures in `public/traces/alert-triage/`. Re-measure
* before re-quoting them after any change to the reward or the traces.
*/
export const narrative: Narrative = {
thesis:
'Almost every alert an analyst reads is a false positive, and the operating budget is built on how ' +
'fast the team clears them. The few that are not are the whole conversation with the regulator. ' +
'This environment puts both numbers in one reward: it pays, in analyst-hours against a reference, for ' +
'clearing an alert with the facts that decide it, and it takes more away for a missed escalation than ' +
'every hour a policy could save. Every alert is synthetic and every one has a planted answer, so the ' +
'grader computes rather than opines — and a policy that never opens the file cannot collect all of ' +
'the counterweight, because the cases built to pass the screen only give themselves away in a lookup.',
anxiety:
'Almost every alert my team reads is a false positive. The handful that are not is the entire conversation with my regulator.',
claims: {
play:
'Work one alert yourself: read the free screen, pay hours for the lookups you think you need, ' +
'and close or escalate it citing the facts. Only then does the board show what the generator ' +
'planted — and what your call cost against the reference.',
watch:
'A recorded run of one of three scripted reference analysts — not a language model; none has been ' +
'run against this environment yet — replayed one reply at a time: every lookup it paid for, every ' +
'reply the engine refused, and the disposition it reached. The note it wrote is shown beside the ' +
'call and never scored.',
reward:
'Three scripted analysts, re-scored under your weights. On the eight recorded seeds the one who ' +
'reads the screen edges the one who checks the hidden tells at the shipped weights, 0.871 to 0.866; ' +
'over the 894-seed probe ladder the order is the other way, 0.894 to 0.846, because eight seeds ' +
'hold one hidden case. Drag the hours weight down one notch and the ranking on these seeds flips ' +
'too. The asymmetry a policy memo can only assert is sitting in a slider here.',
evidence:
'The grader is a short Python function, printed with the hour constants it assumes, the digests ' +
'that prove your browser plays the same alert the environment did, and a plain list of the ways ' +
'this is easier than the real job.',
},
limits: [
{
text:
'Every alert here has a knowable answer, because the generator planted it. A real SAR decision is a ' +
'human judgement, and a filed report is not a proven crime. Where a rationale the grader refuses is ' +
'one an officer would accept, the fix is another alternate in the template — never a judge.',
answeredBy: 'playbook-redline',
},
{
text:
'The label mix is 55 percent benign by design, far from the real false-positive rate, so a small ' +
'seed set meets both labels. "Hours per thousand alerts" is a ratio inside this environment, not ' +
'a production number, and the hour constants are the generator’s config — nobody timed an analyst.',
},
{
text:
'The lookups are idealised. Each returns the decisive fact in one call — a counterparty’s count of ' +
'other alerted accounts, a prior alert’s closing analyst, the party a document names. In production ' +
'those signals sit across systems and are often absent. This tests whether the agent asks, not ' +
'whether the answer exists.',
},
{
text:
'The adversary designed the data to pass the screen and moved the reporting threshold once, on the ' +
'held-out seeds. Nothing responds to the agent while it trains. That is a moved adversary, not an ' +
'adapting one.',
},
{
text:
'Nine scenario templates and three hidden overlays stand in for a typology space in the hundreds, ' +
'and no institution, real case or real jurisdiction is behind any of them.',
},
{
text:
'There is no rule the agent could break while it optimises: no privacy boundary, no tipping-off ' +
'offence, no policy it must satisfy on the way to the score.',
answeredBy: 'denial-appeal',
},
],
};
src/demos/alert-triage/policies.ts
+341
View File
@@ -0,0 +1,341 @@
/**
* The scripted analysts: three shipped references and the probe ladder's cheats.
*
* A port of `envs/alert_triage/alert_triage/policies.py`. Every policy is a
* function `(view) => reply text`, where `view` is exactly what the engine has
* rendered to the model (`Engine.view()`), and nothing else. No policy sees
* the world, the label, the typology, the planted set or the reference. They
* return TEXT so the parser and the rejection path are exercised exactly as a
* model would exercise them.
*
* The browser needs `fast`, `targeted` and `thorough` to recompute
* `reference_minutes` rather than trust the recorded number; the cheats are
* here so a recorded cheat arm can be replayed live too.
*
* fast decides from the screen; pulls `documents` only where the screen
* says a document should explain the alert. Cannot see a hidden signal.
* targeted `fast` on a screen that shows a typology; otherwise the three
* lookups that could carry a hidden signal, then decides. No history.
* thorough always the full procedure, plus itemised history for the two most
* recent months; cites every constituting transaction it saw.
*/
import { INBOUND, NOMINAL_TYPOLOGY, RULE_IDS } from './generator';
import { floorDiv } from './rng';
import type { Counterparty, Document, Family, PriorAlert, Screen, Txn } from './generator';
import { pyJsonDumps } from './protocol';
import type { Lookup, LookupResult } from './protocol';
/** What the engine has shown, and nothing it has not. */
export interface View {
screen: Screen;
lookups: LookupRecord[];
turns_spent: number;
turns_left: number;
rejected: number;
minutes_spent: number;
last_rejection: string | null;
}
export interface LookupRecord {
what: Lookup;
key: string | null;
result: LookupResult;
}
export type Policy = (view: View) => string;
interface Decision {
action: 'close' | 'escalate';
reason?: string;
typology?: string;
cites: string[];
note?: string;
narrative?: string;
}
const FAMILY_OF = new Map<string, Family>();
for (const table of Object.values(RULE_IDS)) {
for (const [fam, rid] of Object.entries(table)) FAMILY_OF.set(rid, fam as Family);
}
export const EXPLAINING_DOCS: readonly string[] = ['processing_agreement', 'closing_statement', 'executor_letter'];
const emit = (obj: unknown): string => pyJsonDumps(obj);
const family = (view: View): Family => FAMILY_OF.get(view.screen.alert.rule) ?? 'STR';
function looked(view: View, what: Lookup, key: string | null = null): LookupResult | null {
for (const l of view.lookups) {
if (l.what === what && (key === null || l.key === key)) return l.result;
}
return null;
}
const asDocuments = (r: LookupResult | null) => (r as { documents: Document[] } | null)?.documents ?? null;
const asAlerts = (r: LookupResult | null) => (r as { alerts: PriorAlert[] } | null)?.alerts ?? null;
function band(view: View, t: Txn): boolean {
const threshold = view.screen.threshold;
return t.channel === 'cash_in' && floorDiv(threshold * 8, 10) <= t.amount && t.amount < threshold;
}
const close = (reason: string, cites: string[], note: string): Decision => ({ action: 'close', reason, cites, note });
const escalate = (typology: string, cites: string[], narrative: string): Decision => ({ action: 'escalate', typology, cites, narrative });
function dedupe<T>(ids: readonly T[]): T[] {
const out: T[] = [];
for (const i of ids) if (!out.includes(i)) out.push(i);
return out;
}
/* -------------------------------------------------------------- decisions */
/** What the screen alone supports. null means "a document should explain this". */
export function screenDecision(view: View): Decision | null {
const screen = view.screen;
const fam = family(view);
const trig = screen.triggering_transactions;
const trigIds = trig.map((t) => t.id);
const kyc = screen.customer;
const maxCash = Math.max(...screen.summary.map((r) => r.cash_in));
const expected = kyc.expected_monthly_cash ?? 0;
if (fam === 'STR') {
const inBand = trig.filter((t) => band(view, t));
const branches = new Set(inBand.map((t) => t.branch));
if (inBand.length >= 5 || (inBand.length >= 4 && branches.size >= 3)) {
return escalate('STRUCTURING', inBand.map((t) => t.id), 'Repeated cash deposits just under the threshold across branches.');
}
if (expected >= maxCash) {
return close('CONSISTENT_WITH_PROFILE', ['kyc.expected_monthly_cash'], 'Cash volume within the expectation on file; band deposits are ordinary takings.');
}
return escalate('STRUCTURING', inBand.length ? inBand.map((t) => t.id) : trigIds, 'Band deposits with no expectation on file to cover them.');
}
if (fam === 'CASH') {
if (expected >= maxCash) {
return close('CONSISTENT_WITH_PROFILE', ['kyc.expected_activity', 'kyc.expected_monthly_cash'], 'Seasonal peak the file anticipated; within expected monthly cash.');
}
return escalate('STRUCTURING', trigIds, 'Cash far above the expectation on file and rising.');
}
if (fam === 'VEL') {
const senders = new Set(trig.filter((t) => t.channel === 'p2p_in').map((t) => t.cp));
if (senders.size >= 3) {
return escalate('MONEY_MULE', trigIds, 'Credits from unrelated individuals moved to an exchange within a day.');
}
return null;
}
if (fam === 'WIRE') {
if ((kyc.remittance_declared ?? 'none declared') !== 'none declared') {
return close('CONSISTENT_WITH_PROFILE', ['kyc.remittance_declared'], 'Destination declared at onboarding; amounts within income.');
}
return escalate('UNKNOWN', trigIds, 'Wires to a monitored jurisdiction with no declared purpose.');
}
return null; // DORM: needs the document
}
/** After `documents`: close on the explaining document, or escalate if there is none. */
export function documentDecision(view: View): Decision {
const docs = asDocuments(looked(view, 'documents'));
const trigIds = view.screen.triggering_transactions.map((t) => t.id);
if (docs) {
for (const d of docs) {
if (EXPLAINING_DOCS.includes(d.kind)) return close('DOCUMENTED_SOURCE_OF_FUNDS', [d.id], `Source of funds documented: ${d.kind}.`);
}
}
return escalate(NOMINAL_TYPOLOGY[family(view)], trigIds, 'No document on file explains the movement.');
}
/** A hidden signal in the lookups, or null. */
export function lookupDecision(view: View): Decision | null {
const screen = view.screen;
const fam = family(view);
const trig = screen.triggering_transactions;
const trigIds = trig.map((t) => t.id);
for (const l of view.lookups) {
if (l.what === 'counterparty' && (l.result as Counterparty).other_alerted_accounts >= 3) {
const r = l.result as Counterparty;
return escalate('FUNNEL_ACCOUNT', [r.id, ...r.transactions], `${r.id} is shared with ${r.other_alerted_accounts} other alerted accounts; funds converge on it.`);
}
}
const alerts = asAlerts(looked(view, 'prior_alerts'));
if (alerts && alerts.length >= 3) {
const rationales = new Set(alerts.map((p) => p.rationale));
if (alerts.every((p) => p.disposition === 'closed' && p.elapsed_days <= 1) && rationales.size === 1) {
return escalate(NOMINAL_TYPOLOGY[fam], [...alerts.map((p) => p.id), ...trigIds], 'Repeat alerts closed within a day on one boilerplate rationale; nobody has looked.');
}
}
const docs = asDocuments(looked(view, 'documents'));
if (docs) {
for (const d of docs) {
if (!d.party) continue;
for (const cpId of dedupe(trig.filter((t) => INBOUND.includes(t.channel) && t.cp).map((t) => t.cp as string))) {
const cp = looked(view, 'counterparty', cpId) as Counterparty | null;
if (cp && cp.name !== d.party) {
const credits = trig.filter((t) => t.cp === cpId && INBOUND.includes(t.channel)).map((t) => t.id);
return escalate(NOMINAL_TYPOLOGY[fam], [d.id, cpId, ...credits], `The document names ${d.party}; the funds came from ${cp.name}.`);
}
}
}
}
return null;
}
/** thorough's final call: the screen rules, with citations widened to everything itemised. */
export function enrichedDecision(view: View): Decision {
const d = screenDecision(view);
const fam = family(view);
const screen = view.screen;
const months = screen.months;
let seen: Txn[] = [...screen.triggering_transactions];
for (const l of view.lookups) {
if (l.what === 'history') seen = seen.concat((l.result as { transactions: Txn[] }).transactions);
}
if (d !== null && d.action === 'escalate') {
if (fam === 'STR') {
const last = months[months.length - 1];
d.cites = dedupe([...seen.filter((t) => band(view, t) && t.month === last).map((t) => t.id), ...d.cites]);
} else if (fam === 'CASH') {
const lastTwo = new Set(months.slice(-2));
d.cites = dedupe([...seen.filter((t) => t.channel === 'cash_in' && lastTwo.has(t.month)).map((t) => t.id), ...d.cites]);
}
return d;
}
if (d === null) return documentDecision(view);
return d;
}
type Plan = [Lookup, string | null][];
function planNext(view: View, plan: Plan): string | null {
for (const [what, key] of plan) {
if (looked(view, what, key) === null) {
if (what === 'history') return emit({ action: 'lookup', what: 'history', month: key });
if (what === 'counterparty') return emit({ action: 'lookup', what: 'counterparty', id: key });
return emit({ action: 'lookup', what });
}
}
return null;
}
/* ---------------------------------------------------------------- shipped */
export function fast(view: View): string {
const d = screenDecision(view);
if (d !== null) return emit(d);
if (looked(view, 'documents') === null) return emit({ action: 'lookup', what: 'documents' });
return emit(documentDecision(view));
}
function investigationPlan(view: View): Plan {
const plan: Plan = [['documents', null], ['prior_alerts', null]];
for (const cp of view.screen.screen_counterparties) plan.push(['counterparty', cp.id]);
return plan;
}
export function targeted(view: View): string {
const d = screenDecision(view);
if (d !== null && d.action === 'escalate') return emit(d);
const next = planNext(view, investigationPlan(view));
if (next) return next;
const found = lookupDecision(view);
if (found) return emit(found);
return emit(d !== null ? d : documentDecision(view));
}
export function thorough(view: View): string {
const months = view.screen.months;
const plan: Plan = [...investigationPlan(view), ['history', months[months.length - 1]!], ['history', months[months.length - 2]!]];
const next = planNext(view, plan);
if (next) return next;
const found = lookupDecision(view);
return emit(found ?? enrichedDecision(view));
}
/** thorough plus a month that carries nothing. Proves the hour meter binds. */
export function thoroughWasteful(view: View): string {
const months = view.screen.months;
const plan: Plan = [
...investigationPlan(view),
['history', months[months.length - 1]!],
['history', months[months.length - 2]!],
['history', months[months.length - 3]!],
];
const next = planNext(view, plan);
if (next) return next;
const found = lookupDecision(view);
return emit(found ?? enrichedDecision(view));
}
/* ----------------------------------------------------------------- cheats */
export function inaction(): string {
return 'Let me reason about this alert carefully before committing to a disposition.';
}
export function malformed(): string {
return emit({ action: 'file_sar', cites: [] });
}
export function closeBlind(): string {
return emit(close('CONSISTENT_WITH_PROFILE', ['kyc.customer_type'], 'Looks fine.'));
}
export function closeCitesAll(view: View): string {
const screen = view.screen;
const record = screen.customer as unknown as Record<string, unknown>;
const cites = Object.keys(record)
.filter((k) => k !== 'id' && k !== 'name' && record[k] !== undefined)
.map((k) => `kyc.${k}`);
cites.push(...screen.triggering_transactions.map((t) => t.id));
return emit(close('CONSISTENT_WITH_PROFILE', cites, 'See everything.'));
}
export function escalateBlind(view: View): string {
const trigIds = view.screen.triggering_transactions.map((t) => t.id);
return emit(escalate(NOMINAL_TYPOLOGY[family(view)], trigIds, 'Filing defensively.'));
}
/** fast where the screen shows a typology; a blind filing everywhere else. */
export function fastDefensive(view: View): string {
const d = screenDecision(view);
if (d !== null && d.action === 'escalate') return emit(d);
return escalateBlind(view);
}
/** Closes every wire-corridor and dormancy alert blind; files on everything else. */
export function ruleTable(view: View): string {
const fam = family(view);
if (fam === 'WIRE') return emit(close('CONSISTENT_WITH_PROFILE', ['kyc.remittance_declared'], 'Wires are always remittances.'));
if (fam === 'DORM') return emit(close('CONSISTENT_WITH_PROFILE', ['kyc.account_opened'], 'Old accounts wake up.'));
return escalateBlind(view);
}
/** Cites a document it never opened; rejected, then falls back to closing blind. */
export function citeUnseen(view: View): string {
if (view.rejected === 0) return emit(close('DOCUMENTED_SOURCE_OF_FUNDS', ['doc.probate_letter'], 'Per the letter on file.'));
return closeBlind();
}
export const POLICIES: Record<string, Policy> = {
inaction,
malformed,
close_blind: closeBlind,
close_cites_all: closeCitesAll,
cite_unseen: citeUnseen,
escalate_blind: escalateBlind,
fast_defensive: fastDefensive,
rule_table: ruleTable,
fast,
targeted,
thorough,
thorough_wasteful: thoroughWasteful,
};
export const SHIPPED = ['fast', 'targeted', 'thorough'] as const;
src/demos/alert-triage/protocol.ts
+495
View File
@@ -0,0 +1,495 @@
/**
* Turning model text into one action, and engine results into text.
*
* A port of `envs/alert_triage/alert_triage/protocol.py`. The one rule, as in
* wordle: **never throw**. A malformed reply is a thing the model did, not an
* error in the harness. It becomes a rejection that costs a turn.
*
* The parse rules are pinned because this is where two runtimes disagree, and
* CI hashes a committed corpus of reply strings through this parser and the
* engine on both sides (digest `protocol` in CONFORMANCE.txt):
*
* 1. If a ```json fence appears anywhere, its contents are the candidate.
* A bare `{` before the fence does not win.
* 2. Otherwise the candidate is the first string-aware balanced `{…}` span:
* braces inside JSON strings do not count, backslash escapes are honoured.
* 3. If the candidate is not valid JSON, or is JSON but not an object, the
* reply is rejected. There is no further scanning for a second candidate.
* 4. Enum values are matched exact-case. Cites are deduplicated in order and
* matched exact-case against what the engine has shown.
*
* Why this file carries its own JSON reader rather than `JSON.parse`: the
* engine's rejection reasons quote the offending value with Python's `repr`
* (`unknown action None`, `month 3.0 is not in the twelve-month window`), and
* those strings are hashed. `JSON.parse` cannot tell `3` from `3.0`, so the
* reader keeps each number's source text, and `pyRepr` formats it the way
* Python would. The reader accepts exactly what `json.loads` accepts — with
* NaN/Infinity refused, as the Python does via `parse_constant`.
*/
import { MAX_TURNS } from './generator';
import type { Counterparty, Document, PriorAlert, Screen, Txn } from './generator';
import { DOC_MINUTES, LOOKUP_MINUTES, SCREEN_MINUTES, TURN_MINUTES } from './score';
export const ACTIONS = ['lookup', 'close', 'escalate'] as const;
export const LOOKUPS = ['history', 'counterparty', 'prior_alerts', 'documents'] as const;
export const CLOSE_REASONS = ['CONSISTENT_WITH_PROFILE', 'DOCUMENTED_SOURCE_OF_FUNDS', 'PREVIOUSLY_REVIEWED', 'RULE_ARTEFACT'] as const;
export const TYPOLOGIES = ['STRUCTURING', 'FUNNEL_ACCOUNT', 'MONEY_MULE', 'RAPID_MOVEMENT', 'UNKNOWN'] as const;
export type Action = (typeof ACTIONS)[number];
export type Lookup = (typeof LOOKUPS)[number];
export type CloseReason = (typeof CLOSE_REASONS)[number];
export type Typology = (typeof TYPOLOGIES)[number];
/* ----------------------------------------------------------- JSON values */
/**
* A number as `json.loads` would hold it: `int` when the token had no
* fraction or exponent, `float` otherwise. `raw` is the token's own text.
*/
export class JsonNumber {
constructor(
readonly raw: string,
readonly value: number,
) {}
get isInt(): boolean {
return !/[.eE]/.test(this.raw);
}
}
/** Objects are Maps so a key named `__proto__` is data, as it is in Python. */
export type JsonValue = null | boolean | string | JsonNumber | JsonValue[] | Map<string, JsonValue>;
export const isJsonObject = (v: JsonValue | undefined): v is Map<string, JsonValue> => v instanceof Map;
class JsonError extends Error {}
/**
* `json.loads`, to the letter that matters: RFC 8259 grammar, `NaN` and
* `Infinity` refused, control characters refused inside strings, lone
* surrogate escapes allowed, trailing content refused. Throws `JsonError`.
*/
export function pyJsonLoads(text: string): JsonValue {
let i = 0;
const n = text.length;
const ws = () => {
while (i < n) {
const c = text.charCodeAt(i);
if (c === 0x20 || c === 0x09 || c === 0x0a || c === 0x0d) i += 1;
else break;
}
};
const fail = (what: string): never => {
throw new JsonError(`${what} at ${i}`);
};
const string = (): string => {
// text[i] is the opening quote
i += 1;
let out = '';
for (;;) {
if (i >= n) fail('unterminated string');
const c = text.charCodeAt(i);
if (c === 0x22) {
i += 1;
return out;
}
if (c === 0x5c) {
const e = text[i + 1];
i += 2;
switch (e) {
case '"': out += '"'; break;
case '\\': out += '\\'; break;
case '/': out += '/'; break;
case 'b': out += '\b'; break;
case 'f': out += '\f'; break;
case 'n': out += '\n'; break;
case 'r': out += '\r'; break;
case 't': out += '\t'; break;
case 'u': {
const hex = text.slice(i, i + 4);
if (!/^[0-9a-fA-F]{4}$/.test(hex)) fail('bad \\u escape');
i += 4;
out += String.fromCharCode(parseInt(hex, 16));
break;
}
default:
fail('bad escape');
}
continue;
}
if (c < 0x20) fail('control character in string');
out += text[i];
i += 1;
}
};
const number = (): JsonNumber => {
const m = /^-?(?:0|[1-9]\d*)(?:\.\d+)?(?:[eE][-+]?\d+)?/.exec(text.slice(i));
if (!m) fail('bad number');
const raw = m![0];
i += raw.length;
return new JsonNumber(raw, Number(raw));
};
const value = (): JsonValue => {
ws();
if (i >= n) fail('unexpected end');
const c = text[i]!;
if (c === '{') {
i += 1;
const obj = new Map<string, JsonValue>();
ws();
if (text[i] === '}') {
i += 1;
return obj;
}
for (;;) {
ws();
if (text[i] !== '"') fail('expected property name');
const key = string();
ws();
if (text[i] !== ':') fail('expected colon');
i += 1;
obj.set(key, value());
ws();
if (text[i] === ',') {
i += 1;
continue;
}
if (text[i] === '}') {
i += 1;
return obj;
}
fail('expected , or }');
}
}
if (c === '[') {
i += 1;
const arr: JsonValue[] = [];
ws();
if (text[i] === ']') {
i += 1;
return arr;
}
for (;;) {
arr.push(value());
ws();
if (text[i] === ',') {
i += 1;
continue;
}
if (text[i] === ']') {
i += 1;
return arr;
}
fail('expected , or ]');
}
}
if (c === '"') return string();
if (text.startsWith('true', i)) {
i += 4;
return true;
}
if (text.startsWith('false', i)) {
i += 5;
return false;
}
if (text.startsWith('null', i)) {
i += 4;
return null;
}
if (c === '-' || (c >= '0' && c <= '9')) return number();
// NaN, Infinity, -Infinity and anything else: json.loads with parse_constant raising.
return fail('unexpected token');
};
const result = value();
ws();
if (i < n) fail('extra data');
return result;
}
/* ------------------------------------------------------------ Python repr */
/** Python's `str.isprintable()` for one code point. The ASCII space is the one printable separator. */
const NOT_PRINTABLE = /[\p{Cc}\p{Cf}\p{Cs}\p{Co}\p{Cn}\p{Zl}\p{Zp}\p{Zs}]/u;
function pyStrRepr(s: string): string {
const quote = s.includes("'") && !s.includes('"') ? '"' : "'";
let out = quote;
for (const ch of s) {
const cp = ch.codePointAt(0)!;
if (ch === quote || ch === '\\') out += `\\${ch}`;
else if (ch === '\n') out += '\\n';
else if (ch === '\r') out += '\\r';
else if (ch === '\t') out += '\\t';
else if (cp !== 0x20 && NOT_PRINTABLE.test(ch)) {
const hex = cp.toString(16);
out += cp < 0x100 ? `\\x${hex.padStart(2, '0')}` : cp < 0x10000 ? `\\u${hex.padStart(4, '0')}` : `\\U${hex.padStart(8, '0')}`;
} else out += ch;
}
return out + quote;
}
/** `repr(float(raw))`: shortest round-trip digits, exponent form outside 1e-4 ≤ |x| < 1e16. */
export function pyFloatRepr(x: number): string {
if (Number.isNaN(x)) return 'nan';
if (x === Infinity) return 'inf';
if (x === -Infinity) return '-inf';
if (x === 0) return Object.is(x, -0) ? '-0.0' : '0.0';
const sign = x < 0 ? '-' : '';
const [mantissa, expText] = Math.abs(x).toExponential().split('e');
const digits = mantissa!.replace('.', '');
const exp = Number(expText);
if (exp < -4 || exp >= 16) {
const m = digits.length > 1 ? `${digits[0]}.${digits.slice(1)}` : digits;
const e = Math.abs(exp).toString().padStart(2, '0');
return `${sign}${m}e${exp < 0 ? '-' : '+'}${e}`;
}
if (exp >= 0) {
const whole = digits.length > exp + 1 ? digits.slice(0, exp + 1) : digits + '0'.repeat(exp + 1 - digits.length);
const frac = digits.length > exp + 1 ? digits.slice(exp + 1) : '0';
return `${sign}${whole}.${frac}`;
}
return `${sign}0.${'0'.repeat(-exp - 1)}${digits}`;
}
/** Python's `repr` of a value `json.loads` produced, or of a missing key. */
export function pyRepr(v: JsonValue | undefined): string {
if (v === null || v === undefined) return 'None';
if (v === true) return 'True';
if (v === false) return 'False';
if (typeof v === 'string') return pyStrRepr(v);
if (v instanceof JsonNumber) return v.isInt ? BigInt(v.raw).toString() : pyFloatRepr(v.value);
if (Array.isArray(v)) return `[${v.map(pyRepr).join(', ')}]`;
return `{${[...v].map(([k, val]) => `${pyStrRepr(k)}: ${pyRepr(val)}`).join(', ')}}`;
}
/* --------------------------------------------------------------- extract */
/**
* Python's `\s` for str patterns is `str.isspace()`, which is not JavaScript's
* `\s`: Python includes U+001C–U+001F and U+0085, JavaScript includes U+FEFF.
*/
const PY_WS = '[\\t\\n\\x0b\\x0c\\r\\x1c-\\x1f \\x85\\xa0\\u1680\\u2000-\\u200a\\u2028\\u2029\\u202f\\u205f\\u3000]';
const FENCE = new RegExp(`\`\`\`json${PY_WS}*([\\s\\S]*?)\`\`\``);
const MONTH = /^\d{4}-\d{2}$/;
export function extractCandidate(text: string): string | null {
const match = FENCE.exec(text);
if (match) return match[1]!;
const start = text.indexOf('{');
if (start < 0) return null;
let depth = 0;
let inString = false;
let escaped = false;
for (let i = start; i < text.length; i += 1) {
const ch = text[i];
if (inString) {
if (escaped) escaped = false;
else if (ch === '\\') escaped = true;
else if (ch === '"') inString = false;
continue;
}
if (ch === '"') inString = true;
else if (ch === '{') depth += 1;
else if (ch === '}') {
depth -= 1;
if (depth === 0) return text.slice(start, i + 1);
}
}
return null;
}
/** `[object, null]` or `[null, reason]`. Validation of fields is the engine's job. */
export function parseReply(text: string | null | undefined): [Map<string, JsonValue> | null, string | null] {
if (!text) return [null, 'no JSON object found in the reply'];
const candidate = extractCandidate(text);
if (candidate === null) return [null, 'no JSON object found in the reply'];
let value: JsonValue;
try {
value = pyJsonLoads(candidate);
} catch {
return [null, 'the JSON object could not be parsed'];
}
if (!isJsonObject(value)) return [null, 'the reply parsed but is not a JSON object'];
return [value, null];
}
/**
* Is `month` a YYYY-MM inside the window? Python's `\d` and `$` are looser
* than JavaScript's, but the membership test that follows makes the regex
* redundant either way: only a month that IS in the window passes.
*/
export function isWindowMonth(month: unknown, months: readonly string[]): month is string {
return typeof month === 'string' && MONTH.test(month) && months.includes(month);
}
/* ------------------------------------------------------------------ emit */
/**
* Python's `json.dumps(obj)` with its defaults: `", "` and `": "` separators,
* `ensure_ascii=True`. The scripted policies emit their replies through this
* so a recorded arm's reply strings are byte-identical on both sides.
*/
export function pyJsonDumps(value: unknown): string {
if (value === null || value === undefined) return 'null';
if (typeof value === 'string') {
// ensure_ascii escapes everything above '~', DEL included, one \uXXXX per UTF-16 unit.
return JSON.stringify(value).replace(/[\u007f-\uffff]/g, (c) => `\\u${c.charCodeAt(0).toString(16).padStart(4, '0')}`);
}
if (typeof value === 'boolean') return value ? 'true' : 'false';
if (typeof value === 'number') {
if (!Number.isInteger(value)) throw new Error(`pyJsonDumps: non-integer number ${value}`);
return String(value);
}
if (Array.isArray(value)) return `[${value.map(pyJsonDumps).join(', ')}]`;
const record = value as Record<string, unknown>;
return `{${Object.keys(record)
.filter((k) => record[k] !== undefined)
.map((k) => `${pyJsonDumps(k)}: ${pyJsonDumps(record[k])}`)
.join(', ')}}`;
}
/* ---------------------------------------------------------------- render */
const hours2 = (minutes: number) => (minutes / 60).toFixed(2);
export function systemPrompt(): string {
const screen = hours2(SCREEN_MINUTES);
const turn = hours2(TURN_MINUTES);
const lookup = hours2(LOOKUP_MINUTES);
const doc = hours2(DOC_MINUTES);
return `You are a transaction-monitoring analyst at a bank. One alert has been assigned to you.
All data is synthetic. No real customer, account, transaction or counterparty exists.
Your screen shows the alert, the transactions that triggered it, the customer's KYC file and a twelve-month summary. Reading it has already cost ${screen} analyst-hours. You have ${MAX_TURNS} replies. Each reply must be exactly ONE JSON object (in a \`\`\`json fence, or as the first {...} in the reply), and every reply costs ${turn} hours whether or not it is accepted.
Lookups (each may be made once):
{"action":"lookup","what":"history","month":"YYYY-MM"} itemised transactions for a month in the window +${lookup} h
{"action":"lookup","what":"counterparty","id":"CP-n"} name, type, jurisdiction, first seen, how many OTHER alerted accounts share it, its transactions +${lookup} h
{"action":"lookup","what":"prior_alerts"} prior alerts on this customer with disposition, analyst and rationale +${lookup} h
{"action":"lookup","what":"documents"} documents on file +${doc} h
Dispositions (end the episode):
{"action":"close","reason":R,"cites":[ids],"note":"..."}
R is one of CONSISTENT_WITH_PROFILE, DOCUMENTED_SOURCE_OF_FUNDS, PREVIOUSLY_REVIEWED, RULE_ARTEFACT
{"action":"escalate","typology":T,"cites":[ids],"narrative":"..."}
T is one of STRUCTURING, FUNNEL_ACCOUNT, MONEY_MULE, RAPID_MOVEMENT, UNKNOWN
\`cites\` must be a non-empty list of ids you have actually been shown: kyc.<field>, T-<n>, CP-<n>, A-<n> (a prior alert), doc.<id>. Citing an id you have not been shown is rejected. The note or narrative is displayed and never scored.
A close on an alert that should have been escalated scores zero. An escalation on a clean customer scores half on the counterweight and zero on the rest. Cite the facts that decide the case, not everything you saw. If you reach your last reply without a disposition the alert stays in the queue and the episode scores zero.`;
}
const commas = (n: number) => {
const s = String(n);
let out = '';
for (let i = 0; i < s.length; i += 1) {
if (i > 0 && (s.length - i) % 3 === 0) out += ',';
out += s[i];
}
return out;
};
export function money(cents: number, currency: string): string {
const sign = cents < 0 ? '-' : '';
const abs = Math.abs(cents);
return `${sign}${commas(Math.floor(abs / 100))}.${String(abs % 100).padStart(2, '0')} ${currency}`;
}
const padEnd = (s: string, w: number) => s.padEnd(w);
const padStart = (s: string, w: number) => s.padStart(w);
function txnLine(t: Txn, currency: string): string {
const where = t.branch ? `branch ${t.branch}` : `${t.cp}`;
return ` ${padEnd(t.id, 6)} ${t.date} ${padEnd(t.channel, 9)} ${padStart(money(t.amount, currency), 18)} ${where}`;
}
export function renderScreen(screen: Screen): string {
const cur = screen.currency;
const c = screen.customer;
const a = screen.alert;
const alertBlock = [
`ALERT ${a.id} rule ${a.rule} — ${a.rule_text}`,
` fired ${a.fired}`,
' triggering transactions:',
...screen.triggering_transactions.map((t) => txnLine(t, cur)),
];
const kycLines = [`KYC FILE ${c.id} ${c.name}`];
const record = c as unknown as Record<string, string | number | boolean | undefined>;
for (const key of Object.keys(record).sort()) {
if (key === 'id' || key === 'name' || record[key] === undefined) continue;
let value: string | number | boolean = record[key]!;
if (key === 'stated_income' || key === 'expected_monthly_cash') value = money(value as number, cur);
else if (typeof value === 'boolean') value = value ? 'True' : 'False';
kycLines.push(` kyc.${key} = ${value}`);
}
const summary = [
`TWELVE-MONTH SUMMARY (${cur})`,
` ${padEnd('month', 8)}${padStart('cash in', 16)}${padStart('cash out', 16)}${padStart('wires in', 16)}${padStart('wires out', 16)}${padStart('ach in', 16)}${padStart('ach out', 16)}${padStart('txns', 6)}`,
];
for (const r of screen.summary) {
const cell = (v: number) => padStart(commas(Math.floor(v / 100)), 16);
summary.push(
` ${padEnd(r.month, 8)}${cell(r.cash_in)}${cell(r.cash_out)}${cell(r.wires_in)}${cell(r.wires_out)}${cell(r.ach_in)}${cell(r.ach_out)}${padStart(String(r.count), 6)}`,
);
}
const cps = ['COUNTERPARTIES on screen (top by twelve-month volume):'];
for (const cp of screen.screen_counterparties) {
cps.push(` ${padEnd(cp.id, 6)} ${padEnd(cp.direction, 4)} ${padStart(String(cp.count), 3)} txns ${money(cp.total, cur)}`);
}
const blocks = screen.screen_order === 'kyc_first' ? [kycLines, alertBlock] : [alertBlock, kycLines];
blocks.push(summary, cps, [
`Reporting threshold in this jurisdiction: ${money(screen.threshold, cur)}`,
'SYNTHETIC DATA — no real customer, account or transaction.',
]);
return blocks.map((b) => b.join('\n')).join('\n');
}
export type LookupResult =
| { month: string; transactions: Txn[] }
| Counterparty
| { alerts: PriorAlert[] }
| { documents: Document[] };
export function renderLookup(what: Lookup, result: LookupResult, currency: string): string {
if (what === 'history') {
const r = result as { month: string; transactions: Txn[] };
return [`HISTORY ${r.month}: ${r.transactions.length} transactions`, ...r.transactions.map((t) => txnLine(t, currency))].join('\n');
}
if (what === 'counterparty') {
const r = result as Counterparty;
return [
`COUNTERPARTY ${r.id} ${r.name}`,
` type ${r.type}; jurisdiction ${r.jurisdiction}; first seen ${r.first_seen}`,
` other alerted accounts sharing this counterparty in twelve months: ${r.other_alerted_accounts}`,
` transactions: ${r.transactions.join(', ')}`,
].join('\n');
}
if (what === 'prior_alerts') {
const r = result as { alerts: PriorAlert[] };
if (!r.alerts.length) return 'PRIOR ALERTS: none on file.';
const lines = ['PRIOR ALERTS:'];
for (const p of r.alerts) {
lines.push(` ${p.id} ${p.rule} fired ${p.fired} ${p.disposition} after ${p.elapsed_days} day(s) by ${p.analyst}`);
lines.push(` ${p.rationale}`);
}
return lines.join('\n');
}
const r = result as { documents: Document[] };
return ['DOCUMENTS ON FILE:', ...r.documents.map((d) => ` ${d.id} [${d.kind}] ${d.description}`)].join('\n');
}
export function renderRejection(reason: string, turnsLeft: number): string {
const plural = turnsLeft === 1 ? 'y' : 'ies';
return `That reply was not accepted: ${reason}. It still cost a turn. You have ${turnsLeft} repl${plural} left.`;
}
export function renderCost(minutesSpent: number, turnsLeft: number): string {
const plural = turnsLeft === 1 ? 'y' : 'ies';
return `Hours spent so far: ${hours2(minutesSpent)}. You have ${turnsLeft} repl${plural} left.`;
}
src/demos/alert-triage/reward.ts
+74
View File
@@ -0,0 +1,74 @@
import type { RewardSpec } from '@/lib/demo-kit';
import rewardSource from '../../../envs/alert_triage/alert_triage/reward.py?raw';
/**
* The reward, mirrored from `envs/alert_triage/alert_triage/reward.py`.
*
* The labels are for the BSA officer, not the variable names. `caught` is a
* fair identifier and says nothing to someone deciding a budget; "nothing
* slipped through" is the same quantity in the words they already use.
*
* The constants are restated here as literals rather than imported, because
* `check-demos` reads the weights off the page beside the code. If reward.py
* changes, the receipt panel shows the change and these must follow it.
*/
export const reward: RewardSpec = {
components: [
{
key: 'caught',
label: 'Nothing slipped through',
description:
'One point for any correct call. Half a point if a clean customer was escalated — nothing slipped, but a report was filed on them. Nothing if a suspicious alert was closed or no call was made. This is the counterweight, and it is over half the reward: a miss also zeroes the other two, so one miss costs more than every hour a policy could save.',
weight: 0.55,
role: 'counterweight',
},
{
key: 'hours',
label: 'Cleared it in the hours budgeted',
description:
'Analyst-hours spent — 0.50 to read the screen, 0.05 per reply, 0.40 per lookup, 0.10 for the documents — as a ratio against the cheapest shipped analyst that got this same alert right. Capped at one. A fast wrong answer saves nothing.',
weight: 0.25,
role: 'objective',
},
{
key: 'evidence',
label: 'Named the facts that decide it',
description:
'F1 between the ids the run cited and the facts the generator planted: the explaining KYC or document fact on a clean alert, the transactions that constitute the typology on a suspicious one. Cite everything and precision pays you what it thinks of that. Zero if the call was wrong. The narrative prose is never read.',
weight: 0.2,
role: 'objective',
},
],
metrics: [
{ key: 'hours_spent', label: 'Hours spent', description: 'Screen, replies and lookups, at the generator’s prices.' },
{
key: 'reference_hours',
label: 'Reference took',
description: 'Hours the cheapest shipped analyst needed to get this same alert right.',
},
{ key: 'lookups', label: 'Lookups made', description: 'History months, counterparties, prior alerts, documents.' },
{ key: 'turns_used', label: 'Replies spent', description: 'Accepted or refused. Eight ends the episode.' },
{
key: 'rejected_replies',
label: 'Replies refused',
description: 'No JSON object, an unknown action, an empty cites list, an id never shown, or a repeated lookup. Costs a turn.',
},
{
key: 'false_escalation',
label: 'Escalated a clean customer',
description: 'One if a report was filed on a benign alert.',
},
{ key: 'missed', label: 'Closed a suspicious alert', description: 'One if the alert should have been escalated and was closed instead.' },
{
key: 'typology_match',
label: 'Named the right typology',
description: 'One if the escalation named what the generator planted. Diagnostic only; not scored, because it would double-count the call with an enum the agent can guess.',
},
],
source: {
path: 'envs/alert_triage/alert_triage/reward.py',
code: rewardSource,
marker: 'reward',
},
};
src/demos/alert-triage/rng.ts
+154
View File
@@ -0,0 +1,154 @@
/**
* Portable randomness and portable dates.
*
* A port of `envs/alert_triage/alert_triage/rng.py`, line for line. Nothing
* in the generator may touch a language built-in for either: a built-in RNG
* picks a different alert for the same seed on the two sides, and `Date` is
* timezone-sensitive, so CI in UTC passes and a visitor elsewhere sees a
* different fire date for the same `?seed=`.
*
* Pure and dependency-free. Runs on the main thread and under `node --test`.
*/
/**
* FNV-1a, 32-bit, over the UTF-8 bytes. Identical to wordle's.
*
* `Math.imul` is what makes the 32-bit multiply exact; a plain `*` overflows
* into a double and silently diverges from Python after the first few bytes.
*/
export function fnv1a32(text: string): number {
let h = 0x811c9dc5;
for (const byte of new TextEncoder().encode(text)) {
h ^= byte;
h = Math.imul(h, 0x01000193) >>> 0;
}
return h >>> 0;
}
/**
* Python's floor division. `Math.floor(a / b)` is right for the positive
* integers this generator deals in, but the subtraction form is exact for
* every integer a double can hold and matches `//` on negatives too.
*/
export function floorDiv(a: number, b: number): number {
return (a - pyMod(a, b)) / b;
}
/** Python's `%`: the result takes the sign of the divisor. */
export function pyMod(a: number, b: number): number {
const r = a % b;
return r !== 0 && r < 0 !== b < 0 ? r + b : r;
}
/**
* Marsaglia xorshift32. Seven lines, and Python has the same seven.
*
* `below(n)` is a plain modulo. It is biased for large `n`, which does not
* matter here (every `n` is tiny) and keeps the port a one-liner.
*/
export class XorShift32 {
state: number;
constructor(seed: number) {
const s = seed >>> 0;
this.state = s === 0 ? 0x9e3779b9 : s;
}
next(): number {
let x = this.state;
x = (x ^ (x << 13)) >>> 0;
x = (x ^ (x >>> 17)) >>> 0;
x = (x ^ (x << 5)) >>> 0;
this.state = x;
return x;
}
below(n: number): number {
return this.next() % n;
}
/** Inclusive on both ends. */
between(lo: number, hi: number): number {
return lo + this.below(hi - lo + 1);
}
chance(percent: number): boolean {
return this.below(100) < percent;
}
pick<T>(items: readonly T[]): T {
return items[this.below(items.length)]!;
}
/** k distinct items, Fisher-Yates over a copy, first k taken. */
sample<T>(items: readonly T[], k: number): T[] {
const pool = [...items];
const out: T[] = [];
const n = Math.min(k, pool.length);
for (let i = 0; i < n; i += 1) {
const j = i + this.below(pool.length - i);
const tmp = pool[i]!;
pool[i] = pool[j]!;
pool[j] = tmp;
out.push(pool[i]!);
}
return out;
}
hex4(): string {
return this.below(65536).toString(16).padStart(4, '0');
}
}
/* ------------------------------------------------------------------ dates */
// Howard Hinnant's civil-from-days / days-from-civil. Integer arithmetic only.
// Day 0 is 1970-01-01.
export function daysFromCivil(y: number, m: number, d: number): number {
y -= m <= 2 ? 1 : 0;
const era = floorDiv(y >= 0 ? y : y - 399, 400);
const yoe = y - era * 400;
const mp = pyMod(m + 9, 12);
const doy = floorDiv(153 * mp + 2, 5) + d - 1;
const doe = yoe * 365 + floorDiv(yoe, 4) - floorDiv(yoe, 100) + doy;
return era * 146097 + doe - 719468;
}
export function civilFromDays(z: number): [number, number, number] {
z += 719468;
const era = floorDiv(z >= 0 ? z : z - 146096, 146097);
const doe = z - era * 146097;
const yoe = floorDiv(doe - floorDiv(doe, 1460) + floorDiv(doe, 36524) - floorDiv(doe, 146096), 365);
const y = yoe + era * 400;
const doy = doe - (365 * yoe + floorDiv(yoe, 4) - floorDiv(yoe, 100));
const mp = floorDiv(5 * doy + 2, 153);
const d = doy - floorDiv(153 * mp + 2, 5) + 1;
const m = mp < 10 ? mp + 3 : mp - 9;
return [y + (m <= 2 ? 1 : 0), m, d];
}
export function daysInMonth(y: number, m: number): number {
if (m === 12) return daysFromCivil(y + 1, 1, 1) - daysFromCivil(y, 12, 1);
return daysFromCivil(y, m + 1, 1) - daysFromCivil(y, m, 1);
}
const pad = (n: number, width: number) => String(n).padStart(width, '0');
export function isoDate(day: number): string {
const [y, m, d] = civilFromDays(day);
return `${pad(y, 4)}-${pad(m, 2)}-${pad(d, 2)}`;
}
export function monthKey(y: number, m: number): string {
return `${pad(y, 4)}-${pad(m, 2)}`;
}
export function monthOfDay(day: number): string {
const [y, m] = civilFromDays(day);
return monthKey(y, m);
}
export function addMonths(y: number, m: number, n: number): [number, number] {
const index = y * 12 + (m - 1) + n;
return [floorDiv(index, 12), pyMod(index, 12) + 1];
}
src/demos/alert-triage/score.ts
+178
View File
@@ -0,0 +1,178 @@
/**
* The reward, in the browser.
*
* A port of `envs/alert_triage/alert_triage/reward.py`. The page quotes the
* Python — its `# region: pig-demo/reward` block is the receipt — and this
* file exists so the browser can re-derive what the environment recorded. CI
* hashes `scoreExact` over a fixed grid on both sides (digest `scorer`).
*
* Three components, weights summing to 1.0. Two pull toward clearing the
* alert quickly with the right citations. The third — the heaviest — pulls
* the other way: it pays for not letting a suspicious customer through.
*
* Hours are integers in MINUTES so that the world, the grader and the browser
* agree to the last digit; the page shows them as hours.
*/
// The generator's config, NOT industry measurements. Nobody timed an analyst.
export const SCREEN_MINUTES = 30; // reading the free screen; charged to everyone
export const TURN_MINUTES = 3; // every reply, accepted or rejected
export const LOOKUP_MINUTES = 24; // history, counterparty, prior alerts
export const DOC_MINUTES = 6; // documents on file
export const CAUGHT_WEIGHT = 0.55;
export const HOURS_WEIGHT = 0.25;
export const EVIDENCE_WEIGHT = 0.2;
/**
* What `caught` pays for filing on a clean customer. Not zero: a defensive
* escalation is a wasted filing, not a missed crime. Not one: otherwise a
* policy that escalates everything takes the counterweight for free.
*/
export const FALSE_ESCALATION_CREDIT = 0.5;
export type ComponentKey = 'caught' | 'hours' | 'evidence';
/** Everything the reward needs from one played alert. */
export interface Episode {
/** What the generator planted. */
label: 'benign' | 'suspicious';
/** Alternates; the best one is scored. */
planted: string[][];
disposition: 'close' | 'escalate' | null;
cites: string[];
minutes_spent: number;
/** Cheapest shipped policy that got this seed right. */
reference_minutes: number | null;
turns_spent: number;
rejected: number;
/** Ended early with no disposition: not scored. */
truncated: boolean;
/** What the agent wrote; diagnostic only. */
typology?: string | null;
/** What the generator planted; diagnostic only. */
true_typology?: string | null;
lookups?: number;
}
/** A close on a benign alert, or an escalation on a suspicious one. */
export function correct(ep: Episode): boolean {
return (ep.label === 'benign' && ep.disposition === 'close') || (ep.label === 'suspicious' && ep.disposition === 'escalate');
}
/**
* Nothing slipped through. The counterweight, and over half the reward.
*
* 1.0 on any correct call. 0.0 if a suspicious alert was closed, or nothing
* was decided. A false escalation earns FALSE_ESCALATION_CREDIT.
*/
export function caught(ep: Episode): number {
if (ep.disposition === null) return 0.0;
if (correct(ep)) return 1.0;
if (ep.label === 'benign') return FALSE_ESCALATION_CREDIT;
return 0.0;
}
/**
* Cleared it in the hours budgeted, as a ratio against a reference.
*
* Capped at 1.0. A fast wrong answer saves nothing. If no shipped policy got
* this seed right there is no reference and this is null — not scored.
*/
export function hours(ep: Episode): number | null {
if (!correct(ep)) return 0.0;
if (ep.reference_minutes === null) return null;
return Math.min(1.0, ep.reference_minutes / Math.max(1, ep.minutes_spent));
}
/** F1 as an exact fraction [numerator, denominator], so both runtimes agree bit for bit. */
export function f1Exact(cites: readonly string[], planted: readonly string[]): [number, number] {
const c = new Set(cites);
const p = new Set(planted);
let common = 0;
for (const id of c) if (p.has(id)) common += 1;
return [2 * common, c.size + p.size];
}
function bestF1(ep: Episode): [number, number] {
let best: [number, number] = [0, 1];
for (const alt of ep.planted) {
const [num, den] = f1Exact(ep.cites, alt);
if (num * best[1] > best[0] * den) best = [num, den];
}
return best;
}
/**
* Named the facts that decide it: F1 between the cites and the planted set.
* Where a template admits two reasonable rationales the best alternate is
* scored. Zero if the disposition was wrong. The narrative prose is never
* read here.
*/
export function evidence(ep: Episode): number {
if (!correct(ep) || !ep.cites.length) return 0.0;
const [num, den] = bestF1(ep);
return num / den;
}
export type Components = Record<ComponentKey, number | null>;
/** The components, unweighted. null means "not scored", never zero. */
export function score(ep: Episode): Components {
if (ep.truncated) return { caught: null, hours: null, evidence: null };
return { caught: caught(ep), hours: hours(ep), evidence: evidence(ep) };
}
export const WEIGHTS: Record<ComponentKey, number> = { caught: CAUGHT_WEIGHT, hours: HOURS_WEIGHT, evidence: EVIDENCE_WEIGHT };
export const ROLES: Record<ComponentKey, 'counterweight' | 'objective'> = { caught: 'counterweight', hours: 'objective', evidence: 'objective' };
export function total(ep: Episode, weights: Record<ComponentKey, number> = WEIGHTS): number | null {
const parts = score(ep);
if (parts.caught === null || parts.hours === null || parts.evidence === null) return null;
return parts.caught * weights.caught + parts.hours * weights.hours + parts.evidence * weights.evidence;
}
/** P(benign) below which a blind escalation out-scores a well-cited close. */
export function escalateBreakEven(evidenceIfClose = 1.0): number {
const closePerBenign = CAUGHT_WEIGHT + HOURS_WEIGHT + EVIDENCE_WEIGHT * evidenceIfClose;
const escPerBenign = CAUGHT_WEIGHT * FALSE_ESCALATION_CREDIT;
const escPerSuspicious = CAUGHT_WEIGHT + HOURS_WEIGHT;
return escPerSuspicious / (closePerBenign - escPerBenign + escPerSuspicious);
}
export type ExactComponents = Record<ComponentKey, [number, number] | null>;
/** The components as exact fractions, for the cross-language digest. */
export function scoreExact(ep: Episode): ExactComponents {
if (ep.truncated) return { caught: null, hours: null, evidence: null };
let c: [number, number];
if (ep.disposition === null) c = [0, 1];
else if (correct(ep)) c = [1, 1];
else if (ep.label === 'benign') c = [1, 2];
else c = [0, 1];
let h: [number, number] | null;
if (!correct(ep)) h = [0, 1];
else if (ep.reference_minutes === null) h = null;
else {
const spent = Math.max(1, ep.minutes_spent);
h = ep.reference_minutes >= spent ? [1, 1] : [ep.reference_minutes, spent];
}
const e: [number, number] = !correct(ep) || !ep.cites.length ? [0, 1] : bestF1(ep);
return { caught: c, hours: h, evidence: e };
}
/** Diagnostics. Weight zero — reported, never summed into the reward. */
export function metrics(ep: Episode): Record<string, number | null> {
return {
hours_spent: ep.minutes_spent / 60,
reference_hours: ep.reference_minutes === null ? null : ep.reference_minutes / 60,
lookups: ep.lookups ?? 0,
turns_used: ep.turns_spent,
rejected_replies: ep.rejected,
false_escalation: ep.label === 'benign' && ep.disposition === 'escalate' ? 1.0 : 0.0,
missed: ep.label === 'suspicious' && ep.disposition === 'close' ? 1.0 : 0.0,
typology_match: ep.disposition !== 'escalate' ? null : (ep.typology ?? null) === (ep.true_typology ?? null) ? 1.0 : 0.0,
};
}
src/demos/alert-triage/surface.tsx
+580
View File
@@ -0,0 +1,580 @@
import { memo, type ReactNode } from 'react';
import { cn } from '@/lib/utils';
import { channelLabel, enumLabel, hours, kycEntries, money, wholeUnits } from './format';
import { bestAlternate } from './grade';
import type { Lookup, Reveal, Screen, Transaction, TriageState } from './types';
/**
* The analyst's screen. One component for all three jobs — the visitor
* playing, the recorded replay, and the timeline thumbnail — because three
* near-identical boards is how they drift.
*
* Two things this board must do that a game board does not:
*
* · Say, on the board itself, that the data is synthetic. Not in a footnote.
* The label is the first row of every render.
* · Show the answer only once the episode is over. `state.reveal` is null
* until then, so the surface cannot leak the label even by accident.
*
* Nothing here interprets the data for the visitor. A counterparty shared with
* nine other alerted accounts is rendered as the number nine, the way the
* model saw it; whether nine is a lot is the analyst's call to make.
*/
const MAX_TURNS = 8;
/* ---------------------------------------------------------------- pieces */
function SyntheticLabel({ compact }: { compact?: boolean }) {
return (
<p
className={cn(
'rounded-md border border-dashed border-warning/60 bg-warning/5 font-semibold uppercase tracking-wide text-warning',
compact ? 'px-1 py-0.5 text-[7px] leading-tight' : 'px-2 py-1 text-[0.65rem]',
)}
>
Synthetic data — no real customer, account or transaction
</p>
);
}
function Id({ children, tone = 'muted' }: { children: ReactNode; tone?: 'muted' | 'fg' }) {
return (
<code className={cn('nums font-mono text-[0.8em]', tone === 'muted' ? 'text-muted' : 'text-fg')}>
{children}
</code>
);
}
function Block({ title, aside, children }: { title: string; aside?: ReactNode; children: ReactNode }) {
return (
<section className="rounded-lg border border-border bg-surface p-3">
<header className="mb-2 flex items-baseline justify-between gap-2">
<h4 className="text-xs font-semibold uppercase tracking-wide text-muted">{title}</h4>
{aside ? <div className="text-xs text-muted">{aside}</div> : null}
</header>
{children}
</section>
);
}
/**
* `contain: inline-size` is load-bearing. The board lives inside a
* shrink-to-fit card, so a table's min-content width would otherwise become
* the board's minimum width and push a 390px phone into horizontal scroll.
* Containment makes the wrapper contribute nothing and scroll instead.
*/
function Scroll({ children }: { children: ReactNode }) {
return <div className="overflow-x-auto [contain:inline-size]">{children}</div>;
}
function TransactionTable({ rows, currency }: { rows: Transaction[]; currency: string }) {
if (rows.length === 0) return <p className="text-xs text-muted">No transactions.</p>;
return (
<Scroll>
<table className="w-full min-w-[26rem] text-xs">
<thead className="text-left text-muted">
<tr>
<th className="pb-1 pr-2 font-medium">id</th>
<th className="pb-1 pr-2 font-medium">date</th>
<th className="pb-1 pr-2 font-medium">channel</th>
<th className="pb-1 pr-2 text-right font-medium">amount</th>
<th className="pb-1 font-medium">where</th>
</tr>
</thead>
<tbody>
{rows.map((t) => (
<tr key={t.id} className="border-t border-border/60">
<td className="py-0.5 pr-2">
<Id tone="fg">{t.id}</Id>
</td>
<td className="nums py-0.5 pr-2 whitespace-nowrap">{t.date}</td>
<td className="py-0.5 pr-2 whitespace-nowrap">{channelLabel(t.channel)}</td>
<td className="nums py-0.5 pr-2 text-right whitespace-nowrap">{money(t.amount, currency)}</td>
<td className="py-0.5 whitespace-nowrap">
{t.branch ? `branch ${t.branch}` : t.cp ? <Id>{t.cp}</Id> : '—'}
</td>
</tr>
))}
</tbody>
</table>
</Scroll>
);
}
function AlertBlock({ screen }: { screen: Screen }) {
const a = screen.alert;
return (
<Block title="Alert" aside={<span className="nums">fired {a.fired}</span>}>
<p className="text-sm">
<Id tone="fg">{a.id}</Id>
<span className="mx-1.5 text-muted">·</span>
<span className="font-mono text-xs font-semibold">{a.rule}</span>
<span className="mx-1.5 text-muted">—</span>
{a.rule_text}
</p>
<p className="mb-1 mt-2 text-xs text-muted">Triggering transactions</p>
<TransactionTable rows={screen.triggering_transactions} currency={screen.currency} />
</Block>
);
}
function kycValue(key: string, value: unknown, currency: string): string {
if (typeof value === 'boolean') return value ? 'yes' : 'no';
if (typeof value === 'number') {
return key === 'stated_income' || key === 'expected_monthly_cash' ? money(value, currency) : String(value);
}
return String(value ?? '');
}
function KycBlock({ screen }: { screen: Screen }) {
const c = screen.customer;
return (
<Block title="KYC file" aside={<Id>{c.id}</Id>}>
<p className="mb-2 text-sm font-medium">{c.name}</p>
<dl className="grid grid-cols-1 gap-x-4 gap-y-1 text-xs sm:grid-cols-2">
{kycEntries(c as unknown as Record<string, unknown>).map(([key, value]) => (
<div key={key} className="flex min-w-0 flex-col">
<dt className="text-muted">
<Id>kyc.{key}</Id>
</dt>
<dd className="nums break-words">{kycValue(key, value, screen.currency)}</dd>
</div>
))}
</dl>
</Block>
);
}
function SummaryBlock({ screen }: { screen: Screen }) {
const cols: [Exclude<keyof Screen['summary'][number], 'month'>, string][] = [
['cash_in', 'cash in'],
['cash_out', 'cash out'],
['wires_in', 'wires in'],
['wires_out', 'wires out'],
['ach_in', 'ACH in'],
['ach_out', 'ACH out'],
];
return (
<Block
title="Twelve-month summary"
aside={
<span>
whole {screen.currency}
{screen.screen_order === 'kyc_first' ? ' · newest first' : ''}
</span>
}
>
<Scroll>
<table className="w-full min-w-[30rem] text-xs">
<thead className="text-left text-muted">
<tr>
<th className="pb-1 pr-2 font-medium">month</th>
{cols.map(([key, label]) => (
<th key={key} className="pb-1 pr-2 text-right font-medium">
{label}
</th>
))}
<th className="pb-1 text-right font-medium">txns</th>
</tr>
</thead>
<tbody>
{screen.summary.map((row) => (
<tr key={row.month} className="border-t border-border/60">
<td className="nums py-0.5 pr-2 whitespace-nowrap">{row.month}</td>
{cols.map(([key]) => (
<td key={key} className="nums py-0.5 pr-2 text-right">
{wholeUnits(row[key])}
</td>
))}
<td className="nums py-0.5 text-right">{row.count}</td>
</tr>
))}
</tbody>
</table>
</Scroll>
</Block>
);
}
function CounterpartyBlock({ screen }: { screen: Screen }) {
return (
<Block title="Counterparties on screen" aside="top by twelve-month volume">
<ul className="space-y-0.5 text-xs">
{screen.screen_counterparties.map((cp) => (
<li key={cp.id} className="flex flex-wrap items-baseline gap-x-2">
<Id tone="fg">{cp.id}</Id>
<span className="text-muted">{cp.direction === 'in' ? 'inbound' : 'outbound'}</span>
<span className="nums">{cp.count} txns</span>
<span className="nums">{money(cp.total, screen.currency)}</span>
</li>
))}
</ul>
<p className="mt-2 text-xs text-muted">
Reporting threshold in this jurisdiction:{' '}
<span className="nums text-fg">{money(screen.threshold, screen.currency)}</span>
</p>
</Block>
);
}
function LookupCard({ lookup, currency }: { lookup: Lookup; currency: string }) {
switch (lookup.what) {
case 'history':
return (
<Block
title={`History ${lookup.result.month}`}
aside={<span className="nums">{lookup.result.transactions.length} transactions</span>}
>
<TransactionTable rows={lookup.result.transactions} currency={currency} />
</Block>
);
case 'counterparty': {
const r = lookup.result;
return (
<Block title={`Counterparty ${r.id}`} aside={<span className="nums">first seen {r.first_seen}</span>}>
<p className="text-sm font-medium">{r.name}</p>
<p className="text-xs text-muted">
{r.type.replace(/_/g, ' ')} · {r.jurisdiction}
</p>
<p className="mt-1 text-xs">
Other alerted accounts sharing this counterparty in twelve months:{' '}
<span className="nums font-semibold">{r.other_alerted_accounts}</span>
</p>
<p className="mt-1 text-xs text-muted">
transactions:{' '}
{r.transactions.map((id) => (
<span key={id} className="mr-1">
<Id>{id}</Id>
</span>
))}
</p>
</Block>
);
}
case 'prior_alerts':
return (
<Block title="Prior alerts" aside={<span className="nums">{lookup.result.alerts.length} on file</span>}>
{lookup.result.alerts.length === 0 ? (
<p className="text-xs text-muted">None on file.</p>
) : (
<ul className="space-y-1.5 text-xs">
{lookup.result.alerts.map((p) => (
<li key={p.id}>
<div className="flex flex-wrap items-baseline gap-x-2">
<Id tone="fg">{p.id}</Id>
<span className="font-mono">{p.rule}</span>
<span className="nums text-muted">fired {p.fired}</span>
<span>
{p.disposition} after {p.elapsed_days} {p.elapsed_days === 1 ? 'day' : 'days'} by {p.analyst}
</span>
</div>
<p className="text-muted">{p.rationale}</p>
</li>
))}
</ul>
)}
</Block>
);
case 'documents':
return (
<Block title="Documents on file" aside={<span className="nums">{lookup.result.documents.length}</span>}>
<ul className="space-y-1 text-xs">
{lookup.result.documents.map((d) => (
<li key={d.id} className="flex flex-col gap-0.5 sm:flex-row sm:gap-2">
<span className="shrink-0">
<Id tone="fg">{d.id}</Id>{' '}
<span className="text-muted">[{d.kind.replace(/_/g, ' ')}]</span>
</span>
<span>{d.description}</span>
</li>
))}
</ul>
</Block>
);
}
}
function CiteChip({ id, hit }: { id: string; hit?: boolean | null }) {
return (
<span
className={cn(
'inline-flex items-center gap-1 rounded-md border px-1.5 py-0.5 font-mono text-[0.7rem]',
hit === true && 'border-positive/50 bg-positive/10 text-fg',
hit === false && 'border-border text-muted line-through decoration-muted/60',
(hit === null || hit === undefined) && 'border-border text-fg',
)}
>
{hit === true ? <span aria-hidden="true">✓</span> : null}
{id}
{hit === true ? <span className="sr-only">, planted</span> : null}
{hit === false ? <span className="sr-only">, not planted</span> : null}
</span>
);
}
function DispositionCard({ state }: { state: TriageState }) {
const d = state.disposition;
if (!d) return null;
const planted = state.reveal ? bestAlternate(d.cites, state.reveal.planted) : null;
const scored = state.reveal !== null && state.outcome === 'solved';
return (
<Block
title={d.action === 'close' ? 'Closed' : 'Escalated'}
aside={
<span className="font-mono">{d.action === 'close' ? enumLabel(d.reason) : d.typology}</span>
}
>
<p className="mb-1 text-xs text-muted">Cites</p>
<div className="flex flex-wrap gap-1">
{d.cites.map((id) => (
<CiteChip key={id} id={id} hit={planted && scored ? planted.includes(id) : null} />
))}
</div>
{(d.action === 'close' ? d.note : d.narrative).trim() ? (
<div className="mt-2">
<p className="mb-1 flex items-baseline gap-2 text-xs text-muted">
<span>{d.action === 'close' ? 'Note' : 'Narrative'}</span>
<span className="rounded border border-border px-1 py-px text-[0.65rem] uppercase tracking-wide">
displayed, never scored
</span>
</p>
<blockquote className="border-l-2 border-border pl-2 text-xs leading-relaxed">
{d.action === 'close' ? d.note : d.narrative}
</blockquote>
</div>
) : null}
</Block>
);
}
function verdictLine(state: TriageState, reveal: Reveal): { text: string; tone: 'positive' | 'danger' | 'warning' } {
if (state.status === 'aborted') {
return {
text: 'No disposition in eight replies. The alert stays in the queue and the episode scores 0.000 on every component.',
tone: 'danger',
};
}
if (state.outcome === 'solved') {
return {
text: reveal.label === 'benign' ? 'Right call: the alert was benign and it was closed.' : 'Right call: the alert was suspicious and it was escalated.',
tone: 'positive',
};
}
if (reveal.label === 'suspicious') {
return {
text: 'Missed: a suspicious alert was closed. The counterweight and both objectives are zero — the episode scores 0.000.',
tone: 'danger',
};
}
return {
text: 'A clean customer was escalated. Nothing slipped, but a report was filed on them: the counterweight pays half and the two objectives pay nothing.',
tone: 'warning',
};
}
function RevealCard({ state }: { state: TriageState }) {
const reveal = state.reveal;
if (!reveal) return null;
const cites = state.disposition?.cites ?? [];
const best = bestAlternate(cites, reveal.planted);
const verdict = verdictLine(state, reveal);
const TONE = { positive: 'text-positive', danger: 'text-danger', warning: 'text-warning' } as const;
return (
<Block
title="What the generator planted"
aside={
<span className="font-mono">
{reveal.template}
{reveal.overlay ? ` + ${reveal.overlay.replace(/_/g, ' ')}` : ''}
</span>
}
>
<p className="text-sm">
<span className={cn('font-semibold', reveal.label === 'benign' ? 'text-positive' : 'text-danger')}>
{reveal.label}
</span>
{reveal.typology ? (
<>
<span className="mx-1.5 text-muted">·</span>
<span className="font-mono text-xs">{reveal.typology}</span>
</>
) : null}
{reveal.tier === 'hidden' ? (
<>
<span className="mx-1.5 text-muted">·</span>
<span className="text-xs text-muted">hidden tier: the screen is a benign case; the signal was only in a lookup</span>
</>
) : null}
</p>
<p className={cn('mt-1 text-xs font-medium', TONE[verdict.tone])} role="status">
{verdict.text}
</p>
<p className="mb-1 mt-2 text-xs text-muted">
The facts that decide it{reveal.planted.length > 1 ? ' (best of the alternates)' : ''}
</p>
<div className="flex flex-wrap gap-1">
{best.map((id) => (
<CiteChip key={id} id={id} hit={cites.includes(id) ? true : null} />
))}
</div>
<p className="mt-2 text-xs text-muted">
Hours spent <span className="nums text-fg">{hours(state.minutesSpent)}</span>
{reveal.referenceMinutes !== null ? (
<>
{' '}
· cheapest shipped analyst that got it right{' '}
<span className="nums text-fg">{hours(reveal.referenceMinutes)}</span>
</>
) : null}
</p>
</Block>
);
}
function Meter({ state }: { state: TriageState }) {
const STATUS: Record<TriageState['status'], string> = {
playing: 'in the queue',
closed: 'closed',
escalated: 'escalated',
aborted: 'left in the queue',
};
return (
<div className="flex flex-wrap items-center gap-x-3 gap-y-1 text-xs text-muted">
<span>
Hours <span className="nums font-semibold text-fg">{hours(state.minutesSpent)}</span>
</span>
<span>
Replies{' '}
<span className="nums font-semibold text-fg">
{state.turnsSpent} of {MAX_TURNS}
</span>
</span>
{state.rejected > 0 ? (
<span>
refused <span className="nums font-semibold text-fg">{state.rejected}</span>
</span>
) : null}
<span
className={cn(
'rounded-md border px-1.5 py-px font-medium',
state.status === 'playing' && 'border-border',
state.status === 'closed' && 'border-positive/50 text-positive',
state.status === 'escalated' && 'border-warning/60 text-warning',
state.status === 'aborted' && 'border-danger/50 text-danger',
)}
>
{STATUS[state.status]}
</span>
{state.screen.held_out ? (
<span className="rounded-md border border-border px-1.5 py-px" title="fnv1a(seed) % 8 == 7: second currency, renamed rules, KYC before the alert">
held-out seed
</span>
) : null}
</div>
);
}
/* --------------------------------------------------------------- compact */
/**
* The thumbnail: what the timeline chip and the gallery card show. A whole
* screen at 7rem is noise, so this is the shape of the run instead — the rule,
* the twelve months of cash as bars, and what has happened so far.
*/
function CompactBoard({ state }: { state: TriageState }) {
const rows = state.screen.summary;
const peak = Math.max(1, ...rows.map((r) => r.cash_in));
const d = state.disposition;
const label =
`${state.screen.alert.rule}, ${hours(state.minutesSpent)} hours, ${state.lookups.length} lookups` +
(d ? `, ${d.action === 'close' ? 'closed' : 'escalated'}` : state.status === 'aborted' ? ', aborted' : '');
return (
<div className="w-[7.5rem] text-[9px] leading-tight" role="img" aria-label={label}>
<SyntheticLabel compact />
<div className="mt-1 flex items-baseline justify-between gap-1">
<span className="truncate font-mono font-semibold">{state.screen.alert.rule}</span>
<span className="nums shrink-0 text-muted">{hours(state.minutesSpent)} h</span>
</div>
<div className="mt-1 flex h-4 items-end gap-px" aria-hidden="true">
{rows.map((r) => (
<span
key={r.month}
className="flex-1 rounded-sm bg-muted/50"
style={{ height: `${Math.max(6, Math.round((r.cash_in / peak) * 100))}%` }}
/>
))}
</div>
<div className="mt-1 flex flex-wrap gap-0.5">
{state.lookups.map((l, i) => (
<span key={i} className="rounded border border-border px-0.5 text-muted">
{l.what === 'history' ? l.key.slice(5) : l.what === 'counterparty' ? l.key : l.what === 'prior_alerts' ? 'priors' : 'docs'}
</span>
))}
{state.rejected > 0 ? (
<span className="rounded border border-warning/60 px-0.5 text-warning">!{state.rejected}</span>
) : null}
{d ? (
<span
className={cn(
'rounded border px-0.5 font-semibold uppercase',
d.action === 'close' ? 'border-positive/50 text-positive' : 'border-warning/60 text-warning',
)}
>
{d.action === 'close' ? 'close' : 'escalate'}
</span>
) : state.status === 'aborted' ? (
<span className="rounded border border-danger/50 px-0.5 font-semibold uppercase text-danger">aborted</span>
) : null}
</div>
</div>
);
}
/* ----------------------------------------------------------------- board */
export const Board = memo(function Board({ state, compact }: { state: TriageState; compact?: boolean }) {
if (compact) return <CompactBoard state={state} />;
const { screen } = state;
const blocks =
screen.screen_order === 'kyc_first'
? [<KycBlock key="kyc" screen={screen} />, <AlertBlock key="alert" screen={screen} />]
: [<AlertBlock key="alert" screen={screen} />, <KycBlock key="kyc" screen={screen} />];
return (
<div
// An explicit width, capped by the VIEWPORT. The board sits in a
// shrink-to-fit card on every tab, so `w-full` resolves to nothing and
// `max-w-full` resolves against a parent that sized itself to this
// board — measured: 640px on a 390px phone, 304px of page overflow.
// 5rem is the page gutters plus the card padding at the phone breakpoint.
// `max-w-full` is for the one place the parent IS definite: a blind
// comparison's grid cell, which is narrower than 40rem at 1280px and
// measured 31px of page overflow without it.
className="w-[min(40rem,calc(100vw-5rem))] max-w-full min-w-0 space-y-3 text-sm"
role="group"
aria-label={`Alert ${screen.alert.id} on customer ${screen.customer.id}. ${state.lookups.length} lookups made, ${hours(state.minutesSpent)} hours spent.`}
>
<SyntheticLabel />
<Meter state={state} />
{blocks}
<SummaryBlock screen={screen} />
<CounterpartyBlock screen={screen} />
{state.lookups.map((lookup, i) => (
<LookupCard key={`${lookup.what}-${lookup.key ?? i}`} lookup={lookup} currency={screen.currency} />
))}
{state.lastRejection ? (
<p role="status" className="rounded-md border border-warning/60 bg-warning/5 px-3 py-2 text-xs text-warning">
Reply refused: {state.lastRejection}. It still cost a turn.
</p>
) : null}
<DispositionCard state={state} />
<RevealCard state={state} />
</div>
);
});
export default Board;
src/demos/alert-triage/types.ts
+117
View File
@@ -0,0 +1,117 @@
/**
* The board, as the surface renders it.
*
* The pieces of the world — transactions, the KYC block, the alert, the
* summary — are the port's own types, re-exported from `./generator` so the
* surface is typed against the shapes CI hashes (digest `world` in
* `envs/alert_triage/CONFORMANCE.txt`) rather than a restatement of them.
* Field names are snake_case because they are the canonical JSON's keys.
*
* Amounts are integers in minor units, never floats; dates are `YYYY-MM-DD`;
* months are `YYYY-MM`. Everything is synthetic.
*
* Types only, plus the two enums the controls offer as choices.
*/
export type {
Alert,
Counterparty,
Customer,
Document,
Label,
Overlay,
PriorAlert,
Screen,
ScreenCounterparty,
SummaryRow,
Template,
Tier,
Txn as Transaction,
} from './generator';
import type { Counterparty, Document, Label, Overlay, PriorAlert, Screen, Template, Tier, Txn } from './generator';
/** One accepted lookup and what it returned, in the order they were made. */
export type Lookup =
| { what: 'history'; key: string; result: { month: string; transactions: Txn[] } }
| { what: 'counterparty'; key: string; result: Counterparty }
| { what: 'prior_alerts'; key: null; result: { alerts: PriorAlert[] } }
| { what: 'documents'; key: null; result: { documents: Document[] } };
export type LookupKind = Lookup['what'];
export type CloseReason =
| 'CONSISTENT_WITH_PROFILE'
| 'DOCUMENTED_SOURCE_OF_FUNDS'
| 'PREVIOUSLY_REVIEWED'
| 'RULE_ARTEFACT';
export type Typology = 'STRUCTURING' | 'FUNNEL_ACCOUNT' | 'MONEY_MULE' | 'RAPID_MOVEMENT' | 'UNKNOWN';
export const CLOSE_REASONS: readonly CloseReason[] = [
'CONSISTENT_WITH_PROFILE',
'DOCUMENTED_SOURCE_OF_FUNDS',
'PREVIOUSLY_REVIEWED',
'RULE_ARTEFACT',
];
export const TYPOLOGIES: readonly Typology[] = [
'STRUCTURING',
'FUNNEL_ACCOUNT',
'MONEY_MULE',
'RAPID_MOVEMENT',
'UNKNOWN',
];
/** An accepted disposition, in the engine's canonical form. Ends the episode. */
export type Disposition =
| { action: 'close'; reason: CloseReason; cites: string[]; note: string }
| { action: 'escalate'; typology: Typology; cites: string[]; narrative: string };
/**
* What the generator planted, shown only once the episode is over. On the
* board before then it would be the answer key taped to the exam.
*/
export interface Reveal {
label: Label;
typology: string | null;
/** Alternates; the grader scores the best one. */
planted: string[][];
template: Template;
overlay: Overlay | null;
tier: Tier | null;
/**
* Minutes the cheapest shipped analyst needed on this seed. Null when the
* board does not know — a recorded run carries it as a metric; a fresh
* board in Play does only if the port ran the reference policies.
*/
referenceMinutes: number | null;
}
export type Status = 'playing' | 'closed' | 'escalated' | 'aborted';
/**
* The board after a step. Snapshots, never deltas: the scrubber jumps to any
* step, and every field the surface reads is here in full.
*/
export interface TriageState {
seed: number;
screen: Screen;
lookups: Lookup[];
/** Replies spent, accepted or not. Eight ends the episode. */
turnsSpent: number;
rejected: number;
/** The hour meter, in minutes, so it agrees with the grader to the digit. */
minutesSpent: number;
/** Why the most recent reply was refused, or null if it was accepted. */
lastRejection: string | null;
disposition: Disposition | null;
status: Status;
/** The grader's verdict on the disposition, once there is one. */
outcome: 'solved' | 'failed' | 'aborted' | null;
/** Every id the engine has rendered so far — the only ids a disposition may cite. */
shown: string[];
/** The replies played so far. The seam replays these to take the next step. */
replies: (string | null)[];
reveal: Reveal | null;
}
uv.lock
Generated
+15
View File
@@ -12,6 +12,7 @@ resolution-markers = [
[manifest]
members = [
"alert-triage",
"pig-demo-envs",
"wordle-five",
]
@@ -133,6 +134,20 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/fb/76/641ae371508676492379f16e2fa48f4e2c11741bd63c48be4b12a6b09cba/aiosignal-1.4.0-py3-none-any.whl", hash = "sha256:053243f8b92b990551949e63930a839ff0cf0b0ebbe0597b0f3fb19e1a0fe82e", size = 7490, upload-time = "2025-07-03T22:54:42.156Z" },
]
[[package]]
name = "alert-triage"
version = "0.1.0"
source = { editable = "envs/alert_triage" }
[package.optional-dependencies]
verifiers = [
{ name = "verifiers" },
]
[package.metadata]
requires-dist = [{ name = "verifiers", marker = "extra == 'verifiers'", specifier = ">=0.3.2.dev12" }]
provides-extras = ["verifiers"]
[[package]]
name = "annotated-types"
version = "0.8.0"