Alert Triage: environment #2, built end to end by the pipeline
The first environment shipped through .claude/workflows/new-environment.js: specification, three adversarial reviews (all 'fixable', none fatal), the Python environment, the TypeScript port, captured rollouts, and the demo page. Eleven agents, no errors. The proof that the platform scales is one line long. Alert Triage has a completely different shape from Word Five — JSON actions, priced lookups, an analyst screen instead of a grid — and the only change under src/components/demo/ is a comment edit, because the isolation lint refused the word "wordle" there. Zero shell code changed. 415 contract checks now pass against two demos, up from 206 against one. The environment is honest by construction. Every alert is synthetic, generated from the seed, and the banner saying so sits inside the board surface. Two of the eleven scenario templates are hidden-suspicious: generated by the same code as their benign twin with the signal overlaid only in lookup data, so the free screen is identically distributed and a screen-only policy STRUCTURALLY cannot tell them apart. The probe ladder measures it: `fast` catches 0.0 of hidden seeds. That is the counterweight made real rather than asserted. Twelve policies, thirteen ladder assertions, a genuine three-way trade: fast 0.846 wins hours (0.85), misses every hidden case targeted 0.894 wins the shipped total thorough 0.820 wins evidence (1.00), spends 2.9 hours None dominates. 92 Python tests, 35 TypeScript tests, 65 fixtures replaying at delta 0, and conformance gated on world + scorer + protocol so the browser shows the same alert for ?seed= that Python generated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019mt6sHQHEnEYrJZvoMCJSB
This commit is contained in:
karti-ai
@@ -0,0 +1,3 @@
|
||||
505ae48b664aae23aea01208879d899701cc2f6babd8e76ccbe36e75f9ba1b34 world sha256 over canonical world JSON, seeds 0-4095, newline-joined, reference_minutes included
|
||||
f370d5f6f28964a070e926e7469ffbdea595fcb9ce45b4dbc7e16f607ed4cc47 scorer sha256 over score_exact() of the fixed grid in conformance.py, newline-joined
|
||||
63dafb1034f19973fd208345f5bec8c7abd9839d8e1c11c2bb1ed8d581f34205 protocol sha256 over protocol_trace() of conformance/replies.json, newline-joined
|
||||
@@ -0,0 +1,259 @@
|
||||
# alert-triage — the environment
|
||||
|
||||
One transaction-monitoring alert on one synthetic customer. The agent reads
|
||||
the analyst's screen, may pay analyst-hours for lookups, and ends the episode
|
||||
by closing the alert with a cited rationale or escalating it with a typology
|
||||
and the transactions that constitute it. The grader compares the disposition
|
||||
and the citations to what the generator planted and charges the hours. No
|
||||
model, human or prose is read anywhere in it.
|
||||
|
||||
`SPEC.md` is the specification as written and reviewed before this code
|
||||
existed. This file records **what was implemented**, including every change
|
||||
the reviewers demanded; where the two differ, this file is what ships.
|
||||
|
||||
```bash
|
||||
uv run python envs/probe.py --taskset alert-triage # the ladder
|
||||
uv run pytest envs/alert_triage/tests # ~30 s; hashes 4096 worlds
|
||||
uv run python -m alert_triage.conformance # the three digests
|
||||
```
|
||||
|
||||
## Layout
|
||||
|
||||
| file | what |
|
||||
|---|---|
|
||||
| `alert_triage/rng.py` | FNV-1a seeding, xorshift32, civil dates. No platform RNG, no platform date. |
|
||||
| `alert_triage/generator.py` | Templates, overlays, the canonical world, `reference_minutes`, digest (a). |
|
||||
| `alert_triage/engine.py` | Screen, lookups, dispositions, shown-id discipline, the minute meter. |
|
||||
| `alert_triage/protocol.py` | The pinned parse rules, the system prompt, text renderers. |
|
||||
| `alert_triage/reward.py` | The three components. The `# region: pig-demo/reward` block is what the page quotes. |
|
||||
| `alert_triage/policies.py` | `fast`, `targeted`, `thorough` and the seven cheats. Read only the engine's view. |
|
||||
| `alert_triage/ladder.py` | The probe rungs and their thirteen assertions. |
|
||||
| `alert_triage/conformance.py` | Digests (b) and (c); writes `CONFORMANCE.txt`. |
|
||||
| `alert_triage/taskset.py` | `play_episode`, `replay`, and the optional verifiers Env. |
|
||||
| `conformance/replies.json` | The reply corpus digest (c) runs. |
|
||||
| `names/` | The synthetic name and word lists, with `PROVENANCE.md`. |
|
||||
|
||||
## The world, and what the reviewers changed
|
||||
|
||||
**Templates.** Six benign builders (B1 seasonal cash, B2 processor
|
||||
settlement, B3 property sale, B4 cash restaurant, B5 family remittance, B6
|
||||
inheritance) and three visible-suspicious ones (S1 structuring, S3 money mule,
|
||||
S6 smurfing over time). The hidden tier is no longer two fixed twins. A
|
||||
hidden alert is any benign builder run to completion — screen frozen — with
|
||||
one of three **overlays** applied to lookup data only:
|
||||
|
||||
| overlay | where it lives | compatible with | planted set |
|
||||
|---|---|---|---|
|
||||
| `funnel` | the counterparty lookup: the top outbound counterparty on screen is shared with 5–14 other alerted accounts | every benign template | `CP-x` + its transactions; alternate: the transactions |
|
||||
| `serial_closer` | the prior-alerts lookup: every prior alert on the same rule, closed within a day on one identical boilerplate, by different analysts | screens showing ≥ 3 prior alerts and no prior report | the `A-` ids + the triggering transactions; alternate: the `A-` ids |
|
||||
| `doc_mismatch` | the documents lookup against the counterparty lookup: the document names a party; the funds came from someone else | B2, B3, B6 | the document + `CP-x` + its triggering credits; alternate: the document + `CP-x` |
|
||||
|
||||
The tier is drawn *before* the template and the screen, and the prior-alert
|
||||
count on the KYC block is drawn before any template code runs, so the screen
|
||||
distribution is the same for hidden and benign by construction, not by
|
||||
effort. `tests/test_triage_screen_leak.py` asserts the screen is
|
||||
byte-identical with every compatible overlay on or off for every seed, and
|
||||
that no screen feature separates the populations beyond four sigma.
|
||||
|
||||
This answers three reviewer findings at once: every rule id now fires on both
|
||||
labels (`R-WIRE-04` and `R-DORM-05` carry funnel and serial-closer twins);
|
||||
each hidden signal lands on several templates so there is no template→lookup
|
||||
mapping to memorise; and no screen-identifiable segment carries a suspicious
|
||||
prior above the overall 15 / (55 + 15) ≈ 21 %.
|
||||
|
||||
**Removed:** B4's standalone `PREVIOUSLY_REVIEWED` alternate. A prior closure
|
||||
never resolves an alert on its own. The reason stays in the enum as legal and
|
||||
never planted, so its use renders as a diagnostic.
|
||||
|
||||
**Documents** carry per-world ids (`doc.D-<4 hex>`), and the engine rejects
|
||||
any citation of an id it has not rendered. `fast` therefore pulls
|
||||
`documents` on the templates a document explains, and the reference on those
|
||||
seeds is 0.70 h, not 0.55 h.
|
||||
|
||||
**The held-out slice** is `fnv1a(str(seed)) % 8 == 7`, about 12.5 % of
|
||||
seeds. It is generated in a **fictional** currency, KRN, with a fictional
|
||||
20,000 KRN reporting threshold (amounts scale ×2), rule ids renamed
|
||||
`TM-A1..A5`, the KYC block before the alert and the summary newest-first. It
|
||||
is never probed and never used to tune anything; the capture stage runs it
|
||||
once, after the arms are frozen, for the side-by-side numbers on the page.
|
||||
No claim about any real jurisdiction's threshold is made anywhere.
|
||||
|
||||
## Canonical world
|
||||
|
||||
Sorted keys, no whitespace, `null` for absent. Every amount is an integer in
|
||||
minor units; every date is a `YYYY-MM-DD` string produced by
|
||||
`rng.iso_date` from a day ordinal. `reference_minutes` and
|
||||
`reference_policy` are **part of the world** and therefore of digest (a):
|
||||
the browser recomputes them by running the ported policies rather than
|
||||
trusting the recorded number, and a mismatch is a verification failure, not
|
||||
an "unverifiable".
|
||||
|
||||
Draw order is the contract. Reading `generator.py` top to bottom in the
|
||||
order the builder calls things is the port; the port must not reorder a
|
||||
single `rng` call.
|
||||
|
||||
## The action set and the pinned parse rules
|
||||
|
||||
One JSON object per reply. A ```` ```json ```` fence anywhere wins; otherwise
|
||||
the first string-aware balanced `{…}` span; an unparseable or non-object
|
||||
candidate is a rejection with no further scanning. Enums exact-case. Cites
|
||||
deduplicated in order, matched exact-case against the shown set. The
|
||||
twelve-month window is the twelve calendar months ending in the fire month
|
||||
inclusive. A repeated lookup is rejected. Only accepted lookups add their
|
||||
price; every turn, accepted or rejected, costs `TURN_MINUTES`.
|
||||
|
||||
`aborted` is exactly `MAX_TURNS = 8` turns spent with no accepted
|
||||
disposition and scores 0.000. A run that stops earlier with no disposition is
|
||||
`truncated` and every component is `null` — rendered "not scored", never
|
||||
zero. Correct disposition → `solved`, wrong → `failed`.
|
||||
|
||||
## Hours
|
||||
|
||||
Integers in minutes, shown as hours. **The generator's config, not
|
||||
measurements** — nobody timed an analyst:
|
||||
|
||||
| constant | minutes | hours |
|
||||
|---|---|---|
|
||||
| `SCREEN_MINUTES` | 30 | 0.50 |
|
||||
| `TURN_MINUTES` | 3 | 0.05 |
|
||||
| `LOOKUP_MINUTES` | 24 | 0.40 |
|
||||
| `DOC_MINUTES` | 6 | 0.10 |
|
||||
|
||||
`LOOKUP_MINUTES` was raised from the spec's 0.25 h to 0.40 h so that the
|
||||
hours spent investigating clean screens genuinely exceed the hours a
|
||||
screen-only analyst loses to misses — the trade-off the reviewers said was
|
||||
manufactured in the spec is real at these constants and the ladder measures
|
||||
it (rung 8).
|
||||
|
||||
## The reward
|
||||
|
||||
| key | weight | role | definition |
|
||||
|---|---|---|---|
|
||||
| `caught` | 0.55 | counterweight | 1.0 on a correct call; 0.0 on a miss or no disposition; **0.5 on a false escalation** ("filed on a clean customer"). |
|
||||
| `hours` | 0.25 | objective | `min(1, reference / spent)` if correct, else 0. `null` if no shipped policy got the seed right (CI asserts this never happens). Capped: faster than the reference earns 1.0, not more. |
|
||||
| `evidence` | 0.20 | objective | F1 of the cites against the best planted alternate; 0 if the call was wrong. |
|
||||
|
||||
The 0.5 credit is the reviewers' fix for segment-blind defensive filing: with
|
||||
it, a blind escalation only beats a well-cited close when P(benign) is below
|
||||
**0.52** (`reward.escalate_break_even()`, printed by the probe), and no
|
||||
benign-looking screen segment is that suspicious.
|
||||
|
||||
## The ladder, measured
|
||||
|
||||
`uv run python envs/probe.py --taskset alert-triage`, 894 main-slice seeds
|
||||
(seeds 0–1023 minus the held-out bucket). These are the numbers the page may
|
||||
quote; the spec's `~0.86 vs ~0.85` text may not.
|
||||
|
||||
```
|
||||
policy caught hours evidence TOTAL hours@0.45
|
||||
inaction 0.0000 0.0000 0.0000 0.0000 0.0000
|
||||
malformed 0.0000 0.0000 0.0000 0.0000 0.0000
|
||||
close_blind 0.5705 0.5705 0.0000 0.4564 0.4868
|
||||
close_cites_all 0.5705 0.5705 0.0815 0.4727 0.4988
|
||||
cite_unseen 0.5705 0.5473 0.0000 0.4506 0.4764
|
||||
escalate_blind 0.7148 0.4295 0.3006 0.5606 0.5257
|
||||
fast_defensive 0.7148 0.4295 0.3006 0.5606 0.5257
|
||||
rule_table 0.7550 0.5593 0.3680 0.6287 0.6102
|
||||
fast 0.8535 0.8535 0.8159 0.8460 0.8480
|
||||
targeted 1.0000 0.6039 0.9625 0.8935 0.8163
|
||||
thorough 1.0000 0.2784 1.0000 0.8196 0.6753
|
||||
thorough_wasteful 1.0000 0.2412 1.0000 0.8103 0.6585
|
||||
```
|
||||
|
||||
Mean hours: fast 0.60, targeted 1.63, thorough 2.92. Caught on hidden seeds:
|
||||
fast 0.0, targeted 1.0, thorough 1.0.
|
||||
|
||||
The three shipped analysts, and the claim the page can make:
|
||||
|
||||
- **`fast`** decides from the screen (plus `documents` where a document
|
||||
should explain the alert). Wins `hours` outright. Misses every hidden
|
||||
seed, and only those.
|
||||
- **`targeted`** is `fast` on a screen that shows a typology, and otherwise
|
||||
pulls exactly the three lookups that could carry a hidden signal. Catches
|
||||
everything. **Wins at shipped weights, 0.894 to 0.846.** It defines
|
||||
`reference_minutes` on hidden seeds.
|
||||
- **`thorough`** always runs the full procedure, including two months of
|
||||
itemised history, and cites every constituting transaction it saw:
|
||||
evidence 1.000 exactly, and the lowest hours.
|
||||
|
||||
Move `hours` to **0.45** (the other two weights keeping their ratio) and
|
||||
`fast` wins, 0.848 to 0.816. Nothing else on the page depends on the
|
||||
direction; rung 8 asserts both margins are at least 0.02.
|
||||
|
||||
Note the weakest honest reading: `thorough` — which gets every call and
|
||||
every citation right — scores *below* `fast` at shipped weights, because
|
||||
spending five times the hours on the 85 % of alerts the screen settles costs
|
||||
more than the misses. That is the environment's argument, not a defect.
|
||||
|
||||
### Rungs a reviewer named
|
||||
|
||||
| rung | the cheat | assertion |
|
||||
|---|---|---|
|
||||
| `fast_defensive` | file blind on every benign-looking screen | total < every good policy |
|
||||
| `cite_unseen` | close citing a document it never opened | rejected on every seed; total < fast |
|
||||
| `rule_table` | close blind on wire/dormancy rules, file blind elsewhere | total < every good policy |
|
||||
| `close_cites_all` | cite every id on the screen | evidence < ½ its caught share; < fast's evidence |
|
||||
| `thorough_wasteful` | pull a month that carries nothing | total < thorough, same caught and evidence |
|
||||
|
||||
## The reference analysts see only the view
|
||||
|
||||
`policies.py` functions take `Engine.view()` and nothing else — the
|
||||
screen, the lookups so far, the turn and minute counters. Tests assert the
|
||||
signature, that the view carries none of `label`, `typology`, `planted`,
|
||||
`overlay`, `tier`, `template`, `reference_*`, and that every policy emits
|
||||
identical replies against a world with those fields deleted.
|
||||
|
||||
## Conformance
|
||||
|
||||
Three digests in `CONFORMANCE.txt`, each a CI gate for the TypeScript port:
|
||||
|
||||
1. **world** — canonical world JSON for seeds 0–4095, reference included.
|
||||
The port must carry the generator, not only the scorer: the Play tab and
|
||||
`?seed=` need the same alert on both sides.
|
||||
2. **scorer** — `score_exact()` over a fixed grid of episodes. Exact
|
||||
fractions, so no float formatting is hashed.
|
||||
3. **protocol** — parse + engine over `conformance/replies.json` on fixed
|
||||
worlds: accepted actions, rejection reasons, and the minute meter after
|
||||
every turn. This is the digest wordle does not need.
|
||||
|
||||
## What this is easier than, for the page
|
||||
|
||||
1. Every alert has a knowable answer; real SAR decisions are judgements, and a
|
||||
filed report is not a proven crime. Nine builders and three overlays stand
|
||||
in for a typology space in the hundreds.
|
||||
2. The 55/30/15 mix is deliberately not the real base rate. The great
|
||||
majority of production alerts close as false positives — no figure is
|
||||
quoted here because none was measured by us. "Hours per thousand alerts"
|
||||
on the page is Σ `hours_spent` over recorded seeds inside this
|
||||
environment, with the constants above beside it.
|
||||
3. The hour constants are assumptions.
|
||||
4. **The lookups are idealised.** Each returns the decisive fact in one
|
||||
call: a counterparty's other-alerted-account count, a prior alert's
|
||||
analyst and rationale, a document's named party. In production those
|
||||
signals are spread across systems and often absent. This environment
|
||||
tests whether the agent asks; it does not test whether the answer exists.
|
||||
5. The narrative prose is unscored.
|
||||
6. The adversary is static: the launderer shaped the data to pass the screen
|
||||
and the held-out slice moves the threshold once. Nothing responds to the
|
||||
agent's policy. The demo must claim exactly that and no more, and the
|
||||
wordle limit that names this demo as its answer should be softened to
|
||||
"nobody designed the data to pass the screen".
|
||||
7. `caught` is 1.0 on every benign alert by construction; the counterweight
|
||||
is argued from the ladder over seeds, never from one run.
|
||||
|
||||
## For the next stages
|
||||
|
||||
- **Port**: `rng.py`, `generator.py`, `engine.py`, `protocol.py`,
|
||||
`reward.py`, `policies.py` (`fast`, `targeted`, `thorough` — verify
|
||||
recomputes `reference_minutes`). Gate on all three digests.
|
||||
- **Capture**: `taskset.play_episode(seed, respond)` is the loop;
|
||||
`taskset.replay(seed, replies)` re-scores a fixture from its reply strings.
|
||||
Scripted arms: `fast`, `targeted`, `thorough`, and at least one cheat.
|
||||
Capture the held-out bucket exactly once, after the arms are frozen.
|
||||
- **Demo**: the board must carry "Synthetic data — no real customer, account
|
||||
or transaction"; the hour constants as config; the narrative marked "not
|
||||
scored"; the label reveal only after a disposition. The vertical card's
|
||||
`reward` line must be amended to say "hours as a ratio against a reference
|
||||
analyst policy", not "hours per thousand alerts".
|
||||
- `SPEC.md` is stage 1's document and was not edited; §3, §4, §6, §7 and
|
||||
§8 are superseded by this file where they differ.
|
||||
@@ -0,0 +1,11 @@
|
||||
"""alert-triage: one synthetic transaction-monitoring alert per seed.
|
||||
|
||||
Public surface, in the order the next stages need it:
|
||||
|
||||
generator.world_for_seed(seed) the world, canonical and portable
|
||||
engine.Engine(world) lookups, dispositions, the hour meter
|
||||
protocol.parse_reply(text) one JSON object -> action or rejection
|
||||
reward.score(episode) the three components, or None
|
||||
policies.POLICIES the scripted analysts
|
||||
taskset.play_episode(seed, fn) the loop capture, probe and tests share
|
||||
"""
|
||||
@@ -0,0 +1,124 @@
|
||||
"""The three cross-language digests. `CONFORMANCE.txt` commits them; CI gates them.
|
||||
|
||||
(a) world SHA-256 over the canonical JSON world for seeds 0–4095,
|
||||
reference minutes included. The browser must show the same
|
||||
alert for ?seed= as Python generated.
|
||||
(b) scorer SHA-256 over the exact-fraction scores of a fixed grid of
|
||||
synthetic episodes: label × disposition × cites × planted ×
|
||||
minutes × reference. No floats are hashed.
|
||||
(c) protocol SHA-256 over what parse + engine make of a committed corpus of
|
||||
reply strings (`conformance/replies.json`) on fixed worlds:
|
||||
accepted actions, rejection reasons and the minute meter after
|
||||
every turn. This is the digest wordle does not need and this
|
||||
environment cannot do without — JSON-in-prose with fences and a
|
||||
brace scan is where two runtimes disagree.
|
||||
|
||||
Run: uv run python -m alert_triage.conformance [--write]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from .engine import Engine
|
||||
from .generator import canonical_json, world_digest, world_for_seed
|
||||
from .reward import Episode, score_exact
|
||||
|
||||
ROOT = Path(__file__).parent.parent
|
||||
CORPUS = ROOT / "conformance" / "replies.json"
|
||||
COMMITTED = ROOT / "CONFORMANCE.txt"
|
||||
|
||||
GRID_LABELS = ("benign", "suspicious")
|
||||
GRID_DISPOSITIONS = ("close", "escalate", "none", "truncated")
|
||||
GRID_CITES = (["a"], ["a", "b"], ["a", "b", "c", "d"], ["x"], ["a", "x"], ["a", "b", "x", "y", "z"], ["a", "a", "b"], ["kyc.z"])
|
||||
GRID_PLANTED = ([["a"]], [["a", "b"]], [["a", "b"], ["a"]], [["c"], ["a", "b", "c", "d"]])
|
||||
GRID_MINUTES = (33, 42, 60, 96, 150, 204)
|
||||
GRID_REFERENCE = (33, 42, 96, None)
|
||||
|
||||
|
||||
def scorer_digest() -> str:
|
||||
digest = hashlib.sha256()
|
||||
for label in GRID_LABELS:
|
||||
for disposition in GRID_DISPOSITIONS:
|
||||
for cites in GRID_CITES:
|
||||
for planted in GRID_PLANTED:
|
||||
for minutes in GRID_MINUTES:
|
||||
for reference in GRID_REFERENCE:
|
||||
ep = Episode(
|
||||
label=label,
|
||||
planted=planted,
|
||||
disposition=None if disposition in ("none", "truncated") else disposition,
|
||||
cites=[] if disposition in ("none", "truncated") else cites,
|
||||
minutes_spent=minutes,
|
||||
reference_minutes=reference,
|
||||
turns_spent=1,
|
||||
rejected=0,
|
||||
truncated=disposition == "truncated",
|
||||
)
|
||||
digest.update(canonical_json(score_exact(ep)).encode())
|
||||
digest.update(b"\n")
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def protocol_trace(corpus: list[dict] | None = None) -> list[dict]:
|
||||
corpus = corpus if corpus is not None else json.loads(CORPUS.read_text())
|
||||
out = []
|
||||
for case in corpus:
|
||||
engine = Engine(world_for_seed(case["seed"]))
|
||||
turns = []
|
||||
for reply in case["replies"]:
|
||||
if engine.done:
|
||||
break
|
||||
step = engine.step(reply)
|
||||
turns.append({"accepted": step.accepted, "action": step.action, "reason": step.reason, "minutes": step.minutes_spent, "turns": step.turns_spent})
|
||||
ep = engine.episode()
|
||||
out.append({"seed": case["seed"], "turns": turns, "outcome": engine.outcome, "truncated": ep.truncated, "score": score_exact(ep)})
|
||||
return out
|
||||
|
||||
|
||||
def protocol_digest() -> str:
|
||||
digest = hashlib.sha256()
|
||||
for entry in protocol_trace():
|
||||
digest.update(canonical_json(entry).encode())
|
||||
digest.update(b"\n")
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def all_digests() -> dict[str, str]:
|
||||
return {"world": world_digest(), "scorer": scorer_digest(), "protocol": protocol_digest()}
|
||||
|
||||
|
||||
def committed() -> dict[str, str]:
|
||||
out = {}
|
||||
for line in COMMITTED.read_text().splitlines():
|
||||
parts = line.split()
|
||||
if len(parts) >= 2:
|
||||
out[parts[1]] = parts[0]
|
||||
return out
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
digests = all_digests()
|
||||
lines = [
|
||||
f"{digests['world']} world sha256 over canonical world JSON, seeds 0-4095, newline-joined, reference_minutes included",
|
||||
f"{digests['scorer']} scorer sha256 over score_exact() of the fixed grid in conformance.py, newline-joined",
|
||||
f"{digests['protocol']} protocol sha256 over protocol_trace() of conformance/replies.json, newline-joined",
|
||||
]
|
||||
if "--write" in argv:
|
||||
COMMITTED.write_text("\n".join(lines) + "\n")
|
||||
print(f"wrote {COMMITTED}")
|
||||
for line in lines:
|
||||
print(line)
|
||||
if COMMITTED.exists():
|
||||
mismatched = [k for k, v in committed().items() if digests.get(k) != v]
|
||||
if mismatched:
|
||||
print(f"MISMATCH against CONFORMANCE.txt: {mismatched}")
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main(sys.argv[1:]))
|
||||
@@ -0,0 +1,229 @@
|
||||
"""One episode: the free screen, up to eight replies, and the hour meter.
|
||||
|
||||
Pure and deterministic in the world it is given. It never raises on input: a
|
||||
bad reply is a rejection with a one-line reason, charged a turn.
|
||||
|
||||
Two discipline rules the reviewers demanded, both enforced here rather than in
|
||||
the grader:
|
||||
|
||||
* **Shown ids.** A disposition may cite only ids the engine has actually
|
||||
rendered — KYC fields and triggering transactions from the screen, plus
|
||||
whatever lookups returned. "Closed per the probate letter on file"
|
||||
without opening the file is a rejected reply, not a scored one.
|
||||
* **Abort versus truncate.** `aborted` is exactly MAX_TURNS turns spent
|
||||
with no accepted disposition, and scores 0.000. Anything that stops
|
||||
earlier with no disposition is `truncated` and is not scored at all.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
from .generator import MAX_TURNS, screen_of
|
||||
from .protocol import (
|
||||
ACTIONS,
|
||||
CLOSE_REASONS,
|
||||
LOOKUPS,
|
||||
TYPOLOGIES,
|
||||
_MONTH,
|
||||
parse_reply,
|
||||
)
|
||||
from .reward import DOC_MINUTES, LOOKUP_MINUTES, SCREEN_MINUTES, TURN_MINUTES, Episode
|
||||
|
||||
LOOKUP_PRICE = {"history": LOOKUP_MINUTES, "counterparty": LOOKUP_MINUTES, "prior_alerts": LOOKUP_MINUTES, "documents": DOC_MINUTES}
|
||||
|
||||
|
||||
@dataclass
|
||||
class Step:
|
||||
accepted: bool
|
||||
action: dict[str, Any] | None # canonical form of an accepted action
|
||||
reason: str | None # why a rejected reply was rejected
|
||||
result: dict[str, Any] | None # what an accepted lookup returned
|
||||
minutes_spent: int
|
||||
turns_spent: int
|
||||
done: bool
|
||||
|
||||
|
||||
@dataclass
|
||||
class Engine:
|
||||
world: dict[str, Any]
|
||||
turns: int = 0
|
||||
rejected: int = 0
|
||||
minutes: int = SCREEN_MINUTES
|
||||
actions: list[dict[str, Any]] = field(default_factory=list)
|
||||
lookups: list[dict[str, Any]] = field(default_factory=list)
|
||||
disposition: dict[str, Any] | None = None
|
||||
shown: set[str] = field(default_factory=set)
|
||||
last_rejection: str | None = None
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
screen = self.screen()
|
||||
self.shown.update(f"kyc.{k}" for k in screen["customer"] if k not in ("id", "name"))
|
||||
self.shown.add(screen["alert"]["id"])
|
||||
for t in screen["triggering_transactions"]:
|
||||
self.shown.add(t["id"])
|
||||
if t["cp"]:
|
||||
self.shown.add(t["cp"])
|
||||
for cp in screen["screen_counterparties"]:
|
||||
self.shown.add(cp["id"])
|
||||
|
||||
# ------------------------------------------------------------ views --
|
||||
|
||||
def screen(self) -> dict[str, Any]:
|
||||
screen = screen_of(self.world)
|
||||
if screen["screen_order"] == "kyc_first":
|
||||
screen["summary"] = list(reversed(screen["summary"]))
|
||||
return screen
|
||||
|
||||
def view(self) -> dict[str, Any]:
|
||||
"""Everything the model has been shown, and nothing it has not.
|
||||
|
||||
The scripted policies take this and only this. There is no label,
|
||||
typology, planted set, overlay or reference in it — `tests/` scans
|
||||
the keys to make sure.
|
||||
"""
|
||||
return {
|
||||
"screen": self.screen(),
|
||||
"lookups": list(self.lookups),
|
||||
"turns_spent": self.turns,
|
||||
"turns_left": MAX_TURNS - self.turns,
|
||||
"rejected": self.rejected,
|
||||
"minutes_spent": self.minutes,
|
||||
"last_rejection": self.last_rejection,
|
||||
}
|
||||
|
||||
@property
|
||||
def done(self) -> bool:
|
||||
return self.disposition is not None or self.turns >= MAX_TURNS
|
||||
|
||||
@property
|
||||
def aborted(self) -> bool:
|
||||
return self.disposition is None and self.turns >= MAX_TURNS
|
||||
|
||||
# ------------------------------------------------------------- step --
|
||||
|
||||
def step(self, reply: str | None) -> Step:
|
||||
if self.done:
|
||||
return Step(False, None, "the episode is over", None, self.minutes, self.turns, True)
|
||||
obj, reason = parse_reply(reply)
|
||||
action = None
|
||||
if obj is not None:
|
||||
action, reason = self._validate(obj)
|
||||
self.turns += 1
|
||||
self.minutes += TURN_MINUTES
|
||||
if action is None:
|
||||
self.rejected += 1
|
||||
self.last_rejection = reason
|
||||
return Step(False, None, reason, None, self.minutes, self.turns, self.done)
|
||||
|
||||
self.last_rejection = None
|
||||
self.actions.append(action)
|
||||
result = None
|
||||
if action["action"] == "lookup":
|
||||
self.minutes += LOOKUP_PRICE[action["what"]]
|
||||
result = self._lookup(action)
|
||||
self.lookups.append({"what": action["what"], "key": action.get("month") or action.get("id"), "result": result})
|
||||
else:
|
||||
self.disposition = action
|
||||
return Step(True, action, None, result, self.minutes, self.turns, self.done)
|
||||
|
||||
def _validate(self, obj: dict[str, Any]) -> tuple[dict[str, Any] | None, str | None]:
|
||||
action = obj.get("action")
|
||||
if action not in ACTIONS:
|
||||
return None, f"unknown action {action!r}; use lookup, close or escalate"
|
||||
if action == "lookup":
|
||||
what = obj.get("what")
|
||||
if what not in LOOKUPS:
|
||||
return None, f"unknown lookup {what!r}; use history, counterparty, prior_alerts or documents"
|
||||
if what == "history":
|
||||
month = obj.get("month")
|
||||
if not isinstance(month, str) or not _MONTH.match(month) or month not in self.world["months"]:
|
||||
return None, f"month {month!r} is not in the twelve-month window"
|
||||
if any(l["what"] == "history" and l["key"] == month for l in self.lookups):
|
||||
return None, f"history for {month} has already been looked up"
|
||||
return {"action": "lookup", "what": "history", "month": month}, None
|
||||
if what == "counterparty":
|
||||
cid = obj.get("id")
|
||||
if not isinstance(cid, str) or cid not in self.shown or not cid.startswith("CP-"):
|
||||
return None, f"counterparty {cid!r} has not appeared on your screen or in a fetched month"
|
||||
if any(l["what"] == "counterparty" and l["key"] == cid for l in self.lookups):
|
||||
return None, f"counterparty {cid} has already been looked up"
|
||||
return {"action": "lookup", "what": "counterparty", "id": cid}, None
|
||||
if any(l["what"] == what for l in self.lookups):
|
||||
return None, f"{what} has already been looked up"
|
||||
return {"action": "lookup", "what": what}, None
|
||||
|
||||
cites = obj.get("cites")
|
||||
if not isinstance(cites, list) or not cites:
|
||||
return None, "cites must be a non-empty list of ids"
|
||||
seen: list[str] = []
|
||||
for c in cites:
|
||||
if not isinstance(c, str):
|
||||
return None, "cites must be a non-empty list of ids"
|
||||
if c not in seen:
|
||||
seen.append(c)
|
||||
for c in seen:
|
||||
if c not in self.shown:
|
||||
return None, f"cites an id you have not been shown: {c}"
|
||||
if action == "close":
|
||||
reason = obj.get("reason")
|
||||
if reason not in CLOSE_REASONS:
|
||||
return None, f"unknown close reason {reason!r}"
|
||||
note = obj.get("note")
|
||||
return {"action": "close", "reason": reason, "cites": seen, "note": note if isinstance(note, str) else ""}, None
|
||||
typology = obj.get("typology")
|
||||
if typology not in TYPOLOGIES:
|
||||
return None, f"unknown typology {typology!r}"
|
||||
narrative = obj.get("narrative")
|
||||
return {"action": "escalate", "typology": typology, "cites": seen, "narrative": narrative if isinstance(narrative, str) else ""}, None
|
||||
|
||||
def _lookup(self, action: dict[str, Any]) -> dict[str, Any]:
|
||||
w = self.world
|
||||
what = action["what"]
|
||||
if what == "history":
|
||||
rows = [t for t in w["transactions"] if t["month"] == action["month"]]
|
||||
for t in rows:
|
||||
self.shown.add(t["id"])
|
||||
if t["cp"]:
|
||||
self.shown.add(t["cp"])
|
||||
return {"month": action["month"], "transactions": rows}
|
||||
if what == "counterparty":
|
||||
cp = next(c for c in w["counterparties"] if c["id"] == action["id"])
|
||||
self.shown.update(cp["transactions"])
|
||||
return dict(cp)
|
||||
if what == "prior_alerts":
|
||||
self.shown.update(p["id"] for p in w["prior_alerts"])
|
||||
return {"alerts": list(w["prior_alerts"])}
|
||||
self.shown.update(d["id"] for d in w["documents"])
|
||||
return {"documents": list(w["documents"])}
|
||||
|
||||
# ---------------------------------------------------------- outcome --
|
||||
|
||||
@property
|
||||
def outcome(self) -> str | None:
|
||||
"""solved | failed | aborted, or None while running / if truncated."""
|
||||
if self.disposition is not None:
|
||||
label = self.world["label"]
|
||||
right = (label == "benign") == (self.disposition["action"] == "close")
|
||||
return "solved" if right else "failed"
|
||||
return "aborted" if self.aborted else None
|
||||
|
||||
def episode(self, truncated: bool | None = None) -> Episode:
|
||||
if truncated is None:
|
||||
truncated = not self.done
|
||||
d = self.disposition
|
||||
return Episode(
|
||||
label=self.world["label"],
|
||||
planted=self.world["planted"],
|
||||
disposition=None if d is None else d["action"],
|
||||
cites=[] if d is None else list(d["cites"]),
|
||||
minutes_spent=self.minutes,
|
||||
reference_minutes=self.world.get("reference_minutes"),
|
||||
turns_spent=self.turns,
|
||||
rejected=self.rejected,
|
||||
truncated=truncated,
|
||||
typology=None if d is None else d.get("typology"),
|
||||
true_typology=self.world.get("typology"),
|
||||
lookups=len(self.lookups),
|
||||
)
|
||||
@@ -0,0 +1,881 @@
|
||||
"""The world for a seed: one alert on one synthetic customer.
|
||||
|
||||
Everything here is synthetic. No customer, account, transaction, counterparty,
|
||||
analyst or institution exists behind any of it — see `names/PROVENANCE.md`.
|
||||
|
||||
Portability is the whole design. `src/demos/alert-triage/` carries a port of
|
||||
this file, and CI hashes the canonical JSON of every world for seeds 0–4095 on
|
||||
both sides. So:
|
||||
|
||||
* every random draw goes through `XorShift32` seeded by FNV-1a over the
|
||||
decimal seed, in the order written here — the order IS the contract;
|
||||
* every amount is an integer in minor units (cents), never a float;
|
||||
* every date is a day ordinal turned into YYYY-MM-DD by our own civil-date
|
||||
function, never a platform date type;
|
||||
* canonical JSON is sorted keys, no whitespace, `null` for absent.
|
||||
|
||||
The hidden tier — a suspicious alert whose free screen is drawn from a benign
|
||||
template — is produced by running the benign builder to completion, freezing
|
||||
the screen, and only then overlaying the suspicious signal onto data that a
|
||||
lookup returns. `screen_of(world)` is byte-identical with the overlay on or
|
||||
off, and `tests/test_screen_leak.py` asserts it for every seed.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from .rng import (
|
||||
XorShift32,
|
||||
add_months,
|
||||
days_from_civil,
|
||||
days_in_month,
|
||||
fnv1a32,
|
||||
iso_date,
|
||||
month_key,
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------ constants --
|
||||
|
||||
MAX_TURNS = 8
|
||||
WINDOW_MONTHS = 12
|
||||
SCREEN_COUNTERPARTIES = 2 # top-N by volume shown on the free screen
|
||||
|
||||
# Mix by seed. Deliberately not the real false-positive base rate — see
|
||||
# README.md, "what this is easier than".
|
||||
BENIGN_PCT = 55
|
||||
VISIBLE_PCT = 30
|
||||
# hidden = the remaining 15
|
||||
|
||||
FAMILIES = ("STR", "VEL", "CASH", "WIRE", "DORM")
|
||||
|
||||
# Rule ids per slice. The held-out slice renames them so a policy that learned
|
||||
# "9,xxx and R-STR-01" rather than the pattern is caught.
|
||||
RULE_IDS = {
|
||||
"main": {"STR": "R-STR-01", "VEL": "R-VEL-02", "CASH": "R-CASH-03", "WIRE": "R-WIRE-04", "DORM": "R-DORM-05"},
|
||||
"held_out": {"STR": "TM-A1", "VEL": "TM-A2", "CASH": "TM-A3", "WIRE": "TM-A4", "DORM": "TM-A5"},
|
||||
}
|
||||
|
||||
RULE_TEXT = {
|
||||
"STR": "Three or more cash deposits just under the reporting threshold within ten days",
|
||||
"VEL": "Funds received and moved out again within 48 hours",
|
||||
"CASH": "Monthly cash deposits materially above the trailing six-month average",
|
||||
"WIRE": "Outbound wires to a jurisdiction on the institution's monitored list",
|
||||
"DORM": "Large credit to an account with no material activity for nine months",
|
||||
}
|
||||
|
||||
# The typology the rule nominally points at — what a defensive filer would
|
||||
# write. Diagnostic only; `typology` is never scored.
|
||||
NOMINAL_TYPOLOGY = {"STR": "STRUCTURING", "VEL": "RAPID_MOVEMENT", "CASH": "STRUCTURING", "WIRE": "UNKNOWN", "DORM": "UNKNOWN"}
|
||||
|
||||
CURRENCIES = {
|
||||
# threshold in minor units; scale multiplies every generated amount
|
||||
"main": {"currency": "USD", "threshold": 1_000_000, "scale": 1},
|
||||
"held_out": {"currency": "KRN", "threshold": 2_000_000, "scale": 2},
|
||||
}
|
||||
|
||||
BENIGN_TEMPLATES = ("B1", "B2", "B3", "B4", "B5", "B6")
|
||||
VISIBLE_TEMPLATES = ("S1", "S3", "S6")
|
||||
OVERLAYS = ("funnel", "serial_closer", "doc_mismatch")
|
||||
DOC_TEMPLATES = ("B2", "B3", "B6") # where a document is the explaining fact
|
||||
|
||||
TEMPLATE_FAMILY = {"B1": "CASH", "B2": "VEL", "B3": "VEL", "B4": "STR", "B5": "WIRE", "B6": "DORM", "S1": "STR", "S3": "VEL", "S6": "CASH"}
|
||||
|
||||
INBOUND = ("cash_in", "wire_in", "ach_in", "p2p_in")
|
||||
|
||||
_NAMES = Path(__file__).parent.parent / "names"
|
||||
|
||||
|
||||
def _load(name: str):
|
||||
return json.loads((_NAMES / name).read_text())
|
||||
|
||||
|
||||
FIRST_NAMES: list[str] = _load("first_names.json")
|
||||
SURNAMES: list[str] = _load("surnames.json")
|
||||
WORDS: dict[str, list[str]] = _load("words.json")
|
||||
JURISDICTIONS: dict[str, Any] = _load("jurisdictions.json")
|
||||
|
||||
BOILERPLATE = "Reviewed. Activity consistent with customer profile. No further action."
|
||||
|
||||
BENIGN_RATIONALE = {
|
||||
"STR": "Closed: deposits are documented daily takings; amounts vary with trading days; within kyc.expected_monthly_cash.",
|
||||
"VEL": "Closed: pass-through matches the settlement terms in the agreement on file; counterparties verified.",
|
||||
"CASH": "Closed: seasonal peak consistent with kyc.expected_activity; volume within stated expectation.",
|
||||
"WIRE": "Closed: destination and amount match the remittance declared at onboarding; within stated income.",
|
||||
"DORM": "Closed: reactivation explained by a documented source of funds; document reviewed and retained.",
|
||||
}
|
||||
|
||||
|
||||
# ----------------------------------------------------------- utilities --
|
||||
|
||||
|
||||
def canonical_json(value: Any) -> str:
|
||||
return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
|
||||
|
||||
|
||||
def _round_to(cents: int, step: int) -> int:
|
||||
return (cents // step) * step
|
||||
|
||||
|
||||
class _Ledger:
|
||||
"""Transactions are collected unordered, then ids are assigned by date.
|
||||
|
||||
A stable sort on the day ordinal, with insertion order breaking ties, is
|
||||
what makes T-ids identical across the two implementations.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.rows: list[dict[str, Any]] = []
|
||||
|
||||
def add(self, day: int, channel: str, amount: int, branch: str | None = None, cp: str | None = None) -> int:
|
||||
self.rows.append({"day": day, "seq": len(self.rows), "channel": channel, "amount": amount, "branch": branch, "cp": cp})
|
||||
return len(self.rows) - 1
|
||||
|
||||
def finish(self) -> tuple[list[dict[str, Any]], dict[int, str]]:
|
||||
ordered = sorted(self.rows, key=lambda r: (r["day"], r["seq"]))
|
||||
id_of: dict[int, str] = {}
|
||||
out = []
|
||||
for i, row in enumerate(ordered):
|
||||
tid = f"T-{i + 1}"
|
||||
id_of[row["seq"]] = tid
|
||||
out.append({
|
||||
"id": tid,
|
||||
"date": iso_date(row["day"]),
|
||||
"month": iso_date(row["day"])[:7],
|
||||
"channel": row["channel"],
|
||||
"amount": row["amount"],
|
||||
"branch": row["branch"],
|
||||
"cp": row["cp"],
|
||||
})
|
||||
return out, id_of
|
||||
|
||||
|
||||
class _Ctx:
|
||||
"""Everything a template builder needs, plus the accumulating world."""
|
||||
|
||||
def __init__(self, seed: int, rng: XorShift32, slice_name: str) -> None:
|
||||
self.seed = seed
|
||||
self.rng = rng
|
||||
self.slice = slice_name
|
||||
cur = CURRENCIES[slice_name]
|
||||
self.currency = cur["currency"]
|
||||
self.threshold = cur["threshold"]
|
||||
self.scale = cur["scale"]
|
||||
self.rule_ids = RULE_IDS[slice_name]
|
||||
self.home = JURISDICTIONS["home"][self.currency]
|
||||
self.ledger = _Ledger()
|
||||
self.cps: list[dict[str, Any]] = []
|
||||
self.customer: dict[str, Any] = {}
|
||||
self.months: list[tuple[int, int]] = []
|
||||
self.fire_day = 0
|
||||
self.fire_y = 0
|
||||
self.fire_m = 0
|
||||
self.documents: list[dict[str, Any]] = []
|
||||
self.doc_party_cp: str | None = None # the CP a party-bearing document names
|
||||
self.branches = rng.sample(WORDS["branches"], 2)
|
||||
|
||||
# money: `usd(n)` is n whole units of the slice currency, in minor units.
|
||||
def usd(self, n: int) -> int:
|
||||
return n * 100 * self.scale
|
||||
|
||||
def money(self, lo: int, hi: int, step: int = 10) -> int:
|
||||
"""A random whole-unit amount in [lo, hi], rounded down to `step` units."""
|
||||
return _round_to(self.usd(self.rng.between(lo, hi)), self.usd(step))
|
||||
|
||||
def set_fire(self, month_choices: list[int], day_lo: int = 12, day_hi: int = 27) -> None:
|
||||
self.fire_y = 2026
|
||||
self.fire_m = self.rng.pick(month_choices)
|
||||
self.fire_day = days_from_civil(self.fire_y, self.fire_m, self.rng.between(day_lo, day_hi))
|
||||
self.months = [add_months(self.fire_y, self.fire_m, -(WINDOW_MONTHS - 1 - i)) for i in range(WINDOW_MONTHS)]
|
||||
|
||||
def fire_dom(self) -> int:
|
||||
return self.fire_day - days_from_civil(self.fire_y, self.fire_m, 1) + 1
|
||||
|
||||
def day_in(self, i: int, lo: int = 1, hi: int | None = None) -> int:
|
||||
"""A day ordinal inside window month i. Nothing is ever dated on or after the fire date."""
|
||||
y, m = self.months[i]
|
||||
last = days_in_month(y, m) if hi is None else hi
|
||||
if i == WINDOW_MONTHS - 1:
|
||||
last = min(last, self.fire_dom() - 1)
|
||||
lo = min(lo, last)
|
||||
return days_from_civil(y, m, self.rng.between(lo, last))
|
||||
|
||||
def days_in(self, i: int, k: int, lo: int = 1, hi: int | None = None) -> list[int]:
|
||||
"""k distinct days in month i, ascending."""
|
||||
y, m = self.months[i]
|
||||
last = days_in_month(y, m) if hi is None else hi
|
||||
if i == WINDOW_MONTHS - 1:
|
||||
last = min(last, self.fire_dom() - 1)
|
||||
lo = min(lo, last)
|
||||
first = days_from_civil(y, m, 1)
|
||||
picks = self.rng.sample(list(range(lo, last + 1)), k)
|
||||
return sorted(first + d - 1 for d in picks)
|
||||
|
||||
def person(self) -> str:
|
||||
return f"{self.rng.pick(FIRST_NAMES)} {self.rng.pick(SURNAMES)}"
|
||||
|
||||
def business(self, noun: str | None = None) -> str:
|
||||
adj = self.rng.pick(WORDS["adjectives"])
|
||||
noun = noun or self.rng.pick(WORDS["nouns"])
|
||||
return f"{adj} {noun} {self.rng.pick(WORDS['suffixes'])}"
|
||||
|
||||
def cp(self, name: str, kind: str, jurisdiction: str | None = None) -> str:
|
||||
cid = f"CP-{len(self.cps) + 1}"
|
||||
self.cps.append({"id": cid, "name": name, "type": kind, "jurisdiction": jurisdiction or self.home})
|
||||
return cid
|
||||
|
||||
def doc(self, kind: str, description: str, party: str | None = None) -> str:
|
||||
while True:
|
||||
did = f"doc.D-{self.rng.hex4()}"
|
||||
if all(d["id"] != did for d in self.documents):
|
||||
break
|
||||
self.documents.append({"id": did, "kind": kind, "description": description, "party": party})
|
||||
return did
|
||||
|
||||
def add(self, day: int, channel: str, amount: int, branch: str | None = None, cp: str | None = None) -> int:
|
||||
return self.ledger.add(day, channel, amount, branch, cp)
|
||||
|
||||
def branch(self) -> str:
|
||||
return self.rng.pick(self.branches)
|
||||
|
||||
def max_month_cash(self) -> int:
|
||||
"""The largest month of cash actually deposited so far — what the KYC expectation must cover."""
|
||||
totals: dict[int, int] = {}
|
||||
for row in self.ledger.rows:
|
||||
if row["channel"] == "cash_in":
|
||||
key = iso_date(row["day"])[:7]
|
||||
idx = int(key[:4]) * 12 + int(key[5:])
|
||||
totals[idx] = totals.get(idx, 0) + row["amount"]
|
||||
return max(totals.values()) if totals else 0
|
||||
|
||||
|
||||
def _kyc_common(ctx: _Ctx, customer_type: str) -> None:
|
||||
"""The KYC draws every template makes, in the same order.
|
||||
|
||||
Prior-alert and prior-SAR counts are drawn HERE, before any template or
|
||||
label-specific code runs, from one distribution. The serial-closer overlay
|
||||
rewrites what the prior alerts say; it can never change how many there are,
|
||||
which is the number the screen shows.
|
||||
"""
|
||||
rng = ctx.rng
|
||||
ctx.customer = {
|
||||
"id": f"C-{rng.between(1000, 9999)}",
|
||||
"name": "",
|
||||
"customer_type": customer_type,
|
||||
"country": ctx.home,
|
||||
"pep": rng.chance(3),
|
||||
"prior_alerts": 0,
|
||||
"prior_sars": 0,
|
||||
"remittance_declared": "none declared",
|
||||
}
|
||||
roll = rng.below(100)
|
||||
ctx.customer["prior_alerts"] = 0 if roll < 35 else 1 if roll < 55 else 2 if roll < 70 else 3 if roll < 88 else 4
|
||||
ctx.customer["prior_sars"] = 1 if ctx.customer["prior_alerts"] >= 1 and rng.chance(8) else 0
|
||||
opened_years = rng.between(2, 15)
|
||||
ctx.customer["account_opened"] = iso_date(ctx.fire_day - opened_years * 365 - rng.between(0, 300))
|
||||
ctx.customer["kyc_refreshed"] = iso_date(ctx.fire_day - rng.between(40, 500))
|
||||
|
||||
|
||||
def _salary(ctx: _Ctx, monthly: int, employer: str) -> None:
|
||||
for i in range(WINDOW_MONTHS):
|
||||
amount = monthly + ctx.usd(ctx.rng.between(-60, 60))
|
||||
ctx.add(ctx.day_in(i, 1, 5), "ach_in", amount, cp=employer)
|
||||
|
||||
|
||||
def _small_cash_out(ctx: _Ctx, lo: int, hi: int, per_month_lo: int, per_month_hi: int) -> None:
|
||||
for i in range(WINDOW_MONTHS):
|
||||
n = ctx.rng.between(per_month_lo, per_month_hi)
|
||||
for day in ctx.days_in(i, n):
|
||||
ctx.add(day, "cash_out", ctx.money(lo, hi), branch=ctx.branch())
|
||||
|
||||
|
||||
def _cash_deposits(ctx: _Ctx, i: int, total: int, n: int, lo_cap: int, hi_cap: int, days: list[int] | None = None) -> list[int]:
|
||||
"""n cash deposits in month i summing roughly to `total`, each clamped."""
|
||||
days = days if days is not None else ctx.days_in(i, n)
|
||||
out = []
|
||||
for day in days:
|
||||
amount = total // n * ctx.rng.between(75, 125) // 100
|
||||
amount = max(lo_cap, min(hi_cap, _round_to(amount, ctx.usd(10))))
|
||||
out.append(ctx.add(day, "cash_in", amount, branch=ctx.branch()))
|
||||
return out
|
||||
|
||||
|
||||
# --------------------------------------------------------------- benign --
|
||||
|
||||
|
||||
def _build_B1(ctx: _Ctx) -> dict[str, Any]:
|
||||
"""Seasonal cash business. CASH fires on the first peak month; the KYC file said it would."""
|
||||
rng = ctx.rng
|
||||
kind = rng.pick([
|
||||
("landscaping services", "Landscaping", [4, 5, 6, 7, 8, 9], [4, 5], "April to September"),
|
||||
("tax preparation", "Tax Services", [2, 3, 4], [3], "February to April"),
|
||||
("pool maintenance", "Pool Care", [5, 6, 7, 8], [5, 6], "May to August"),
|
||||
])
|
||||
business_type, noun, peak, fire_choices, peak_text = kind
|
||||
ctx.set_fire(fire_choices)
|
||||
_kyc_common(ctx, "business")
|
||||
ctx.customer["name"] = ctx.business(noun)
|
||||
ctx.customer["business_type"] = business_type
|
||||
off = ctx.money(3000, 7000, 100)
|
||||
mult = rng.between(30, 40)
|
||||
supplier = ctx.cp(ctx.business("Supply"), "business")
|
||||
equipment = ctx.cp(ctx.business("Equipment"), "business")
|
||||
for i, (y, m) in enumerate(ctx.months):
|
||||
in_peak = m in peak
|
||||
total = off * mult // 10 if in_peak else off
|
||||
total = total * rng.between(85, 115) // 100
|
||||
n = rng.between(8, 12) if in_peak else rng.between(3, 5)
|
||||
_cash_deposits(ctx, i, total, n, ctx.usd(200), ctx.threshold * 8 // 10 - ctx.usd(100))
|
||||
ctx.add(ctx.day_in(i, 20, 28), "ach_out", total * rng.between(20, 35) // 100, cp=supplier)
|
||||
if m % 3 == 0:
|
||||
ctx.add(ctx.day_in(i, 6, 12), "wire_out", ctx.money(2500, 9000, 100), cp=equipment)
|
||||
expected = _round_to(ctx.max_month_cash() * rng.between(110, 130) // 100, ctx.usd(1000)) + ctx.usd(1000)
|
||||
ctx.customer["expected_monthly_cash"] = expected
|
||||
ctx.customer["expected_activity"] = f"Seasonal business; cash receipts peak {peak_text}."
|
||||
ctx.customer["stated_income"] = _round_to(expected * 7, ctx.usd(1000))
|
||||
ctx.doc("registration", f"Business registration — {ctx.customer['name']}")
|
||||
ctx.doc("licence", f"Trade licence, {business_type}, renewed annually")
|
||||
return {
|
||||
"family": "CASH",
|
||||
"triggering": "fire_month_cash",
|
||||
"planted": [["kyc.expected_activity", "kyc.expected_monthly_cash"], ["kyc.expected_monthly_cash"]],
|
||||
}
|
||||
|
||||
|
||||
def _build_B4(ctx: _Ctx) -> dict[str, Any]:
|
||||
"""Cash restaurant. Three of the month's deposits land in the band; the file expected more cash than this."""
|
||||
rng = ctx.rng
|
||||
business_type, noun = rng.pick([("restaurant", "Bistro"), ("diner", "Diner"), ("pizzeria", "Kitchen"), ("cafe", "Grill")])
|
||||
ctx.set_fire([3, 4, 5, 6], 14, 27)
|
||||
_kyc_common(ctx, "business")
|
||||
ctx.customer["name"] = ctx.business(noun)
|
||||
ctx.customer["business_type"] = business_type
|
||||
base = ctx.money(40000, 75000, 1000)
|
||||
produce = ctx.cp(ctx.business("Produce"), "business")
|
||||
foods = ctx.cp(ctx.business("Foods"), "business")
|
||||
band_lo = ctx.threshold * 8 // 10
|
||||
for i in range(WINDOW_MONTHS):
|
||||
total = base * rng.between(90, 110) // 100
|
||||
n = rng.between(11, 14)
|
||||
if i == WINDOW_MONTHS - 1:
|
||||
# Three window deposits in the band, on distinct days; the rest ordinary.
|
||||
fire_dom = ctx.fire_dom()
|
||||
window_days = ctx.days_in(i, 3, max(1, fire_dom - 10), fire_dom - 1)
|
||||
for day in window_days:
|
||||
ctx.add(day, "cash_in", ctx.money(band_lo // ctx.usd(1) + 100, ctx.threshold // ctx.usd(1) - 100), branch=ctx.branch())
|
||||
n -= 3
|
||||
_cash_deposits(ctx, i, total, n, ctx.usd(1500), band_lo - ctx.usd(100))
|
||||
for day in ctx.days_in(i, 4, 1, 28):
|
||||
ctx.add(day, "ach_out", total * rng.between(4, 7) // 100, cp=produce)
|
||||
ctx.add(ctx.day_in(i, 1, 10), "ach_out", total * rng.between(10, 16) // 100, cp=foods)
|
||||
expected = _round_to(ctx.max_month_cash() * rng.between(110, 130) // 100, ctx.usd(1000)) + ctx.usd(1000)
|
||||
ctx.customer["expected_monthly_cash"] = expected
|
||||
ctx.customer["expected_activity"] = "Cash-intensive food service; daily takings deposited most trading days."
|
||||
ctx.customer["stated_income"] = _round_to(base * 12, ctx.usd(1000))
|
||||
ctx.doc("registration", f"Business registration — {ctx.customer['name']}")
|
||||
ctx.doc("licence", "Food service licence, renewed annually")
|
||||
return {
|
||||
"family": "STR",
|
||||
"triggering": "band_window",
|
||||
"planted": [["kyc.expected_monthly_cash"], ["kyc.expected_monthly_cash", "kyc.business_type"]],
|
||||
}
|
||||
|
||||
|
||||
def _build_B2(ctx: _Ctx) -> dict[str, Any]:
|
||||
"""Processor settlement. Wires in from a processor, contractors paid within two days; the agreement is on file."""
|
||||
rng = ctx.rng
|
||||
business_type, noun = rng.pick([("staffing agency", "Staffing"), ("freight brokerage", "Freight"), ("event production", "Events"), ("general contractor", "Contracting")])
|
||||
ctx.set_fire([3, 4, 5, 6], 14, 27)
|
||||
_kyc_common(ctx, "business")
|
||||
ctx.customer["name"] = ctx.business(noun)
|
||||
ctx.customer["business_type"] = business_type
|
||||
processor_name = f"{rng.pick(WORDS['adjectives'])} Payments LLC"
|
||||
processor = ctx.cp(processor_name, "processor")
|
||||
ctx.doc_party_cp = processor
|
||||
contractors = [ctx.cp(ctx.business(), "business") for _ in range(2)]
|
||||
revenue = 0
|
||||
window: list[int] = []
|
||||
for i in range(WINDOW_MONTHS):
|
||||
k = rng.between(3, 4)
|
||||
days = ctx.days_in(i, k, 1, 24)
|
||||
if i == WINDOW_MONTHS - 1:
|
||||
# Guarantee one settlement inside the 14-day window with its payouts before the fire date.
|
||||
fire_dom = ctx.fire_dom()
|
||||
days = ctx.days_in(i, k - 1, 1, max(1, fire_dom - 8)) + [ctx.fire_day - rng.between(3, 6)]
|
||||
for day in days:
|
||||
amount = ctx.money(15000, 60000, 100)
|
||||
revenue += amount
|
||||
t_in = ctx.add(day, "wire_in", amount, cp=processor)
|
||||
share = rng.between(70, 90)
|
||||
first = amount * share // 100 * rng.between(50, 70) // 100
|
||||
second = amount * share // 100 - first
|
||||
t_a = ctx.add(day + 1, "wire_out", _round_to(first, ctx.usd(10)), cp=contractors[0])
|
||||
t_b = ctx.add(day + rng.between(1, 2), "ach_out", _round_to(second, ctx.usd(10)), cp=contractors[1])
|
||||
if i == WINDOW_MONTHS - 1 and day >= ctx.fire_day - 14:
|
||||
window += [t_in, t_a, t_b]
|
||||
ctx.customer["expected_monthly_cash"] = ctx.usd(0)
|
||||
ctx.customer["expected_activity"] = "Card settlements from a payment processor; subcontractor payouts within days of settlement."
|
||||
ctx.customer["stated_income"] = _round_to(revenue, ctx.usd(10000))
|
||||
ctx.doc("registration", f"Business registration — {ctx.customer['name']}")
|
||||
ctx.doc("processing_agreement", f"Merchant processing agreement between {ctx.customer['name']} and {processor_name}; settlement to this account on T+1.", party=processor_name)
|
||||
return {"family": "VEL", "triggering": window, "planted": [["DOC"], ["DOC", "kyc.business_type"]]}
|
||||
|
||||
|
||||
def _build_B3(ctx: _Ctx) -> dict[str, Any]:
|
||||
"""Property sale. One title-company wire in, moved to a brokerage; closing statement on file."""
|
||||
rng = ctx.rng
|
||||
ctx.set_fire([3, 4, 5, 6], 14, 27)
|
||||
_kyc_common(ctx, "individual")
|
||||
ctx.customer["name"] = ctx.person()
|
||||
ctx.customer["occupation"] = rng.pick(["teacher", "software engineer", "pharmacist", "electrician", "accountant"])
|
||||
salary = ctx.money(4000, 9000, 100)
|
||||
employer = ctx.cp(ctx.business(), "business")
|
||||
_salary(ctx, salary, employer)
|
||||
_small_cash_out(ctx, 100, 400, 2, 4)
|
||||
title_name = f"{rng.pick(SURNAMES)} Title & Escrow"
|
||||
title = ctx.cp(title_name, "title_company")
|
||||
ctx.doc_party_cp = title
|
||||
brokerage = ctx.cp(f"{rng.pick(WORDS['adjectives'])} Securities", "brokerage")
|
||||
proceeds = ctx.money(180000, 650000, 1000)
|
||||
day_in = ctx.fire_day - rng.between(3, 6)
|
||||
t_in = ctx.add(day_in, "wire_in", proceeds, cp=title)
|
||||
t_out = ctx.add(day_in + rng.between(1, 2), "wire_out", _round_to(proceeds * rng.between(85, 100) // 100, ctx.usd(100)), cp=brokerage)
|
||||
ctx.customer["expected_monthly_cash"] = ctx.usd(0)
|
||||
ctx.customer["expected_activity"] = "Salaried individual; payroll deposits and routine spending."
|
||||
ctx.customer["stated_income"] = _round_to(salary * 12, ctx.usd(1000))
|
||||
ctx.doc("identity", "Government-issued identity document, verified at onboarding")
|
||||
address = f"{rng.between(12, 980)} {rng.pick(WORDS['streets'])}"
|
||||
ctx.doc("closing_statement", f"Closing statement — sale of {address}; net proceeds to seller disbursed by {title_name}.", party=title_name)
|
||||
return {"family": "VEL", "triggering": [t_in, t_out], "planted": [["DOC"], ["DOC", "kyc.occupation"]]}
|
||||
|
||||
|
||||
def _build_B5(ctx: _Ctx) -> dict[str, Any]:
|
||||
"""Family remittance. A monthly wire to one relative in a monitored jurisdiction, declared at onboarding."""
|
||||
rng = ctx.rng
|
||||
ctx.set_fire([3, 4, 5, 6])
|
||||
_kyc_common(ctx, "individual")
|
||||
surname = rng.pick(SURNAMES)
|
||||
ctx.customer["name"] = f"{rng.pick(FIRST_NAMES)} {surname}"
|
||||
ctx.customer["occupation"] = rng.pick(["registered nurse", "civil engineer", "teacher", "pharmacist"])
|
||||
salary = ctx.money(4500, 8500, 100)
|
||||
employer = ctx.cp(ctx.business(), "business")
|
||||
_salary(ctx, salary, employer)
|
||||
_small_cash_out(ctx, 100, 400, 2, 4)
|
||||
jurisdiction = rng.pick(JURISDICTIONS["monitored"])
|
||||
relation = rng.pick(["mother", "father", "brother", "sister"])
|
||||
relative = ctx.cp(f"{rng.pick(FIRST_NAMES)} {surname}", "individual", jurisdiction)
|
||||
monthly = ctx.money(800, 2500, 50)
|
||||
wires = []
|
||||
for i in range(WINDOW_MONTHS):
|
||||
wires.append(ctx.add(ctx.day_in(i, 1, 7), "wire_out", monthly + ctx.usd(rng.between(-50, 50)), cp=relative))
|
||||
ctx.customer["expected_monthly_cash"] = ctx.usd(0)
|
||||
ctx.customer["expected_activity"] = "Salaried individual; payroll deposits and routine spending."
|
||||
ctx.customer["remittance_declared"] = f"Monthly support to {relation} in {jurisdiction}, about {monthly // 100 // ctx.scale * ctx.scale:,} {ctx.currency} per month."
|
||||
ctx.customer["stated_income"] = _round_to(salary * 12, ctx.usd(1000))
|
||||
ctx.doc("identity", "Government-issued identity document, verified at onboarding")
|
||||
ctx.doc("address", "Proof of address, utility statement")
|
||||
return {"family": "WIRE", "triggering": wires[-3:], "planted": [["kyc.remittance_declared"], ["kyc.remittance_declared", "kyc.stated_income"]]}
|
||||
|
||||
|
||||
def _build_B6(ctx: _Ctx) -> dict[str, Any]:
|
||||
"""Inheritance. A dormant account receives an estate distribution; the executor's letter is on file."""
|
||||
rng = ctx.rng
|
||||
ctx.set_fire([3, 4, 5, 6], 14, 27)
|
||||
_kyc_common(ctx, "individual")
|
||||
ctx.customer["name"] = ctx.person()
|
||||
ctx.customer["occupation"] = "retired"
|
||||
ctx.customer["account_opened"] = iso_date(ctx.fire_day - rng.between(8, 20) * 365 - rng.between(0, 300))
|
||||
for i in range(WINDOW_MONTHS - 1):
|
||||
if rng.chance(35):
|
||||
ctx.add(ctx.day_in(i), "cash_out", ctx.money(40, 200), branch=ctx.branch())
|
||||
firm = f"{rng.pick(SURNAMES)} & {rng.pick(SURNAMES)} LLP, client trust account"
|
||||
trust = ctx.cp(firm, "law_firm_trust")
|
||||
ctx.doc_party_cp = trust
|
||||
brokerage = ctx.cp(f"{rng.pick(WORDS['adjectives'])} Securities", "brokerage")
|
||||
amount = ctx.money(60000, 400000, 1000)
|
||||
day_in = ctx.fire_day - rng.between(4, 7)
|
||||
t_in = ctx.add(day_in, "wire_in", amount, cp=trust)
|
||||
t_out = ctx.add(day_in + rng.between(1, 3), "wire_out", _round_to(amount * rng.between(50, 90) // 100, ctx.usd(100)), cp=brokerage)
|
||||
ctx.customer["expected_monthly_cash"] = ctx.usd(0)
|
||||
ctx.customer["expected_activity"] = "Retired; low activity expected."
|
||||
ctx.customer["stated_income"] = ctx.money(20000, 40000, 1000)
|
||||
ctx.doc("identity", "Government-issued identity document, verified at onboarding")
|
||||
executor, decedent = ctx.person(), ctx.person()
|
||||
ctx.doc("executor_letter", f"Letter from {executor}, executor, via {firm}: distribution from the estate of {decedent} to the named beneficiary.", party=firm)
|
||||
return {"family": "DORM", "triggering": [t_in, t_out], "planted": [["DOC"]]}
|
||||
|
||||
|
||||
# -------------------------------------------------------------- visible --
|
||||
|
||||
|
||||
def _build_S1(ctx: _Ctx) -> dict[str, Any]:
|
||||
"""Structuring. Eight deposits just under the threshold in twelve days across three branches, on a modest income."""
|
||||
rng = ctx.rng
|
||||
ctx.set_fire([3, 4, 5, 6], 14, 27)
|
||||
_kyc_common(ctx, "individual")
|
||||
ctx.customer["name"] = ctx.person()
|
||||
ctx.customer["occupation"] = rng.pick(["warehouse associate", "delivery driver", "retail supervisor", "line cook"])
|
||||
ctx.branches = rng.sample(WORDS["branches"], 3)
|
||||
salary = ctx.money(2600, 3600, 100)
|
||||
employer = ctx.cp(ctx.business(), "business")
|
||||
_salary(ctx, salary, employer)
|
||||
_small_cash_out(ctx, 60, 300, 2, 4)
|
||||
fire_dom = ctx.fire_dom()
|
||||
days = ctx.days_in(WINDOW_MONTHS - 1, 8, max(1, fire_dom - 12), fire_dom - 2)
|
||||
lo = ctx.threshold // ctx.usd(1) * 92 // 100
|
||||
hi = ctx.threshold // ctx.usd(1) * 99 // 100
|
||||
deposits = []
|
||||
total = 0
|
||||
for j, day in enumerate(days):
|
||||
amount = ctx.money(lo, hi)
|
||||
total += amount
|
||||
deposits.append(ctx.add(day, "cash_in", amount, branch=ctx.branches[j % 3]))
|
||||
holdings = ctx.cp(ctx.business("Holdings"), "business")
|
||||
ctx.add(ctx.fire_day - 1, "wire_out", _round_to(total * rng.between(85, 95) // 100, ctx.usd(100)), cp=holdings)
|
||||
ctx.customer["expected_monthly_cash"] = ctx.money(0, 500, 100)
|
||||
ctx.customer["expected_activity"] = "Salaried individual; payroll deposits and routine spending."
|
||||
ctx.customer["stated_income"] = _round_to(salary * 12 + ctx.usd(rng.between(0, 6000)), ctx.usd(1000))
|
||||
ctx.doc("identity", "Government-issued identity document, verified at onboarding")
|
||||
return {"family": "STR", "triggering": "band_window", "planted": [deposits], "typology": "STRUCTURING"}
|
||||
|
||||
|
||||
def _build_S3(ctx: _Ctx) -> dict[str, Any]:
|
||||
"""Money mule. Four P2P credits from unrelated individuals, each wired to an exchange within a day."""
|
||||
rng = ctx.rng
|
||||
ctx.set_fire([3, 4, 5, 6], 16, 27)
|
||||
_kyc_common(ctx, "individual")
|
||||
ctx.customer["name"] = ctx.person()
|
||||
ctx.customer["occupation"] = rng.pick(["student", "retired", "part-time cashier"])
|
||||
stipend = ctx.money(900, 1800, 50)
|
||||
source = ctx.cp(ctx.business(), "business")
|
||||
_salary(ctx, stipend, source)
|
||||
_small_cash_out(ctx, 40, 200, 2, 4)
|
||||
exchange = ctx.cp(f"{rng.pick(WORDS['adjectives'])} Digital Assets", "exchange")
|
||||
fire_dom = ctx.fire_dom()
|
||||
days = ctx.days_in(WINDOW_MONTHS - 1, 4, max(1, fire_dom - 14), fire_dom - 2)
|
||||
pairs = []
|
||||
for day in days:
|
||||
sender = ctx.cp(ctx.person(), "individual")
|
||||
amount = ctx.money(2000, 4900, 10)
|
||||
t_in = ctx.add(day, "p2p_in", amount, cp=sender)
|
||||
t_out = ctx.add(day + rng.between(0, 1), "wire_out", _round_to(amount * rng.between(90, 97) // 100, ctx.usd(10)), cp=exchange)
|
||||
pairs += [t_in, t_out]
|
||||
ctx.customer["expected_monthly_cash"] = ctx.money(0, 300, 100)
|
||||
ctx.customer["expected_activity"] = "Low-income individual; small regular credits and routine spending."
|
||||
ctx.customer["stated_income"] = _round_to(stipend * 12, ctx.usd(1000))
|
||||
ctx.doc("identity", "Government-issued identity document, verified at onboarding")
|
||||
return {"family": "VEL", "triggering": pairs, "planted": [pairs], "typology": "MONEY_MULE"}
|
||||
|
||||
|
||||
def _build_S6(ctx: _Ctx) -> dict[str, Any]:
|
||||
"""Smurfing over time. Monthly cash triples across the year and no single deposit ever reaches the threshold."""
|
||||
rng = ctx.rng
|
||||
business_type, noun = rng.pick([("vending route", "Vending"), ("car wash", "Car Wash"), ("laundromat", "Laundry"), ("convenience store", "Market")])
|
||||
ctx.set_fire([3, 4, 5, 6], 14, 27)
|
||||
_kyc_common(ctx, "business")
|
||||
ctx.customer["name"] = ctx.business(noun)
|
||||
ctx.customer["business_type"] = business_type
|
||||
c0 = ctx.money(9000, 14000, 500)
|
||||
supplier = ctx.cp(ctx.business("Supply"), "business")
|
||||
cap = ctx.threshold - ctx.usd(100)
|
||||
for i in range(WINDOW_MONTHS):
|
||||
total = c0 * (11 + 2 * i) // 11 * rng.between(97, 103) // 100
|
||||
n = rng.between(6, 9)
|
||||
_cash_deposits(ctx, i, total, n, ctx.usd(2000), cap)
|
||||
ctx.add(ctx.day_in(i, 20, 28), "ach_out", total * rng.between(15, 25) // 100, cp=supplier)
|
||||
ctx.customer["expected_monthly_cash"] = _round_to(c0 * 12 // 10, ctx.usd(500))
|
||||
ctx.customer["expected_activity"] = "Cash-intensive retail; steady volume expected."
|
||||
ctx.customer["stated_income"] = _round_to(c0 * 14, ctx.usd(1000))
|
||||
ctx.doc("registration", f"Business registration — {ctx.customer['name']}")
|
||||
return {"family": "CASH", "triggering": "fire_month_cash", "planted": "last_two_months", "typology": "STRUCTURING"}
|
||||
|
||||
|
||||
BUILDERS = {"B1": _build_B1, "B2": _build_B2, "B3": _build_B3, "B4": _build_B4, "B5": _build_B5, "B6": _build_B6, "S1": _build_S1, "S3": _build_S3, "S6": _build_S6}
|
||||
|
||||
|
||||
# ------------------------------------------------------------- assembly --
|
||||
|
||||
|
||||
def _prior_alerts(ctx: _Ctx, family: str) -> list[dict[str, Any]]:
|
||||
"""Benign prior alerts: mixed rules, specific rationales, a few days each."""
|
||||
rng = ctx.rng
|
||||
out: list[dict[str, Any]] = []
|
||||
used: set[str] = set()
|
||||
for j in range(ctx.customer["prior_alerts"]):
|
||||
while True:
|
||||
aid = f"A-{rng.between(1000, 4999)}"
|
||||
if aid not in used:
|
||||
used.add(aid)
|
||||
break
|
||||
fam = rng.pick(list(FAMILIES))
|
||||
fired = ctx.fire_day - rng.between(30, 720)
|
||||
escalated = j == 0 and ctx.customer["prior_sars"] >= 1
|
||||
out.append({
|
||||
"id": aid,
|
||||
"rule": ctx.rule_ids[fam],
|
||||
"fired": iso_date(fired),
|
||||
"disposition": "escalated — report filed" if escalated else "closed",
|
||||
"analyst": ctx.person(),
|
||||
"rationale": "Escalated: pattern not explained by the file at the time." if escalated else BENIGN_RATIONALE[fam],
|
||||
"elapsed_days": rng.between(2, 9),
|
||||
})
|
||||
out.sort(key=lambda a: a["fired"])
|
||||
return out
|
||||
|
||||
|
||||
def _assemble(ctx: _Ctx, template: str, spec: dict[str, Any]) -> dict[str, Any]:
|
||||
transactions, id_of = ctx.ledger.finish()
|
||||
family = spec["family"]
|
||||
|
||||
# Triggering set, from the rule where the rule is mechanical.
|
||||
if spec["triggering"] == "band_window":
|
||||
lo, hi = ctx.threshold * 8 // 10, ctx.threshold - 1
|
||||
triggering = [t["id"] for t in transactions
|
||||
if t["channel"] == "cash_in" and lo <= t["amount"] <= hi
|
||||
and ctx.fire_day - 10 <= days_from_civil(*map(int, t["date"].split("-"))) <= ctx.fire_day - 1]
|
||||
elif spec["triggering"] == "fire_month_cash":
|
||||
fire_month = month_key(ctx.fire_y, ctx.fire_m)
|
||||
triggering = [t["id"] for t in transactions if t["channel"] == "cash_in" and t["month"] == fire_month]
|
||||
else:
|
||||
triggering = sorted((id_of[s] for s in spec["triggering"]), key=lambda x: int(x[2:]))
|
||||
|
||||
# Counterparties: transactions and first-seen from the ledger, direction from the channel.
|
||||
by_cp: dict[str, list[dict[str, Any]]] = {c["id"]: [] for c in ctx.cps}
|
||||
for t in transactions:
|
||||
if t["cp"]:
|
||||
by_cp[t["cp"]].append(t)
|
||||
counterparties = []
|
||||
for c in ctx.cps:
|
||||
rows = by_cp[c["id"]]
|
||||
counterparties.append({
|
||||
**c,
|
||||
"first_seen": rows[0]["date"] if rows else iso_date(ctx.fire_day),
|
||||
"other_alerted_accounts": 1 if ctx.rng.chance(15) else 0,
|
||||
"transactions": [t["id"] for t in rows],
|
||||
})
|
||||
ranked = sorted(counterparties, key=lambda c: (-sum(t["amount"] for t in by_cp[c["id"]]), int(c["id"][3:])))
|
||||
screen_cps = []
|
||||
for c in ranked[:SCREEN_COUNTERPARTIES]:
|
||||
rows = by_cp[c["id"]]
|
||||
screen_cps.append({
|
||||
"id": c["id"],
|
||||
"direction": "in" if rows and rows[0]["channel"] in INBOUND else "out",
|
||||
"count": len(rows),
|
||||
"total": sum(t["amount"] for t in rows),
|
||||
})
|
||||
|
||||
summary = []
|
||||
for (y, m) in ctx.months:
|
||||
key = month_key(y, m)
|
||||
row = {"month": key, "cash_in": 0, "cash_out": 0, "wires_in": 0, "wires_out": 0, "ach_in": 0, "ach_out": 0, "count": 0}
|
||||
for t in transactions:
|
||||
if t["month"] != key:
|
||||
continue
|
||||
row["count"] += 1
|
||||
ch = t["channel"]
|
||||
col = {"cash_in": "cash_in", "cash_out": "cash_out", "wire_in": "wires_in", "wire_out": "wires_out",
|
||||
"ach_in": "ach_in", "ach_out": "ach_out", "p2p_in": "wires_in", "p2p_out": "wires_out"}[ch]
|
||||
row[col] += t["amount"]
|
||||
summary.append(row)
|
||||
|
||||
# Planted citation sets. "DOC" is the template's explaining document.
|
||||
explaining = next((d["id"] for d in ctx.documents if d["party"]), None)
|
||||
if spec["planted"] == "last_two_months":
|
||||
last_two = {month_key(*ctx.months[-1]), month_key(*ctx.months[-2])}
|
||||
planted = [[t["id"] for t in transactions if t["channel"] == "cash_in" and t["month"] in last_two]]
|
||||
else:
|
||||
planted = []
|
||||
for alt in spec["planted"]:
|
||||
planted.append([
|
||||
(explaining if x == "DOC" else id_of[x] if isinstance(x, int) else x) for x in alt
|
||||
])
|
||||
|
||||
world = {
|
||||
"seed": ctx.seed,
|
||||
"held_out": ctx.slice == "held_out",
|
||||
"currency": ctx.currency,
|
||||
"threshold": ctx.threshold,
|
||||
"screen_order": "kyc_first" if ctx.slice == "held_out" else "alert_first",
|
||||
"customer": ctx.customer,
|
||||
"months": [month_key(y, m) for (y, m) in ctx.months],
|
||||
"transactions": transactions,
|
||||
"counterparties": counterparties,
|
||||
"screen_counterparties": screen_cps,
|
||||
"alert": {
|
||||
"id": f"A-{ctx.rng.between(5000, 9999)}",
|
||||
"rule": ctx.rule_ids[family],
|
||||
"family": family,
|
||||
"rule_text": RULE_TEXT[family],
|
||||
"fired": iso_date(ctx.fire_day),
|
||||
"triggering": triggering,
|
||||
},
|
||||
"prior_alerts": _prior_alerts(ctx, family),
|
||||
"documents": ctx.documents,
|
||||
"summary": summary,
|
||||
"template": template,
|
||||
"overlay": None,
|
||||
"label": "suspicious" if template.startswith("S") else "benign",
|
||||
"typology": spec.get("typology"),
|
||||
"planted": planted,
|
||||
"doc_party_cp": ctx.doc_party_cp,
|
||||
}
|
||||
return world
|
||||
|
||||
|
||||
# -------------------------------------------------------------- overlay --
|
||||
|
||||
|
||||
def compatible_overlays(world: dict[str, Any]) -> list[str]:
|
||||
"""Which hidden signals this benign world can carry. Depends on the screen, never on the label."""
|
||||
kinds = ["funnel"]
|
||||
if world["customer"]["prior_alerts"] >= 3 and world["customer"]["prior_sars"] == 0:
|
||||
kinds.append("serial_closer")
|
||||
if world["template"] in DOC_TEMPLATES:
|
||||
kinds.append("doc_mismatch")
|
||||
return kinds
|
||||
|
||||
|
||||
def _overlay_urn(kinds: list[str]) -> list[str]:
|
||||
"""funnel is always available, so the other two are drawn twice as often where they can be."""
|
||||
urn: list[str] = []
|
||||
for kind in kinds:
|
||||
urn += [kind] * (1 if kind == "funnel" else 2)
|
||||
return urn
|
||||
|
||||
|
||||
def _outbound_target(world: dict[str, Any]) -> str:
|
||||
"""The highest-volume outbound counterparty on the screen. Every benign template has one."""
|
||||
for c in world["screen_counterparties"]:
|
||||
if c["direction"] == "out":
|
||||
return c["id"]
|
||||
raise AssertionError(f"template {world['template']} has no outbound counterparty on screen")
|
||||
|
||||
|
||||
def apply_overlay(world: dict[str, Any], kind: str, rng: XorShift32) -> None:
|
||||
"""Overlay a suspicious signal onto lookup data only. Mutates `world`.
|
||||
|
||||
The screen — alert, triggering set, KYC block, summary, screen
|
||||
counterparties — is not touched, and `tests/test_screen_leak.py` proves it.
|
||||
"""
|
||||
family = world["alert"]["family"]
|
||||
triggering = world["alert"]["triggering"]
|
||||
world["overlay"] = kind
|
||||
world["label"] = "suspicious"
|
||||
|
||||
if kind == "funnel":
|
||||
target = _outbound_target(world)
|
||||
cp = next(c for c in world["counterparties"] if c["id"] == target)
|
||||
cp["other_alerted_accounts"] = rng.between(5, 14)
|
||||
world["typology"] = "FUNNEL_ACCOUNT"
|
||||
world["planted"] = [[target] + cp["transactions"], list(cp["transactions"])]
|
||||
|
||||
elif kind == "serial_closer":
|
||||
priors = world["prior_alerts"]
|
||||
analysts = []
|
||||
while len(analysts) < len(priors):
|
||||
name = f"{rng.pick(FIRST_NAMES)} {rng.pick(SURNAMES)}"
|
||||
if name not in analysts:
|
||||
analysts.append(name)
|
||||
fire_day = days_from_civil(*map(int, world["alert"]["fired"].split("-")))
|
||||
days = sorted(rng.sample(list(range(fire_day - 330, fire_day - 30)), len(priors)))
|
||||
for j, prior in enumerate(priors):
|
||||
prior["rule"] = world["alert"]["rule"]
|
||||
prior["fired"] = iso_date(days[j])
|
||||
prior["disposition"] = "closed"
|
||||
prior["analyst"] = analysts[j]
|
||||
prior["rationale"] = BOILERPLATE
|
||||
prior["elapsed_days"] = rng.between(0, 1)
|
||||
world["typology"] = NOMINAL_TYPOLOGY[family]
|
||||
ids = [p["id"] for p in priors]
|
||||
world["planted"] = [ids + triggering, ids]
|
||||
|
||||
elif kind == "doc_mismatch":
|
||||
target = world["doc_party_cp"]
|
||||
cp = next(c for c in world["counterparties"] if c["id"] == target)
|
||||
cp["name"] = f"{rng.pick(WORDS['adjectives'])} {rng.pick(['Holdings', 'Ventures', 'Trading'])} {rng.pick(WORDS['suffixes'])}"
|
||||
doc = next(d["id"] for d in world["documents"] if d["party"])
|
||||
inbound = [t["id"] for t in world["transactions"] if t["id"] in triggering and t["cp"] == target]
|
||||
world["typology"] = NOMINAL_TYPOLOGY[family]
|
||||
world["planted"] = [[doc, target] + inbound, [doc, target]]
|
||||
else:
|
||||
raise ValueError(kind)
|
||||
|
||||
|
||||
# -------------------------------------------------------------- public --
|
||||
|
||||
|
||||
def is_held_out(seed: int) -> bool:
|
||||
return fnv1a32(str(seed)) % 8 == 7
|
||||
|
||||
|
||||
def generate(seed: int, overlay: str | None = "auto") -> dict[str, Any]:
|
||||
"""The world before reference hours. `overlay` is for tests:
|
||||
"auto" follows the seed; None forces the benign twin; a kind forces it on."""
|
||||
h = fnv1a32(str(seed))
|
||||
slice_name = "held_out" if h % 8 == 7 else "main"
|
||||
rng = XorShift32(h)
|
||||
ctx = _Ctx(seed, rng, slice_name)
|
||||
|
||||
roll = rng.below(100)
|
||||
tier = "benign" if roll < BENIGN_PCT else "visible" if roll < BENIGN_PCT + VISIBLE_PCT else "hidden"
|
||||
if tier == "visible":
|
||||
template = VISIBLE_TEMPLATES[rng.below(len(VISIBLE_TEMPLATES))]
|
||||
else:
|
||||
template = BENIGN_TEMPLATES[rng.below(len(BENIGN_TEMPLATES))]
|
||||
|
||||
spec = BUILDERS[template](ctx)
|
||||
world = _assemble(ctx, template, spec)
|
||||
world["tier"] = tier
|
||||
|
||||
if tier != "visible":
|
||||
kinds = compatible_overlays(world)
|
||||
if overlay == "auto":
|
||||
if tier == "hidden":
|
||||
apply_overlay(world, rng.pick(_overlay_urn(kinds)), rng)
|
||||
elif overlay is not None:
|
||||
apply_overlay(world, overlay, rng)
|
||||
del world["doc_party_cp"]
|
||||
return world
|
||||
|
||||
|
||||
def world_for_seed(seed: int) -> dict[str, Any]:
|
||||
"""The world with `reference_minutes` attached — part of the digest.
|
||||
|
||||
The reference is the cheapest shipped policy that reached the correct
|
||||
disposition on this seed, found by running them through the engine. The
|
||||
policies read only the engine's view, never this dict, so attaching their
|
||||
result here is not circular; it is the same move as wordle's
|
||||
`reference_depth`, made part of the world so the browser recomputes it
|
||||
instead of trusting the recorded number.
|
||||
"""
|
||||
from .policies import reference_for # local import: policies -> engine -> generator
|
||||
|
||||
world = generate(seed)
|
||||
minutes, policy = reference_for(world)
|
||||
world["reference_minutes"] = minutes
|
||||
world["reference_policy"] = policy
|
||||
return world
|
||||
|
||||
|
||||
SCREEN_KEYS = ("seed", "held_out", "currency", "threshold", "screen_order", "customer", "months",
|
||||
"summary", "screen_counterparties", "alert")
|
||||
|
||||
|
||||
def screen_of(world: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Exactly what turn 0 shows, including the triggering transactions in full."""
|
||||
triggering = set(world["alert"]["triggering"])
|
||||
screen = {k: world[k] for k in SCREEN_KEYS}
|
||||
screen["triggering_transactions"] = [t for t in world["transactions"] if t["id"] in triggering]
|
||||
return screen
|
||||
|
||||
|
||||
def world_digest(seeds=range(4096)) -> str:
|
||||
"""Digest (a): SHA-256 over the canonical world for each seed, newline-joined."""
|
||||
digest = hashlib.sha256()
|
||||
for seed in seeds:
|
||||
digest.update(canonical_json(world_for_seed(seed)).encode())
|
||||
digest.update(b"\n")
|
||||
return digest.hexdigest()
|
||||
@@ -0,0 +1,165 @@
|
||||
"""The probe ladder for alert-triage. Invoked by `envs/probe.py`.
|
||||
|
||||
Every rung is a policy from `policies.py`, played over the main-slice seeds
|
||||
(held-out seeds are never probed). The assertions are the reward's claims,
|
||||
checked per weighted component rather than as one blended float. Any cheat a
|
||||
reviewer named is a rung here with an assertion that it loses — a cheat that
|
||||
is measured is a cheat that cannot come back unnoticed.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from .generator import is_held_out, world_for_seed
|
||||
from .policies import POLICIES, run_policy
|
||||
from .reward import (
|
||||
CAUGHT_WEIGHT,
|
||||
EVIDENCE_WEIGHT,
|
||||
ROLES,
|
||||
WEIGHTS,
|
||||
escalate_break_even,
|
||||
metrics,
|
||||
score,
|
||||
total,
|
||||
)
|
||||
|
||||
# Main-slice seeds only: fnv1a(seed) % 8 == 7 is never probed. 1024 seeds
|
||||
# because the margins asserted in rung 8 are population claims, and a
|
||||
# 200-seed sample can draw 11% hidden instead of 15% and turn one into noise.
|
||||
SEEDS = [s for s in range(1024) if not is_held_out(s)]
|
||||
COMPONENTS = ("caught", "hours", "evidence")
|
||||
GOOD = ("fast", "targeted", "thorough")
|
||||
CHEATS = ("close_blind", "close_cites_all", "cite_unseen", "escalate_blind", "fast_defensive", "rule_table")
|
||||
|
||||
# The one-slider counterfactual the page names: move `hours` here and the
|
||||
# screen-only analyst wins. The other two weights keep their shipped ratio.
|
||||
COUNTERFACTUAL_HOURS = 0.45
|
||||
MARGIN = 0.02
|
||||
|
||||
|
||||
def counterfactual_weights(hours_weight: float) -> dict[str, float]:
|
||||
rest = 1.0 - hours_weight
|
||||
share = CAUGHT_WEIGHT + EVIDENCE_WEIGHT
|
||||
return {"caught": rest * CAUGHT_WEIGHT / share, "hours": hours_weight, "evidence": rest * EVIDENCE_WEIGHT / share}
|
||||
|
||||
|
||||
def run(seeds: list[int] | None = None) -> int:
|
||||
seeds = list(seeds or SEEDS)
|
||||
worlds = [world_for_seed(s) for s in seeds]
|
||||
tiers = {w["tier"] for w in worlds}
|
||||
print(f"Probing alert-triage over {len(seeds)} main-slice seeds "
|
||||
f"({sum(w['tier'] == 'benign' for w in worlds)} benign, "
|
||||
f"{sum(w['tier'] == 'visible' for w in worlds)} visible, "
|
||||
f"{sum(w['tier'] == 'hidden' for w in worlds)} hidden)\n")
|
||||
|
||||
results: dict[str, dict[str, float]] = {}
|
||||
cf: dict[str, float] = {}
|
||||
per_seed: dict[str, list] = {}
|
||||
for name, policy in POLICIES.items():
|
||||
episodes = [run_policy(policy, w).episode() for w in worlds]
|
||||
scores = [score(e) for e in episodes]
|
||||
assert all(v is not None for s in scores for v in s.values()), f"{name}: a component was None on a probe seed"
|
||||
results[name] = {c: sum(s[c] for s in scores) / len(scores) for c in COMPONENTS} # type: ignore[misc]
|
||||
results[name]["total"] = sum(total(e) for e in episodes) / len(episodes) # type: ignore[misc]
|
||||
cf[name] = sum(total(e, counterfactual_weights(COUNTERFACTUAL_HOURS)) for e in episodes) / len(episodes) # type: ignore[misc]
|
||||
per_seed[name] = episodes
|
||||
|
||||
width = max(len(n) for n in POLICIES)
|
||||
header = f"{'policy':<{width}} " + " ".join(f"{c:>10}" for c in COMPONENTS) + f" {'TOTAL':>8} {'hours@' + str(COUNTERFACTUAL_HOURS):>10}"
|
||||
print(header)
|
||||
print("-" * len(header))
|
||||
for name in POLICIES:
|
||||
row = results[name]
|
||||
cells = " ".join(f"{row[c]:>10.4f}" for c in COMPONENTS)
|
||||
print(f"{name:<{width}} {cells} {row['total']:>8.4f} {cf[name]:>10.4f}")
|
||||
|
||||
print(f"\nweights: {WEIGHTS}")
|
||||
print(f"roles: {ROLES}")
|
||||
print(f"blind-escalation break-even: escalate whenever P(benign) < {escalate_break_even():.2f}")
|
||||
mean_hours = {n: round(sum(metrics(e)['hours_spent'] for e in per_seed[n]) / len(seeds), 2) for n in GOOD}
|
||||
print(f"hours spent (mean): {mean_hours}")
|
||||
hidden = [i for i, w in enumerate(worlds) if w["tier"] == "hidden"]
|
||||
print("caught on hidden seeds:", {n: round(sum(score(per_seed[n][i])['caught'] for i in hidden) / max(1, len(hidden)), 3) for n in GOOD}) # type: ignore[arg-type]
|
||||
|
||||
failures: list[str] = []
|
||||
r = results
|
||||
|
||||
# 0. The seed set must contain every tier, or the assertions below measure noise.
|
||||
if tiers != {"benign", "visible", "hidden"}:
|
||||
failures.append(f"probe seeds cover tiers {sorted(tiers)}; need all three")
|
||||
if any(w["reference_minutes"] is None for w in worlds):
|
||||
failures.append("a probe seed has no reference_minutes — no shipped policy got it right")
|
||||
|
||||
# 1. Doing nothing, and doing it malformed, must measure exactly nothing.
|
||||
for name in ("inaction", "malformed"):
|
||||
if abs(r[name]["total"]) > 1e-12:
|
||||
failures.append(f"{name} scored {r[name]['total']:.6f}, must be 0.000")
|
||||
if any(e.rejected != 8 or e.disposition is not None for e in per_seed[name]):
|
||||
failures.append(f"{name} must be rejected on all eight turns and abort")
|
||||
|
||||
# 2. The counterweight is attainable: the investigating references catch everything.
|
||||
for name in ("targeted", "thorough"):
|
||||
if abs(r[name]["caught"] - 1.0) > 1e-12:
|
||||
failures.append(f"{name} caught {r[name]['caught']:.4f}, must be exactly 1.0")
|
||||
|
||||
# 3. And it bites the policy that defines the hours denominator on most seeds.
|
||||
if r["fast"]["caught"] >= 1.0 - 1e-12:
|
||||
failures.append("fast caught 1.0 — the hidden tier leaked into the screen")
|
||||
|
||||
# 4. Paying the counterweight's price is a real price.
|
||||
if not (r["fast"]["hours"] > r["targeted"]["hours"] > r["thorough"]["hours"]):
|
||||
failures.append("hours must order fast > targeted > thorough")
|
||||
|
||||
# 5. No good policy dominates another on every component.
|
||||
for a, b in (("fast", "targeted"), ("fast", "thorough"), ("targeted", "thorough")):
|
||||
if all(r[a][c] >= r[b][c] - 1e-12 for c in COMPONENTS):
|
||||
failures.append(f"{a} dominates {b} on every component — no real trade-off")
|
||||
if all(r[b][c] >= r[a][c] - 1e-12 for c in COMPONENTS):
|
||||
failures.append(f"{b} dominates {a} on every component — no real trade-off")
|
||||
|
||||
# 6. Every named cheat loses to every good policy.
|
||||
for cheat in CHEATS:
|
||||
for good in GOOD:
|
||||
if r[cheat]["total"] >= r[good]["total"] - 1e-12:
|
||||
failures.append(f"{cheat} ({r[cheat]['total']:.4f}) >= {good} ({r[good]['total']:.4f})")
|
||||
|
||||
# 7. The hour meter binds: a wasted month costs, and nothing else changes.
|
||||
if r["thorough_wasteful"]["total"] >= r["thorough"]["total"] - 1e-12:
|
||||
failures.append("thorough_wasteful >= thorough — the hour budget does not bind")
|
||||
for c in ("caught", "evidence"):
|
||||
if abs(r["thorough_wasteful"][c] - r["thorough"][c]) > 1e-12:
|
||||
failures.append(f"thorough_wasteful differs from thorough on {c}")
|
||||
|
||||
# 8. The headline claim, measured: targeted wins at shipped weights by a
|
||||
# stated margin, and the one-slider counterfactual hands it to fast.
|
||||
if r["targeted"]["total"] - r["fast"]["total"] < MARGIN:
|
||||
failures.append(f"targeted beats fast by only {r['targeted']['total'] - r['fast']['total']:.4f} at shipped weights (need {MARGIN})")
|
||||
if cf["fast"] - cf["targeted"] < MARGIN:
|
||||
failures.append(f"fast beats targeted by only {cf['fast'] - cf['targeted']:.4f} at hours={COUNTERFACTUAL_HOURS} (need {MARGIN})")
|
||||
|
||||
# 9. Citing an unopened document is rejected, every time.
|
||||
if any(e.rejected < 1 for e in per_seed["cite_unseen"]):
|
||||
failures.append("cite_unseen was not rejected on some seed")
|
||||
|
||||
# 10. F1's precision term works: cite-everything earns less than half of its caught share in evidence.
|
||||
if r["close_cites_all"]["evidence"] >= 0.5 * r["close_cites_all"]["caught"]:
|
||||
failures.append("close_cites_all earns too much evidence — precision is not biting")
|
||||
if r["close_cites_all"]["evidence"] >= r["fast"]["evidence"]:
|
||||
failures.append("close_cites_all evidence >= fast evidence")
|
||||
|
||||
# 11. History buys evidence on the visible typologies, and nothing else.
|
||||
if r["thorough"]["evidence"] <= r["fast"]["evidence"]:
|
||||
failures.append("thorough evidence <= fast evidence — itemised history buys nothing")
|
||||
|
||||
# 12. No weighted component may be flat across the ladder.
|
||||
for component in COMPONENTS:
|
||||
values = [r[n][component] for n in POLICIES]
|
||||
if max(values) - min(values) < 1e-9:
|
||||
failures.append(f"component '{component}' is flat across the ladder — it measures nothing")
|
||||
|
||||
print()
|
||||
if failures:
|
||||
for f in failures:
|
||||
print(f"FAIL: {f}")
|
||||
return 1
|
||||
print("All 13 ladder assertions hold.")
|
||||
return 0
|
||||
@@ -0,0 +1,346 @@
|
||||
"""The scripted analysts: three shipped references and the probe ladder's cheats.
|
||||
|
||||
Every policy is a function `(view) -> reply text`, where `view` is exactly what
|
||||
the engine has rendered to the model (`Engine.view()`), and nothing else. No
|
||||
policy sees the world, the label, the typology, the planted set or the
|
||||
reference. `tests/test_policies.py` enforces the signature and scans the view
|
||||
for hidden keys. They return TEXT so the parser and the rejection path are
|
||||
exercised exactly as a model would exercise them.
|
||||
|
||||
The decision rules below are heuristics, written down so a reviewer can see
|
||||
that the reference "analysts" are players, not oracles:
|
||||
|
||||
fast decides from the screen; pulls `documents` only where the screen
|
||||
says a document should explain the alert (a processor, a title
|
||||
company, an estate). One or two turns. Cannot see a hidden signal.
|
||||
targeted `fast` on a screen that shows a typology; otherwise the three
|
||||
lookups that could carry a hidden signal — documents, prior
|
||||
alerts, each counterparty on screen — then decides. No history.
|
||||
thorough always the full procedure: documents, prior alerts, each
|
||||
counterparty on screen, itemised history for the two most recent
|
||||
months, then decides; cites every constituting transaction it saw.
|
||||
|
||||
Hidden-signal rules, shared by targeted and thorough:
|
||||
* a counterparty shared with ≥ 3 other alerted accounts → FUNNEL_ACCOUNT,
|
||||
cite the counterparty and its transactions;
|
||||
* ≥ 3 prior alerts, all closed within a day on one identical rationale
|
||||
→ the rule's typology,
|
||||
cite the prior alerts and the triggering transactions;
|
||||
* a document naming a party whose counterparty lookup carries a different
|
||||
name → the rule's typology,
|
||||
cite the document, the counterparty and its triggering credits.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any, Callable
|
||||
|
||||
from .generator import INBOUND, NOMINAL_TYPOLOGY, RULE_IDS
|
||||
|
||||
Policy = Callable[[dict[str, Any]], str]
|
||||
|
||||
_FAMILY_OF = {rid: fam for table in RULE_IDS.values() for fam, rid in table.items()}
|
||||
EXPLAINING_DOCS = ("processing_agreement", "closing_statement", "executor_letter")
|
||||
|
||||
|
||||
def _emit(obj: dict[str, Any]) -> str:
|
||||
return json.dumps(obj)
|
||||
|
||||
|
||||
def _family(view: dict[str, Any]) -> str:
|
||||
return _FAMILY_OF.get(view["screen"]["alert"]["rule"], "STR")
|
||||
|
||||
|
||||
def _looked(view: dict[str, Any], what: str, key: str | None = None) -> dict[str, Any] | None:
|
||||
for l in view["lookups"]:
|
||||
if l["what"] == what and (key is None or l["key"] == key):
|
||||
return l["result"]
|
||||
return None
|
||||
|
||||
|
||||
def _band(view: dict[str, Any], t: dict[str, Any]) -> bool:
|
||||
threshold = view["screen"]["threshold"]
|
||||
return t["channel"] == "cash_in" and threshold * 8 // 10 <= t["amount"] < threshold
|
||||
|
||||
|
||||
def _close(reason: str, cites: list[str], note: str) -> dict[str, Any]:
|
||||
return {"action": "close", "reason": reason, "cites": cites, "note": note}
|
||||
|
||||
|
||||
def _escalate(typology: str, cites: list[str], narrative: str) -> dict[str, Any]:
|
||||
return {"action": "escalate", "typology": typology, "cites": cites, "narrative": narrative}
|
||||
|
||||
|
||||
def _dedupe(ids: list[str]) -> list[str]:
|
||||
out: list[str] = []
|
||||
for i in ids:
|
||||
if i not in out:
|
||||
out.append(i)
|
||||
return out
|
||||
|
||||
|
||||
# ------------------------------------------------------------ decisions --
|
||||
|
||||
|
||||
def screen_decision(view: dict[str, Any]) -> dict[str, Any] | None:
|
||||
"""What the screen alone supports. None means "a document should explain this"."""
|
||||
screen = view["screen"]
|
||||
fam = _family(view)
|
||||
trig = screen["triggering_transactions"]
|
||||
trig_ids = [t["id"] for t in trig]
|
||||
kyc = screen["customer"]
|
||||
max_cash = max(r["cash_in"] for r in screen["summary"])
|
||||
|
||||
if fam == "STR":
|
||||
band = [t for t in trig if _band(view, t)]
|
||||
branches = {t["branch"] for t in band}
|
||||
if len(band) >= 5 or (len(band) >= 4 and len(branches) >= 3):
|
||||
return _escalate("STRUCTURING", [t["id"] for t in band], "Repeated cash deposits just under the threshold across branches.")
|
||||
if kyc["expected_monthly_cash"] >= max_cash:
|
||||
return _close("CONSISTENT_WITH_PROFILE", ["kyc.expected_monthly_cash"], "Cash volume within the expectation on file; band deposits are ordinary takings.")
|
||||
return _escalate("STRUCTURING", [t["id"] for t in band] or trig_ids, "Band deposits with no expectation on file to cover them.")
|
||||
|
||||
if fam == "CASH":
|
||||
if kyc["expected_monthly_cash"] >= max_cash:
|
||||
return _close("CONSISTENT_WITH_PROFILE", ["kyc.expected_activity", "kyc.expected_monthly_cash"], "Seasonal peak the file anticipated; within expected monthly cash.")
|
||||
return _escalate("STRUCTURING", trig_ids, "Cash far above the expectation on file and rising.")
|
||||
|
||||
if fam == "VEL":
|
||||
senders = {t["cp"] for t in trig if t["channel"] == "p2p_in"}
|
||||
if len(senders) >= 3:
|
||||
return _escalate("MONEY_MULE", trig_ids, "Credits from unrelated individuals moved to an exchange within a day.")
|
||||
return None
|
||||
|
||||
if fam == "WIRE":
|
||||
if kyc.get("remittance_declared", "none declared") != "none declared":
|
||||
return _close("CONSISTENT_WITH_PROFILE", ["kyc.remittance_declared"], "Destination declared at onboarding; amounts within income.")
|
||||
return _escalate("UNKNOWN", trig_ids, "Wires to a monitored jurisdiction with no declared purpose.")
|
||||
|
||||
return None # DORM: needs the document
|
||||
|
||||
|
||||
def document_decision(view: dict[str, Any]) -> dict[str, Any]:
|
||||
"""After `documents`: close on the explaining document, or escalate if there is none."""
|
||||
docs = _looked(view, "documents")
|
||||
trig_ids = [t["id"] for t in view["screen"]["triggering_transactions"]]
|
||||
if docs:
|
||||
for d in docs["documents"]:
|
||||
if d["kind"] in EXPLAINING_DOCS:
|
||||
return _close("DOCUMENTED_SOURCE_OF_FUNDS", [d["id"]], f"Source of funds documented: {d['kind']}.")
|
||||
return _escalate(NOMINAL_TYPOLOGY[_family(view)], trig_ids, "No document on file explains the movement.")
|
||||
|
||||
|
||||
def lookup_decision(view: dict[str, Any]) -> dict[str, Any] | None:
|
||||
"""A hidden signal in the lookups, or None."""
|
||||
screen = view["screen"]
|
||||
fam = _family(view)
|
||||
trig = screen["triggering_transactions"]
|
||||
trig_ids = [t["id"] for t in trig]
|
||||
|
||||
for l in view["lookups"]:
|
||||
if l["what"] == "counterparty" and l["result"]["other_alerted_accounts"] >= 3:
|
||||
r = l["result"]
|
||||
return _escalate("FUNNEL_ACCOUNT", [r["id"]] + list(r["transactions"]),
|
||||
f"{r['id']} is shared with {r['other_alerted_accounts']} other alerted accounts; funds converge on it.")
|
||||
|
||||
priors = _looked(view, "prior_alerts")
|
||||
if priors and len(priors["alerts"]) >= 3:
|
||||
alerts = priors["alerts"]
|
||||
rationales = {p["rationale"] for p in alerts}
|
||||
if all(p["disposition"] == "closed" and p["elapsed_days"] <= 1 for p in alerts) and len(rationales) == 1:
|
||||
return _escalate(NOMINAL_TYPOLOGY[fam], [p["id"] for p in alerts] + trig_ids,
|
||||
"Repeat alerts closed within a day on one boilerplate rationale; nobody has looked.")
|
||||
|
||||
docs = _looked(view, "documents")
|
||||
if docs:
|
||||
for d in docs["documents"]:
|
||||
if not d["party"]:
|
||||
continue
|
||||
for cp_id in _dedupe([t["cp"] for t in trig if t["channel"] in INBOUND and t["cp"]]):
|
||||
cp = _looked(view, "counterparty", cp_id)
|
||||
if cp and cp["name"] != d["party"]:
|
||||
credits = [t["id"] for t in trig if t["cp"] == cp_id and t["channel"] in INBOUND]
|
||||
return _escalate(NOMINAL_TYPOLOGY[fam], [d["id"], cp_id] + credits,
|
||||
f"The document names {d['party']}; the funds came from {cp['name']}.")
|
||||
return None
|
||||
|
||||
|
||||
def enriched_decision(view: dict[str, Any]) -> dict[str, Any]:
|
||||
"""thorough's final call: the screen rules, with citations widened to everything itemised."""
|
||||
d = screen_decision(view)
|
||||
fam = _family(view)
|
||||
screen = view["screen"]
|
||||
months = screen["months"]
|
||||
seen: list[dict[str, Any]] = list(screen["triggering_transactions"])
|
||||
for l in view["lookups"]:
|
||||
if l["what"] == "history":
|
||||
seen += l["result"]["transactions"]
|
||||
if d is not None and d["action"] == "escalate":
|
||||
if fam == "STR":
|
||||
d["cites"] = _dedupe([t["id"] for t in seen if _band(view, t) and t["month"] == months[-1]] + d["cites"])
|
||||
elif fam == "CASH":
|
||||
last_two = set(months[-2:])
|
||||
d["cites"] = _dedupe([t["id"] for t in seen if t["channel"] == "cash_in" and t["month"] in last_two] + d["cites"])
|
||||
return d
|
||||
if d is None:
|
||||
return document_decision(view)
|
||||
return d
|
||||
|
||||
|
||||
def _plan_next(view: dict[str, Any], plan: list[tuple[str, str | None]]) -> str | None:
|
||||
for what, key in plan:
|
||||
if _looked(view, what, key) is None:
|
||||
if what == "history":
|
||||
return _emit({"action": "lookup", "what": "history", "month": key})
|
||||
if what == "counterparty":
|
||||
return _emit({"action": "lookup", "what": "counterparty", "id": key})
|
||||
return _emit({"action": "lookup", "what": what})
|
||||
return None
|
||||
|
||||
|
||||
# ------------------------------------------------------------- shipped --
|
||||
|
||||
|
||||
def fast(view: dict[str, Any]) -> str:
|
||||
d = screen_decision(view)
|
||||
if d is not None:
|
||||
return _emit(d)
|
||||
if _looked(view, "documents") is None:
|
||||
return _emit({"action": "lookup", "what": "documents"})
|
||||
return _emit(document_decision(view))
|
||||
|
||||
|
||||
def _investigation_plan(view: dict[str, Any]) -> list[tuple[str, str | None]]:
|
||||
plan: list[tuple[str, str | None]] = [("documents", None), ("prior_alerts", None)]
|
||||
plan += [("counterparty", cp["id"]) for cp in view["screen"]["screen_counterparties"]]
|
||||
return plan
|
||||
|
||||
|
||||
def targeted(view: dict[str, Any]) -> str:
|
||||
d = screen_decision(view)
|
||||
if d is not None and d["action"] == "escalate":
|
||||
return _emit(d)
|
||||
nxt = _plan_next(view, _investigation_plan(view))
|
||||
if nxt:
|
||||
return nxt
|
||||
found = lookup_decision(view)
|
||||
if found:
|
||||
return _emit(found)
|
||||
return _emit(d if d is not None else document_decision(view))
|
||||
|
||||
|
||||
def thorough(view: dict[str, Any]) -> str:
|
||||
months = view["screen"]["months"]
|
||||
plan = _investigation_plan(view) + [("history", months[-1]), ("history", months[-2])]
|
||||
nxt = _plan_next(view, plan)
|
||||
if nxt:
|
||||
return nxt
|
||||
found = lookup_decision(view)
|
||||
return _emit(found if found else enriched_decision(view))
|
||||
|
||||
|
||||
def thorough_wasteful(view: dict[str, Any]) -> str:
|
||||
"""thorough plus a month that carries nothing. Proves the hour meter binds."""
|
||||
months = view["screen"]["months"]
|
||||
plan = _investigation_plan(view) + [("history", months[-1]), ("history", months[-2]), ("history", months[-3])]
|
||||
nxt = _plan_next(view, plan)
|
||||
if nxt:
|
||||
return nxt
|
||||
found = lookup_decision(view)
|
||||
return _emit(found if found else enriched_decision(view))
|
||||
|
||||
|
||||
# -------------------------------------------------------------- cheats --
|
||||
|
||||
|
||||
def inaction(view: dict[str, Any]) -> str:
|
||||
return "Let me reason about this alert carefully before committing to a disposition."
|
||||
|
||||
|
||||
def malformed(view: dict[str, Any]) -> str:
|
||||
return _emit({"action": "file_sar", "cites": []})
|
||||
|
||||
|
||||
def close_blind(view: dict[str, Any]) -> str:
|
||||
return _emit(_close("CONSISTENT_WITH_PROFILE", ["kyc.customer_type"], "Looks fine."))
|
||||
|
||||
|
||||
def close_cites_all(view: dict[str, Any]) -> str:
|
||||
screen = view["screen"]
|
||||
cites = [f"kyc.{k}" for k in screen["customer"] if k not in ("id", "name")]
|
||||
cites += [t["id"] for t in screen["triggering_transactions"]]
|
||||
return _emit(_close("CONSISTENT_WITH_PROFILE", cites, "See everything."))
|
||||
|
||||
|
||||
def escalate_blind(view: dict[str, Any]) -> str:
|
||||
trig_ids = [t["id"] for t in view["screen"]["triggering_transactions"]]
|
||||
return _emit(_escalate(NOMINAL_TYPOLOGY[_family(view)], trig_ids, "Filing defensively."))
|
||||
|
||||
|
||||
def fast_defensive(view: dict[str, Any]) -> str:
|
||||
"""fast where the screen shows a typology; a blind filing everywhere else."""
|
||||
d = screen_decision(view)
|
||||
if d is not None and d["action"] == "escalate":
|
||||
return _emit(d)
|
||||
return escalate_blind(view)
|
||||
|
||||
|
||||
def rule_table(view: dict[str, Any]) -> str:
|
||||
"""Closes every wire-corridor and dormancy alert blind; files on everything else."""
|
||||
fam = _family(view)
|
||||
if fam == "WIRE":
|
||||
return _emit(_close("CONSISTENT_WITH_PROFILE", ["kyc.remittance_declared"], "Wires are always remittances."))
|
||||
if fam == "DORM":
|
||||
return _emit(_close("CONSISTENT_WITH_PROFILE", ["kyc.account_opened"], "Old accounts wake up."))
|
||||
return escalate_blind(view)
|
||||
|
||||
|
||||
def cite_unseen(view: dict[str, Any]) -> str:
|
||||
"""Cites a document it never opened; rejected, then falls back to closing blind."""
|
||||
if view["rejected"] == 0:
|
||||
return _emit(_close("DOCUMENTED_SOURCE_OF_FUNDS", ["doc.probate_letter"], "Per the letter on file."))
|
||||
return close_blind(view)
|
||||
|
||||
|
||||
POLICIES: dict[str, Policy] = {
|
||||
"inaction": inaction,
|
||||
"malformed": malformed,
|
||||
"close_blind": close_blind,
|
||||
"close_cites_all": close_cites_all,
|
||||
"cite_unseen": cite_unseen,
|
||||
"escalate_blind": escalate_blind,
|
||||
"fast_defensive": fast_defensive,
|
||||
"rule_table": rule_table,
|
||||
"fast": fast,
|
||||
"targeted": targeted,
|
||||
"thorough": thorough,
|
||||
"thorough_wasteful": thorough_wasteful,
|
||||
}
|
||||
|
||||
SHIPPED = ("fast", "targeted", "thorough")
|
||||
|
||||
|
||||
def run_policy(policy: Policy, world: dict[str, Any]):
|
||||
"""Play a world to the end under one policy. Returns the finished Engine."""
|
||||
from .engine import Engine
|
||||
|
||||
engine = Engine(world)
|
||||
while not engine.done:
|
||||
engine.step(policy(engine.view()))
|
||||
return engine
|
||||
|
||||
|
||||
def reference_for(world: dict[str, Any]) -> tuple[int | None, str | None]:
|
||||
"""The cheapest shipped policy that reaches the correct disposition, in minutes.
|
||||
|
||||
(None, None) if none does — which `tests/test_reference.py` asserts never
|
||||
happens for seeds 0–4095, so a null here is a CI failure, not a silent
|
||||
"not scored" on the page.
|
||||
"""
|
||||
best: tuple[int, str] | None = None
|
||||
for name in SHIPPED:
|
||||
engine = run_policy(POLICIES[name], world)
|
||||
if engine.outcome == "solved" and (best is None or engine.minutes < best[0]):
|
||||
best = (engine.minutes, name)
|
||||
return best if best else (None, None)
|
||||
@@ -0,0 +1,209 @@
|
||||
"""Turning model text into one action, and engine results into text.
|
||||
|
||||
The one rule, as in wordle_five: **never raise**. A malformed reply is a
|
||||
thing the model did, not an error in the harness. It becomes a rejection that
|
||||
costs a turn and shows up in `rejected_replies`.
|
||||
|
||||
The parse rules are pinned here because this is where two runtimes disagree,
|
||||
and CI hashes a committed corpus of reply strings through this parser and the
|
||||
engine on both sides (digest (c) in CONFORMANCE.txt):
|
||||
|
||||
1. If a ```json fence appears anywhere, its contents are the candidate.
|
||||
A bare `{` before the fence does not win.
|
||||
2. Otherwise the candidate is the first string-aware balanced `{…}` span:
|
||||
braces inside JSON strings do not count, backslash escapes are honoured.
|
||||
3. If the candidate is not valid JSON, or is JSON but not an object, the
|
||||
reply is rejected. There is no further scanning for a second candidate.
|
||||
4. Enum values are matched exact-case. Cites are deduplicated in order and
|
||||
matched exact-case against what the engine has shown.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
from .generator import MAX_TURNS
|
||||
|
||||
ACTIONS = ("lookup", "close", "escalate")
|
||||
LOOKUPS = ("history", "counterparty", "prior_alerts", "documents")
|
||||
CLOSE_REASONS = ("CONSISTENT_WITH_PROFILE", "DOCUMENTED_SOURCE_OF_FUNDS", "PREVIOUSLY_REVIEWED", "RULE_ARTEFACT")
|
||||
TYPOLOGIES = ("STRUCTURING", "FUNNEL_ACCOUNT", "MONEY_MULE", "RAPID_MOVEMENT", "UNKNOWN")
|
||||
|
||||
_FENCE = re.compile(r"```json\s*(.*?)```", re.S)
|
||||
_MONTH = re.compile(r"^\d{4}-\d{2}$")
|
||||
|
||||
|
||||
def _reject_constant(name: str):
|
||||
# Python's json accepts NaN/Infinity by default; JSON.parse does not.
|
||||
raise ValueError(name)
|
||||
|
||||
|
||||
def extract_candidate(text: str) -> str | None:
|
||||
match = _FENCE.search(text)
|
||||
if match:
|
||||
return match.group(1)
|
||||
start = text.find("{")
|
||||
if start < 0:
|
||||
return None
|
||||
depth = 0
|
||||
in_string = False
|
||||
escaped = False
|
||||
for i in range(start, len(text)):
|
||||
ch = text[i]
|
||||
if in_string:
|
||||
if escaped:
|
||||
escaped = False
|
||||
elif ch == "\\":
|
||||
escaped = True
|
||||
elif ch == '"':
|
||||
in_string = False
|
||||
continue
|
||||
if ch == '"':
|
||||
in_string = True
|
||||
elif ch == "{":
|
||||
depth += 1
|
||||
elif ch == "}":
|
||||
depth -= 1
|
||||
if depth == 0:
|
||||
return text[start : i + 1]
|
||||
return None
|
||||
|
||||
|
||||
def parse_reply(text: str | None) -> tuple[dict[str, Any] | None, str | None]:
|
||||
"""(object, None) or (None, reason). Validation of fields is the engine's job."""
|
||||
if not text:
|
||||
return None, "no JSON object found in the reply"
|
||||
candidate = extract_candidate(text)
|
||||
if candidate is None:
|
||||
return None, "no JSON object found in the reply"
|
||||
try:
|
||||
value = json.loads(candidate, parse_constant=_reject_constant)
|
||||
except (ValueError, RecursionError):
|
||||
return None, "the JSON object could not be parsed"
|
||||
if not isinstance(value, dict):
|
||||
return None, "the reply parsed but is not a JSON object"
|
||||
return value, None
|
||||
|
||||
|
||||
# -------------------------------------------------------------- render --
|
||||
|
||||
SYSTEM_PROMPT = """You are a transaction-monitoring analyst at a bank. One alert has been assigned to you.
|
||||
|
||||
All data is synthetic. No real customer, account, transaction or counterparty exists.
|
||||
|
||||
Your screen shows the alert, the transactions that triggered it, the customer's KYC file and a twelve-month summary. Reading it has already cost {screen} analyst-hours. You have {turns} replies. Each reply must be exactly ONE JSON object (in a ```json fence, or as the first {{...}} in the reply), and every reply costs {turn} hours whether or not it is accepted.
|
||||
|
||||
Lookups (each may be made once):
|
||||
{{"action":"lookup","what":"history","month":"YYYY-MM"}} itemised transactions for a month in the window +{lookup} h
|
||||
{{"action":"lookup","what":"counterparty","id":"CP-n"}} name, type, jurisdiction, first seen, how many OTHER alerted accounts share it, its transactions +{lookup} h
|
||||
{{"action":"lookup","what":"prior_alerts"}} prior alerts on this customer with disposition, analyst and rationale +{lookup} h
|
||||
{{"action":"lookup","what":"documents"}} documents on file +{doc} h
|
||||
|
||||
Dispositions (end the episode):
|
||||
{{"action":"close","reason":R,"cites":[ids],"note":"..."}}
|
||||
R is one of CONSISTENT_WITH_PROFILE, DOCUMENTED_SOURCE_OF_FUNDS, PREVIOUSLY_REVIEWED, RULE_ARTEFACT
|
||||
{{"action":"escalate","typology":T,"cites":[ids],"narrative":"..."}}
|
||||
T is one of STRUCTURING, FUNNEL_ACCOUNT, MONEY_MULE, RAPID_MOVEMENT, UNKNOWN
|
||||
|
||||
`cites` must be a non-empty list of ids you have actually been shown: kyc.<field>, T-<n>, CP-<n>, A-<n> (a prior alert), doc.<id>. Citing an id you have not been shown is rejected. The note or narrative is displayed and never scored.
|
||||
|
||||
A close on an alert that should have been escalated scores zero. An escalation on a clean customer scores half on the counterweight and zero on the rest. Cite the facts that decide the case, not everything you saw. If you reach your last reply without a disposition the alert stays in the queue and the episode scores zero."""
|
||||
|
||||
|
||||
def system_prompt() -> str:
|
||||
from .reward import DOC_MINUTES, LOOKUP_MINUTES, SCREEN_MINUTES, TURN_MINUTES
|
||||
|
||||
return SYSTEM_PROMPT.format(
|
||||
screen=f"{SCREEN_MINUTES / 60:.2f}",
|
||||
turn=f"{TURN_MINUTES / 60:.2f}",
|
||||
lookup=f"{LOOKUP_MINUTES / 60:.2f}",
|
||||
doc=f"{DOC_MINUTES / 60:.2f}",
|
||||
turns=MAX_TURNS,
|
||||
)
|
||||
|
||||
|
||||
def money(cents: int, currency: str) -> str:
|
||||
sign = "-" if cents < 0 else ""
|
||||
cents = abs(cents)
|
||||
return f"{sign}{cents // 100:,}.{cents % 100:02d} {currency}"
|
||||
|
||||
|
||||
def _txn_line(t: dict[str, Any], currency: str) -> str:
|
||||
where = f"branch {t['branch']}" if t["branch"] else f"{t['cp']}"
|
||||
return f" {t['id']:<6} {t['date']} {t['channel']:<9} {money(t['amount'], currency):>18} {where}"
|
||||
|
||||
|
||||
def render_screen(screen: dict[str, Any]) -> str:
|
||||
cur = screen["currency"]
|
||||
c = screen["customer"]
|
||||
a = screen["alert"]
|
||||
alert_block = [
|
||||
f"ALERT {a['id']} rule {a['rule']} — {a['rule_text']}",
|
||||
f" fired {a['fired']}",
|
||||
" triggering transactions:",
|
||||
*[_txn_line(t, cur) for t in screen["triggering_transactions"]],
|
||||
]
|
||||
kyc_lines = [f"KYC FILE {c['id']} {c['name']}"]
|
||||
for key in sorted(c):
|
||||
if key in ("id", "name"):
|
||||
continue
|
||||
value = c[key]
|
||||
if key in ("stated_income", "expected_monthly_cash"):
|
||||
value = money(value, cur)
|
||||
kyc_lines.append(f" kyc.{key} = {value}")
|
||||
rows = screen["summary"]
|
||||
summary = ["TWELVE-MONTH SUMMARY (" + cur + ")",
|
||||
f" {'month':<8}{'cash in':>16}{'cash out':>16}{'wires in':>16}{'wires out':>16}{'ach in':>16}{'ach out':>16}{'txns':>6}"]
|
||||
for r in rows:
|
||||
summary.append(
|
||||
f" {r['month']:<8}{r['cash_in'] // 100:>16,}{r['cash_out'] // 100:>16,}{r['wires_in'] // 100:>16,}"
|
||||
f"{r['wires_out'] // 100:>16,}{r['ach_in'] // 100:>16,}{r['ach_out'] // 100:>16,}{r['count']:>6}"
|
||||
)
|
||||
cps = ["COUNTERPARTIES on screen (top by twelve-month volume):"]
|
||||
for cp in screen["screen_counterparties"]:
|
||||
cps.append(f" {cp['id']:<6} {cp['direction']:<4} {cp['count']:>3} txns {money(cp['total'], cur)}")
|
||||
blocks = [kyc_lines, alert_block] if screen["screen_order"] == "kyc_first" else [alert_block, kyc_lines]
|
||||
blocks += [summary, cps, [f"Reporting threshold in this jurisdiction: {money(screen['threshold'], cur)}",
|
||||
"SYNTHETIC DATA — no real customer, account or transaction."]]
|
||||
return "\n".join("\n".join(b) for b in blocks)
|
||||
|
||||
|
||||
def render_lookup(what: str, result: dict[str, Any], currency: str) -> str:
|
||||
if what == "history":
|
||||
lines = [f"HISTORY {result['month']}: {len(result['transactions'])} transactions"]
|
||||
lines += [_txn_line(t, currency) for t in result["transactions"]]
|
||||
return "\n".join(lines)
|
||||
if what == "counterparty":
|
||||
r = result
|
||||
return "\n".join([
|
||||
f"COUNTERPARTY {r['id']} {r['name']}",
|
||||
f" type {r['type']}; jurisdiction {r['jurisdiction']}; first seen {r['first_seen']}",
|
||||
f" other alerted accounts sharing this counterparty in twelve months: {r['other_alerted_accounts']}",
|
||||
f" transactions: {', '.join(r['transactions'])}",
|
||||
])
|
||||
if what == "prior_alerts":
|
||||
if not result["alerts"]:
|
||||
return "PRIOR ALERTS: none on file."
|
||||
lines = ["PRIOR ALERTS:"]
|
||||
for p in result["alerts"]:
|
||||
lines.append(f" {p['id']} {p['rule']} fired {p['fired']} {p['disposition']} after {p['elapsed_days']} day(s) by {p['analyst']}")
|
||||
lines.append(f" {p['rationale']}")
|
||||
return "\n".join(lines)
|
||||
if what == "documents":
|
||||
lines = ["DOCUMENTS ON FILE:"]
|
||||
for d in result["documents"]:
|
||||
lines.append(f" {d['id']} [{d['kind']}] {d['description']}")
|
||||
return "\n".join(lines)
|
||||
return json.dumps(result)
|
||||
|
||||
|
||||
def render_rejection(reason: str, turns_left: int) -> str:
|
||||
plural = "y" if turns_left == 1 else "ies"
|
||||
return f"That reply was not accepted: {reason}. It still cost a turn. You have {turns_left} repl{plural} left."
|
||||
|
||||
|
||||
def render_cost(minutes_spent: int, turns_left: int) -> str:
|
||||
plural = "y" if turns_left == 1 else "ies"
|
||||
return f"Hours spent so far: {minutes_spent / 60:.2f}. You have {turns_left} repl{plural} left."
|
||||
@@ -0,0 +1,210 @@
|
||||
"""The reward. Quoted verbatim on the demo page, so written to be read by
|
||||
someone who does not write Python.
|
||||
|
||||
Three components, weights summing to 1.0. Two pull toward clearing the alert
|
||||
quickly with the right citations. The third — the heaviest — pulls the other
|
||||
way: it pays for not letting a suspicious customer through, and a policy that
|
||||
never investigates cannot collect it.
|
||||
|
||||
Hours are integers in MINUTES so that the world, the grader and the browser
|
||||
agree to the last digit; the page shows them as hours.
|
||||
|
||||
# region: pig-demo/reward
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
# The generator's config, NOT industry measurements. Nobody timed an analyst.
|
||||
SCREEN_MINUTES = 30 # reading the free screen; charged to everyone
|
||||
TURN_MINUTES = 3 # every reply, accepted or rejected
|
||||
LOOKUP_MINUTES = 24 # history, counterparty, prior alerts
|
||||
DOC_MINUTES = 6 # documents on file
|
||||
|
||||
CAUGHT_WEIGHT = 0.55
|
||||
HOURS_WEIGHT = 0.25
|
||||
EVIDENCE_WEIGHT = 0.20
|
||||
|
||||
# What `caught` pays for filing on a clean customer. Not zero: a defensive
|
||||
# escalation is a wasted filing, not a missed crime. Not one: otherwise a
|
||||
# policy that escalates everything takes the counterweight for free.
|
||||
FALSE_ESCALATION_CREDIT = 0.5
|
||||
|
||||
|
||||
@dataclass
|
||||
class Episode:
|
||||
"""Everything the reward needs from one played alert."""
|
||||
|
||||
label: str # "benign" | "suspicious" — what the generator planted
|
||||
planted: list[list[str]] # alternates; the best one is scored
|
||||
disposition: str | None # "close" | "escalate" | None
|
||||
cites: list[str]
|
||||
minutes_spent: int
|
||||
reference_minutes: int | None # cheapest shipped policy that got this seed right
|
||||
turns_spent: int
|
||||
rejected: int
|
||||
truncated: bool = False # ended early with no disposition: not scored
|
||||
typology: str | None = None # what the agent wrote; diagnostic only
|
||||
true_typology: str | None = None # what the generator planted; diagnostic only
|
||||
lookups: int = 0
|
||||
|
||||
|
||||
def correct(ep: Episode) -> bool:
|
||||
"""A close on a benign alert, or an escalation on a suspicious one."""
|
||||
return (ep.label == "benign" and ep.disposition == "close") or (
|
||||
ep.label == "suspicious" and ep.disposition == "escalate"
|
||||
)
|
||||
|
||||
|
||||
def caught(ep: Episode) -> float:
|
||||
"""Nothing slipped through. The counterweight, and over half the reward.
|
||||
|
||||
1.0 on any correct call. 0.0 if a suspicious alert was closed, or nothing
|
||||
was decided. A false escalation earns FALSE_ESCALATION_CREDIT: the
|
||||
customer was clean, so nothing slipped, but a report was filed on them.
|
||||
|
||||
Read the asymmetry off the weights: a miss zeroes this AND the two
|
||||
objectives below (the call was wrong), so the episode scores 0.000. The
|
||||
most any policy can gain by being fast is `hours`' 0.25. One miss costs
|
||||
more than every hour saved.
|
||||
"""
|
||||
if ep.disposition is None:
|
||||
return 0.0
|
||||
if correct(ep):
|
||||
return 1.0
|
||||
if ep.label == "benign":
|
||||
return FALSE_ESCALATION_CREDIT
|
||||
return 0.0
|
||||
|
||||
|
||||
def hours(ep: Episode) -> float | None:
|
||||
"""Cleared it in the hours budgeted, as a ratio against a reference.
|
||||
|
||||
The reference is the cheapest SHIPPED policy that reached the correct
|
||||
disposition on this same alert — `fast` where the screen was enough,
|
||||
`targeted` where a lookup was needed. Never a true optimum: grade against
|
||||
an optimum and the shipped policy fails its own probe on some seeds.
|
||||
|
||||
Capped at 1.0. A correct call faster than the reference is not paid
|
||||
extra; the ceiling is our heuristic, not an optimum. A fast wrong answer
|
||||
saves nothing. If no shipped policy got this seed right there is no
|
||||
reference and this is None — not scored — which CI asserts never happens.
|
||||
"""
|
||||
if not correct(ep):
|
||||
return 0.0
|
||||
if ep.reference_minutes is None:
|
||||
return None
|
||||
return min(1.0, ep.reference_minutes / max(1, ep.minutes_spent))
|
||||
|
||||
|
||||
def f1_exact(cites: list[str], planted: list[str]) -> tuple[int, int]:
|
||||
"""F1 as an exact fraction (numerator, denominator), so both runtimes agree bit for bit."""
|
||||
common = len(set(cites) & set(planted))
|
||||
return 2 * common, len(set(cites)) + len(set(planted))
|
||||
|
||||
|
||||
def evidence(ep: Episode) -> float:
|
||||
"""Named the facts that decide it: F1 between the cites and the planted set.
|
||||
|
||||
The planted set is what the generator buried — the explaining KYC or
|
||||
document fact on a benign alert, the transactions that constitute the
|
||||
typology on a suspicious one. Where a template admits two reasonable
|
||||
rationales the best alternate is scored.
|
||||
|
||||
F1's precision term is what stops cite-everything: one planted id among
|
||||
eight cited scores 0.22. Its recall term is what stops cite-one-thing on
|
||||
a pattern of eight deposits. Zero if the disposition was wrong; a false
|
||||
escalation is checked against an empty set. The narrative prose is never
|
||||
read here.
|
||||
"""
|
||||
if not correct(ep) or not ep.cites:
|
||||
return 0.0
|
||||
best_num, best_den = 0, 1
|
||||
for alt in ep.planted:
|
||||
num, den = f1_exact(ep.cites, alt)
|
||||
if num * best_den > best_num * den:
|
||||
best_num, best_den = num, den
|
||||
return best_num / best_den
|
||||
|
||||
|
||||
def score(ep: Episode) -> dict[str, float | None]:
|
||||
"""The components, unweighted. None means "not scored", never zero."""
|
||||
if ep.truncated:
|
||||
return {"caught": None, "hours": None, "evidence": None}
|
||||
return {"caught": caught(ep), "hours": hours(ep), "evidence": evidence(ep)}
|
||||
|
||||
|
||||
def total(ep: Episode, weights: dict[str, float] | None = None) -> float | None:
|
||||
parts = score(ep)
|
||||
if any(v is None for v in parts.values()):
|
||||
return None
|
||||
w = weights or WEIGHTS
|
||||
return parts["caught"] * w["caught"] + parts["hours"] * w["hours"] + parts["evidence"] * w["evidence"] # type: ignore[operator]
|
||||
|
||||
|
||||
WEIGHTS = {"caught": CAUGHT_WEIGHT, "hours": HOURS_WEIGHT, "evidence": EVIDENCE_WEIGHT}
|
||||
ROLES = {"caught": "counterweight", "hours": "objective", "evidence": "objective"}
|
||||
|
||||
# endregion: pig-demo/reward
|
||||
|
||||
|
||||
def escalate_break_even(evidence_if_close: float = 1.0) -> float:
|
||||
"""P(benign) below which a blind escalation out-scores a well-cited close.
|
||||
|
||||
Escalating blind pays CAUGHT_WEIGHT × (credit × p_b + (1 − p_b)) plus, on
|
||||
the suspicious share, hours (paid 1.0: the blind filer is fastest) and
|
||||
nothing for evidence. Closing pays p_b × (CAUGHT + HOURS + EVIDENCE × e).
|
||||
Printed on the page so a BSA officer can read the threshold the weights
|
||||
encode rather than infer it.
|
||||
"""
|
||||
close_per_benign = CAUGHT_WEIGHT + HOURS_WEIGHT + EVIDENCE_WEIGHT * evidence_if_close
|
||||
esc_per_benign = CAUGHT_WEIGHT * FALSE_ESCALATION_CREDIT
|
||||
esc_per_suspicious = CAUGHT_WEIGHT + HOURS_WEIGHT
|
||||
# p_b · close = p_b · esc_b + (1 − p_b) · esc_s → p_b = esc_s / (close − esc_b + esc_s)
|
||||
return esc_per_suspicious / (close_per_benign - esc_per_benign + esc_per_suspicious)
|
||||
|
||||
|
||||
def score_exact(ep: Episode) -> dict[str, list[int] | None]:
|
||||
"""The components as exact fractions, for the cross-language digest."""
|
||||
if ep.truncated:
|
||||
return {"caught": None, "hours": None, "evidence": None}
|
||||
if ep.disposition is None:
|
||||
c: list[int] = [0, 1]
|
||||
elif correct(ep):
|
||||
c = [1, 1]
|
||||
elif ep.label == "benign":
|
||||
c = [1, 2]
|
||||
else:
|
||||
c = [0, 1]
|
||||
if not correct(ep):
|
||||
h: list[int] | None = [0, 1]
|
||||
elif ep.reference_minutes is None:
|
||||
h = None
|
||||
else:
|
||||
spent = max(1, ep.minutes_spent)
|
||||
h = [1, 1] if ep.reference_minutes >= spent else [ep.reference_minutes, spent]
|
||||
if not correct(ep) or not ep.cites:
|
||||
e = [0, 1]
|
||||
else:
|
||||
best = (0, 1)
|
||||
for alt in ep.planted:
|
||||
num, den = f1_exact(ep.cites, alt)
|
||||
if num * best[1] > best[0] * den:
|
||||
best = (num, den)
|
||||
e = [best[0], best[1]]
|
||||
return {"caught": c, "hours": h, "evidence": e}
|
||||
|
||||
|
||||
def metrics(ep: Episode) -> dict[str, float | None]:
|
||||
"""Diagnostics. Weight zero — reported, never summed into the reward."""
|
||||
return {
|
||||
"hours_spent": ep.minutes_spent / 60,
|
||||
"reference_hours": None if ep.reference_minutes is None else ep.reference_minutes / 60,
|
||||
"lookups": float(ep.lookups),
|
||||
"turns_used": float(ep.turns_spent),
|
||||
"rejected_replies": float(ep.rejected),
|
||||
"false_escalation": 1.0 if ep.label == "benign" and ep.disposition == "escalate" else 0.0,
|
||||
"missed": 1.0 if ep.label == "suspicious" and ep.disposition == "close" else 0.0,
|
||||
"typology_match": None if ep.disposition != "escalate" else (1.0 if ep.typology == ep.true_typology else 0.0),
|
||||
}
|
||||
@@ -0,0 +1,120 @@
|
||||
"""Portable randomness and portable dates.
|
||||
|
||||
Nothing in the generator may touch a language built-in for either. A built-in
|
||||
RNG picks a different alert for the same seed in the browser, and a built-in
|
||||
date type is timezone-sensitive in JavaScript — CI in UTC passes and a
|
||||
visitor in another timezone sees a different fire date for the same `?seed=`.
|
||||
Both are hand-written here and ported line for line to TypeScript.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
MASK = 0xFFFFFFFF
|
||||
|
||||
|
||||
def fnv1a32(text: str) -> int:
|
||||
"""FNV-1a, 32-bit, over the UTF-8 bytes. Identical to wordle_five's."""
|
||||
h = 0x811C9DC5
|
||||
for byte in text.encode():
|
||||
h ^= byte
|
||||
h = (h * 0x01000193) & MASK
|
||||
return h
|
||||
|
||||
|
||||
class XorShift32:
|
||||
"""Marsaglia xorshift32. Seven lines, and the browser has the same seven.
|
||||
|
||||
`below(n)` is a plain modulo. It is biased for large `n`, which does not
|
||||
matter here (every `n` is tiny) and keeps the port a one-liner —
|
||||
`Math.imul`-free, since the state is only ever shifted and xored.
|
||||
"""
|
||||
|
||||
def __init__(self, seed: int) -> None:
|
||||
self.state = seed & MASK or 0x9E3779B9
|
||||
|
||||
def next(self) -> int:
|
||||
x = self.state
|
||||
x ^= (x << 13) & MASK
|
||||
x ^= x >> 17
|
||||
x ^= (x << 5) & MASK
|
||||
self.state = x
|
||||
return x
|
||||
|
||||
def below(self, n: int) -> int:
|
||||
return self.next() % n
|
||||
|
||||
def between(self, lo: int, hi: int) -> int:
|
||||
"""Inclusive on both ends."""
|
||||
return lo + self.below(hi - lo + 1)
|
||||
|
||||
def chance(self, percent: int) -> bool:
|
||||
return self.below(100) < percent
|
||||
|
||||
def pick(self, items):
|
||||
return items[self.below(len(items))]
|
||||
|
||||
def sample(self, items, k: int) -> list:
|
||||
"""k distinct items, Fisher-Yates over a copy, first k taken."""
|
||||
pool = list(items)
|
||||
out = []
|
||||
for i in range(min(k, len(pool))):
|
||||
j = i + self.below(len(pool) - i)
|
||||
pool[i], pool[j] = pool[j], pool[i]
|
||||
out.append(pool[i])
|
||||
return out
|
||||
|
||||
def hex4(self) -> str:
|
||||
return format(self.below(65536), "04x")
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ dates --
|
||||
# Howard Hinnant's civil-from-days / days-from-civil. Integer arithmetic only.
|
||||
# Day 0 is 1970-01-01.
|
||||
|
||||
|
||||
def days_from_civil(y: int, m: int, d: int) -> int:
|
||||
y -= m <= 2
|
||||
era = (y if y >= 0 else y - 399) // 400
|
||||
yoe = y - era * 400
|
||||
mp = (m + 9) % 12
|
||||
doy = (153 * mp + 2) // 5 + d - 1
|
||||
doe = yoe * 365 + yoe // 4 - yoe // 100 + doy
|
||||
return era * 146097 + doe - 719468
|
||||
|
||||
|
||||
def civil_from_days(z: int) -> tuple[int, int, int]:
|
||||
z += 719468
|
||||
era = (z if z >= 0 else z - 146096) // 146097
|
||||
doe = z - era * 146097
|
||||
yoe = (doe - doe // 1460 + doe // 36524 - doe // 146096) // 365
|
||||
y = yoe + era * 400
|
||||
doy = doe - (365 * yoe + yoe // 4 - yoe // 100)
|
||||
mp = (5 * doy + 2) // 153
|
||||
d = doy - (153 * mp + 2) // 5 + 1
|
||||
m = mp + 3 if mp < 10 else mp - 9
|
||||
return (y + (m <= 2), m, d)
|
||||
|
||||
|
||||
def days_in_month(y: int, m: int) -> int:
|
||||
if m == 12:
|
||||
return days_from_civil(y + 1, 1, 1) - days_from_civil(y, 12, 1)
|
||||
return days_from_civil(y, m + 1, 1) - days_from_civil(y, m, 1)
|
||||
|
||||
|
||||
def iso_date(day: int) -> str:
|
||||
y, m, d = civil_from_days(day)
|
||||
return f"{y:04d}-{m:02d}-{d:02d}"
|
||||
|
||||
|
||||
def month_key(y: int, m: int) -> str:
|
||||
return f"{y:04d}-{m:02d}"
|
||||
|
||||
|
||||
def month_of_day(day: int) -> str:
|
||||
y, m, _ = civil_from_days(day)
|
||||
return month_key(y, m)
|
||||
|
||||
|
||||
def add_months(y: int, m: int, n: int) -> tuple[int, int]:
|
||||
index = y * 12 + (m - 1) + n
|
||||
return (index // 12, index % 12 + 1)
|
||||
@@ -0,0 +1,116 @@
|
||||
"""The episode loop everything shares, and the optional verifiers taskset.
|
||||
|
||||
`play_episode` is the whole game: capture, the probe, the tests and the
|
||||
fixture verifier all drive it, so the thing evaluated and the thing tested
|
||||
cannot drift apart. The verifiers Env is a thin adapter over the same engine.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Callable
|
||||
|
||||
from .engine import Engine
|
||||
from .generator import MAX_TURNS, world_for_seed
|
||||
from .protocol import render_cost, render_lookup, render_rejection, render_screen, system_prompt
|
||||
from .reward import ROLES, WEIGHTS, metrics, score
|
||||
|
||||
TASKSET_ID = "alert-triage"
|
||||
|
||||
Responder = Callable[[str, list[dict[str, Any]], dict[str, Any]], str | None]
|
||||
|
||||
|
||||
def opening_message(engine: Engine) -> str:
|
||||
return render_screen(engine.screen()) + "\n\n" + render_cost(engine.minutes, MAX_TURNS - engine.turns)
|
||||
|
||||
|
||||
def observation_for(engine: Engine, step) -> str:
|
||||
"""The text the model sees after a reply."""
|
||||
left = MAX_TURNS - engine.turns
|
||||
if not step.accepted:
|
||||
return render_rejection(step.reason or "rejected", left)
|
||||
if step.action["action"] == "lookup":
|
||||
return render_lookup(step.action["what"], step.result, engine.world["currency"]) + "\n\n" + render_cost(engine.minutes, left)
|
||||
return "Disposition recorded. The episode is over."
|
||||
|
||||
|
||||
def play_episode(seed: int, respond: Responder, world: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
"""Drive one alert against a callable `(prompt, transcript, view) -> reply text`.
|
||||
|
||||
`view` is the structured observation (what scripted policies read);
|
||||
`transcript` is the text history (what a model reads). A `None` reply is
|
||||
recorded as a rejected turn — a dropped model call is a thing that
|
||||
happened, not a turn to discard.
|
||||
"""
|
||||
world = world or world_for_seed(seed)
|
||||
engine = Engine(world)
|
||||
prompt = system_prompt()
|
||||
transcript: list[dict[str, Any]] = [{"reply": None, "action": None, "observation": opening_message(engine)}]
|
||||
|
||||
while not engine.done:
|
||||
reply = respond(prompt, transcript, engine.view())
|
||||
step = engine.step(reply)
|
||||
transcript.append({"reply": reply, "action": step.action, "rejection": step.reason, "observation": observation_for(engine, step)})
|
||||
|
||||
episode = engine.episode()
|
||||
return {
|
||||
"seed": seed,
|
||||
"outcome": engine.outcome,
|
||||
"truncated": episode.truncated,
|
||||
"transcript": transcript,
|
||||
"rewards": score(episode),
|
||||
"weights": WEIGHTS,
|
||||
"roles": ROLES,
|
||||
"metrics": metrics(episode),
|
||||
"info": {
|
||||
"label": world["label"],
|
||||
"template": world["template"],
|
||||
"overlay": world["overlay"],
|
||||
"typology": world["typology"],
|
||||
"planted": world["planted"],
|
||||
"reference_minutes": world["reference_minutes"],
|
||||
"reference_policy": world["reference_policy"],
|
||||
"held_out": world["held_out"],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def replay(seed: int, replies: list[str | None], world: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
"""Re-score a recorded run from its reply strings alone. What `verify` does."""
|
||||
replies = list(replies)
|
||||
engine = Engine(world or world_for_seed(seed))
|
||||
for reply in replies:
|
||||
if engine.done:
|
||||
break
|
||||
engine.step(reply)
|
||||
episode = engine.episode()
|
||||
return {"rewards": score(episode), "metrics": metrics(episode), "outcome": engine.outcome, "truncated": episode.truncated}
|
||||
|
||||
|
||||
def build_taskset(): # pragma: no cover - requires the optional extra
|
||||
"""The verifiers v1 Taskset. Imported lazily so the module stays optional."""
|
||||
import verifiers as vf
|
||||
|
||||
class AlertTriageEnv(vf.Env):
|
||||
"""One agent seat, `analyst`. The engine plays the user role: every
|
||||
observation is computed, never generated, which is why the reward is
|
||||
verifiable at all."""
|
||||
|
||||
async def run(self, task, agent, trace):
|
||||
world = world_for_seed(task.data.info["seed"])
|
||||
engine = Engine(world)
|
||||
async with agent.interaction(system=system_prompt()) as session:
|
||||
message = opening_message(engine)
|
||||
while not engine.done:
|
||||
reply = await session.turn(message)
|
||||
step = engine.step(reply)
|
||||
message = observation_for(engine, step)
|
||||
episode = engine.episode()
|
||||
for key, value in score(episode).items():
|
||||
trace.record_reward(key, 0.0 if value is None else value, WEIGHTS[key])
|
||||
for key, value in metrics(episode).items():
|
||||
if value is not None:
|
||||
trace.record_reward(key, value, 0.0)
|
||||
trace.info["label"] = world["label"]
|
||||
trace.info["outcome"] = engine.outcome
|
||||
|
||||
return AlertTriageEnv
|
||||
@@ -0,0 +1,88 @@
|
||||
[
|
||||
{
|
||||
"seed": 0,
|
||||
"note": "well-formed lookups, a fenced disposition after a bare brace, brace inside a string",
|
||||
"replies": [
|
||||
"{\"action\":\"lookup\",\"what\":\"documents\"}",
|
||||
"I will check priors first. {\"action\":\"lookup\",\"what\":\"prior_alerts\"} and then decide.",
|
||||
"{\"action\":\"lookup\",\"what\":\"history\",\"month\":\"BADMONTH\"}",
|
||||
"Let me look at CP-1 {not json} ```json\n{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-1\"}\n```",
|
||||
"{\"action\":\"close\",\"reason\":\"CONSISTENT_WITH_PROFILE\",\"cites\":[\"kyc.customer_type\",\"kyc.customer_type\",\"kyc.country\"],\"note\":\"braces {inside} a string \\\" and an escaped quote\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"seed": 1,
|
||||
"note": "invalid first span then valid: rejected, no further scanning; lowercase enums; non-string cites",
|
||||
"replies": [
|
||||
"{\"action\":\"lookup\",\"what\":\"documents\",} {\"action\":\"lookup\",\"what\":\"documents\"}",
|
||||
"{\"action\":\"close\",\"reason\":\"consistent_with_profile\",\"cites\":[\"kyc.country\"]}",
|
||||
"{\"action\":\"Lookup\",\"what\":\"documents\"}",
|
||||
"{\"action\":\"close\",\"reason\":\"CONSISTENT_WITH_PROFILE\",\"cites\":[1,2]}",
|
||||
"{\"action\":\"close\",\"reason\":\"CONSISTENT_WITH_PROFILE\",\"cites\":\"kyc.country\"}",
|
||||
"{\"action\":\"escalate\",\"typology\":\"structuring\",\"cites\":[\"kyc.country\"]}",
|
||||
"{\"action\":\"escalate\",\"typology\":\"STRUCTURING\",\"cites\":[],\"narrative\":\"x\"}",
|
||||
"{\"action\":\"escalate\",\"typology\":\"STRUCTURING\",\"cites\":[\"kyc.country\"],\"narrative\":42}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"seed": 2,
|
||||
"note": "repeat lookups, nonexistent and unseen counterparties, out-of-window months, an unseen cite",
|
||||
"replies": [
|
||||
"{\"action\":\"lookup\",\"what\":\"prior_alerts\"}",
|
||||
"{\"action\":\"lookup\",\"what\":\"prior_alerts\"}",
|
||||
"{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-99\"}",
|
||||
"{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"cp-1\"}",
|
||||
"{\"action\":\"lookup\",\"what\":\"history\",\"month\":\"2024-01\"}",
|
||||
"{\"action\":\"lookup\",\"what\":\"history\",\"month\":\"2026-1\"}",
|
||||
"{\"action\":\"close\",\"reason\":\"DOCUMENTED_SOURCE_OF_FUNDS\",\"cites\":[\"doc.probate_letter\"]}",
|
||||
"{\"action\":\"close\",\"reason\":\"PREVIOUSLY_REVIEWED\",\"cites\":[\"kyc.prior_alerts\"]}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"seed": 3,
|
||||
"note": "prose only, empty, null, NaN, a fenced array, nested objects, unbalanced braces; runs out the budget -> aborted",
|
||||
"replies": [
|
||||
"I would like to think about this alert a little longer before acting.",
|
||||
"",
|
||||
null,
|
||||
"{\"action\":\"lookup\",\"what\":\"documents\",\"weight\":NaN}",
|
||||
"```json\n[{\"action\":\"lookup\",\"what\":\"documents\"}]\n```",
|
||||
"{\"outer\":{\"action\":\"lookup\",\"what\":\"documents\"}}",
|
||||
"{\"action\":\"lookup\",\"what\":\"documents\"",
|
||||
"{\"what\":\"documents\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"seed": 4,
|
||||
"note": "a full investigation that itemises the two most recent months and escalates on the last turn",
|
||||
"replies": [
|
||||
"{\"action\":\"lookup\",\"what\":\"documents\"}",
|
||||
"{\"action\":\"lookup\",\"what\":\"prior_alerts\"}",
|
||||
"{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-2\"}",
|
||||
"{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-3\"}",
|
||||
"{\"action\":\"lookup\",\"what\":\"history\",\"month\":\"2026-03\"}",
|
||||
"{\"action\":\"lookup\",\"what\":\"history\",\"month\":\"2026-04\"}",
|
||||
"{\"action\":\"lookup\",\"what\":\"history\",\"month\":\"2026-05\"}",
|
||||
"{\"action\":\"escalate\",\"typology\":\"UNKNOWN\",\"cites\":[\"kyc.country\",\"kyc.pep\"],\"narrative\":\"eighth turn\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"seed": 5,
|
||||
"note": "a disposition citing every shown KYC field; extra keys ignored; the episode ends and later replies are not counted",
|
||||
"replies": [
|
||||
"```json\n{\"action\": \"close\", \"reason\": \"RULE_ARTEFACT\", \"cites\": [\"kyc.customer_type\", \"kyc.country\", \"kyc.pep\", \"kyc.prior_alerts\", \"kyc.prior_sars\", \"kyc.stated_income\", \"kyc.expected_monthly_cash\", \"kyc.expected_activity\", \"kyc.account_opened\", \"kyc.kyc_refreshed\", \"kyc.remittance_declared\"], \"note\": \"\", \"confidence\": 0.9}\n```",
|
||||
"{\"action\":\"lookup\",\"what\":\"documents\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"seed": 6,
|
||||
"note": "stops after three turns with no disposition: truncated, not aborted; an unknown lookup kind",
|
||||
"replies": [
|
||||
"{\"action\":\"lookup\",\"what\":\"documents\"}",
|
||||
"thinking",
|
||||
"{\"action\":\"lookup\",\"what\":\"prior_alerts\"}",
|
||||
"{\"action\":\"lookup\",\"what\":\"emails\"}",
|
||||
"[{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":\"CP-1\"}] as a bare array: the inner object is the first balanced span"
|
||||
]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,31 @@
|
||||
# Where these names came from
|
||||
|
||||
Every list in this directory was written by hand for this environment. None
|
||||
is derived from a customer file, a sanctions list, a leaked dataset, a public
|
||||
records scrape or any other collection of real people or real businesses.
|
||||
|
||||
- **`first_names.json`, `surnames.json`** — 40 given names and 40 surnames,
|
||||
chosen to be common across many languages so that no generated customer
|
||||
reads as belonging to one region. A generated name such as "Priya Whitlock"
|
||||
or "Dmitri Okafor" is a random pairing; any resemblance to a real person is
|
||||
the ordinary consequence of common names being common. Analysts who "closed"
|
||||
prior alerts are drawn from the same two lists.
|
||||
- **`words.json`** — business names are composed as `adjective + noun +
|
||||
suffix` ("Harbor Lane Bistro LLC" is the shape). Branch codes and street
|
||||
names are invented.
|
||||
- **`jurisdictions.json`** — every counterparty jurisdiction is **fictional**.
|
||||
The "monitored" list is the synthetic institution's own high-risk corridor
|
||||
list; it names no real country and makes no claim about any real
|
||||
jurisdiction's risk rating. The customer's home country is the United States
|
||||
in the main seed slice and the fictional "Karnland" (currency KRN) in the
|
||||
held-out slice, where the reporting threshold is a fictional 20,000 KRN
|
||||
rather than the public 10,000 USD currency-transaction-report threshold.
|
||||
|
||||
Licence: these lists are part of the repository and carry its Apache-2.0
|
||||
licence.
|
||||
|
||||
The scenario *templates* in `../alert_triage/generator.py` are written from
|
||||
public typology descriptions — FinCEN's SAR narrative guidance and the FATF
|
||||
typology reports — which are cited on the demo's evidence tab as references.
|
||||
Nothing from those documents is shipped as data, and no template is a real
|
||||
case.
|
||||
@@ -0,0 +1,4 @@
|
||||
["Aaron","Alicia","Amara","Andre","Beatriz","Bilal","Camille","Carlos","Chidi","Dana",
|
||||
"Dmitri","Elena","Emeka","Farah","Felix","Gabriela","Hana","Hiro","Ines","Ivan",
|
||||
"Jamal","Jonas","Keiko","Lars","Leila","Luca","Malik","Marta","Mateo","Mira",
|
||||
"Naomi","Nikolai","Olu","Priya","Rafael","Rosa","Samir","Sofia","Tomas","Yara"]
|
||||
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"home": {"USD": "United States", "KRN": "Karnland"},
|
||||
"monitored": ["Marovia","Kestrel Islands","Nordwall","Sarenne"],
|
||||
"other": ["Ostmark","Vellan","Tirsa","Halmond","Brixa","Corvale"]
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
["Abara","Bergstrom","Calloway","Delacroix","Eskildsen","Farrow","Galloway","Haddad","Iwata","Jankowski",
|
||||
"Kavanagh","Lindqvist","Marchetti","Nakamura","Oduya","Petrov","Quintero","Rasmussen","Sandoval","Tanaka",
|
||||
"Ubaldi","Vasquez","Whitlock","Xiang","Yilmaz","Zapata","Okafor","Brennan","Castellano","Dubois",
|
||||
"Ferreira","Grimaldi","Halvorsen","Ibarra","Jensen","Kowalczyk","Laurent","Moreau","Novak","Pereira"]
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"adjectives": ["Harbor","Northgate","Summit","Blue Ridge","Cedar","Ironwood","Lakeside","Meridian","Copperfield","Silverline",
|
||||
"Westbrook","Granite","Oakhill","Redfern","Stonebridge","Brightwater","Highland","Riverbend","Fairmont","Ashgrove",
|
||||
"Kingsway","Maple Lane","Pinecrest","Eastfield"],
|
||||
"nouns": ["Bistro","Grill","Kitchen","Diner","Landscaping","Tax Services","Pool Care","Staffing","Freight","Events",
|
||||
"Contracting","Foods","Produce","Supply","Equipment","Payroll","Logistics","Holdings","Trading","Ventures",
|
||||
"Vending","Car Wash","Laundry","Market"],
|
||||
"suffixes": ["LLC","Inc.","Co.","Group"],
|
||||
"branches": ["BR-014","BR-027","BR-031","BR-052","BR-068","BR-073","BR-089","BR-102"],
|
||||
"streets": ["Alder Street","Birch Avenue","Cypress Road","Dogwood Lane","Elm Court","Fir Street","Hazel Way","Juniper Drive"]
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
[project]
|
||||
name = "alert-triage"
|
||||
version = "0.1.0"
|
||||
description = "One synthetic transaction-monitoring alert, closed or escalated with citations, scored by a pure grader"
|
||||
requires-python = ">=3.11,<3.14"
|
||||
dependencies = []
|
||||
|
||||
[project.optional-dependencies]
|
||||
# verifiers is optional for the same reason as wordle_five: the generator, the
|
||||
# engine, the reward and the probe are gated in CI without an RL stack.
|
||||
verifiers = ["verifiers>=0.3.2.dev12"]
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["alert_triage"]
|
||||
|
||||
[tool.hatch.build]
|
||||
include = ["alert_triage/**", "names/**", "conformance/**", "pyproject.toml", "README.md"]
|
||||
@@ -0,0 +1,32 @@
|
||||
"""The cross-language gates. If any of these changes, the TypeScript port must be re-verified."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from alert_triage.conformance import committed, protocol_digest, protocol_trace, scorer_digest
|
||||
from alert_triage.generator import world_digest
|
||||
|
||||
|
||||
def test_scorer_digest_matches_the_committed_value() -> None:
|
||||
assert scorer_digest() == committed()["scorer"]
|
||||
|
||||
|
||||
def test_protocol_digest_matches_the_committed_value() -> None:
|
||||
assert protocol_digest() == committed()["protocol"]
|
||||
|
||||
|
||||
def test_protocol_corpus_exercises_every_rejection_path() -> None:
|
||||
trace = protocol_trace()
|
||||
reasons = {t["reason"] for case in trace for t in case["turns"] if not t["accepted"]}
|
||||
fragments = ["no JSON object", "could not be parsed", "not a JSON object", "unknown action", "unknown lookup",
|
||||
"not in the twelve-month window", "already been looked up", "has not appeared", "cites must be",
|
||||
"not been shown", "unknown close reason", "unknown typology"]
|
||||
for fragment in fragments:
|
||||
assert any(fragment in r for r in reasons), fragment
|
||||
outcomes = {case["outcome"] for case in trace}
|
||||
assert "aborted" in outcomes
|
||||
assert any(case["truncated"] for case in trace)
|
||||
assert any(len(case["turns"]) == 8 and case["outcome"] in ("solved", "failed") for case in trace)
|
||||
|
||||
|
||||
def test_world_digest_matches_the_committed_value() -> None:
|
||||
assert world_digest() == committed()["world"]
|
||||
@@ -0,0 +1,124 @@
|
||||
"""Rejections, charges, shown ids, and the abort/truncate line."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
from alert_triage.engine import Engine
|
||||
from alert_triage.generator import MAX_TURNS, world_for_seed
|
||||
from alert_triage.reward import DOC_MINUTES, LOOKUP_MINUTES, SCREEN_MINUTES, TURN_MINUTES, score
|
||||
|
||||
JUNK = ["", " ", None, "no json here", "{", "}", "{}", "[]", "null", "{\"action\":null}", "{\"action\":\"lookup\"}",
|
||||
"{\"action\":\"lookup\",\"what\":\"history\"}", "{\"action\":\"lookup\",\"what\":\"history\",\"month\":7}",
|
||||
"{\"action\":\"lookup\",\"what\":\"counterparty\"}", "{\"action\":\"lookup\",\"what\":\"counterparty\",\"id\":[]}",
|
||||
"{\"action\":\"close\"}", "{\"action\":\"close\",\"reason\":\"RULE_ARTEFACT\"}",
|
||||
"{\"action\":\"close\",\"reason\":\"RULE_ARTEFACT\",\"cites\":{}}", "{\"action\":\"escalate\",\"cites\":[\"kyc.country\"]}",
|
||||
"{\"action\":\"lookup\",\"what\":\"documents\",\"x\":NaN}", "\x00\x01", "{" * 50]
|
||||
|
||||
|
||||
def _e(seed: int = 0) -> Engine:
|
||||
return Engine(world_for_seed(seed))
|
||||
|
||||
|
||||
def test_screen_costs_and_first_turn() -> None:
|
||||
e = _e()
|
||||
assert e.minutes == SCREEN_MINUTES and e.turns == 0 and not e.done
|
||||
|
||||
|
||||
def test_junk_never_raises_and_costs_one_turn_each() -> None:
|
||||
for junk in JUNK:
|
||||
e = _e()
|
||||
step = e.step(junk)
|
||||
assert not step.accepted and step.reason and e.turns == 1 and e.rejected == 1
|
||||
assert e.minutes == SCREEN_MINUTES + TURN_MINUTES
|
||||
|
||||
|
||||
def test_trailing_garbage_after_a_balanced_object_is_fine() -> None:
|
||||
e = _e()
|
||||
assert e.step('{"action":"lookup","what":"documents"}' + "}" * 50 + " and more prose").accepted
|
||||
|
||||
|
||||
def test_lookup_charges_its_price_only_when_accepted() -> None:
|
||||
e = _e()
|
||||
e.step('{"action":"lookup","what":"documents"}')
|
||||
assert e.minutes == SCREEN_MINUTES + TURN_MINUTES + DOC_MINUTES
|
||||
e.step('{"action":"lookup","what":"documents"}') # repeat: rejected
|
||||
assert e.minutes == SCREEN_MINUTES + 2 * TURN_MINUTES + DOC_MINUTES and e.rejected == 1
|
||||
e.step('{"action":"lookup","what":"prior_alerts"}')
|
||||
assert e.minutes == SCREEN_MINUTES + 3 * TURN_MINUTES + DOC_MINUTES + LOOKUP_MINUTES
|
||||
|
||||
|
||||
def test_history_window_is_the_twelve_months_ending_in_the_fire_month() -> None:
|
||||
e = _e()
|
||||
months = e.world["months"]
|
||||
assert len(months) == 12 and months[-1] == e.world["alert"]["fired"][:7]
|
||||
assert e.step(json.dumps({"action": "lookup", "what": "history", "month": months[0]})).accepted
|
||||
before = months[0][:4] + "-" + f"{int(months[0][5:]) - 1:02d}" if months[0][5:] != "01" else f"{int(months[0][:4]) - 1}-12"
|
||||
assert not e.step(json.dumps({"action": "lookup", "what": "history", "month": before})).accepted
|
||||
|
||||
|
||||
def test_counterparty_must_have_been_shown() -> None:
|
||||
e = _e()
|
||||
on_screen = e.world["screen_counterparties"][0]["id"]
|
||||
assert e.step(json.dumps({"action": "lookup", "what": "counterparty", "id": on_screen})).accepted
|
||||
assert not e.step('{"action":"lookup","what":"counterparty","id":"CP-999"}').accepted
|
||||
assert not e.step(json.dumps({"action": "lookup", "what": "counterparty", "id": on_screen})).accepted
|
||||
|
||||
|
||||
def test_cites_must_be_shown_and_are_deduplicated() -> None:
|
||||
e = _e()
|
||||
step = e.step('{"action":"close","reason":"RULE_ARTEFACT","cites":["kyc.country","doc.D-0000"]}')
|
||||
assert not step.accepted and "not been shown" in step.reason
|
||||
step = e.step('{"action":"close","reason":"RULE_ARTEFACT","cites":["kyc.country","kyc.country","kyc.pep"]}')
|
||||
assert step.accepted and step.action["cites"] == ["kyc.country", "kyc.pep"] and e.done
|
||||
|
||||
|
||||
def test_documents_become_citable_after_the_lookup() -> None:
|
||||
e = _e()
|
||||
doc = e.world["documents"][0]["id"]
|
||||
assert not e.step(json.dumps({"action": "close", "reason": "DOCUMENTED_SOURCE_OF_FUNDS", "cites": [doc]})).accepted
|
||||
assert e.step('{"action":"lookup","what":"documents"}').accepted
|
||||
assert e.step(json.dumps({"action": "close", "reason": "DOCUMENTED_SOURCE_OF_FUNDS", "cites": [doc]})).accepted
|
||||
|
||||
|
||||
def test_enums_are_exact_case() -> None:
|
||||
e = _e()
|
||||
assert not e.step('{"action":"close","reason":"rule_artefact","cites":["kyc.country"]}').accepted
|
||||
assert not e.step('{"action":"ESCALATE","typology":"UNKNOWN","cites":["kyc.country"]}').accepted
|
||||
assert e.step('{"action":"escalate","typology":"UNKNOWN","cites":["kyc.country"]}').accepted
|
||||
|
||||
|
||||
def test_eight_rejections_abort_and_score_zero() -> None:
|
||||
e = _e()
|
||||
for _ in range(MAX_TURNS):
|
||||
e.step("nothing")
|
||||
assert e.done and e.aborted and e.outcome == "aborted"
|
||||
assert e.step("{}").reason == "the episode is over" and e.turns == MAX_TURNS
|
||||
assert score(e.episode()) == {"caught": 0.0, "hours": 0.0, "evidence": 0.0}
|
||||
|
||||
|
||||
def test_a_disposition_on_the_eighth_turn_is_accepted() -> None:
|
||||
e = _e()
|
||||
for _ in range(MAX_TURNS - 1):
|
||||
e.step("nothing")
|
||||
assert e.step('{"action":"close","reason":"RULE_ARTEFACT","cites":["kyc.country"]}').accepted
|
||||
assert e.outcome in ("solved", "failed")
|
||||
|
||||
|
||||
def test_stopping_early_is_truncated_not_aborted() -> None:
|
||||
e = _e()
|
||||
e.step("nothing")
|
||||
e.step('{"action":"lookup","what":"documents"}')
|
||||
assert not e.done and e.outcome is None
|
||||
ep = e.episode()
|
||||
assert ep.truncated
|
||||
assert score(ep) == {"caught": None, "hours": None, "evidence": None}
|
||||
|
||||
|
||||
def test_view_never_shows_hidden_fields() -> None:
|
||||
e = _e(5)
|
||||
e.step('{"action":"lookup","what":"documents"}')
|
||||
e.step('{"action":"lookup","what":"prior_alerts"}')
|
||||
text = json.dumps(e.view())
|
||||
for key in ("label", "planted", "typology", "overlay", "tier", "template", "reference_minutes", "reference_policy"):
|
||||
assert f'"{key}"' not in text, key
|
||||
@@ -0,0 +1,164 @@
|
||||
"""What the generator promises about every world, checked over every digested seed."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections import Counter
|
||||
|
||||
import pytest
|
||||
|
||||
from alert_triage.generator import (
|
||||
BENIGN_PCT,
|
||||
RULE_IDS,
|
||||
VISIBLE_PCT,
|
||||
canonical_json,
|
||||
generate,
|
||||
is_held_out,
|
||||
screen_of,
|
||||
)
|
||||
from alert_triage.rng import days_from_civil
|
||||
|
||||
ALL = range(4096)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def worlds():
|
||||
return [generate(s) for s in ALL]
|
||||
|
||||
|
||||
def _day(iso: str) -> int:
|
||||
y, m, d = (int(x) for x in iso.split("-"))
|
||||
return days_from_civil(y, m, d)
|
||||
|
||||
|
||||
def test_deterministic() -> None:
|
||||
assert canonical_json(generate(17)) == canonical_json(generate(17))
|
||||
assert canonical_json(generate(17)) != canonical_json(generate(18))
|
||||
|
||||
|
||||
def test_canonical_json_has_no_floats(worlds) -> None:
|
||||
def walk(v):
|
||||
if isinstance(v, float):
|
||||
raise AssertionError("float in world")
|
||||
if isinstance(v, dict):
|
||||
for x in v.values():
|
||||
walk(x)
|
||||
if isinstance(v, list):
|
||||
for x in v:
|
||||
walk(x)
|
||||
for w in worlds[:512]:
|
||||
walk(w)
|
||||
assert json.loads(canonical_json(w)) == w
|
||||
|
||||
|
||||
def test_mix_by_seed_is_close_to_declared(worlds) -> None:
|
||||
tiers = Counter(w["tier"] for w in worlds)
|
||||
n = len(worlds)
|
||||
assert abs(tiers["benign"] / n - BENIGN_PCT / 100) < 0.03
|
||||
assert abs(tiers["visible"] / n - VISIBLE_PCT / 100) < 0.03
|
||||
assert abs(tiers["hidden"] / n - (100 - BENIGN_PCT - VISIBLE_PCT) / 100) < 0.03
|
||||
|
||||
|
||||
def test_every_rule_carries_both_labels(worlds) -> None:
|
||||
"""CHEAT 3: no rule id may be a label. Each fires on benign AND suspicious seeds."""
|
||||
seen = Counter((w["alert"]["rule"], w["label"]) for w in worlds)
|
||||
for table in RULE_IDS.values():
|
||||
for rid in table.values():
|
||||
assert seen[(rid, "benign")] > 0, rid
|
||||
assert seen[(rid, "suspicious")] > 0, rid
|
||||
|
||||
|
||||
def test_every_hidden_signal_lands_on_several_templates(worlds) -> None:
|
||||
"""CHEAT 4: no fixed template -> lookup mapping to memorise."""
|
||||
pairs = Counter((w["overlay"], w["template"]) for w in worlds if w["overlay"])
|
||||
by_kind: dict[str, set[str]] = {}
|
||||
for (kind, template), _ in pairs.items():
|
||||
by_kind.setdefault(kind, set()).add(template)
|
||||
assert len(by_kind["funnel"]) == 6
|
||||
assert len(by_kind["serial_closer"]) >= 4
|
||||
assert len(by_kind["doc_mismatch"]) == 3
|
||||
|
||||
|
||||
def test_overlay_only_on_benign_screens(worlds) -> None:
|
||||
for w in worlds:
|
||||
if w["overlay"]:
|
||||
assert w["template"].startswith("B") and w["tier"] == "hidden" and w["label"] == "suspicious"
|
||||
if w["template"].startswith("S"):
|
||||
assert w["label"] == "suspicious" and w["overlay"] is None
|
||||
|
||||
|
||||
def test_nothing_is_dated_on_or_after_the_fire_date(worlds) -> None:
|
||||
for w in worlds:
|
||||
fired = _day(w["alert"]["fired"])
|
||||
assert all(_day(t["date"]) < fired for t in w["transactions"]), w["seed"]
|
||||
|
||||
|
||||
def test_transaction_ids_are_chronological_and_dense(worlds) -> None:
|
||||
for w in worlds[:512]:
|
||||
ids = [t["id"] for t in w["transactions"]]
|
||||
assert ids == [f"T-{i + 1}" for i in range(len(ids))]
|
||||
days = [_day(t["date"]) for t in w["transactions"]]
|
||||
assert days == sorted(days)
|
||||
|
||||
|
||||
def test_triggering_set_is_non_empty_and_shown(worlds) -> None:
|
||||
for w in worlds:
|
||||
ids = {t["id"] for t in w["transactions"]}
|
||||
assert w["alert"]["triggering"], w["seed"]
|
||||
assert set(w["alert"]["triggering"]) <= ids
|
||||
|
||||
|
||||
def test_planted_is_citable_after_the_right_lookups(worlds) -> None:
|
||||
"""Every planted id exists in the world: a kyc field, a T-id, a CP-id, a doc or a prior alert."""
|
||||
for w in worlds:
|
||||
universe = {f"kyc.{k}" for k in w["customer"]} | {t["id"] for t in w["transactions"]}
|
||||
universe |= {c["id"] for c in w["counterparties"]} | {d["id"] for d in w["documents"]} | {p["id"] for p in w["prior_alerts"]}
|
||||
assert w["planted"], w["seed"]
|
||||
for alt in w["planted"]:
|
||||
assert alt and set(alt) <= universe, (w["seed"], alt)
|
||||
|
||||
|
||||
def test_benign_screens_always_carry_an_outbound_counterparty(worlds) -> None:
|
||||
for w in worlds:
|
||||
if w["template"].startswith("B"):
|
||||
assert any(c["direction"] == "out" for c in w["screen_counterparties"]), w["seed"]
|
||||
assert len(w["screen_counterparties"]) == 2
|
||||
assert 1 <= len(w["screen_counterparties"]) <= 2
|
||||
|
||||
|
||||
def test_prior_alert_count_on_screen_matches_the_lookup(worlds) -> None:
|
||||
for w in worlds:
|
||||
assert len(w["prior_alerts"]) == w["customer"]["prior_alerts"]
|
||||
|
||||
|
||||
def test_held_out_slice_moves_the_threshold_and_the_screen(worlds) -> None:
|
||||
held = [w for w in worlds if w["held_out"]]
|
||||
main = [w for w in worlds if not w["held_out"]]
|
||||
assert 0.10 < len(held) / len(worlds) < 0.15
|
||||
for w in held:
|
||||
assert is_held_out(w["seed"])
|
||||
assert w["currency"] == "KRN" and w["threshold"] == 2_000_000 and w["screen_order"] == "kyc_first"
|
||||
assert w["alert"]["rule"] in RULE_IDS["held_out"].values()
|
||||
assert w["customer"]["country"] == "Karnland"
|
||||
for w in main:
|
||||
assert w["currency"] == "USD" and w["threshold"] == 1_000_000 and w["screen_order"] == "alert_first"
|
||||
assert w["alert"]["rule"] in RULE_IDS["main"].values()
|
||||
|
||||
|
||||
def test_structuring_band_is_threshold_relative(worlds) -> None:
|
||||
for w in worlds:
|
||||
if w["template"] == "S1":
|
||||
lo, hi = w["threshold"] * 92 // 100, w["threshold"] - 1
|
||||
deposits = [t for t in w["transactions"] if t["id"] in w["planted"][0]]
|
||||
assert len(deposits) == 8
|
||||
assert all(lo <= t["amount"] <= hi for t in deposits)
|
||||
assert len({t["branch"] for t in deposits}) == 3
|
||||
|
||||
|
||||
def test_screen_carries_no_hidden_field(worlds) -> None:
|
||||
hidden = {"label", "typology", "planted", "overlay", "tier", "template", "reference_minutes", "reference_policy",
|
||||
"prior_alerts_list", "documents", "counterparties", "transactions", "other_alerted_accounts", "party", "rationale"}
|
||||
for w in worlds[:256]:
|
||||
text = canonical_json(screen_of(w))
|
||||
for key in hidden:
|
||||
assert f'"{key}"' not in text, key
|
||||
@@ -0,0 +1,84 @@
|
||||
"""The reference analysts are players, not oracles, and the reference exists everywhere."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import inspect
|
||||
|
||||
from alert_triage.engine import Engine
|
||||
from alert_triage.generator import MAX_TURNS, canonical_json, generate, world_for_seed
|
||||
from alert_triage.policies import POLICIES, SHIPPED, reference_for, run_policy
|
||||
|
||||
ALL = range(4096)
|
||||
|
||||
|
||||
def test_every_policy_takes_only_the_view() -> None:
|
||||
for name, policy in POLICIES.items():
|
||||
params = list(inspect.signature(policy).parameters)
|
||||
assert params == ["view"], name
|
||||
|
||||
|
||||
def test_policies_never_mutate_the_world() -> None:
|
||||
for seed in range(32):
|
||||
world = world_for_seed(seed)
|
||||
before = canonical_json(world)
|
||||
for policy in POLICIES.values():
|
||||
run_policy(policy, world)
|
||||
assert canonical_json(world) == before
|
||||
|
||||
|
||||
def test_shipped_policies_are_never_rejected() -> None:
|
||||
for seed in range(256):
|
||||
world = world_for_seed(seed)
|
||||
for name in SHIPPED + ("thorough_wasteful",):
|
||||
engine = run_policy(POLICIES[name], world)
|
||||
assert engine.rejected == 0, (seed, name)
|
||||
assert engine.disposition is not None, (seed, name)
|
||||
|
||||
|
||||
def test_turn_budgets_are_as_documented() -> None:
|
||||
for seed in range(128):
|
||||
world = world_for_seed(seed)
|
||||
assert run_policy(POLICIES["fast"], world).turns <= 2
|
||||
assert run_policy(POLICIES["targeted"], world).turns <= 5
|
||||
thorough = run_policy(POLICIES["thorough"], world).turns
|
||||
assert thorough <= MAX_TURNS - 1
|
||||
assert run_policy(POLICIES["thorough_wasteful"], world).turns == thorough + 1
|
||||
|
||||
|
||||
def test_reference_exists_and_the_investigators_catch_everything_on_every_seed() -> None:
|
||||
"""Lens 0: `reference_minutes` is never null. Lens 1/2: caught 1.0 over 4096, not 24."""
|
||||
for seed in ALL:
|
||||
world = generate(seed)
|
||||
minutes, policy = reference_for(world)
|
||||
assert minutes is not None and policy is not None, seed
|
||||
for name in ("targeted", "thorough"):
|
||||
assert run_policy(POLICIES[name], world).outcome == "solved", (seed, name)
|
||||
|
||||
|
||||
def test_fast_misses_exactly_the_hidden_tier() -> None:
|
||||
for seed in range(512):
|
||||
world = world_for_seed(seed)
|
||||
outcome = run_policy(POLICIES["fast"], world).outcome
|
||||
assert (outcome == "solved") == (world["tier"] != "hidden"), seed
|
||||
|
||||
|
||||
def test_reference_is_fast_where_the_screen_suffices() -> None:
|
||||
for seed in range(512):
|
||||
world = world_for_seed(seed)
|
||||
assert world["reference_policy"] == ("targeted" if world["tier"] == "hidden" else "fast"), seed
|
||||
|
||||
|
||||
def test_policies_cannot_see_the_answer() -> None:
|
||||
"""Run every policy against a view whose world has the hidden fields deleted: identical replies."""
|
||||
world = world_for_seed(9)
|
||||
stripped = copy.deepcopy(world)
|
||||
for key in ("label", "typology", "planted", "overlay", "tier", "template", "reference_minutes", "reference_policy"):
|
||||
stripped[key] = None if key != "planted" else []
|
||||
for name, policy in POLICIES.items():
|
||||
a, b = Engine(world), Engine(stripped)
|
||||
while not a.done:
|
||||
ra, rb = policy(a.view()), policy(b.view())
|
||||
assert ra == rb, name
|
||||
a.step(ra)
|
||||
b.step(rb)
|
||||
@@ -0,0 +1,48 @@
|
||||
"""The pinned parse rules — the ones two runtimes disagree on."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from alert_triage.protocol import extract_candidate, parse_reply
|
||||
|
||||
FENCE = "```"
|
||||
|
||||
|
||||
def test_fence_wins_over_an_earlier_bare_brace() -> None:
|
||||
text = 'first {not it} then ' + FENCE + 'json\n{"action":"lookup","what":"documents"}\n' + FENCE
|
||||
assert extract_candidate(text) == '{"action":"lookup","what":"documents"}\n'
|
||||
obj, reason = parse_reply(text)
|
||||
assert obj == {"action": "lookup", "what": "documents"} and reason is None
|
||||
|
||||
|
||||
def test_first_balanced_span_is_string_aware() -> None:
|
||||
text = 'ok {"note":"a } inside \\" quotes","action":"close"} trailing {"x":1}'
|
||||
obj, _ = parse_reply(text)
|
||||
assert obj == {"note": 'a } inside " quotes', "action": "close"}
|
||||
|
||||
|
||||
def test_invalid_first_span_is_rejected_without_further_scanning() -> None:
|
||||
obj, reason = parse_reply('{"a":1,} {"action":"lookup","what":"documents"}')
|
||||
assert obj is None and reason
|
||||
|
||||
|
||||
def test_non_object_json_is_rejected() -> None:
|
||||
assert parse_reply("[1,2]")[0] is None
|
||||
assert parse_reply(FENCE + "json\n[{\"action\":\"lookup\"}]\n" + FENCE)[0] is None
|
||||
assert parse_reply("42")[0] is None
|
||||
|
||||
|
||||
def test_nan_and_infinity_are_rejected_like_json_parse_does() -> None:
|
||||
assert parse_reply('{"x":NaN}')[0] is None
|
||||
assert parse_reply('{"x":Infinity}')[0] is None
|
||||
|
||||
|
||||
def test_unbalanced_or_absent_braces() -> None:
|
||||
assert parse_reply('{"action":"lookup"')[0] is None
|
||||
assert parse_reply("no braces at all")[0] is None
|
||||
assert parse_reply("")[0] is None
|
||||
assert parse_reply(None)[0] is None
|
||||
|
||||
|
||||
def test_nested_object_is_the_outer_one() -> None:
|
||||
obj, _ = parse_reply('{"outer":{"action":"lookup"}}')
|
||||
assert obj == {"outer": {"action": "lookup"}}
|
||||
@@ -0,0 +1,94 @@
|
||||
"""Worked examples of the reward, and the marker region the page quotes."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from alert_triage.reward import (
|
||||
FALSE_ESCALATION_CREDIT,
|
||||
WEIGHTS,
|
||||
Episode,
|
||||
escalate_break_even,
|
||||
metrics,
|
||||
score,
|
||||
score_exact,
|
||||
total,
|
||||
)
|
||||
|
||||
|
||||
def ep(**kw) -> Episode:
|
||||
base = dict(label="benign", planted=[["kyc.a"]], disposition="close", cites=["kyc.a"], minutes_spent=33,
|
||||
reference_minutes=33, turns_spent=1, rejected=0)
|
||||
base.update(kw)
|
||||
return Episode(**base)
|
||||
|
||||
|
||||
def test_weights_sum_to_one_and_have_a_counterweight() -> None:
|
||||
assert abs(sum(WEIGHTS.values()) - 1.0) < 1e-12
|
||||
from alert_triage.reward import ROLES
|
||||
assert "counterweight" in ROLES.values()
|
||||
|
||||
|
||||
def test_perfect_close() -> None:
|
||||
assert score(ep()) == {"caught": 1.0, "hours": 1.0, "evidence": 1.0}
|
||||
assert abs(total(ep()) - 1.0) < 1e-12
|
||||
|
||||
|
||||
def test_a_miss_scores_exactly_zero() -> None:
|
||||
e = ep(label="suspicious", planted=[["T-1", "T-2"]], disposition="close", cites=["kyc.a"])
|
||||
assert score(e) == {"caught": 0.0, "hours": 0.0, "evidence": 0.0}
|
||||
assert total(e) == 0.0
|
||||
|
||||
|
||||
def test_a_false_escalation_takes_half_the_counterweight_and_nothing_else() -> None:
|
||||
e = ep(disposition="escalate", cites=["kyc.a"])
|
||||
assert score(e) == {"caught": FALSE_ESCALATION_CREDIT, "hours": 0.0, "evidence": 0.0}
|
||||
|
||||
|
||||
def test_no_disposition_is_zero_and_truncated_is_none() -> None:
|
||||
assert score(ep(disposition=None, cites=[])) == {"caught": 0.0, "hours": 0.0, "evidence": 0.0}
|
||||
assert score(ep(truncated=True)) == {"caught": None, "hours": None, "evidence": None}
|
||||
assert total(ep(truncated=True)) is None
|
||||
|
||||
|
||||
def test_hours_is_capped_and_ratio_against_the_reference() -> None:
|
||||
assert score(ep(minutes_spent=20))["hours"] == 1.0
|
||||
assert abs(score(ep(minutes_spent=66))["hours"] - 0.5) < 1e-12
|
||||
assert score(ep(reference_minutes=None))["hours"] is None
|
||||
assert total(ep(reference_minutes=None)) is None
|
||||
|
||||
|
||||
def test_evidence_is_f1_over_the_best_alternate() -> None:
|
||||
e = ep(planted=[["a", "b"], ["a"]], cites=["a"])
|
||||
assert score(e)["evidence"] == 1.0
|
||||
e = ep(planted=[["a", "b"]], cites=["a", "x", "y"])
|
||||
assert abs(score(e)["evidence"] - 0.4) < 1e-12 # 2·1 / (3+2)
|
||||
e = ep(planted=[["a"]], cites=["a", "b", "c", "d", "e", "f", "g", "h"])
|
||||
assert abs(score(e)["evidence"] - 2 / 9) < 1e-12 # one planted among eight cited
|
||||
assert score_exact(e)["evidence"] == [2, 9]
|
||||
|
||||
|
||||
def test_exact_and_float_scores_agree() -> None:
|
||||
for e in (ep(), ep(minutes_spent=66), ep(planted=[["a", "b"]], cites=["a", "x", "y"]), ep(disposition="escalate")):
|
||||
exact = score_exact(e)
|
||||
approx = score(e)
|
||||
for k in approx:
|
||||
assert abs(exact[k][0] / exact[k][1] - approx[k]) < 1e-12
|
||||
|
||||
|
||||
def test_break_even_is_between_zero_and_one() -> None:
|
||||
assert 0.3 < escalate_break_even() < 0.7
|
||||
|
||||
|
||||
def test_metrics_are_diagnostics() -> None:
|
||||
m = metrics(ep(disposition="escalate", typology="UNKNOWN", true_typology="UNKNOWN", label="suspicious"))
|
||||
assert m["typology_match"] == 1.0 and m["false_escalation"] == 0.0
|
||||
assert metrics(ep())["typology_match"] is None
|
||||
|
||||
|
||||
def test_exactly_one_marker_pair() -> None:
|
||||
text = (Path(__file__).parent.parent / "alert_triage" / "reward.py").read_text()
|
||||
assert text.count("# region: pig-demo/reward") == 1
|
||||
assert text.count("# endregion: pig-demo/reward") == 1
|
||||
@@ -0,0 +1,52 @@
|
||||
"""The portable primitives, pinned so the TypeScript port has vectors to hit."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from alert_triage.rng import (
|
||||
XorShift32,
|
||||
civil_from_days,
|
||||
days_from_civil,
|
||||
days_in_month,
|
||||
fnv1a32,
|
||||
iso_date,
|
||||
)
|
||||
|
||||
# fnv1a32 over the decimal seed — identical to wordle_five's and to engine.ts.
|
||||
FNV_VECTORS = {"0": 0x350CA8AF, "1": 0x340CA71C, "42": 0x87E38583, "4095": 0x43875F5F}
|
||||
|
||||
|
||||
def test_fnv1a_vectors() -> None:
|
||||
for text, expect in FNV_VECTORS.items():
|
||||
assert fnv1a32(text) == expect, text
|
||||
|
||||
|
||||
def test_xorshift_sequence_is_pinned() -> None:
|
||||
rng = XorShift32(fnv1a32("0"))
|
||||
first = [rng.next() for _ in range(5)]
|
||||
assert first == XORSHIFT_FROM_SEED_0
|
||||
|
||||
|
||||
XORSHIFT_FROM_SEED_0 = [2738490563, 3068243922, 3765331391, 3085691315, 2439018365]
|
||||
|
||||
|
||||
def test_zero_seed_is_replaced() -> None:
|
||||
assert XorShift32(0).state != 0
|
||||
|
||||
|
||||
def test_sample_is_distinct_and_partial_shuffle() -> None:
|
||||
rng = XorShift32(7)
|
||||
out = rng.sample(list(range(10)), 4)
|
||||
assert len(out) == 4 and len(set(out)) == 4
|
||||
|
||||
|
||||
def test_civil_dates_round_trip() -> None:
|
||||
for day in range(days_from_civil(1999, 12, 25), days_from_civil(2030, 3, 2)):
|
||||
y, m, d = civil_from_days(day)
|
||||
assert days_from_civil(y, m, d) == day
|
||||
|
||||
|
||||
def test_known_dates() -> None:
|
||||
assert days_from_civil(1970, 1, 1) == 0
|
||||
assert iso_date(days_from_civil(2026, 3, 1)) == "2026-03-01"
|
||||
assert days_in_month(2024, 2) == 29 and days_in_month(2026, 2) == 28
|
||||
assert days_in_month(2026, 12) == 31
|
||||
@@ -0,0 +1,87 @@
|
||||
"""The hidden tier is hidden only if the generator keeps it so. This is the gate.
|
||||
|
||||
Two checks. The first is structural and exact: for every seed the free screen
|
||||
is byte-identical with the overlay on or off, for every overlay the screen can
|
||||
carry. The second is distributional: across seeds, the screen features a
|
||||
policy could key on do not separate hidden from benign beyond noise. The
|
||||
generator draws the tier BEFORE the template and the screen, so the two
|
||||
populations are the same draw; this test is what would catch a regression that
|
||||
made them differ.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
|
||||
import pytest
|
||||
|
||||
from alert_triage.generator import canonical_json, compatible_overlays, generate, screen_of
|
||||
|
||||
ALL = range(4096)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def pairs():
|
||||
out = []
|
||||
for s in ALL:
|
||||
base = generate(s, overlay=None)
|
||||
if base["template"].startswith("S"):
|
||||
continue
|
||||
out.append((s, base))
|
||||
return out
|
||||
|
||||
|
||||
def test_overlay_never_touches_the_screen(pairs) -> None:
|
||||
for seed, base in pairs:
|
||||
before = canonical_json(screen_of(base))
|
||||
for kind in compatible_overlays(base):
|
||||
overlaid = generate(seed, overlay=kind)
|
||||
assert overlaid["label"] == "suspicious" and overlaid["overlay"] == kind
|
||||
assert canonical_json(screen_of(overlaid)) == before, (seed, kind)
|
||||
|
||||
|
||||
def test_the_seed_overlay_is_one_of_the_compatible_kinds(pairs) -> None:
|
||||
for seed, base in pairs:
|
||||
w = generate(seed)
|
||||
if w["overlay"]:
|
||||
assert w["overlay"] in compatible_overlays(base)
|
||||
|
||||
|
||||
def _features(w) -> list[float]:
|
||||
c = w["customer"]
|
||||
screen = screen_of(w)
|
||||
trig = screen["triggering_transactions"]
|
||||
amounts = [t["amount"] for t in trig]
|
||||
return [
|
||||
float(c["prior_alerts"]),
|
||||
float(c["prior_sars"]),
|
||||
float(c["pep"]),
|
||||
float(sum(1 for cp in screen["screen_counterparties"] if cp["direction"] == "out")),
|
||||
float(sum(1 for a in amounts if a % 100000 == 0)) / max(1, len(amounts)),
|
||||
float(len(trig)),
|
||||
float(len({t["branch"] for t in trig if t["branch"]})),
|
||||
float(sum(r["wires_out"] > 0 for r in w["summary"])),
|
||||
float(screen["alert"]["rule"] == "R-STR-01"),
|
||||
float(c["expected_monthly_cash"] > 0),
|
||||
]
|
||||
|
||||
|
||||
def test_screen_features_do_not_separate_hidden_from_benign() -> None:
|
||||
"""Per-feature z-test between the two populations; nothing beyond 4 sigma."""
|
||||
benign, hidden = [], []
|
||||
for s in ALL:
|
||||
w = generate(s)
|
||||
if w["tier"] == "benign":
|
||||
benign.append(_features(w))
|
||||
elif w["tier"] == "hidden":
|
||||
hidden.append(_features(w))
|
||||
assert len(hidden) > 400
|
||||
for j in range(len(benign[0])):
|
||||
b = [f[j] for f in benign]
|
||||
h = [f[j] for f in hidden]
|
||||
mb, mh = sum(b) / len(b), sum(h) / len(h)
|
||||
vb = sum((x - mb) ** 2 for x in b) / max(1, len(b) - 1)
|
||||
vh = sum((x - mh) ** 2 for x in h) / max(1, len(h) - 1)
|
||||
se = math.sqrt(vb / len(b) + vh / len(h)) or 1e-9
|
||||
z = abs(mb - mh) / se
|
||||
assert z < 4.0, f"feature {j}: benign {mb:.3f} vs hidden {mh:.3f}, z={z:.1f}"
|
||||
+13
-4
@@ -13,11 +13,18 @@ from pathlib import Path
|
||||
|
||||
TRACES = Path(__file__).parent.parent / "public" / "traces"
|
||||
|
||||
# Arm ids are shared across tasksets where they mean the same thing (the two
|
||||
# model arms); the generated arms are per-taskset and simply do not collide.
|
||||
ARMS = {
|
||||
"base-off": ("Out of the box", "recorded"),
|
||||
"base-on": ("Allowed to think", "intervened"),
|
||||
# wordle
|
||||
"solver": ("Best-known play", "generated"),
|
||||
"cautious": ("Never wastes a guess", "generated"),
|
||||
# alert-triage: the shipped scripted analysts from alert_triage/policies.py
|
||||
"fast": ("Reads the screen", "generated"),
|
||||
"targeted": ("Checks the hidden tells", "generated"),
|
||||
"thorough": ("Runs the full procedure", "generated"),
|
||||
}
|
||||
|
||||
# What was done to the run, for arms that had something done to them. Required
|
||||
@@ -27,7 +34,7 @@ INTERVENTIONS = {
|
||||
"base-on": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning.",
|
||||
}
|
||||
|
||||
ORDER = ["base-off", "base-on", "solver", "cautious"]
|
||||
ORDER = ["base-off", "base-on", "solver", "cautious", "fast", "targeted", "thorough"]
|
||||
|
||||
|
||||
def main() -> int:
|
||||
@@ -67,11 +74,13 @@ def main() -> int:
|
||||
by_arm.setdefault(r["id"].rsplit("-s", 1)[0], []).append(r)
|
||||
print(f"{slug}: {len(runs)} runs")
|
||||
for arm, group in by_arm.items():
|
||||
solved = 0
|
||||
outcomes: dict[str, int] = {}
|
||||
for r in group:
|
||||
data = json.loads((TRACES.parent / r["path"].lstrip("/")).read_text())
|
||||
solved += 1 if data["outcome"] == "solved" else 0
|
||||
print(f" {arm:<10} {len(group)} runs, solved {solved}/{len(group)}")
|
||||
outcomes[data["outcome"]] = outcomes.get(data["outcome"], 0) + 1
|
||||
solved = outcomes.get("solved", 0)
|
||||
rest = ", ".join(f"{k} {v}" for k, v in sorted(outcomes.items()) if k != "solved")
|
||||
print(f" {arm:<10} {len(group)} runs, solved {solved}/{len(group)}" + (f" ({rest})" if rest else ""))
|
||||
return 0
|
||||
|
||||
|
||||
|
||||
+158
-20
@@ -12,6 +12,13 @@ number it shows from the recorded moves, and says so. See verify.ts.
|
||||
Usage:
|
||||
uv run python envs/capture.py --arm base-off --seeds 0-7
|
||||
uv run python envs/capture.py --arm solver --seeds 0-7 # no model needed
|
||||
uv run python envs/capture.py --taskset alert-triage --arm base-on --seeds 1-7,9
|
||||
uv run python envs/capture.py --taskset alert-triage --arm targeted --seeds 1-7,9
|
||||
|
||||
spark-1 serves one model, single-stream: run the model arms one after another,
|
||||
never concurrently. A thinking arm on alert-triage takes minutes per seed —
|
||||
run it in the background with stdout redirected to a file and poll the
|
||||
fixtures being written rather than the log.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -27,9 +34,14 @@ import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent / "wordle_five"))
|
||||
sys.path.insert(0, str(Path(__file__).parent / "alert_triage"))
|
||||
|
||||
import numpy as np # noqa: E402
|
||||
|
||||
from alert_triage import policies as AT # noqa: E402
|
||||
from alert_triage import taskset as AT_taskset # noqa: E402
|
||||
from alert_triage.generator import is_held_out, world_for_seed # noqa: E402
|
||||
|
||||
from wordle_five import solver as S # noqa: E402
|
||||
from wordle_five.engine import MAX_GUESSES, Game, answers # noqa: E402
|
||||
from wordle_five.protocol import ( # noqa: E402
|
||||
@@ -40,7 +52,8 @@ from wordle_five.protocol import ( # noqa: E402
|
||||
)
|
||||
from wordle_five.reward import Episode, metrics, score # noqa: E402
|
||||
|
||||
OUT = Path(__file__).parent.parent / "public" / "traces" / "wordle"
|
||||
TRACES = Path(__file__).parent.parent / "public" / "traces"
|
||||
OUT = TRACES / "wordle"
|
||||
ENDPOINT = os.environ.get("PIG_DEMO_INFERENCE", "http://100.127.247.67:8001/v1/chat/completions")
|
||||
MODEL = os.environ.get("PIG_DEMO_MODEL", "brain-qwen38-dspark")
|
||||
|
||||
@@ -52,7 +65,7 @@ ARMS = {
|
||||
}
|
||||
|
||||
|
||||
def call_model(messages: list[dict], thinking: bool) -> dict:
|
||||
def call_model(messages: list[dict], thinking: bool, max_tokens: int | None = None, timeout: int = 300) -> dict:
|
||||
"""One completion. Returns reply, reasoning and the real call metrics.
|
||||
|
||||
Never raises on an upstream failure — a dropped call becomes a turn with a
|
||||
@@ -64,7 +77,7 @@ def call_model(messages: list[dict], thinking: bool) -> dict:
|
||||
"model": MODEL,
|
||||
"messages": messages,
|
||||
"temperature": 0.7,
|
||||
"max_tokens": 2048 if thinking else 512,
|
||||
"max_tokens": max_tokens or (2048 if thinking else 512),
|
||||
"chat_template_kwargs": {"enable_thinking": bool(thinking)},
|
||||
}
|
||||
request = urllib.request.Request(
|
||||
@@ -74,7 +87,7 @@ def call_model(messages: list[dict], thinking: bool) -> dict:
|
||||
)
|
||||
started = time.time()
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=300) as response:
|
||||
with urllib.request.urlopen(request, timeout=timeout) as response:
|
||||
payload = json.loads(response.read())
|
||||
except (urllib.error.URLError, TimeoutError, OSError) as exc:
|
||||
return {
|
||||
@@ -245,33 +258,158 @@ def capture(arm: str, seed: int) -> dict:
|
||||
}
|
||||
|
||||
|
||||
# ------------------------------------------------------------ alert-triage --
|
||||
#
|
||||
# The same fixture shape as wordle — runId, seed, model, capturedAt, rewards,
|
||||
# metrics, truncated, outcome, turns[{reply, reasoning, call, info}] — driven
|
||||
# through `alert_triage.taskset.play_episode`, the loop the probe and the
|
||||
# tests share. The browser regenerates the world from the seed and replays the
|
||||
# reply strings; nothing else in the fixture is trusted by the page.
|
||||
|
||||
AT_ARMS = {
|
||||
"base-off": {"label": "Out of the box", "thinking": False, "max_tokens": 1024, "timeout": 300},
|
||||
"base-on": {"label": "Allowed to think", "thinking": True, "max_tokens": 4096, "timeout": 900},
|
||||
"fast": {"label": "Reads the screen", "thinking": None, "policy": "fast"},
|
||||
"targeted": {"label": "Checks the hidden tells", "thinking": None, "policy": "targeted"},
|
||||
"thorough": {"label": "Runs the full procedure", "thinking": None, "policy": "thorough"},
|
||||
}
|
||||
AT_MODEL_NAME = {"fast": "fast-analyst", "targeted": "targeted-analyst", "thorough": "thorough-analyst"}
|
||||
|
||||
# A thinking budget the model exhausts is recorded as finishReason "length"
|
||||
# with whatever content survived (usually none, which the engine rejects).
|
||||
# That is a thing the model did under the budget it was given, not a capture
|
||||
# error, and the fixture says so rather than retrying until it looks better.
|
||||
|
||||
|
||||
def _messages_from(prompt: str, transcript: list[dict]) -> list[dict]:
|
||||
"""The chat a model sees: the system prompt, the screen, then each reply and what it got back."""
|
||||
messages = [{"role": "system", "content": prompt}, {"role": "user", "content": transcript[0]["observation"]}]
|
||||
for entry in transcript[1:]:
|
||||
messages.append({"role": "assistant", "content": entry["reply"] or ""})
|
||||
messages.append({"role": "user", "content": entry["observation"]})
|
||||
return messages
|
||||
|
||||
|
||||
def _generated_call() -> dict:
|
||||
return {"promptTokens": None, "completionTokens": None, "reasoningTokens": None, "durationMs": None, "finishReason": "generated"}
|
||||
|
||||
|
||||
def capture_alert_triage(arm: str, seed: int) -> dict:
|
||||
config = AT_ARMS[arm]
|
||||
calls: list[dict] = []
|
||||
|
||||
if config["thinking"] is None:
|
||||
policy = AT.POLICIES[config["policy"]]
|
||||
|
||||
def respond(prompt: str, transcript: list[dict], view: dict) -> str | None:
|
||||
reply = policy(view)
|
||||
calls.append({"reply": reply, "reasoning": None, "call": _generated_call()})
|
||||
return reply
|
||||
else:
|
||||
def respond(prompt: str, transcript: list[dict], view: dict) -> str | None:
|
||||
result = call_model(_messages_from(prompt, transcript), bool(config["thinking"]),
|
||||
max_tokens=config["max_tokens"], timeout=config["timeout"])
|
||||
calls.append(result)
|
||||
return result["reply"]
|
||||
|
||||
world = world_for_seed(seed)
|
||||
played = AT_taskset.play_episode(seed, respond, world)
|
||||
steps = played["transcript"][1:]
|
||||
assert len(steps) == len(calls), "one model call per engine step"
|
||||
|
||||
turns = [
|
||||
{
|
||||
**call,
|
||||
"info": {"action": step["action"], "rejection": step["rejection"], "observation": step["observation"]},
|
||||
}
|
||||
for call, step in zip(calls, steps)
|
||||
]
|
||||
return {
|
||||
"runId": f"{arm}-s{seed}",
|
||||
"seed": seed,
|
||||
"model": AT_MODEL_NAME.get(arm, MODEL),
|
||||
"capturedAt": time.strftime("%Y-%m-%d"),
|
||||
"rewards": played["rewards"],
|
||||
"metrics": played["metrics"],
|
||||
"truncated": played["truncated"],
|
||||
"outcome": played["outcome"],
|
||||
"info": {**played["info"], "tier": world["tier"], "screen": played["transcript"][0]["observation"]},
|
||||
"turns": turns,
|
||||
}
|
||||
|
||||
|
||||
def _at_summary(episode: dict) -> str:
|
||||
r = episode["rewards"]
|
||||
fmt = lambda v: " -- " if v is None else f"{v:.2f}" # noqa: E731
|
||||
actions = []
|
||||
for t in episode["turns"]:
|
||||
a = t["info"]["action"]
|
||||
if a is None:
|
||||
actions.append("REJ")
|
||||
elif a["action"] == "lookup":
|
||||
actions.append(a.get("month") or a.get("id") or a["what"])
|
||||
else:
|
||||
actions.append(a["action"].upper())
|
||||
return (
|
||||
f"{episode['info']['tier']:<8}{episode['info']['template']:<3} {episode['outcome']:<8}"
|
||||
f" caught={fmt(r['caught'])} hours={fmt(r['hours'])} evid={fmt(r['evidence'])}"
|
||||
f" {episode['metrics']['hours_spent']:.2f}h {actions}"
|
||||
)
|
||||
|
||||
|
||||
TASKSETS = {
|
||||
"wordle-five": {"arms": ARMS, "out": TRACES / "wordle", "capture": capture},
|
||||
"alert-triage": {"arms": AT_ARMS, "out": TRACES / "alert-triage", "capture": capture_alert_triage},
|
||||
}
|
||||
|
||||
|
||||
def parse_seeds(spec: str) -> list[int]:
|
||||
if "-" in spec:
|
||||
lo, hi = spec.split("-")
|
||||
return list(range(int(lo), int(hi) + 1))
|
||||
return [int(s) for s in spec.split(",")]
|
||||
""""0-7", "1,3", or a mix: "1-7,9"."""
|
||||
seeds: list[int] = []
|
||||
for part in spec.split(","):
|
||||
if "-" in part:
|
||||
lo, hi = part.split("-")
|
||||
seeds.extend(range(int(lo), int(hi) + 1))
|
||||
else:
|
||||
seeds.append(int(part))
|
||||
return seeds
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--arm", required=True, choices=sorted(ARMS))
|
||||
parser.add_argument("--taskset", default="wordle-five", choices=sorted(TASKSETS))
|
||||
parser.add_argument("--arm", required=True)
|
||||
parser.add_argument("--seeds", default="0-7")
|
||||
args = parser.parse_args()
|
||||
|
||||
OUT.mkdir(parents=True, exist_ok=True)
|
||||
taskset = TASKSETS[args.taskset]
|
||||
if args.arm not in taskset["arms"]:
|
||||
parser.error(f"--arm must be one of {sorted(taskset['arms'])} for {args.taskset}")
|
||||
|
||||
out: Path = taskset["out"]
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
for seed in parse_seeds(args.seeds):
|
||||
if args.taskset == "alert-triage" and is_held_out(seed):
|
||||
# The held-out bucket is never captured, probed or trained on. A
|
||||
# fixture for one would put a held-out alert behind a permalink.
|
||||
print(f"{args.arm}-s{seed}: seed {seed} is held out — skipped", flush=True)
|
||||
continue
|
||||
started = time.time()
|
||||
episode = capture(args.arm, seed)
|
||||
path = OUT / f"{episode['runId']}.json"
|
||||
episode = taskset["capture"](args.arm, seed)
|
||||
path = out / f"{episode['runId']}.json"
|
||||
path.write_text(json.dumps(episode, indent=2) + "\n")
|
||||
guesses = [t["info"]["guess"] for t in episode["turns"]]
|
||||
print(
|
||||
f"{episode['runId']:>16} {episode['answer']} {episode['outcome']:<7}"
|
||||
f" solved={episode['rewards']['solved']:.0f}"
|
||||
f" econ={episode['rewards']['economy']:.2f}"
|
||||
f" cons={episode['rewards']['consistency']:.2f}"
|
||||
f" {time.time()-started:5.1f}s {guesses}"
|
||||
)
|
||||
if args.taskset == "wordle-five":
|
||||
guesses = [t["info"]["guess"] for t in episode["turns"]]
|
||||
print(
|
||||
f"{episode['runId']:>16} {episode['answer']} {episode['outcome']:<7}"
|
||||
f" solved={episode['rewards']['solved']:.0f}"
|
||||
f" econ={episode['rewards']['economy']:.2f}"
|
||||
f" cons={episode['rewards']['consistency']:.2f}"
|
||||
f" {time.time()-started:5.1f}s {guesses}",
|
||||
flush=True,
|
||||
)
|
||||
else:
|
||||
print(f"{episode['runId']:>16} {_at_summary(episode)} {time.time()-started:6.1f}s", flush=True)
|
||||
return 0
|
||||
|
||||
|
||||
|
||||
+31
-4
@@ -10,8 +10,9 @@ A blended float hides the failure this is built to catch: a component that is
|
||||
flat across every policy is measuring nothing, and it will still move the total
|
||||
because the other components move.
|
||||
|
||||
Run: uv run python envs/probe.py
|
||||
Exits non-zero on any violated assertion.
|
||||
Run: uv run python envs/probe.py [--taskset wordle-five|alert-triage]
|
||||
Runs every registered taskset by default. Exits non-zero on any violated
|
||||
assertion.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -20,6 +21,7 @@ import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent / "wordle_five"))
|
||||
sys.path.insert(0, str(Path(__file__).parent / "alert_triage"))
|
||||
|
||||
from wordle_five import solver as S # noqa: E402
|
||||
from wordle_five.engine import ( # noqa: E402
|
||||
@@ -32,6 +34,8 @@ from wordle_five.engine import ( # noqa: E402
|
||||
from wordle_five.protocol import parse_guess # noqa: E402
|
||||
from wordle_five.reward import ROLES, WEIGHTS, Episode, metrics, score, total # noqa: E402
|
||||
|
||||
from alert_triage import ladder as alert_triage_ladder # noqa: E402
|
||||
|
||||
SEEDS = list(range(16))
|
||||
COMPONENTS = ("solved", "economy", "consistency")
|
||||
|
||||
@@ -162,7 +166,7 @@ def play(policy, seed: int) -> Episode:
|
||||
)
|
||||
|
||||
|
||||
def run() -> int:
|
||||
def run_wordle() -> int:
|
||||
print(f"Probing wordle-five over {len(SEEDS)} seeds\n")
|
||||
results: dict[str, dict[str, float]] = {}
|
||||
extra: dict[str, dict[str, float]] = {}
|
||||
@@ -243,5 +247,28 @@ def run() -> int:
|
||||
return 0
|
||||
|
||||
|
||||
# Each taskset registers its ladder here. The alert-triage rungs live in
|
||||
# envs/alert_triage/alert_triage/ladder.py beside the policies they probe.
|
||||
TASKSETS = {
|
||||
"wordle-five": run_wordle,
|
||||
"alert-triage": alert_triage_ladder.run,
|
||||
}
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
import argparse
|
||||
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--taskset", choices=sorted(TASKSETS), action="append")
|
||||
args = parser.parse_args(argv)
|
||||
chosen = args.taskset or list(TASKSETS)
|
||||
status = 0
|
||||
for i, name in enumerate(chosen):
|
||||
if i:
|
||||
print("\n" + "=" * 78 + "\n")
|
||||
status |= TASKSETS[name]()
|
||||
return status
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(run())
|
||||
raise SystemExit(main(sys.argv[1:]))
|
||||
|
||||
+45
-8
@@ -16,17 +16,20 @@ import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent / "wordle_five"))
|
||||
sys.path.insert(0, str(Path(__file__).parent / "alert_triage"))
|
||||
|
||||
from alert_triage import taskset as AT # noqa: E402
|
||||
from alert_triage.generator import is_held_out # noqa: E402
|
||||
from wordle_five import solver as S # noqa: E402
|
||||
from wordle_five.engine import Game # noqa: E402
|
||||
from wordle_five.protocol import parse_guess # noqa: E402
|
||||
from wordle_five.reward import Episode, score # noqa: E402
|
||||
|
||||
TRACES = Path(__file__).parent.parent / "public" / "traces" / "wordle"
|
||||
TRACES = Path(__file__).parent.parent / "public" / "traces"
|
||||
TOLERANCE = 1e-9
|
||||
|
||||
|
||||
def replay(fixture: dict) -> dict[str, float]:
|
||||
def replay_wordle(fixture: dict) -> dict[str, float]:
|
||||
game = Game(seed=fixture["seed"])
|
||||
if game.answer != fixture["answer"]:
|
||||
raise AssertionError(
|
||||
@@ -49,27 +52,61 @@ def replay(fixture: dict) -> dict[str, float]:
|
||||
return score(episode)
|
||||
|
||||
|
||||
def replay_alert_triage(fixture: dict) -> dict[str, float | None]:
|
||||
"""The world comes from the seed, the score from the reply strings. Nothing
|
||||
else in the fixture is trusted — the same rule the browser's verify uses."""
|
||||
seed = fixture["seed"]
|
||||
if is_held_out(seed):
|
||||
raise AssertionError(f"seed {seed} is in the held-out bucket and must never be captured")
|
||||
played = AT.replay(seed, [turn["reply"] for turn in fixture["turns"]])
|
||||
if played["outcome"] != fixture["outcome"]:
|
||||
raise AssertionError(f"replay gives outcome {played['outcome']!r}, fixture claims {fixture['outcome']!r}")
|
||||
return played["rewards"]
|
||||
|
||||
|
||||
REPLAY = {"wordle": replay_wordle, "alert-triage": replay_alert_triage}
|
||||
|
||||
|
||||
def _delta(recomputed, recorded) -> float:
|
||||
"""None is "not scored". It matches only None; against a number it is a mismatch, not a zero."""
|
||||
if recomputed is None or recorded is None:
|
||||
return 0.0 if recomputed is recorded else float("inf")
|
||||
return abs(recomputed - recorded)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
fixtures = sorted(TRACES.glob("*.json"))
|
||||
fixtures = [f for f in fixtures if f.name != "manifest.json"]
|
||||
fixtures = [
|
||||
f for demo in sorted(p for p in TRACES.iterdir() if p.is_dir())
|
||||
for f in sorted(demo.glob("*.json"))
|
||||
]
|
||||
if not fixtures:
|
||||
print("no fixtures found — nothing to verify")
|
||||
return 0
|
||||
|
||||
failures = 0
|
||||
for path in fixtures:
|
||||
demo = path.parent.name
|
||||
if demo not in REPLAY:
|
||||
failures += 1
|
||||
print(f"{path.parent.name}/{path.name:<20} NO REPLAYER for demo '{demo}'")
|
||||
continue
|
||||
data = json.loads(path.read_text())
|
||||
recomputed = replay(data)
|
||||
try:
|
||||
recomputed = REPLAY[demo](data)
|
||||
except AssertionError as exc:
|
||||
failures += 1
|
||||
print(f"{path.parent.name}/{path.name:<20} MISMATCH {exc}")
|
||||
continue
|
||||
recorded = data["rewards"]
|
||||
deltas = {k: abs(recomputed[k] - recorded[k]) for k in recorded}
|
||||
deltas = {k: _delta(recomputed[k], recorded[k]) for k in recorded}
|
||||
worst = max(deltas.values())
|
||||
status = "ok" if worst <= TOLERANCE else "MISMATCH"
|
||||
if worst > TOLERANCE:
|
||||
failures += 1
|
||||
culprit = max(deltas, key=deltas.get)
|
||||
print(f"{path.name:>20} {status} worst delta {worst:.3g} on '{culprit}'")
|
||||
print(f"{path.parent.name}/{path.name:<20} {status} worst delta {worst:.3g} on '{culprit}'")
|
||||
else:
|
||||
print(f"{path.name:>20} {status} delta {worst:.1e}")
|
||||
print(f"{path.parent.name}/{path.name:<20} {status} delta {worst:.1e}")
|
||||
|
||||
print(f"\n{len(fixtures)} fixtures, {failures} mismatched")
|
||||
return 1 if failures else 0
|
||||
|
||||
Reference in New Issue
Block a user