karti-ai
2dfa96939e
The first environment shipped through .claude/workflows/new-environment.js: specification, three adversarial reviews (all 'fixable', none fatal), the Python environment, the TypeScript port, captured rollouts, and the demo page. Eleven agents, no errors. The proof that the platform scales is one line long. Alert Triage has a completely different shape from Word Five — JSON actions, priced lookups, an analyst screen instead of a grid — and the only change under src/components/demo/ is a comment edit, because the isolation lint refused the word "wordle" there. Zero shell code changed. 415 contract checks now pass against two demos, up from 206 against one. The environment is honest by construction. Every alert is synthetic, generated from the seed, and the banner saying so sits inside the board surface. Two of the eleven scenario templates are hidden-suspicious: generated by the same code as their benign twin with the signal overlaid only in lookup data, so the free screen is identically distributed and a screen-only policy STRUCTURALLY cannot tell them apart. The probe ladder measures it: `fast` catches 0.0 of hidden seeds. That is the counterweight made real rather than asserted. Twelve policies, thirteen ladder assertions, a genuine three-way trade: fast 0.846 wins hours (0.85), misses every hidden case targeted 0.894 wins the shipped total thorough 0.820 wins evidence (1.00), spends 2.9 hours None dominates. 92 Python tests, 35 TypeScript tests, 65 fixtures replaying at delta 0, and conformance gated on world + scorer + protocol so the browser shows the same alert for ?seed= that Python generated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019mt6sHQHEnEYrJZvoMCJSB
75 lines
3.4 KiB
TypeScript
75 lines
3.4 KiB
TypeScript
import type { RewardSpec } from '@/lib/demo-kit';
|
||
import rewardSource from '../../../envs/alert_triage/alert_triage/reward.py?raw';
|
||
|
||
/**
|
||
* The reward, mirrored from `envs/alert_triage/alert_triage/reward.py`.
|
||
*
|
||
* The labels are for the BSA officer, not the variable names. `caught` is a
|
||
* fair identifier and says nothing to someone deciding a budget; "nothing
|
||
* slipped through" is the same quantity in the words they already use.
|
||
*
|
||
* The constants are restated here as literals rather than imported, because
|
||
* `check-demos` reads the weights off the page beside the code. If reward.py
|
||
* changes, the receipt panel shows the change and these must follow it.
|
||
*/
|
||
|
||
export const reward: RewardSpec = {
|
||
components: [
|
||
{
|
||
key: 'caught',
|
||
label: 'Nothing slipped through',
|
||
description:
|
||
'One point for any correct call. Half a point if a clean customer was escalated — nothing slipped, but a report was filed on them. Nothing if a suspicious alert was closed or no call was made. This is the counterweight, and it is over half the reward: a miss also zeroes the other two, so one miss costs more than every hour a policy could save.',
|
||
weight: 0.55,
|
||
role: 'counterweight',
|
||
},
|
||
{
|
||
key: 'hours',
|
||
label: 'Cleared it in the hours budgeted',
|
||
description:
|
||
'Analyst-hours spent — 0.50 to read the screen, 0.05 per reply, 0.40 per lookup, 0.10 for the documents — as a ratio against the cheapest shipped analyst that got this same alert right. Capped at one. A fast wrong answer saves nothing.',
|
||
weight: 0.25,
|
||
role: 'objective',
|
||
},
|
||
{
|
||
key: 'evidence',
|
||
label: 'Named the facts that decide it',
|
||
description:
|
||
'F1 between the ids the run cited and the facts the generator planted: the explaining KYC or document fact on a clean alert, the transactions that constitute the typology on a suspicious one. Cite everything and precision pays you what it thinks of that. Zero if the call was wrong. The narrative prose is never read.',
|
||
weight: 0.2,
|
||
role: 'objective',
|
||
},
|
||
],
|
||
metrics: [
|
||
{ key: 'hours_spent', label: 'Hours spent', description: 'Screen, replies and lookups, at the generator’s prices.' },
|
||
{
|
||
key: 'reference_hours',
|
||
label: 'Reference took',
|
||
description: 'Hours the cheapest shipped analyst needed to get this same alert right.',
|
||
},
|
||
{ key: 'lookups', label: 'Lookups made', description: 'History months, counterparties, prior alerts, documents.' },
|
||
{ key: 'turns_used', label: 'Replies spent', description: 'Accepted or refused. Eight ends the episode.' },
|
||
{
|
||
key: 'rejected_replies',
|
||
label: 'Replies refused',
|
||
description: 'No JSON object, an unknown action, an empty cites list, an id never shown, or a repeated lookup. Costs a turn.',
|
||
},
|
||
{
|
||
key: 'false_escalation',
|
||
label: 'Escalated a clean customer',
|
||
description: 'One if a report was filed on a benign alert.',
|
||
},
|
||
{ key: 'missed', label: 'Closed a suspicious alert', description: 'One if the alert should have been escalated and was closed instead.' },
|
||
{
|
||
key: 'typology_match',
|
||
label: 'Named the right typology',
|
||
description: 'One if the escalation named what the generator planted. Diagnostic only; not scored, because it would double-count the call with an enum the agent can guess.',
|
||
},
|
||
],
|
||
source: {
|
||
path: 'envs/alert_triage/alert_triage/reward.py',
|
||
code: rewardSource,
|
||
marker: 'reward',
|
||
},
|
||
};
|