Alert Triage: the thinking-on model arm, all eight seeds
ci / web (push) Failing after 1m47s
ci / python (push) Successful in 3m31s

base-on solved 6/8 with 2 aborted — the model ran out its eight-turn budget
without reaching a disposition, which scores exactly 0.000 and is recorded as
such rather than dropped. base-off solved 7/8. Both arms sit below the scripted
policies on the shipped weights, and above them on nothing, which is the honest
shape of a model that has not been trained on this task.

72 fixtures across both environments replay at delta 0.
This commit is contained in:
karti-ai
2026-08-28 20:03:26 -07:00
parent c075ac57cd
commit af15ba584e
7 changed files with 805 additions and 0 deletions
+60
View File
@@ -92,6 +92,66 @@
"seed": 2,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "base-on-s3",
"label": "Allowed to think",
"path": "/traces/alert-triage/base-on-s3.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 3,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "base-on-s4",
"label": "Allowed to think",
"path": "/traces/alert-triage/base-on-s4.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 4,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "base-on-s5",
"label": "Allowed to think",
"path": "/traces/alert-triage/base-on-s5.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 5,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "base-on-s6",
"label": "Allowed to think",
"path": "/traces/alert-triage/base-on-s6.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 6,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "base-on-s7",
"label": "Allowed to think",
"path": "/traces/alert-triage/base-on-s7.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 7,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "base-on-s9",
"label": "Allowed to think",
"path": "/traces/alert-triage/base-on-s9.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 9,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "fast-s1",
"label": "Reads the screen",