Alert Triage: the thinking-on model arm, all eight seeds
base-on solved 6/8 with 2 aborted — the model ran out its eight-turn budget without reaching a disposition, which scores exactly 0.000 and is recorded as such rather than dropped. base-off solved 7/8. Both arms sit below the scripted policies on the shipped weights, and above them on nothing, which is the honest shape of a model that has not been trained on this task. 72 fixtures across both environments replay at delta 0.
This commit is contained in:
@@ -92,6 +92,66 @@
|
||||
"seed": 2,
|
||||
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
|
||||
},
|
||||
{
|
||||
"id": "base-on-s3",
|
||||
"label": "Allowed to think",
|
||||
"path": "/traces/alert-triage/base-on-s3.json",
|
||||
"kind": "intervened",
|
||||
"model": "brain-qwen38-dspark",
|
||||
"capturedAt": "2026-08-28",
|
||||
"seed": 3,
|
||||
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
|
||||
},
|
||||
{
|
||||
"id": "base-on-s4",
|
||||
"label": "Allowed to think",
|
||||
"path": "/traces/alert-triage/base-on-s4.json",
|
||||
"kind": "intervened",
|
||||
"model": "brain-qwen38-dspark",
|
||||
"capturedAt": "2026-08-28",
|
||||
"seed": 4,
|
||||
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
|
||||
},
|
||||
{
|
||||
"id": "base-on-s5",
|
||||
"label": "Allowed to think",
|
||||
"path": "/traces/alert-triage/base-on-s5.json",
|
||||
"kind": "intervened",
|
||||
"model": "brain-qwen38-dspark",
|
||||
"capturedAt": "2026-08-28",
|
||||
"seed": 5,
|
||||
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
|
||||
},
|
||||
{
|
||||
"id": "base-on-s6",
|
||||
"label": "Allowed to think",
|
||||
"path": "/traces/alert-triage/base-on-s6.json",
|
||||
"kind": "intervened",
|
||||
"model": "brain-qwen38-dspark",
|
||||
"capturedAt": "2026-08-28",
|
||||
"seed": 6,
|
||||
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
|
||||
},
|
||||
{
|
||||
"id": "base-on-s7",
|
||||
"label": "Allowed to think",
|
||||
"path": "/traces/alert-triage/base-on-s7.json",
|
||||
"kind": "intervened",
|
||||
"model": "brain-qwen38-dspark",
|
||||
"capturedAt": "2026-08-28",
|
||||
"seed": 7,
|
||||
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
|
||||
},
|
||||
{
|
||||
"id": "base-on-s9",
|
||||
"label": "Allowed to think",
|
||||
"path": "/traces/alert-triage/base-on-s9.json",
|
||||
"kind": "intervened",
|
||||
"model": "brain-qwen38-dspark",
|
||||
"capturedAt": "2026-08-28",
|
||||
"seed": 9,
|
||||
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
|
||||
},
|
||||
{
|
||||
"id": "fast-s1",
|
||||
"label": "Reads the screen",
|
||||
|
||||
Reference in New Issue
Block a user