PIG/PIG-Demo
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
4bec0724dba6aa2c60ba26ccb7feb4f95e8cef95
PIG-Demo/public/traces/manifest.json
T
karti-ai af15ba584e
ci / web (push) Failing after 1m47s
Details
ci / python (push) Successful in 3m31s
Details
Alert Triage: the thinking-on model arm, all eight seeds 
base-on solved 6/8 with 2 aborted — the model ran out its eight-turn budget
without reaching a disposition, which scores exactly 0.000 and is recorded as
such rather than dropped. base-off solved 7/8. Both arms sit below the scripted
policies on the shipped weights, and above them on nothing, which is the honest
shape of a model that has not been trained on this task.

72 fixtures across both environments replay at delta 0.
2026-08-28 20:03:26 -07:00

671 lines
19 KiB
JSON
Raw Blame History

{
"alert-triage": [
{
"id": "base-off-s1",
"label": "Out of the box",
"path": "/traces/alert-triage/base-off-s1.json",
"kind": "recorded",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 1
},
{
"id": "base-off-s2",
"label": "Out of the box",
"path": "/traces/alert-triage/base-off-s2.json",
"kind": "recorded",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 2
},
{
"id": "base-off-s3",
"label": "Out of the box",
"path": "/traces/alert-triage/base-off-s3.json",
"kind": "recorded",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 3
},
{
"id": "base-off-s4",
"label": "Out of the box",
"path": "/traces/alert-triage/base-off-s4.json",
"kind": "recorded",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 4
},
{
"id": "base-off-s5",
"label": "Out of the box",
"path": "/traces/alert-triage/base-off-s5.json",
"kind": "recorded",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 5
},
{
"id": "base-off-s6",
"label": "Out of the box",
"path": "/traces/alert-triage/base-off-s6.json",
"kind": "recorded",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 6
},
{
"id": "base-off-s7",
"label": "Out of the box",
"path": "/traces/alert-triage/base-off-s7.json",
"kind": "recorded",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 7
},
{
"id": "base-off-s9",
"label": "Out of the box",
"path": "/traces/alert-triage/base-off-s9.json",
"kind": "recorded",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 9
},
{
"id": "base-on-s1",
"label": "Allowed to think",
"path": "/traces/alert-triage/base-on-s1.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 1,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "base-on-s2",
"label": "Allowed to think",
"path": "/traces/alert-triage/base-on-s2.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 2,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "base-on-s3",
"label": "Allowed to think",
"path": "/traces/alert-triage/base-on-s3.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 3,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "base-on-s4",
"label": "Allowed to think",
"path": "/traces/alert-triage/base-on-s4.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 4,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "base-on-s5",
"label": "Allowed to think",
"path": "/traces/alert-triage/base-on-s5.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 5,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "base-on-s6",
"label": "Allowed to think",
"path": "/traces/alert-triage/base-on-s6.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 6,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "base-on-s7",
"label": "Allowed to think",
"path": "/traces/alert-triage/base-on-s7.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 7,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "base-on-s9",
"label": "Allowed to think",
"path": "/traces/alert-triage/base-on-s9.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 9,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "fast-s1",
"label": "Reads the screen",
"path": "/traces/alert-triage/fast-s1.json",
"kind": "generated",
"model": "fast-analyst",
"capturedAt": "2026-08-28",
"seed": 1
},
{
"id": "fast-s2",
"label": "Reads the screen",
"path": "/traces/alert-triage/fast-s2.json",
"kind": "generated",
"model": "fast-analyst",
"capturedAt": "2026-08-28",
"seed": 2
},
{
"id": "fast-s3",
"label": "Reads the screen",
"path": "/traces/alert-triage/fast-s3.json",
"kind": "generated",
"model": "fast-analyst",
"capturedAt": "2026-08-28",
"seed": 3
},
{
"id": "fast-s4",
"label": "Reads the screen",
"path": "/traces/alert-triage/fast-s4.json",
"kind": "generated",
"model": "fast-analyst",
"capturedAt": "2026-08-28",
"seed": 4
},
{
"id": "fast-s5",
"label": "Reads the screen",
"path": "/traces/alert-triage/fast-s5.json",
"kind": "generated",
"model": "fast-analyst",
"capturedAt": "2026-08-28",
"seed": 5
},
{
"id": "fast-s6",
"label": "Reads the screen",
"path": "/traces/alert-triage/fast-s6.json",
"kind": "generated",
"model": "fast-analyst",
"capturedAt": "2026-08-28",
"seed": 6
},
{
"id": "fast-s7",
"label": "Reads the screen",
"path": "/traces/alert-triage/fast-s7.json",
"kind": "generated",
"model": "fast-analyst",
"capturedAt": "2026-08-28",
"seed": 7
},
{
"id": "fast-s9",
"label": "Reads the screen",
"path": "/traces/alert-triage/fast-s9.json",
"kind": "generated",
"model": "fast-analyst",
"capturedAt": "2026-08-28",
"seed": 9
},
{
"id": "targeted-s1",
"label": "Checks the hidden tells",
"path": "/traces/alert-triage/targeted-s1.json",
"kind": "generated",
"model": "targeted-analyst",
"capturedAt": "2026-08-28",
"seed": 1
},
{
"id": "targeted-s2",
"label": "Checks the hidden tells",
"path": "/traces/alert-triage/targeted-s2.json",
"kind": "generated",
"model": "targeted-analyst",
"capturedAt": "2026-08-28",
"seed": 2
},
{
"id": "targeted-s3",
"label": "Checks the hidden tells",
"path": "/traces/alert-triage/targeted-s3.json",
"kind": "generated",
"model": "targeted-analyst",
"capturedAt": "2026-08-28",
"seed": 3
},
{
"id": "targeted-s4",
"label": "Checks the hidden tells",
"path": "/traces/alert-triage/targeted-s4.json",
"kind": "generated",
"model": "targeted-analyst",
"capturedAt": "2026-08-28",
"seed": 4
},
{
"id": "targeted-s5",
"label": "Checks the hidden tells",
"path": "/traces/alert-triage/targeted-s5.json",
"kind": "generated",
"model": "targeted-analyst",
"capturedAt": "2026-08-28",
"seed": 5
},
{
"id": "targeted-s6",
"label": "Checks the hidden tells",
"path": "/traces/alert-triage/targeted-s6.json",
"kind": "generated",
"model": "targeted-analyst",
"capturedAt": "2026-08-28",
"seed": 6
},
{
"id": "targeted-s7",
"label": "Checks the hidden tells",
"path": "/traces/alert-triage/targeted-s7.json",
"kind": "generated",
"model": "targeted-analyst",
"capturedAt": "2026-08-28",
"seed": 7
},
{
"id": "targeted-s9",
"label": "Checks the hidden tells",
"path": "/traces/alert-triage/targeted-s9.json",
"kind": "generated",
"model": "targeted-analyst",
"capturedAt": "2026-08-28",
"seed": 9
},
{
"id": "thorough-s1",
"label": "Runs the full procedure",
"path": "/traces/alert-triage/thorough-s1.json",
"kind": "generated",
"model": "thorough-analyst",
"capturedAt": "2026-08-28",
"seed": 1
},
{
"id": "thorough-s2",
"label": "Runs the full procedure",
"path": "/traces/alert-triage/thorough-s2.json",
"kind": "generated",
"model": "thorough-analyst",
"capturedAt": "2026-08-28",
"seed": 2
},
{
"id": "thorough-s3",
"label": "Runs the full procedure",
"path": "/traces/alert-triage/thorough-s3.json",
"kind": "generated",
"model": "thorough-analyst",
"capturedAt": "2026-08-28",
"seed": 3
},
{
"id": "thorough-s4",
"label": "Runs the full procedure",
"path": "/traces/alert-triage/thorough-s4.json",
"kind": "generated",
"model": "thorough-analyst",
"capturedAt": "2026-08-28",
"seed": 4
},
{
"id": "thorough-s5",
"label": "Runs the full procedure",
"path": "/traces/alert-triage/thorough-s5.json",
"kind": "generated",
"model": "thorough-analyst",
"capturedAt": "2026-08-28",
"seed": 5
},
{
"id": "thorough-s6",
"label": "Runs the full procedure",
"path": "/traces/alert-triage/thorough-s6.json",
"kind": "generated",
"model": "thorough-analyst",
"capturedAt": "2026-08-28",
"seed": 6
},
{
"id": "thorough-s7",
"label": "Runs the full procedure",
"path": "/traces/alert-triage/thorough-s7.json",
"kind": "generated",
"model": "thorough-analyst",
"capturedAt": "2026-08-28",
"seed": 7
},
{
"id": "thorough-s9",
"label": "Runs the full procedure",
"path": "/traces/alert-triage/thorough-s9.json",
"kind": "generated",
"model": "thorough-analyst",
"capturedAt": "2026-08-28",
"seed": 9
}
],
"wordle": [
{
"id": "base-off-s0",
"label": "Out of the box",
"path": "/traces/wordle/base-off-s0.json",
"kind": "recorded",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 0
},
{
"id": "base-off-s1",
"label": "Out of the box",
"path": "/traces/wordle/base-off-s1.json",
"kind": "recorded",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 1
},
{
"id": "base-off-s2",
"label": "Out of the box",
"path": "/traces/wordle/base-off-s2.json",
"kind": "recorded",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 2
},
{
"id": "base-off-s3",
"label": "Out of the box",
"path": "/traces/wordle/base-off-s3.json",
"kind": "recorded",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 3
},
{
"id": "base-off-s4",
"label": "Out of the box",
"path": "/traces/wordle/base-off-s4.json",
"kind": "recorded",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 4
},
{
"id": "base-off-s5",
"label": "Out of the box",
"path": "/traces/wordle/base-off-s5.json",
"kind": "recorded",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 5
},
{
"id": "base-off-s6",
"label": "Out of the box",
"path": "/traces/wordle/base-off-s6.json",
"kind": "recorded",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 6
},
{
"id": "base-off-s7",
"label": "Out of the box",
"path": "/traces/wordle/base-off-s7.json",
"kind": "recorded",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 7
},
{
"id": "base-on-s0",
"label": "Allowed to think",
"path": "/traces/wordle/base-on-s0.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 0,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "base-on-s1",
"label": "Allowed to think",
"path": "/traces/wordle/base-on-s1.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 1,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "base-on-s2",
"label": "Allowed to think",
"path": "/traces/wordle/base-on-s2.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 2,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "base-on-s3",
"label": "Allowed to think",
"path": "/traces/wordle/base-on-s3.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 3,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "base-on-s4",
"label": "Allowed to think",
"path": "/traces/wordle/base-on-s4.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 4,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "base-on-s5",
"label": "Allowed to think",
"path": "/traces/wordle/base-on-s5.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 5,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "base-on-s6",
"label": "Allowed to think",
"path": "/traces/wordle/base-on-s6.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 6,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "base-on-s7",
"label": "Allowed to think",
"path": "/traces/wordle/base-on-s7.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 7,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "solver-s0",
"label": "Best-known play",
"path": "/traces/wordle/solver-s0.json",
"kind": "generated",
"model": "entropy-solver",
"capturedAt": "2026-08-28",
"seed": 0
},
{
"id": "solver-s1",
"label": "Best-known play",
"path": "/traces/wordle/solver-s1.json",
"kind": "generated",
"model": "entropy-solver",
"capturedAt": "2026-08-28",
"seed": 1
},
{
"id": "solver-s2",
"label": "Best-known play",
"path": "/traces/wordle/solver-s2.json",
"kind": "generated",
"model": "entropy-solver",
"capturedAt": "2026-08-28",
"seed": 2
},
{
"id": "solver-s3",
"label": "Best-known play",
"path": "/traces/wordle/solver-s3.json",
"kind": "generated",
"model": "entropy-solver",
"capturedAt": "2026-08-28",
"seed": 3
},
{
"id": "solver-s4",
"label": "Best-known play",
"path": "/traces/wordle/solver-s4.json",
"kind": "generated",
"model": "entropy-solver",
"capturedAt": "2026-08-28",
"seed": 4
},
{
"id": "solver-s5",
"label": "Best-known play",
"path": "/traces/wordle/solver-s5.json",
"kind": "generated",
"model": "entropy-solver",
"capturedAt": "2026-08-28",
"seed": 5
},
{
"id": "solver-s6",
"label": "Best-known play",
"path": "/traces/wordle/solver-s6.json",
"kind": "generated",
"model": "entropy-solver",
"capturedAt": "2026-08-28",
"seed": 6
},
{
"id": "solver-s7",
"label": "Best-known play",
"path": "/traces/wordle/solver-s7.json",
"kind": "generated",
"model": "entropy-solver",
"capturedAt": "2026-08-28",
"seed": 7
},
{
"id": "cautious-s0",
"label": "Never wastes a guess",
"path": "/traces/wordle/cautious-s0.json",
"kind": "generated",
"model": "candidate-only-solver",
"capturedAt": "2026-08-28",
"seed": 0
},
{
"id": "cautious-s1",
"label": "Never wastes a guess",
"path": "/traces/wordle/cautious-s1.json",
"kind": "generated",
"model": "candidate-only-solver",
"capturedAt": "2026-08-28",
"seed": 1
},
{
"id": "cautious-s2",
"label": "Never wastes a guess",
"path": "/traces/wordle/cautious-s2.json",
"kind": "generated",
"model": "candidate-only-solver",
"capturedAt": "2026-08-28",
"seed": 2
},
{
"id": "cautious-s3",
"label": "Never wastes a guess",
"path": "/traces/wordle/cautious-s3.json",
"kind": "generated",
"model": "candidate-only-solver",
"capturedAt": "2026-08-28",
"seed": 3
},
{
"id": "cautious-s4",
"label": "Never wastes a guess",
"path": "/traces/wordle/cautious-s4.json",
"kind": "generated",
"model": "candidate-only-solver",
"capturedAt": "2026-08-28",
"seed": 4
},
{
"id": "cautious-s5",
"label": "Never wastes a guess",
"path": "/traces/wordle/cautious-s5.json",
"kind": "generated",
"model": "candidate-only-solver",
"capturedAt": "2026-08-28",
"seed": 5
},
{
"id": "cautious-s6",
"label": "Never wastes a guess",
"path": "/traces/wordle/cautious-s6.json",
"kind": "generated",
"model": "candidate-only-solver",
"capturedAt": "2026-08-28",
"seed": 6
},
{
"id": "cautious-s7",
"label": "Never wastes a guess",
"path": "/traces/wordle/cautious-s7.json",
"kind": "generated",
"model": "candidate-only-solver",
"capturedAt": "2026-08-28",
"seed": 7
}
]
}
Reference in New Issue View Git Blame Copy Permalink