PIG/PIG-Demo
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

All four arms captured: the number moves from 0/8 to 4/8

Browse Source 
32 recorded rollouts, 8 seeds x 4 agents, every one replaying through the Python
engine to a delta of exactly 0.

  arm          solved   economy  consistency    total
  base-off       0/8      0.000        0.122   0.0244
  base-on        4/8      0.469        0.479   0.4865
  cautious       8/8      0.944        1.000   0.9831
  solver         8/8      1.000        0.719   0.9437

Thinking-off solves none of eight. The same model on the same seeds, sampled
with thinking on, solves four. That is the headline, it is ours, and it needed
no training — which is also why the run is labelled an intervention and names
what was done to it, so a sampling change can never read as a training result.

`cautious` is new and it exists to make the reward editor honest. Until now the
page invited you to move a slider and watch the ranking change, and no slider
changed anything, because the solver dominated a model that solved nothing. A
candidate-only player — most informative guess among words that could still win
— takes consistency outright and pays for it in turns. Now:

  only speed matters      solver 0.9859  cautious 0.9606  -> solver
  shipped weights         solver 0.9437  cautious 0.9831  -> cautious
  punish contradictions   solver 0.8312  cautious 0.9972  -> cautious

The ranking really does flip, on recorded data, with one slider. The first
version of this arm picked the alphabetically-first candidate and opened on
'abaci', which made the policy the counterweight exists to reward look like a
straw man.

190 contract checks pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019mt6sHQHEnEYrJZvoMCJSB
This commit is contained in:
karti-ai
2026-08-28 16:42:52 -07:00
parent 331b46b114
commit 2ef62c5670
11 changed files with 1027 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
public/traces/manifest.json
+152
View File
@@ -72,6 +72,86 @@
"capturedAt": "2026-08-28",
"seed": 7
},
{
"id": "base-on-s0",
"label": "Allowed to think",
"path": "/traces/wordle/base-on-s0.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 0,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "base-on-s1",
"label": "Allowed to think",
"path": "/traces/wordle/base-on-s1.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 1,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "base-on-s2",
"label": "Allowed to think",
"path": "/traces/wordle/base-on-s2.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 2,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "base-on-s3",
"label": "Allowed to think",
"path": "/traces/wordle/base-on-s3.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 3,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "base-on-s4",
"label": "Allowed to think",
"path": "/traces/wordle/base-on-s4.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 4,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "base-on-s5",
"label": "Allowed to think",
"path": "/traces/wordle/base-on-s5.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 5,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "base-on-s6",
"label": "Allowed to think",
"path": "/traces/wordle/base-on-s6.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 6,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "base-on-s7",
"label": "Allowed to think",
"path": "/traces/wordle/base-on-s7.json",
"kind": "intervened",
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"seed": 7,
"intervention": "Same model, same seeds, sampled with thinking enabled. No training, no fine-tuning."
},
{
"id": "solver-s0",
"label": "Best-known play",
@@ -143,6 +223,78 @@
"model": "entropy-solver",
"capturedAt": "2026-08-28",
"seed": 7
},
{
"id": "cautious-s0",
"label": "Never wastes a guess",
"path": "/traces/wordle/cautious-s0.json",
"kind": "generated",
"model": "candidate-only-solver",
"capturedAt": "2026-08-28",
"seed": 0
},
{
"id": "cautious-s1",
"label": "Never wastes a guess",
"path": "/traces/wordle/cautious-s1.json",
"kind": "generated",
"model": "candidate-only-solver",
"capturedAt": "2026-08-28",
"seed": 1
},
{
"id": "cautious-s2",
"label": "Never wastes a guess",
"path": "/traces/wordle/cautious-s2.json",
"kind": "generated",
"model": "candidate-only-solver",
"capturedAt": "2026-08-28",
"seed": 2
},
{
"id": "cautious-s3",
"label": "Never wastes a guess",
"path": "/traces/wordle/cautious-s3.json",
"kind": "generated",
"model": "candidate-only-solver",
"capturedAt": "2026-08-28",
"seed": 3
},
{
"id": "cautious-s4",
"label": "Never wastes a guess",
"path": "/traces/wordle/cautious-s4.json",
"kind": "generated",
"model": "candidate-only-solver",
"capturedAt": "2026-08-28",
"seed": 4
},
{
"id": "cautious-s5",
"label": "Never wastes a guess",
"path": "/traces/wordle/cautious-s5.json",
"kind": "generated",
"model": "candidate-only-solver",
"capturedAt": "2026-08-28",
"seed": 5
},
{
"id": "cautious-s6",
"label": "Never wastes a guess",
"path": "/traces/wordle/cautious-s6.json",
"kind": "generated",
"model": "candidate-only-solver",
"capturedAt": "2026-08-28",
"seed": 6
},
{
"id": "cautious-s7",
"label": "Never wastes a guess",
"path": "/traces/wordle/cautious-s7.json",
"kind": "generated",
"model": "candidate-only-solver",
"capturedAt": "2026-08-28",
"seed": 7
}
]
}
public/traces/wordle/base-on-s6.json
+143
View File
File diff suppressed because one or more lines are too long
public/traces/wordle/base-on-s7.json
+98
View File
File diff suppressed because one or more lines are too long
public/traces/wordle/cautious-s0.json
+98
View File
@@ -0,0 +1,98 @@
{
"runId": "cautious-s0",
"seed": 0,
"model": "candidate-only-solver",
"capturedAt": "2026-08-28",
"rewards": {
"solved": 1.0,
"economy": 0.8,
"consistency": 1.0
},
"metrics": {
"guesses_used": 5.0,
"rejected_replies": 0.0,
"reference_depth": 4.0,
"turns_granted": 6.0,
"inconsistent_guesses": 0.0
},
"truncated": false,
"outcome": "solved",
"answer": "wants",
"turns": [
{
"reply": "[tares]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "tares",
"observation": "T A R E S\nY G X X G\n\nYou have 5 guesses left."
}
},
{
"reply": "[casts]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "casts",
"observation": "C A S T S\nX G X G G\n\nYou have 4 guesses left."
}
},
{
"reply": "[waits]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "waits",
"observation": "W A I T S\nG G X G G\n\nYou have 3 guesses left."
}
},
{
"reply": "[wafts]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "wafts",
"observation": "W A F T S\nG G X G G\n\nYou have 2 guesses left."
}
},
{
"reply": "[wants]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "wants",
"observation": "W A N T S\nG G G G G\n\nYou have 1 guess left."
}
}
]
}
public/traces/wordle/cautious-s1.json
+53
View File
@@ -0,0 +1,53 @@
{
"runId": "cautious-s1",
"seed": 1,
"model": "candidate-only-solver",
"capturedAt": "2026-08-28",
"rewards": {
"solved": 1.0,
"economy": 1.0,
"consistency": 1.0
},
"metrics": {
"guesses_used": 2.0,
"rejected_replies": 0.0,
"reference_depth": 3.0,
"turns_granted": 6.0,
"inconsistent_guesses": 0.0
},
"truncated": false,
"outcome": "solved",
"answer": "amber",
"turns": [
{
"reply": "[tares]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "tares",
"observation": "T A R E S\nX Y Y G X\n\nYou have 5 guesses left."
}
},
{
"reply": "[amber]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "amber",
"observation": "A M B E R\nG G G G G\n\nYou have 4 guesses left."
}
}
]
}
public/traces/wordle/cautious-s2.json
+83
View File
@@ -0,0 +1,83 @@
{
"runId": "cautious-s2",
"seed": 2,
"model": "candidate-only-solver",
"capturedAt": "2026-08-28",
"rewards": {
"solved": 1.0,
"economy": 1.0,
"consistency": 1.0
},
"metrics": {
"guesses_used": 4.0,
"rejected_replies": 0.0,
"reference_depth": 4.0,
"turns_granted": 6.0,
"inconsistent_guesses": 0.0
},
"truncated": false,
"outcome": "solved",
"answer": "spume",
"turns": [
{
"reply": "[tares]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "tares",
"observation": "T A R E S\nX X X Y Y\n\nYou have 5 guesses left."
}
},
{
"reply": "[spine]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "spine",
"observation": "S P I N E\nG G X X G\n\nYou have 4 guesses left."
}
},
{
"reply": "[spoke]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "spoke",
"observation": "S P O K E\nG G X X G\n\nYou have 3 guesses left."
}
},
{
"reply": "[spume]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "spume",
"observation": "S P U M E\nG G G G G\n\nYou have 2 guesses left."
}
}
]
}
public/traces/wordle/cautious-s3.json
+68
View File
@@ -0,0 +1,68 @@
{
"runId": "cautious-s3",
"seed": 3,
"model": "candidate-only-solver",
"capturedAt": "2026-08-28",
"rewards": {
"solved": 1.0,
"economy": 1.0,
"consistency": 1.0
},
"metrics": {
"guesses_used": 3.0,
"rejected_replies": 0.0,
"reference_depth": 3.0,
"turns_granted": 6.0,
"inconsistent_guesses": 0.0
},
"truncated": false,
"outcome": "solved",
"answer": "toady",
"turns": [
{
"reply": "[tares]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "tares",
"observation": "T A R E S\nG Y X X X\n\nYou have 5 guesses left."
}
},
{
"reply": "[tidal]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "tidal",
"observation": "T I D A L\nG X Y Y X\n\nYou have 4 guesses left."
}
},
{
"reply": "[toady]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "toady",
"observation": "T O A D Y\nG G G G G\n\nYou have 3 guesses left."
}
}
]
}
public/traces/wordle/cautious-s4.json
+83
View File
@@ -0,0 +1,83 @@
{
"runId": "cautious-s4",
"seed": 4,
"model": "candidate-only-solver",
"capturedAt": "2026-08-28",
"rewards": {
"solved": 1.0,
"economy": 1.0,
"consistency": 1.0
},
"metrics": {
"guesses_used": 4.0,
"rejected_replies": 0.0,
"reference_depth": 4.0,
"turns_granted": 6.0,
"inconsistent_guesses": 0.0
},
"truncated": false,
"outcome": "solved",
"answer": "divot",
"turns": [
{
"reply": "[tares]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "tares",
"observation": "T A R E S\nY X X X X\n\nYou have 5 guesses left."
}
},
{
"reply": "[count]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "count",
"observation": "C O U N T\nX Y X X G\n\nYou have 4 guesses left."
}
},
{
"reply": "[pivot]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "pivot",
"observation": "P I V O T\nX G G G G\n\nYou have 3 guesses left."
}
},
{
"reply": "[divot]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "divot",
"observation": "D I V O T\nG G G G G\n\nYou have 2 guesses left."
}
}
]
}
public/traces/wordle/cautious-s5.json
+83
View File
@@ -0,0 +1,83 @@
{
"runId": "cautious-s5",
"seed": 5,
"model": "candidate-only-solver",
"capturedAt": "2026-08-28",
"rewards": {
"solved": 1.0,
"economy": 1.0,
"consistency": 1.0
},
"metrics": {
"guesses_used": 4.0,
"rejected_replies": 0.0,
"reference_depth": 4.0,
"turns_granted": 6.0,
"inconsistent_guesses": 0.0
},
"truncated": false,
"outcome": "solved",
"answer": "filly",
"turns": [
{
"reply": "[tares]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "tares",
"observation": "T A R E S\nX X X X X\n\nYou have 5 guesses left."
}
},
{
"reply": "[doily]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "doily",
"observation": "D O I L Y\nX X Y G G\n\nYou have 4 guesses left."
}
},
{
"reply": "[billy]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "billy",
"observation": "B I L L Y\nX G G G G\n\nYou have 3 guesses left."
}
},
{
"reply": "[filly]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "filly",
"observation": "F I L L Y\nG G G G G\n\nYou have 2 guesses left."
}
}
]
}
public/traces/wordle/cautious-s6.json
+83
View File
@@ -0,0 +1,83 @@
{
"runId": "cautious-s6",
"seed": 6,
"model": "candidate-only-solver",
"capturedAt": "2026-08-28",
"rewards": {
"solved": 1.0,
"economy": 1.0,
"consistency": 1.0
},
"metrics": {
"guesses_used": 4.0,
"rejected_replies": 0.0,
"reference_depth": 4.0,
"turns_granted": 6.0,
"inconsistent_guesses": 0.0
},
"truncated": false,
"outcome": "solved",
"answer": "bobby",
"turns": [
{
"reply": "[tares]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "tares",
"observation": "T A R E S\nX X X X X\n\nYou have 5 guesses left."
}
},
{
"reply": "[doily]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "doily",
"observation": "D O I L Y\nX G X X G\n\nYou have 4 guesses left."
}
},
{
"reply": "[boozy]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "boozy",
"observation": "B O O Z Y\nG G X X G\n\nYou have 3 guesses left."
}
},
{
"reply": "[bobby]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "bobby",
"observation": "B O B B Y\nG G G G G\n\nYou have 2 guesses left."
}
}
]
}
public/traces/wordle/cautious-s7.json
+83
View File
@@ -0,0 +1,83 @@
{
"runId": "cautious-s7",
"seed": 7,
"model": "candidate-only-solver",
"capturedAt": "2026-08-28",
"rewards": {
"solved": 1.0,
"economy": 0.75,
"consistency": 1.0
},
"metrics": {
"guesses_used": 4.0,
"rejected_replies": 0.0,
"reference_depth": 3.0,
"turns_granted": 6.0,
"inconsistent_guesses": 0.0
},
"truncated": false,
"outcome": "solved",
"answer": "clews",
"turns": [
{
"reply": "[tares]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "tares",
"observation": "T A R E S\nX X X Y G\n\nYou have 5 guesses left."
}
},
{
"reply": "[sleds]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "sleds",
"observation": "S L E D S\nX G G X G\n\nYou have 4 guesses left."
}
},
{
"reply": "[clefs]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "clefs",
"observation": "C L E F S\nG G G X G\n\nYou have 3 guesses left."
}
},
{
"reply": "[clews]",
"reasoning": null,
"call": {
"promptTokens": null,
"completionTokens": null,
"reasoningTokens": null,
"durationMs": null,
"finishReason": "generated"
},
"info": {
"guess": "clews",
"observation": "C L E W S\nG G G G G\n\nYou have 2 guesses left."
}
}
]
}