Capture harness, fixture verification, CI, and the public README

The site does no live inference. Rollouts are captured once against spark-1 and
replayed at their recorded wall-clock — a public demo with no auth cannot hold
an API key, and a recorded run can be scrubbed, permalinked, blind-compared and
verified in ways a live one cannot. What stops it being a video is that the
browser re-derives every number from the recorded moves.

verify_fixtures.py is the Python half of that: it replays every committed
fixture through the engine and reproduces its own rewards. All 16 land at
delta 0.0. A fixture that cannot be regenerated is a claim with no receipt.

First real measurement, thinking off, 8 seeds: solved 0/8. The model repeats
guesses it has already played, invents words (trape, slith, postt, boomy),
and contradicts its own feedback — consistency 0.09 to 0.17. That is the
published failure taxonomy showing up in our own data on the first run, and it
is why `consistency` is a reward component rather than a footnote.

A capture failure is recorded as a turn with a null reply, never dropped. A
capture that silently discarded failed turns would be reporting a better model
than the one that ran.

CI gates both halves and four things that fail silently in production: the word
lists must rebuild byte-identically, the prerendered routes must carry their own
baked og tags (crawlers do not run JS, so without them every shared link
previews as the homepage), no blob: URL may reach the bundle (the site's CSP has
no worker-src, so it falls back to default-src 'self' and a blob worker is
blocked with no error), and the conformance digest must match across languages.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019mt6sHQHEnEYrJZvoMCJSB
This commit is contained in:
karti-ai
2026-08-28 15:47:31 -07:00
parent 69607fbfe9
commit 408ce4a525
43 changed files with 5279 additions and 136 deletions
+188
View File
@@ -0,0 +1,188 @@
{
"runId": "base-off-s4",
"seed": 4,
"model": "brain-qwen38-dspark",
"capturedAt": "2026-08-28",
"rewards": {
"solved": 0.0,
"economy": 0.0,
"consistency": 0.09090909090909091
},
"metrics": {
"guesses_used": 5.0,
"rejected_replies": 6.0,
"reference_depth": 4.0,
"turns_granted": 6.0,
"inconsistent_guesses": 5.0
},
"truncated": false,
"outcome": "failed",
"answer": "divot",
"turns": [
{
"reply": "[crane]",
"reasoning": null,
"call": {
"promptTokens": 147,
"completionTokens": 5,
"reasoningTokens": null,
"durationMs": 952,
"finishReason": "stop"
},
"info": {
"guess": "crane",
"observation": "C R A N E\nX X X X X\n\nYou have 5 guesses left."
}
},
{
"reply": "[slith]",
"reasoning": null,
"call": {
"promptTokens": 184,
"completionTokens": 5,
"reasoningTokens": null,
"durationMs": 1501,
"finishReason": "stop"
},
"info": {
"guess": "slith",
"observation": "That guess was not accepted: 'slith' is not in the word list.\nIt did not use up a turn. You have 5 guesses left."
}
},
{
"reply": "[slant]",
"reasoning": null,
"call": {
"promptTokens": 235,
"completionTokens": 5,
"reasoningTokens": null,
"durationMs": 1363,
"finishReason": "stop"
},
"info": {
"guess": "slant",
"observation": "S L A N T\nX X X X G\n\nYou have 4 guesses left."
}
},
{
"reply": "[roast]",
"reasoning": null,
"call": {
"promptTokens": 272,
"completionTokens": 5,
"reasoningTokens": null,
"durationMs": 1609,
"finishReason": "stop"
},
"info": {
"guess": "roast",
"observation": "R O A S T\nX Y X X G\n\nYou have 3 guesses left."
}
},
{
"reply": "[toast]",
"reasoning": null,
"call": {
"promptTokens": 309,
"completionTokens": 4,
"reasoningTokens": null,
"durationMs": 1240,
"finishReason": "stop"
},
"info": {
"guess": "toast",
"observation": "T O A S T\nX Y X X G\n\nYou have 2 guesses left."
}
},
{
"reply": "[plout]",
"reasoning": null,
"call": {
"promptTokens": 345,
"completionTokens": 5,
"reasoningTokens": null,
"durationMs": 1613,
"finishReason": "stop"
},
"info": {
"guess": "plout",
"observation": "That guess was not accepted: 'plout' is not in the word list.\nIt did not use up a turn. You have 2 guesses left."
}
},
{
"reply": "[postt]",
"reasoning": null,
"call": {
"promptTokens": 396,
"completionTokens": 5,
"reasoningTokens": null,
"durationMs": 1349,
"finishReason": "stop"
},
"info": {
"guess": "postt",
"observation": "That guess was not accepted: 'postt' is not in the word list.\nIt did not use up a turn. You have 2 guesses left."
}
},
{
"reply": "[roost]",
"reasoning": null,
"call": {
"promptTokens": 447,
"completionTokens": 5,
"reasoningTokens": null,
"durationMs": 868,
"finishReason": "stop"
},
"info": {
"guess": "roost",
"observation": "R O O S T\nX Y X X G\n\nYou have 1 guess left."
}
},
{
"reply": "[plout]",
"reasoning": null,
"call": {
"promptTokens": 484,
"completionTokens": 5,
"reasoningTokens": null,
"durationMs": 946,
"finishReason": "stop"
},
"info": {
"guess": "plout",
"observation": "That guess was not accepted: 'plout' is not in the word list.\nIt did not use up a turn. You have 1 guess left."
}
},
{
"reply": "[toast]",
"reasoning": null,
"call": {
"promptTokens": 535,
"completionTokens": 4,
"reasoningTokens": null,
"durationMs": 498,
"finishReason": "stop"
},
"info": {
"guess": "toast",
"observation": "That guess was not accepted: 'toast' has already been guessed.\nIt did not use up a turn. You have 1 guess left."
}
},
{
"reply": "[most]",
"reasoning": null,
"call": {
"promptTokens": 582,
"completionTokens": 4,
"reasoningTokens": null,
"durationMs": 490,
"finishReason": "stop"
},
"info": {
"guess": "most",
"observation": "That guess was not accepted: 'most' is not 5 letters.\nIt did not use up a turn. You have 1 guess left."
}
}
]
}