karti-ai
408ce4a525
The site does no live inference. Rollouts are captured once against spark-1 and replayed at their recorded wall-clock — a public demo with no auth cannot hold an API key, and a recorded run can be scrubbed, permalinked, blind-compared and verified in ways a live one cannot. What stops it being a video is that the browser re-derives every number from the recorded moves. verify_fixtures.py is the Python half of that: it replays every committed fixture through the engine and reproduces its own rewards. All 16 land at delta 0.0. A fixture that cannot be regenerated is a claim with no receipt. First real measurement, thinking off, 8 seeds: solved 0/8. The model repeats guesses it has already played, invents words (trape, slith, postt, boomy), and contradicts its own feedback — consistency 0.09 to 0.17. That is the published failure taxonomy showing up in our own data on the first run, and it is why `consistency` is a reward component rather than a footnote. A capture failure is recorded as a turn with a null reply, never dropped. A capture that silently discarded failed turns would be reporting a better model than the one that ran. CI gates both halves and four things that fail silently in production: the word lists must rebuild byte-identically, the prerendered routes must carry their own baked og tags (crawlers do not run JS, so without them every shared link previews as the homepage), no blob: URL may reach the bundle (the site's CSP has no worker-src, so it falls back to default-src 'self' and a blob worker is blocked with no error), and the conformance digest must match across languages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019mt6sHQHEnEYrJZvoMCJSB
189 lines
4.7 KiB
JSON
189 lines
4.7 KiB
JSON
{
|
|
"runId": "base-off-s2",
|
|
"seed": 2,
|
|
"model": "brain-qwen38-dspark",
|
|
"capturedAt": "2026-08-28",
|
|
"rewards": {
|
|
"solved": 0.0,
|
|
"economy": 0.0,
|
|
"consistency": 0.09090909090909091
|
|
},
|
|
"metrics": {
|
|
"guesses_used": 5.0,
|
|
"rejected_replies": 6.0,
|
|
"reference_depth": 4.0,
|
|
"turns_granted": 6.0,
|
|
"inconsistent_guesses": 5.0
|
|
},
|
|
"truncated": false,
|
|
"outcome": "failed",
|
|
"answer": "spume",
|
|
"turns": [
|
|
{
|
|
"reply": "[crane]",
|
|
"reasoning": null,
|
|
"call": {
|
|
"promptTokens": 147,
|
|
"completionTokens": 5,
|
|
"reasoningTokens": null,
|
|
"durationMs": 301,
|
|
"finishReason": "stop"
|
|
},
|
|
"info": {
|
|
"guess": "crane",
|
|
"observation": "C R A N E\nX X X X G\n\nYou have 5 guesses left."
|
|
}
|
|
},
|
|
{
|
|
"reply": "[shine]",
|
|
"reasoning": null,
|
|
"call": {
|
|
"promptTokens": 184,
|
|
"completionTokens": 4,
|
|
"reasoningTokens": null,
|
|
"durationMs": 444,
|
|
"finishReason": "stop"
|
|
},
|
|
"info": {
|
|
"guess": "shine",
|
|
"observation": "S H I N E\nG X X X G\n\nYou have 4 guesses left."
|
|
}
|
|
},
|
|
{
|
|
"reply": "[shone]",
|
|
"reasoning": null,
|
|
"call": {
|
|
"promptTokens": 220,
|
|
"completionTokens": 5,
|
|
"reasoningTokens": null,
|
|
"durationMs": 496,
|
|
"finishReason": "stop"
|
|
},
|
|
"info": {
|
|
"guess": "shone",
|
|
"observation": "S H O N E\nG X X X G\n\nYou have 3 guesses left."
|
|
}
|
|
},
|
|
{
|
|
"reply": "[swine]",
|
|
"reasoning": null,
|
|
"call": {
|
|
"promptTokens": 257,
|
|
"completionTokens": 5,
|
|
"reasoningTokens": null,
|
|
"durationMs": 606,
|
|
"finishReason": "stop"
|
|
},
|
|
"info": {
|
|
"guess": "swine",
|
|
"observation": "S W I N E\nG X X X G\n\nYou have 2 guesses left."
|
|
}
|
|
},
|
|
{
|
|
"reply": "[shine]",
|
|
"reasoning": null,
|
|
"call": {
|
|
"promptTokens": 294,
|
|
"completionTokens": 4,
|
|
"reasoningTokens": null,
|
|
"durationMs": 439,
|
|
"finishReason": "stop"
|
|
},
|
|
"info": {
|
|
"guess": "shine",
|
|
"observation": "That guess was not accepted: 'shine' has already been guessed.\nIt did not use up a turn. You have 2 guesses left."
|
|
}
|
|
},
|
|
{
|
|
"reply": "[spine]",
|
|
"reasoning": null,
|
|
"call": {
|
|
"promptTokens": 341,
|
|
"completionTokens": 5,
|
|
"reasoningTokens": null,
|
|
"durationMs": 384,
|
|
"finishReason": "stop"
|
|
},
|
|
"info": {
|
|
"guess": "spine",
|
|
"observation": "S P I N E\nG G X X G\n\nYou have 1 guess left."
|
|
}
|
|
},
|
|
{
|
|
"reply": "[spine]",
|
|
"reasoning": null,
|
|
"call": {
|
|
"promptTokens": 378,
|
|
"completionTokens": 5,
|
|
"reasoningTokens": null,
|
|
"durationMs": 331,
|
|
"finishReason": "stop"
|
|
},
|
|
"info": {
|
|
"guess": "spine",
|
|
"observation": "That guess was not accepted: 'spine' has already been guessed.\nIt did not use up a turn. You have 1 guess left."
|
|
}
|
|
},
|
|
{
|
|
"reply": "[spine]",
|
|
"reasoning": null,
|
|
"call": {
|
|
"promptTokens": 427,
|
|
"completionTokens": 5,
|
|
"reasoningTokens": null,
|
|
"durationMs": 389,
|
|
"finishReason": "stop"
|
|
},
|
|
"info": {
|
|
"guess": "spine",
|
|
"observation": "That guess was not accepted: 'spine' has already been guessed.\nIt did not use up a turn. You have 1 guess left."
|
|
}
|
|
},
|
|
{
|
|
"reply": "[spine]",
|
|
"reasoning": null,
|
|
"call": {
|
|
"promptTokens": 476,
|
|
"completionTokens": 5,
|
|
"reasoningTokens": null,
|
|
"durationMs": 386,
|
|
"finishReason": "stop"
|
|
},
|
|
"info": {
|
|
"guess": "spine",
|
|
"observation": "That guess was not accepted: 'spine' has already been guessed.\nIt did not use up a turn. You have 1 guess left."
|
|
}
|
|
},
|
|
{
|
|
"reply": "[spine]",
|
|
"reasoning": null,
|
|
"call": {
|
|
"promptTokens": 525,
|
|
"completionTokens": 5,
|
|
"reasoningTokens": null,
|
|
"durationMs": 379,
|
|
"finishReason": "stop"
|
|
},
|
|
"info": {
|
|
"guess": "spine",
|
|
"observation": "That guess was not accepted: 'spine' has already been guessed.\nIt did not use up a turn. You have 1 guess left."
|
|
}
|
|
},
|
|
{
|
|
"reply": "[spine]",
|
|
"reasoning": null,
|
|
"call": {
|
|
"promptTokens": 574,
|
|
"completionTokens": 5,
|
|
"reasoningTokens": null,
|
|
"durationMs": 334,
|
|
"finishReason": "stop"
|
|
},
|
|
"info": {
|
|
"guess": "spine",
|
|
"observation": "That guess was not accepted: 'spine' has already been guessed.\nIt did not use up a turn. You have 1 guess left."
|
|
}
|
|
}
|
|
]
|
|
}
|