Capture harness, fixture verification, CI, and the public README

The site does no live inference. Rollouts are captured once against spark-1 and replayed at their recorded wall-clock — a public demo with no auth cannot hold an API key, and a recorded run can be scrubbed, permalinked, blind-compared and verified in ways a live one cannot. What stops it being a video is that the browser re-derives every number from the recorded moves. verify_fixtures.py is the Python half of that: it replays every committed fixture through the engine and reproduces its own rewards. All 16 land at delta 0.0. A fixture that cannot be regenerated is a claim with no receipt. First real measurement, thinking off, 8 seeds: solved 0/8. The model repeats guesses it has already played, invents words (trape, slith, postt, boomy), and contradicts its own feedback — consistency 0.09 to 0.17. That is the published failure taxonomy showing up in our own data on the first run, and it is why `consistency` is a reward component rather than a footnote. A capture failure is recorded as a turn with a null reply, never dropped. A capture that silently discarded failed turns would be reporting a better model than the one that ran. CI gates both halves and four things that fail silently in production: the word lists must rebuild byte-identically, the prerendered routes must carry their own baked og tags (crawlers do not run JS, so without them every shared link previews as the homepage), no blob: URL may reach the bundle (the site's CSP has no worker-src, so it falls back to default-src 'self' and a blob worker is blocked with no error), and the conformance digest must match across languages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019mt6sHQHEnEYrJZvoMCJSB
2026-08-28 15:47:31 -07:00
parent 69607fbfe9
commit 408ce4a525
43 changed files with 5279 additions and 136 deletions
@@ -0,0 +1,113 @@
+{
+  "runId": "base-off-s1",
+  "seed": 1,
+  "model": "brain-qwen38-dspark",
+  "capturedAt": "2026-08-28",
+  "rewards": {
+    "solved": 0.0,
+    "economy": 0.0,
+    "consistency": 0.16666666666666666
+  },
+  "metrics": {
+    "guesses_used": 6.0,
+    "rejected_replies": 0.0,
+    "reference_depth": 3.0,
+    "turns_granted": 6.0,
+    "inconsistent_guesses": 5.0
+  },
+  "truncated": false,
+  "outcome": "failed",
+  "answer": "amber",
+  "turns": [
+    {
+      "reply": "[crane]",
+      "reasoning": null,
+      "call": {
+        "promptTokens": 147,
+        "completionTokens": 5,
+        "reasoningTokens": null,
+        "durationMs": 385,
+        "finishReason": "stop"
+      },
+      "info": {
+        "guess": "crane",
+        "observation": "C R A N E\nX Y Y X Y\n\nYou have 5 guesses left."
+      }
+    },
+    {
+      "reply": "[slate]",
+      "reasoning": null,
+      "call": {
+        "promptTokens": 184,
+        "completionTokens": 5,
+        "reasoningTokens": null,
+        "durationMs": 443,
+        "finishReason": "stop"
+      },
+      "info": {
+        "guess": "slate",
+        "observation": "S L A T E\nX X Y X Y\n\nYou have 4 guesses left."
+      }
+    },
+    {
+      "reply": "[plume]",
+      "reasoning": null,
+      "call": {
+        "promptTokens": 221,
+        "completionTokens": 5,
+        "reasoningTokens": null,
+        "durationMs": 499,
+        "finishReason": "stop"
+      },
+      "info": {
+        "guess": "plume",
+        "observation": "P L U M E\nX X X Y Y\n\nYou have 3 guesses left."
+      }
+    },
+    {
+      "reply": "[grape]",
+      "reasoning": null,
+      "call": {
+        "promptTokens": 258,
+        "completionTokens": 5,
+        "reasoningTokens": null,
+        "durationMs": 488,
+        "finishReason": "stop"
+      },
+      "info": {
+        "guess": "grape",
+        "observation": "G R A P E\nX Y Y X Y\n\nYou have 2 guesses left."
+      }
+    },
+    {
+      "reply": "[stare]",
+      "reasoning": null,
+      "call": {
+        "promptTokens": 295,
+        "completionTokens": 5,
+        "reasoningTokens": null,
+        "durationMs": 439,
+        "finishReason": "stop"
+      },
+      "info": {
+        "guess": "stare",
+        "observation": "S T A R E\nX X Y Y Y\n\nYou have 1 guess left."
+      }
+    },
+    {
+      "reply": "[crate]",
+      "reasoning": null,
+      "call": {
+        "promptTokens": 332,
+        "completionTokens": 4,
+        "reasoningTokens": null,
+        "durationMs": 493,
+        "finishReason": "stop"
+      },
+      "info": {
+        "guess": "crate",
+        "observation": "C R A T E\nX Y Y X Y\n\nYou have 0 guesses left."
+      }
+    }
+  ]
+}