Capture harness, fixture verification, CI, and the public README

The site does no live inference. Rollouts are captured once against spark-1 and replayed at their recorded wall-clock — a public demo with no auth cannot hold an API key, and a recorded run can be scrubbed, permalinked, blind-compared and verified in ways a live one cannot. What stops it being a video is that the browser re-derives every number from the recorded moves. verify_fixtures.py is the Python half of that: it replays every committed fixture through the engine and reproduces its own rewards. All 16 land at delta 0.0. A fixture that cannot be regenerated is a claim with no receipt. First real measurement, thinking off, 8 seeds: solved 0/8. The model repeats guesses it has already played, invents words (trape, slith, postt, boomy), and contradicts its own feedback — consistency 0.09 to 0.17. That is the published failure taxonomy showing up in our own data on the first run, and it is why `consistency` is a reward component rather than a footnote. A capture failure is recorded as a turn with a null reply, never dropped. A capture that silently discarded failed turns would be reporting a better model than the one that ran. CI gates both halves and four things that fail silently in production: the word lists must rebuild byte-identically, the prerendered routes must carry their own baked og tags (crawlers do not run JS, so without them every shared link previews as the homepage), no blob: URL may reach the bundle (the site's CSP has no worker-src, so it falls back to default-src 'self' and a blob worker is blocked with no error), and the conformance digest must match across languages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019mt6sHQHEnEYrJZvoMCJSB
2026-08-28 15:47:31 -07:00
parent 69607fbfe9
commit 408ce4a525
43 changed files with 5279 additions and 136 deletions
@@ -0,0 +1,86 @@
+name: ci
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+concurrency:
+  group: ci-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  web:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v5
+      - uses: pnpm/action-setup@v4
+        with: { version: 11.21.0 }
+      - uses: actions/setup-node@v5
+        with: { node-version: 22, cache: pnpm }
+
+      # Asserts the toolchain matches what package.json pins, rather than
+      # discovering a mismatch three steps later as an unrelated build error.
+      - name: preflight
+        run: |
+          test "$(pnpm -v)" = "11.21.0" || { echo "pnpm $(pnpm -v) != 11.21.0"; exit 1; }
+
+      - run: CI=true pnpm install --frozen-lockfile
+      - run: pnpm typecheck
+      - run: pnpm check
+      - run: pnpm test
+      - run: pnpm build
+      - run: node scripts/bundle-budget.mjs
+
+      # The prerender pass writes a real HTML file per route. Crawlers do not
+      # run JavaScript, so without these every shared link previews as the
+      # homepage — assert the baked tags actually landed.
+      - name: prerendered head is real
+        run: |
+          test -f dist/demos/wordle/index.html || { echo "no prerendered demo route"; exit 1; }
+          grep -q 'og:title' dist/demos/wordle/index.html || { echo "og tags missing"; exit 1; }
+          grep -qv 'PIG Demo — RL environments you can play</title>' dist/demos/wordle/index.html \
+            || { echo "demo route kept the homepage title"; exit 1; }
+          test -f dist/404.html || { echo "no 404.html"; exit 1; }
+
+      # A blob-backed worker is blocked in production and nowhere else: the
+      # site's CSP has no worker-src, so it falls back to default-src 'self'.
+      # The failure is silent — the solver simply never boots.
+      - name: no inline workers
+        run: |
+          ! grep -rqE "createObjectURL|blob:" dist/assets/*.js \
+            || { echo "a blob: URL reached the bundle; CSP will block it in prod"; exit 1; }
+
+  python:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v5
+      - uses: astral-sh/setup-uv@v6
+        with: { enable-cache: true }
+
+      - run: uv sync --all-packages
+
+      # Rebuild the word lists from their committed sources and assert the
+      # output is byte-identical. If it is not, every downstream number —
+      # the conformance digest included — is describing a different game.
+      - name: word lists rebuild identically
+        run: |
+          uv run python envs/wordle_five/words/build_words.py
+          git diff --exit-code envs/wordle_five/words/*.json
+
+      - run: uv run pytest envs/wordle_five/tests -q
+      - run: uv run python envs/probe.py
+
+      # The cross-language gate. Both halves score all 21.2M (guess, answer)
+      # pairs; the digests must match each other and the committed value.
+      - uses: pnpm/action-setup@v4
+        with: { version: 11.21.0 }
+      - uses: actions/setup-node@v5
+        with: { node-version: 22, cache: pnpm }
+      - run: CI=true pnpm install --frozen-lockfile
+      - run: pnpm conformance
+
+      # Every committed fixture must replay through the Python engine and
+      # reproduce its own recorded rewards. A fixture that cannot be
+      # regenerated is a claim with no receipt behind it.
+      - run: uv run python envs/verify_fixtures.py