32 recorded rollouts, 8 seeds x 4 agents, every one replaying through the Python
engine to a delta of exactly 0.
arm solved economy consistency total
base-off 0/8 0.000 0.122 0.0244
base-on 4/8 0.469 0.479 0.4865
cautious 8/8 0.944 1.000 0.9831
solver 8/8 1.000 0.719 0.9437
Thinking-off solves none of eight. The same model on the same seeds, sampled
with thinking on, solves four. That is the headline, it is ours, and it needed
no training — which is also why the run is labelled an intervention and names
what was done to it, so a sampling change can never read as a training result.
`cautious` is new and it exists to make the reward editor honest. Until now the
page invited you to move a slider and watch the ranking change, and no slider
changed anything, because the solver dominated a model that solved nothing. A
candidate-only player — most informative guess among words that could still win
— takes consistency outright and pays for it in turns. Now:
only speed matters solver 0.9859 cautious 0.9606 -> solver
shipped weights solver 0.9437 cautious 0.9831 -> cautious
punish contradictions solver 0.8312 cautious 0.9972 -> cautious
The ranking really does flip, on recorded data, with one slider. The first
version of this arm picked the alphabetically-first candidate and opened on
'abaci', which made the policy the counterweight exists to reward look like a
straw man.
190 contract checks pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019mt6sHQHEnEYrJZvoMCJSB
Three things the build was quietly missing.
The interactive board was never mounted. The contract has `interactive.init` and
`interactive.Controls`, the demo implemented both, and the shell's `split-play`
beat rendered only the replay — so the beat titled "you and the model get the
same word" showed one board. PlayYourself now renders the visitor's attempt from
the same seed as the run beside it, generically: it knows only the contract, so
any demo shipping an interactive mode gets it and one that does not renders
nothing rather than an empty pane.
solver.worker.ts was dead code — nothing constructed it, which is how CI caught
it: `new Worker(` appeared nowhere in the bundle. It is wired now behind "what
would the best player guess?", and it answers in 92ms from a real worker on
boards no recording covers. That is the difference between a demo and a video.
It also surfaces the moment the solver picks a word that CANNOT win, which is
the counterweight visible in one line instead of explained in a paragraph.
The CI check that found it was itself wrong: it grepped every bundled file for
`blob:`, which React's own code contains in a scheme check, so it failed on a
risk that was not present. It now greps for worker construction from a blob,
which is the thing production CSP actually blocks in silence.
And the run switcher was thirty buttons carrying four distinct labels. Split
into arm and seed, holding the seed across an arm change — comparing two agents
means comparing them on the same hidden word, and silently jumping seeds would
break that while looking fine.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019mt6sHQHEnEYrJZvoMCJSB
pnpm build ends in prerender, which drives a real chromium. Without the install
step the build failed on its very last action, after typecheck, the contract
gates and the tests had all gone green — the most expensive place to discover a
missing dependency.
Two silent bugs in the prerender pass, and the second was caused by the fix for
the first.
`waitForSelector('#root > *')` defaults to waiting for VISIBILITY, and the app's
first child is the skip link, which is hidden until focused. So it burned the
full 30s timeout on every one of 17 routes — twelve minutes of a script that
printed nothing, because its output was buffered behind a pipe — while the page
had rendered the whole time. Switching to `state: 'attached'` then fired too
early instead: useSeo writes the head from an effect, so the title was still
index.html's for a tick, and every route would have baked the homepage's head.
That is the exact bug this script exists to prevent. It now waits for `main`,
then for readyState, then settles.
The board's yellow was --warning, 32 95% 31% — darkened until white text cleared
4.5:1, and at that lightness it renders BROWN. On a board where people arrive
knowing this square should be yellow, a brown square reads as a bug in the
scorer, which on a page arguing "the grader is correct" is the worst thing it
could look like. The fill is now a real yellow and the glyph went dark: more
expected AND higher contrast, 10.02:1 against 5.03:1.
Also measured something the Honesty page had honestly declined to claim. It said
our word list is easier than the original's because our dictionary rule keeps
plurals the original's editor removed by hand. Running the same greedy solver
over both pools, 250 sampled words each: original 2,315 needs 3.552 guesses
(opener RAISE, worst 5), ours 4,603 needs 3.700 (opener TARES, worst 6). The
doubled pool outweighs the plurals. Ours is harder, and the page now says so
with the table.
177 gate checks pass. Entry chunk 106.9 kB gzipped against 160 kB.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019mt6sHQHEnEYrJZvoMCJSB
Five parallel lanes plus an integration pass. The header, gallery, router and
sitemap are all generated from the demo registry, so adding src/demos/<slug>/
puts a demo everywhere with zero edits to shared files — which is the whole
reason demo nine cannot break demo one.
check-demos enforces the twelve contract rules: 142 checks over one live demo.
Two worth naming. The shell may not mention a specific slug, because an
'if (slug === wordle)' in src/components/demo/ is a contract bug wearing a
patch. And a spec-status demo must ship a real specification — task, actions,
grader, counterweight, eval command — since a coming-soon card reads worse than
an honest empty gallery.
Bundle budget holds: entry 108.79 kB gzipped against a 160 kB ceiling, the demo
chunk 21.15 kB against 90 kB. recharts is 108 kB gzipped and lives behind a lazy
import so it never touches the entry.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019mt6sHQHEnEYrJZvoMCJSB
The browser engine is a port of the Python one and CI proves it: all 21.2M
(guess, answer) pairs hashed on both sides to the same SHA-256. Six TS tests,
including the duplicate-letter table and the twelve pinned seed vectors that
keep ?seed= permalinks pointing at the same word the recording used.
Word lists are split by how they are used. answers.json is inlined because the
board needs it before first paint to turn a seed into a word, and a fetch there
means a visibly empty board on a cold cache. guesses.json is fetched, because it
is three times larger and only needed the first time somebody presses Enter;
until it lands, validation falls back to the answer list, which accepts strictly
fewer words. The failure mode is 'your real word was briefly rejected', not 'a
non-word was accepted' — the right way round.
The solver runs in a worker constructed from a same-origin module URL, never
Vite's ?worker&inline: that yields a blob:, and production CSP has no
worker-src, so it falls back to default-src 'self' and the worker is blocked
with no console error. It would fail in production only.
deploy.sh smoke-tests the real public hostname from the deploying machine and
fails on a body under 1 kB, because the bind bug's signature is a valid
certificate over an empty 200 and a local --resolve check passes anyway.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019mt6sHQHEnEYrJZvoMCJSB
CONTRACT.md states the twelve rules and, for each, the failure it prevents.
AGENTS.md lists the three directories a demo agent may touch and the seven traps
that have already cost time here — pnpm 11's settings move, the non-portable
RNG, the two-pass rule, the two reward-denominator subtleties, the silent CSP
worker block, and Caddy's silent bind.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019mt6sHQHEnEYrJZvoMCJSB
The site does no live inference. Rollouts are captured once against spark-1 and
replayed at their recorded wall-clock — a public demo with no auth cannot hold
an API key, and a recorded run can be scrubbed, permalinked, blind-compared and
verified in ways a live one cannot. What stops it being a video is that the
browser re-derives every number from the recorded moves.
verify_fixtures.py is the Python half of that: it replays every committed
fixture through the engine and reproduces its own rewards. All 16 land at
delta 0.0. A fixture that cannot be regenerated is a claim with no receipt.
First real measurement, thinking off, 8 seeds: solved 0/8. The model repeats
guesses it has already played, invents words (trape, slith, postt, boomy),
and contradicts its own feedback — consistency 0.09 to 0.17. That is the
published failure taxonomy showing up in our own data on the first run, and it
is why `consistency` is a reward component rather than a footnote.
A capture failure is recorded as a turn with a null reply, never dropped. A
capture that silently discarded failed turns would be reporting a better model
than the one that ran.
CI gates both halves and four things that fail silently in production: the word
lists must rebuild byte-identically, the prerendered routes must carry their own
baked og tags (crawlers do not run JS, so without them every shared link
previews as the homepage), no blob: URL may reach the bundle (the site's CSP has
no worker-src, so it falls back to default-src 'self' and a blob worker is
blocked with no error), and the conformance digest must match across languages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019mt6sHQHEnEYrJZvoMCJSB
engine.ts used mulberry32 and engine.py used random.Random(seed). Same seed,
different word — so every ?seed= permalink on the site would have shown a
different puzzle than the recorded run it claimed to be replaying, and nobody
would have noticed until someone checked one by hand.
Both now derive the index from FNV-1a 32-bit over the decimal seed. A hash
rather than a PRNG because there is no honest one-line JavaScript equivalent of
Mersenne Twister, and this way there is nothing to keep in step: both sides
compute the same integer from the same string. Math.imul on the JS side is
load-bearing — a plain multiply overflows into a double and diverges after the
first few bytes.
Twelve seeds are pinned as a vector in both test suites.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019mt6sHQHEnEYrJZvoMCJSB
The Python is the source of truth; src/demos/wordle/engine.ts will be a port of
it, and CI gates the two against a SHA-256 over all 21.2M (guess, answer)
pattern pairs rather than a hand-picked vector file — a vector file only ever
catches the cases somebody thought of.
The reward is three weighted components, and the third one is the reason this
demo is worth building. `solved` and `economy` pull toward winning. `consistency`
pulls against them, because a player maximising information deliberately guesses
words that cannot win — a word that splits the remaining candidates evenly
teaches more than a word that might happen to be right. That is good play, and
it costs consistency.
The probe ladder proves the tension is real rather than asserted:
inaction 0.0000 crude 0.0111 plausible 0.1224
candidate_only 0.8925 exhaustive 0.9031 oracle 0.9458
The two good policies are 0.05 apart and neither dominates — the entropy oracle
takes 1.00 economy and 0.73 consistency, the candidate-only player takes 0.75
and 1.00. Which one wins is a decision about what you want, which is the whole
argument the site exists to make. probe.py fails CI if either starts dominating.
Two traps found by building it. `consistency` is scored over turns SPENT, not
guesses accepted: counting only legal guesses hands a free 1.0 to a policy that
plays one word and then jams the parser five times — one guess, no
contradictions, perfect score. And `economy`'s denominator is the depth the
SHIPPED solver reaches, not a depth-optimal search: entropy-greedy is not
depth-optimal, so grading it against an exact optimum would make the oracle
rung fail its own assertion on some seeds.
The word lists are built from Wordnik (MIT) intersected with SCOWL, never from
the original game's 2,315 answers. 4,603 answers makes this materially harder
than the original, so the published SALET/3.4212 results are cited as belonging
to that list and our own reference player's TARES/3.72 is measured here.
verifiers is an optional extra. The engine, reward, solver and probe all run —
and gate — without an RL stack resolvable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019mt6sHQHEnEYrJZvoMCJSB
Foundation for a gallery of RL-environment demos. Three decisions worth
recording here rather than in a commit nobody reads:
The word lists are built, not copied. `envs/wordle_five/words/build_words.py`
intersects Wordnik (MIT, 11,846 five-letter words) with SCOWL's common-American
tier to produce 4,603 answers. The intersection is the point: the list is
derived from two permissive sources by a stated rule rather than copied from
anyone's editorial selection, and both inputs are committed so a rebuild is
byte-identical.
The design tokens are PIG's, inlined as literals. PIG writes its accent onto
the root at runtime because a user picks it; this site has no such choice, so
the runtime theme layer would be a moving part buying nothing. Board tiles get
their own named tokens with measured contrast ratios, because the board is the
one place where colour carries meaning.
pnpm 11 no longer reads the "pnpm" field in package.json. Settings live in
pnpm-workspace.yaml, and an unapproved build script makes `pnpm install` exit 1
rather than warn — so this would have failed CI on a clean checkout, not here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019mt6sHQHEnEYrJZvoMCJSB
karti-ai