PIG/PIG-Demo
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Capture harness, fixture verification, CI, and the public README

Browse Source 
The site does no live inference. Rollouts are captured once against spark-1 and
replayed at their recorded wall-clock — a public demo with no auth cannot hold
an API key, and a recorded run can be scrubbed, permalinked, blind-compared and
verified in ways a live one cannot. What stops it being a video is that the
browser re-derives every number from the recorded moves.

verify_fixtures.py is the Python half of that: it replays every committed
fixture through the engine and reproduces its own rewards. All 16 land at
delta 0.0. A fixture that cannot be regenerated is a claim with no receipt.

First real measurement, thinking off, 8 seeds: solved 0/8. The model repeats
guesses it has already played, invents words (trape, slith, postt, boomy),
and contradicts its own feedback — consistency 0.09 to 0.17. That is the
published failure taxonomy showing up in our own data on the first run, and it
is why `consistency` is a reward component rather than a footnote.

A capture failure is recorded as a turn with a null reply, never dropped. A
capture that silently discarded failed turns would be reporting a better model
than the one that ran.

CI gates both halves and four things that fail silently in production: the word
lists must rebuild byte-identically, the prerendered routes must carry their own
baked og tags (crawlers do not run JS, so without them every shared link
previews as the homepage), no blob: URL may reach the bundle (the site's CSP has
no worker-src, so it falls back to default-src 'self' and a blob worker is
blocked with no error), and the conformance digest must match across languages.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019mt6sHQHEnEYrJZvoMCJSB
This commit is contained in:
karti-ai
2026-08-28 15:47:31 -07:00
parent 69607fbfe9
commit 408ce4a525
43 changed files with 5279 additions and 136 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+144
View File
@@ -0,0 +1,144 @@
# PIG-Demo
Interactive demos of reinforcement-learning **environments**, built for people
who sign the budget rather than write the training loop.
**https://demo.primeintellectgrowth.com**
An environment is four things: a task, a set of legal moves, a grader that
cannot be argued with, and a score that moves. Every demo here renders those
four boxes, plays a real recorded rollout against them, and then lets you
change what "good" means and watch the ranking flip.
Sibling project to [PIG](https://primeintellectgrowth.com).
---
## Why a word game first
Because it is **Prime Intellect's own hello-world**, and that is checkable in
three public repositories in about thirty seconds:
- one of five `basic` examples in [prime-rl](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/basic/wordle)
- an environment in [verifiers](https://github.com/PrimeIntellect-ai/verifiers/tree/main/environments/wordle)
- the environment used in the official [lab-cookbook](https://github.com/PrimeIntellect-ai/lab-cookbook) GEPA prompt-optimisation tutorial
We didn't pick a game. We picked theirs.
It also needs zero domain knowledge, which means an executive learns the
*machine* — task, moves, grader, score — before any vertical vocabulary gets in
the way. Every demo after it renders the same four boxes with a different
grader.
## What is actually in here
```
envs/wordle_five/ A real environment: engine, reward, reference solver, tests
envs/probe.py The ladder that proves the reward measures something
src/demos/wordle/ The browser half — a port of the same engine
public/traces/wordle/ Recorded rollouts, committed. The evidence.
```
The site does **no live inference**. Rollouts are captured once against a real
model and replayed at their recorded wall-clock. That is a deliberate choice: a
public demo with no auth cannot hold an API key, a live call is slow and flaky
on conference wifi, and a recorded run can be scrubbed, permalinked,
blind-compared and *verified* in ways a live one cannot.
What keeps it from being a video is that the browser re-derives every number it
shows. It re-runs the recorded moves through its own copy of the engine,
rescores them, and prints the delta against what the environment recorded.
## The three commands that check the claims
```bash
uv sync --all-packages
uv run pytest envs/wordle_five/tests # 25 tests, incl. the duplicate-letter cases
uv run python envs/probe.py # the reward ladder, 7 hard assertions
pnpm install && pnpm check && pnpm build
```
### The cross-language gate
`engine.py` and `engine.ts` must agree on every tile. CI scores **all 4,603² =
21.2 million** (guess, answer) pairs through both and compares a SHA-256:
```
69f4e8dfbdc492176d7b7f04b080d8d3cccb5f18aca2d592e622938a5aeec8e2
```
A hand-picked vector file only ever catches the cases somebody thought of. This
catches all of them, and it is what lets the page claim it *verified* a run.
The subtlety it protects is the duplicate-letter rule. Scoring runs in two
passes — every green in the word is resolved before any yellow is assigned —
because a letter may be marked non-grey at most as many times as it occurs in
the answer, and greens have first claim. `SASSY` against `BASIS` is `YGGXX`:
three S's guessed, two available, so one yellow and one grey. Getting this
wrong is the most common bug in implementations of this game, and it is the bug
that put a correction video on the most-watched explanation of it ever made.
## The reward, and why the third component is the point
| component | weight | role | measures |
|---|---|---|---|
| `solved` | 0.50 | objective | did it win |
| `economy` | 0.30 | objective | turns used, as a ratio against the reference player on the same word |
| `consistency` | 0.20 | **counterweight** | share of turns spent on a move that could still have won |
A counterweight has to be in genuine tension with the objective, or it is a
gate wearing a counterweight's name. This one is: a player maximising
information deliberately guesses words that **cannot** win, because a word
splitting the candidates evenly teaches more than a word that might happen to
be right. That is good play, and it costs consistency.
The probe ladder measures the trade rather than asserting it:
```
policy solved economy consistency TOTAL
inaction 0.0000 0.0000 0.0000 0.0000
crude 0.0000 0.0000 0.0556 0.0111
plausible 0.1250 0.1000 0.1493 0.1224
candidate_only 0.9375 0.7458 1.0000 0.8925
exhaustive 1.0000 0.9375 0.6094 0.9031
oracle 1.0000 1.0000 0.7292 0.9458
```
The two good policies are 0.05 apart and **neither dominates the other**. Which
one wins is a decision about what you actually want — which is the whole
argument this site exists to make, and it is why the reward editor on the page
can flip the ranking with one slider. `probe.py` fails CI if either policy
starts dominating, if any weighted component goes flat, or if doing nothing
scores above zero.
## Honesty
- Every rollout is **recorded, not live**, and labelled as such on the page.
- The reward editor **re-scores recorded attempts**. It does not retrain
anything, and it says so where you use it.
- The word list is [our own construction](envs/wordle_five/words/PROVENANCE.md)
from Wordnik (MIT) and SCOWL. Our answer pool is 4,603 — roughly twice the
original game's — so this is **harder** than the original and our numbers are
not comparable to published figures for it. The famous SALET / 3.4212-guess
optimum belongs to that list, not ours, and is cited as such.
- The verticals listed on the site are **our proposals**. They are not Prime
Intellect's roadmap and not a customer list. No customer logos, ever.
- Not affiliated with The New York Times. Independent implementation, own word
lists, own palette, own reward.
Full accounting: [demo.primeintellectgrowth.com/honesty](https://demo.primeintellectgrowth.com/honesty)
## Adding a demo
```bash
pnpm demo:new <slug>
```
Copies the templates and wires nothing — the registry finds demos by existence,
so the header, the gallery, the router and the sitemap all pick it up with zero
edits to shared files. `pnpm check` enforces the contract; see
[CONTRACT.md](CONTRACT.md).
## Licence
Apache-2.0. Third-party attributions in [NOTICE](NOTICE).