karti-ai
ba9d7c874e
The site is demo.primeintellectgrowth.com and nothing else. The Caddy block, the deploy smoke test, the mirrored Caddyfile and the docs all carried the second name; all four no longer do.
151 lines
6.7 KiB
Markdown
151 lines
6.7 KiB
Markdown
# PIG-Demo
|
|
|
|
Interactive demos of reinforcement-learning **environments**, built for people
|
|
who sign the budget rather than write the training loop.
|
|
|
|
**https://demo.primeintellectgrowth.com**
|
|
|
|
An environment is four things: a task, a set of legal moves, a grader that
|
|
cannot be argued with, and a score that moves. Every demo here renders those
|
|
four boxes, plays a real recorded rollout against them, and then lets you
|
|
change what "good" means and watch the ranking flip.
|
|
|
|
Sibling project to [PIG](https://primeintellectgrowth.com).
|
|
|
|
---
|
|
|
|
## Why a word game first
|
|
|
|
Because it is **Prime Intellect's own hello-world**, and that is checkable in
|
|
three public repositories in about thirty seconds:
|
|
|
|
- one of five `basic` examples in [prime-rl](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/basic/wordle)
|
|
- an environment in [verifiers](https://github.com/PrimeIntellect-ai/verifiers/tree/main/environments/wordle)
|
|
- the environment used in the official [lab-cookbook](https://github.com/PrimeIntellect-ai/lab-cookbook) GEPA prompt-optimisation tutorial
|
|
|
|
We didn't pick a game. We picked theirs.
|
|
|
|
It also needs zero domain knowledge, which means an executive learns the
|
|
*machine* — task, moves, grader, score — before any vertical vocabulary gets in
|
|
the way. Every demo after it renders the same four boxes with a different
|
|
grader.
|
|
|
|
## What is actually in here
|
|
|
|
```
|
|
envs/wordle_five/ A real environment: engine, reward, reference solver, tests
|
|
envs/probe.py The ladder that proves the reward measures something
|
|
src/demos/wordle/ The browser half — a port of the same engine
|
|
public/traces/wordle/ Recorded rollouts, committed. The evidence.
|
|
```
|
|
|
|
The site does **no live inference**. Rollouts are captured once against a real
|
|
model and replayed at their recorded wall-clock. That is a deliberate choice: a
|
|
public demo with no auth cannot hold an API key, a live call is slow and flaky
|
|
on conference wifi, and a recorded run can be scrubbed, permalinked,
|
|
blind-compared and *verified* in ways a live one cannot.
|
|
|
|
What keeps it from being a video is that the browser re-derives every number it
|
|
shows. It re-runs the recorded moves through its own copy of the engine,
|
|
rescores them, and prints the delta against what the environment recorded.
|
|
|
|
## The three commands that check the claims
|
|
|
|
```bash
|
|
uv sync --all-packages
|
|
uv run pytest envs/wordle_five/tests # 25 tests, incl. the duplicate-letter cases
|
|
uv run python envs/probe.py # the reward ladder, 7 hard assertions
|
|
pnpm install && pnpm check && pnpm build
|
|
```
|
|
|
|
### The cross-language gate
|
|
|
|
`engine.py` and `engine.ts` must agree on every tile. CI scores **all 4,603² =
|
|
21.2 million** (guess, answer) pairs through both and compares a SHA-256:
|
|
|
|
```
|
|
69f4e8dfbdc492176d7b7f04b080d8d3cccb5f18aca2d592e622938a5aeec8e2
|
|
```
|
|
|
|
A hand-picked vector file only ever catches the cases somebody thought of. This
|
|
catches all of them, and it is what lets the page claim it *verified* a run.
|
|
|
|
The subtlety it protects is the duplicate-letter rule. Scoring runs in two
|
|
passes — every green in the word is resolved before any yellow is assigned —
|
|
because a letter may be marked non-grey at most as many times as it occurs in
|
|
the answer, and greens have first claim. `SASSY` against `BASIS` is `YGGXX`:
|
|
three S's guessed, two available, so one yellow and one grey. Getting this
|
|
wrong is the most common bug in implementations of this game, and it is the bug
|
|
that put a correction video on the most-watched explanation of it ever made.
|
|
|
|
## The reward, and why the third component is the point
|
|
|
|
| component | weight | role | measures |
|
|
|---|---|---|---|
|
|
| `solved` | 0.50 | objective | did it win |
|
|
| `economy` | 0.30 | objective | turns used, as a ratio against the reference player on the same word |
|
|
| `consistency` | 0.20 | **counterweight** | share of turns spent on a move that could still have won |
|
|
|
|
A counterweight has to be in genuine tension with the objective, or it is a
|
|
gate wearing a counterweight's name. This one is: a player maximising
|
|
information deliberately guesses words that **cannot** win, because a word
|
|
splitting the candidates evenly teaches more than a word that might happen to
|
|
be right. That is good play, and it costs consistency.
|
|
|
|
The probe ladder measures the trade rather than asserting it:
|
|
|
|
```
|
|
policy solved economy consistency TOTAL
|
|
inaction 0.0000 0.0000 0.0000 0.0000
|
|
crude 0.0000 0.0000 0.0556 0.0111
|
|
plausible 0.1250 0.1000 0.1493 0.1224
|
|
candidate_only 0.9375 0.7458 1.0000 0.8925
|
|
exhaustive 1.0000 0.9375 0.6094 0.9031
|
|
oracle 1.0000 1.0000 0.7292 0.9458
|
|
```
|
|
|
|
The two good policies are 0.05 apart and **neither dominates the other**. Which
|
|
one wins is a decision about what you actually want — which is the whole
|
|
argument this site exists to make, and it is why the reward editor on the page
|
|
can flip the ranking with one slider. `probe.py` fails CI if either policy
|
|
starts dominating, if any weighted component goes flat, or if doing nothing
|
|
scores above zero.
|
|
|
|
## Honesty
|
|
|
|
- Every rollout is **recorded, not live**, and labelled as such on the page.
|
|
- The reward editor **re-scores recorded attempts**. It does not retrain
|
|
anything, and it says so where you use it.
|
|
- The word list is [our own construction](envs/wordle_five/words/PROVENANCE.md)
|
|
from Wordnik (MIT) and SCOWL. Our answer pool is 4,603 — roughly twice the
|
|
original game's — so this is **harder** than the original and our numbers are
|
|
not comparable to published figures for it. The famous SALET / 3.4212-guess
|
|
optimum belongs to that list, not ours, and is cited as such.
|
|
- The verticals listed on the site are **our proposals**. They are not Prime
|
|
Intellect's roadmap and not a customer list. No customer logos, ever.
|
|
- Not affiliated with The New York Times. Independent implementation, own word
|
|
lists, own palette, own reward.
|
|
|
|
Full accounting: [demo.primeintellectgrowth.com/honesty](https://demo.primeintellectgrowth.com/honesty)
|
|
|
|
## Adding a demo
|
|
|
|
```bash
|
|
pnpm demo:new <slug>
|
|
```
|
|
|
|
The full recipe — specification, environment, port, capture, page — is in
|
|
`.claude/skills/new-environment/SKILL.md`, and
|
|
`.claude/workflows/new-environment.js` runs it as a pipeline with adversarial
|
|
review between the stages. The specifications for the eleven environments not
|
|
yet built are already written, in `src/content/verticals.ts`.
|
|
|
|
Copies the templates and wires nothing — the registry finds demos by existence,
|
|
so the header, the gallery, the router and the sitemap all pick it up with zero
|
|
edits to shared files. `pnpm check` enforces the contract; see
|
|
[CONTRACT.md](CONTRACT.md).
|
|
|
|
## Licence
|
|
|
|
Apache-2.0. Third-party attributions in [NOTICE](NOTICE).
|