karti-ai
5bbf913664
The environment page was a linear scroll of eight narrative beats. That is an essay, and it is the wrong shape for somebody who has just chosen an environment and wants to use it. It is now four tabs — Play, Watch, Reward, Evidence — opening on Play, with the board above the fold at 390x844 and the anatomy strip directly beneath it. The landing page leads with the picker instead of burying it under the thesis. The contract changed rather than layering tabs over beats. `Narrative.beats` is gone; `claims: Record<DemoTabId, string>` replaces it, one required sentence per tab. Writing the claim is how an author discovers whether a tab has anything to say — a tab whose claim is hard to write is usually a tab with nothing in it. Doing this now costs one migration; doing it after eleven more environments costs twelve. Tabs are derived, never declared: Play iff the demo ships an `interactive` mode, Watch iff it has recorded runs. A demo that could name its own tabs would mean environment seven inventing a fifth one and the site ceasing to be one product. One thing the browser caught that no gate would have. The header stat strip describes the RECORDED RUN, and on Play it sat above the visitor's own empty board reading "Outcome: failed" — which parses as your game having already failed before you touch a key. It now renders only on the tabs whose subject is that run, which also moved the board 54px up the page. The picker is honest about the shape of the lineup by construction: one built environment gets its own block and the demo's real board as its thumbnail, twelve written specifications render dimmed with a Spec badge, and every count on the page is derived from the data rather than typed. 206 contract checks pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019mt6sHQHEnEYrJZvoMCJSB
154 lines
6.9 KiB
Markdown
154 lines
6.9 KiB
Markdown
# PIG-Demo
|
|
|
|
Interactive demos of reinforcement-learning **environments**, built for people
|
|
who sign the budget rather than write the training loop.
|
|
|
|
**https://demo.primeintellectgrowth.com**
|
|
|
|
An environment is four things: a task, a set of legal moves, a grader that
|
|
cannot be argued with, and a score that moves. Every demo here renders those
|
|
four boxes, plays a real recorded rollout against them, and then lets you
|
|
change what "good" means and watch the ranking flip.
|
|
|
|
Sibling project to [PIG](https://primeintellectgrowth.com).
|
|
|
|
---
|
|
|
|
## Why a word game first
|
|
|
|
Because it is **Prime Intellect's own hello-world**, and that is checkable in
|
|
three public repositories in about thirty seconds:
|
|
|
|
- one of five `basic` examples in [prime-rl](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/basic/wordle)
|
|
- an environment in [verifiers](https://github.com/PrimeIntellect-ai/verifiers/tree/main/environments/wordle)
|
|
- the environment used in the official [lab-cookbook](https://github.com/PrimeIntellect-ai/lab-cookbook) GEPA prompt-optimisation tutorial
|
|
|
|
We didn't pick a game. We picked theirs.
|
|
|
|
It also needs zero domain knowledge, which means an executive learns the
|
|
*machine* — task, moves, grader, score — before any vertical vocabulary gets in
|
|
the way. Every demo after it renders the same four boxes with a different
|
|
grader.
|
|
|
|
## What is actually in here
|
|
|
|
```
|
|
envs/wordle_five/ A real environment: engine, reward, reference solver, tests
|
|
envs/probe.py The ladder that proves the reward measures something
|
|
src/demos/wordle/ The browser half — a port of the same engine
|
|
public/traces/wordle/ Recorded rollouts, committed. The evidence.
|
|
```
|
|
|
|
The site does **no live inference**. Rollouts are captured once against a real
|
|
model and replayed at their recorded wall-clock. That is a deliberate choice: a
|
|
public demo with no auth cannot hold an API key, a live call is slow and flaky
|
|
on conference wifi, and a recorded run can be scrubbed, permalinked,
|
|
blind-compared and *verified* in ways a live one cannot.
|
|
|
|
What keeps it from being a video is that the browser re-derives every number it
|
|
shows. It re-runs the recorded moves through its own copy of the engine,
|
|
rescores them, and prints the delta against what the environment recorded.
|
|
|
|
## The three commands that check the claims
|
|
|
|
```bash
|
|
uv sync --all-packages
|
|
uv run pytest envs/wordle_five/tests # 25 tests, incl. the duplicate-letter cases
|
|
uv run python envs/probe.py # the reward ladder, 7 hard assertions
|
|
pnpm install && pnpm check && pnpm build
|
|
```
|
|
|
|
### The cross-language gate
|
|
|
|
`engine.py` and `engine.ts` must agree on every tile. CI scores **all 4,603² =
|
|
21.2 million** (guess, answer) pairs through both and compares a SHA-256:
|
|
|
|
```
|
|
69f4e8dfbdc492176d7b7f04b080d8d3cccb5f18aca2d592e622938a5aeec8e2
|
|
```
|
|
|
|
A hand-picked vector file only ever catches the cases somebody thought of. This
|
|
catches all of them, and it is what lets the page claim it *verified* a run.
|
|
|
|
The subtlety it protects is the duplicate-letter rule. Scoring runs in two
|
|
passes — every green in the word is resolved before any yellow is assigned —
|
|
because a letter may be marked non-grey at most as many times as it occurs in
|
|
the answer, and greens have first claim. `SASSY` against `BASIS` is `YGGXX`:
|
|
three S's guessed, two available, so one yellow and one grey. Getting this
|
|
wrong is the most common bug in implementations of this game, and it is the bug
|
|
that put a correction video on the most-watched explanation of it ever made.
|
|
|
|
## The reward, and why the third component is the point
|
|
|
|
| component | weight | role | measures |
|
|
|---|---|---|---|
|
|
| `solved` | 0.50 | objective | did it win |
|
|
| `economy` | 0.30 | objective | turns used, as a ratio against the reference player on the same word |
|
|
| `consistency` | 0.20 | **counterweight** | share of turns spent on a move that could still have won |
|
|
|
|
A counterweight has to be in genuine tension with the objective, or it is a
|
|
gate wearing a counterweight's name. This one is: a player maximising
|
|
information deliberately guesses words that **cannot** win, because a word
|
|
splitting the candidates evenly teaches more than a word that might happen to
|
|
be right. That is good play, and it costs consistency.
|
|
|
|
The probe ladder measures the trade rather than asserting it:
|
|
|
|
```
|
|
policy solved economy consistency TOTAL
|
|
inaction 0.0000 0.0000 0.0000 0.0000
|
|
crude 0.0000 0.0000 0.0556 0.0111
|
|
plausible 0.1250 0.1000 0.1493 0.1224
|
|
candidate_only 0.9375 0.7458 1.0000 0.8925
|
|
exhaustive 1.0000 0.9375 0.6094 0.9031
|
|
oracle 1.0000 1.0000 0.7292 0.9458
|
|
```
|
|
|
|
The two good policies are 0.05 apart and **neither dominates the other**. Which
|
|
one wins is a decision about what you actually want — which is the whole
|
|
argument this site exists to make, and it is why the reward editor on the page
|
|
can flip the ranking with one slider. `probe.py` fails CI if either policy
|
|
starts dominating, if any weighted component goes flat, or if doing nothing
|
|
scores above zero.
|
|
|
|
## Honesty
|
|
|
|
- Every rollout is **recorded, not live**, and labelled as such on the page.
|
|
- The reward editor **re-scores recorded attempts**. It does not retrain
|
|
anything, and it says so where you use it.
|
|
- The word list is [our own construction](envs/wordle_five/words/PROVENANCE.md)
|
|
from Wordnik (MIT) and SCOWL. Our answer pool is 4,603 — roughly twice the
|
|
original game's — so this is **harder** than the original and our numbers are
|
|
not comparable to published figures for it. The famous SALET / 3.4212-guess
|
|
optimum belongs to that list, not ours, and is cited as such.
|
|
- The verticals listed on the site are **our proposals**. They are not Prime
|
|
Intellect's roadmap and not a customer list. No customer logos, ever.
|
|
- Not affiliated with The New York Times. Independent implementation, own word
|
|
lists, own palette, own reward.
|
|
|
|
Full accounting: [demo.primeintellectgrowth.com/honesty](https://demo.primeintellectgrowth.com/honesty)
|
|
|
|
Also served at [demo.lumbridgecorp.com](https://demo.lumbridgecorp.com); the
|
|
canonical name is the first one and every page says so in its `<link rel=canonical>`.
|
|
|
|
## Adding a demo
|
|
|
|
```bash
|
|
pnpm demo:new <slug>
|
|
```
|
|
|
|
The full recipe — specification, environment, port, capture, page — is in
|
|
`.claude/skills/new-environment/SKILL.md`, and
|
|
`.claude/workflows/new-environment.js` runs it as a pipeline with adversarial
|
|
review between the stages. The specifications for the eleven environments not
|
|
yet built are already written, in `src/content/verticals.ts`.
|
|
|
|
Copies the templates and wires nothing — the registry finds demos by existence,
|
|
so the header, the gallery, the router and the sitemap all pick it up with zero
|
|
edits to shared files. `pnpm check` enforces the contract; see
|
|
[CONTRACT.md](CONTRACT.md).
|
|
|
|
## Licence
|
|
|
|
Apache-2.0. Third-party attributions in [NOTICE](NOTICE).
|