diff --git a/.claude/skills/new-environment/SKILL.md b/.claude/skills/new-environment/SKILL.md new file mode 100644 index 0000000..64841d1 --- /dev/null +++ b/.claude/skills/new-environment/SKILL.md @@ -0,0 +1,137 @@ +--- +name: new-environment +description: Build a new RL environment demo for this repo, end to end — specification, Python environment, TypeScript port, captured rollouts, and the demo page. Use when adding an environment to demo.primeintellectgrowth.com, promoting a vertical from spec to live, or when asked to "build environment N" / "add the claims demo" / "make the support one real". +--- + +# Building an environment + +The scarce thing here is not code. It is a **defensible environment**: a task +with a grader that computes rather than opines, and a reward that cannot be +maximised by doing the crude thing. Everything in this repository — the probe +ladder, the conformance digest, the contract validator — exists to stop a +plausible-looking fake from shipping. + +Work in this order. Each stage has a gate, and a stage that cannot pass its gate +is telling you the environment is wrong, not that the gate is too strict. + +## 0. The specification already exists + +`src/content/verticals.ts` carries a reviewed entry for every planned +environment: `task`, `reward`, `counterweight`, `persona`, `anxiety`, and a +`rank` (1 is the one to build next). **Start from that entry.** If you are +building something not in that file, add the entry first and have it reviewed +before writing a line of Python — the spec is the cheapest place to discover +that an environment does not work. + +## 1. Write `envs//SPEC.md` before any code + +Concrete answers, not restatements of the vertical entry: + +- **The episode.** Exactly what one instance is, and what ends it. +- **The data.** Where it comes from. If synthetic, say so on the page — never + imply real customer data. If real, name the licence. +- **The action set.** Legal moves, and what happens to a malformed reply. + (It costs a turn and is recorded. It never crashes the rollout.) +- **The grader.** How the score is computed from the transcript, deterministically. +- **The reward components**, with weights summing to 1.0 and a `counterweight`. +- **The probe ladder**: the policies, and what each must score. +- **The held-out slice.** + +⚠️ **If the grader needs an LLM judge, stop.** A judge is the exact thing a +verifiable reward is positioned against, and shipping one here would undercut +every other environment on the site. Either find the computable signal, or ship +the environment as a `spec` with the caveat stated — `legal-playbook-redline` +is in the lineup for precisely this reason and is honest about it. + +**Gate:** the spec is reviewed adversarially before implementation. Ask of it: +could a competent sceptic call this grader subjective? Could the reward be +maxed by doing something the buyer would hate? + +## 2. The Python environment + +`envs//` — `engine.py` (pure, seeded, never raises on bad input), +`protocol.py` (parse/render; a parse failure is a rejected move, not an +exception), `reward.py` (with `# region: pig-demo/reward` markers around the +part the page quotes), a reference policy, and `tests/`. + +Add the policies to `envs/probe.py` and register them for your taskset. + +**Gate:** `uv run python envs/probe.py` passes every assertion. Specifically: +doing nothing scores exactly `0.000`; no weighted component is flat across the +ladder; and **your two good policies do not dominate each other** — if one wins +on every component, your counterweight is decorative and the reward editor on +the page would be theatre. + +Two mistakes already made here, both worth avoiding twice: +- Score the counterweight over turns **spent**, not moves accepted. Otherwise a + policy that acts once and then jams the parser scores a perfect 1.0 on it. +- The efficiency component's denominator is the depth **your shipped reference + policy** reaches, not a true optimum. Grade against an optimum and your own + oracle rung fails its assertion on some seeds. + +## 3. The TypeScript port + +`src/demos//engine.ts` mirrors the Python. Both sides hash the full +cross-product of the task set and the digest is committed and gated in CI. + +⚠️ Never use a language's built-in RNG for seeding. `random.Random(seed)` and +any JS PRNG disagree, so the same seed picks a different instance on each side +and every `?seed=` permalink silently shows a different task than the run it +claims to replay. Use FNV-1a over the decimal seed, as `engine.py` does. + +**Gate:** `pnpm conformance` — the two digests match. + +## 4. Capture + +`uv run python envs/capture.py --arm --seeds 0-7` + +⚠️ **spark-1 serves one model and is single-stream. Capture is a serialised +queue, never parallel.** A thinking-enabled arm can take minutes per seed. +Front-end work does not wait on it — build against `mock.tsx` instead. + +Capture at least: a weak arm, a strong arm, and a generated reference policy. +Include at least one run the agent **loses**; a demo where the agent always wins +teaches nothing about the reward. A run that had anything done to it is +`kind: 'intervened'` and must name the `intervention`, so a sampling change can +never be read as a training result. + +**Gate:** `uv run python envs/verify_fixtures.py` — every fixture replays through +the engine and reproduces its own recorded rewards. + +## 5. The demo module + +`pnpm demo:new ` scaffolds it. Fill in `meta.ts`, `narrative.ts` +(`claims` for all four tabs), `surface.tsx`, `adapter.ts`, `reward.ts`, and +`interactive` if the task can be played by a human. Read `src/demos/wordle/` as +the reference implementation. + +Where the demo needs something the shell does not offer, that is a contract bug +to fix in `demo-kit` or expose as a `SlotRegion` — never an `if (slug === ...)` +in `src/components/demo/`. `check-demos` fails on it. + +**Gate:** `pnpm check && pnpm build`. + +## 6. Ship + +Add the OG card (`node scripts/og.mjs` — x86 only), rebuild, and +`bash deploy/deploy.sh`. Flip the vertical's status so the picker stops +advertising it as a specification. + +## The honesty rules, which are not negotiable + +- Rollouts are **recorded, not live**, and the page says so. +- The reward editor **re-scores recorded attempts**; it does not retrain. +- Synthetic data is labelled synthetic, on the page, not in a footnote. +- No customer logos and no named companies as customers, ever. +- Cite someone else's number as theirs, with a link, and never re-quote a figure + across a version change without re-measuring it. +- If the environment is materially easier or harder than the real task it + stands for, say which and by how much. + +## Running it as a pipeline + +`.claude/workflows/new-environment.js` runs stages 1–5 as a workflow with +adversarial review between them. Invoke with +`Workflow({ name: 'new-environment', args: { slug, vertical, seeds } })`. +Read this file first regardless — the workflow encodes the sequence, this +encodes the judgement. diff --git a/.claude/workflows/new-environment.js b/.claude/workflows/new-environment.js new file mode 100644 index 0000000..78eef85 --- /dev/null +++ b/.claude/workflows/new-environment.js @@ -0,0 +1,182 @@ +export const meta = { + name: 'new-environment', + description: 'Promote a vertical from published spec to a live environment: specification, Python env, TypeScript port, rollouts, demo page', + whenToUse: 'Adding an environment to demo.primeintellectgrowth.com. Pass { slug, vertical, seeds }.', + phases: [ + { title: 'Specify', detail: 'design the environment, then attack the design' }, + { title: 'Implement', detail: 'Python environment and its probe ladder' }, + { title: 'Port', detail: 'TypeScript engine and the demo module, in parallel' }, + { title: 'Capture', detail: 'real rollouts — serialised, spark-1 is single-stream' }, + { title: 'Verify', detail: 'adversarial review across four lenses' }, + ], +} + +const REPO = '/home/kartios/repos/gitea/PIG-Demo' + +const slug = (args && args.slug) || 'support-resolution' +const vertical = (args && args.vertical) || 'customer-support-resolution' +const pkg = slug.replace(/-/g, '_') +const seeds = (args && args.seeds) || '0-7' + +const BASE = `You are building environment "${slug}" for the repo at ${REPO}, promoting the vertical "${vertical}" from a published specification to a live, playable demo at demo.primeintellectgrowth.com. + +**Read \`.claude/skills/new-environment/SKILL.md\` first, in full.** It carries the judgement; this prompt carries only your stage. Also read \`CONTRACT.md\`, \`AGENTS.md\`, and the reference implementation in \`envs/wordle_five/\` and \`src/demos/wordle/\`. + +The specification for this vertical — task, reward, counterweight, persona, anxiety — is already written and reviewed in \`src/content/verticals.ts\`. Start from that entry; do not invent a different task. + +Package: \`envs/${pkg}/\`. Demo: \`src/demos/${slug}/\`. Traces: \`public/traces/${slug}/\`. + +Do NOT run git. Do NOT edit another stage's files.` + +const SPEC_SCHEMA = { + type: 'object', additionalProperties: false, + required: ['episode', 'data', 'actions', 'grader', 'components', 'ladder', 'heldOut', 'risks'], + properties: { + episode: { type: 'string' }, + data: { type: 'string', description: 'Source, synthetic-or-real, licence' }, + actions: { type: 'string' }, + grader: { type: 'string', description: 'How the score is computed, deterministically' }, + components: { type: 'array', items: { type: 'object', additionalProperties: false, + required: ['key','label','weight','role','definition'], + properties: { key:{type:'string'}, label:{type:'string'}, weight:{type:'number'}, + role:{type:'string', enum:['objective','counterweight','gate']}, definition:{type:'string'} } } }, + ladder: { type: 'array', items: { type: 'object', additionalProperties: false, + required: ['policy','behaviour','expected'], + properties: { policy:{type:'string'}, behaviour:{type:'string'}, expected:{type:'string'} } } }, + heldOut: { type: 'string' }, + risks: { type: 'array', items: { type: 'string' } }, + }, +} + +phase('Specify') +const spec = await agent(`${BASE} + +## YOUR STAGE: the specification. No code. + +Design the environment concretely and write \`envs/${pkg}/SPEC.md\`. Answer the seven questions in the skill's stage 1 with specifics, not restatements of the vertical entry. + +The two decisions that make or break this: + +**The grader must compute, not opine.** If you find yourself reaching for an LLM judge, you have not found the environment yet. Look for the signal that already exists in the business: a downstream system's response, a settled number, a later observed outcome. If there genuinely is not one, say so plainly in \`risks\` — shipping it as a spec with the caveat stated is a legitimate outcome and better than faking it. + +**The counterweight must be in genuine tension with the objective.** Name the crude policy that maxes the objective, and the component that stops it. Then name a good policy that pays the counterweight's price — if you cannot, the counterweight is a gate wearing the wrong label. + +Also decide the data honestly. Synthetic is fine and often the only option; it must be labelled synthetic on the page, and it must not imply real customer records. + +Return the structured spec AND write SPEC.md.`, { label: 'spec', phase: 'Specify', schema: SPEC_SCHEMA, effort: 'high' }) + +const specText = spec ? JSON.stringify(spec, null, 2) : '(the spec stage returned nothing)' + +const LENSES = [ + 'DETERMINISM: is this grader really computable from the transcript, every time, with no human or model judgement anywhere in it? Hunt for a step that quietly needs an opinion.', + 'GAMEABILITY: find the policy that maximises this reward while doing something the buyer would be appalled by. If you find one, the counterweight is insufficient. Try at least four distinct cheats.', + 'HONESTY: would an executive be misled by this? Check the data provenance, whether the task is materially easier than the real job, and whether anything here implies a customer, a result, or a capability we have not got.', +] + +const REVIEW_SCHEMA = { + type: 'object', additionalProperties: false, + required: ['verdict', 'findings'], + properties: { + verdict: { type: 'string', enum: ['sound', 'fixable', 'fatal'] }, + findings: { type: 'array', items: { type: 'object', additionalProperties: false, + required: ['problem','fix'], properties: { problem:{type:'string'}, fix:{type:'string'} } } }, + }, +} + +const reviews = (await parallel(LENSES.map((lens, i) => () => + agent(`Attack this environment specification through ONE lens. Be adversarial: your job is to find the reason it should not ship, not to approve it. + +LENS: ${lens} + +${BASE} + +=== THE SPECIFICATION === +${specText}`, { label: `attack:${i}`, phase: 'Specify', schema: REVIEW_SCHEMA, effort: 'high' }) +))).filter(Boolean) + +const reviewText = reviews.map((r, i) => `--- LENS ${i} — verdict: ${r.verdict}\n${r.findings.map(f => ` ! ${f.problem}\n -> ${f.fix}`).join('\n')}`).join('\n\n') + +if (reviews.some(r => r.verdict === 'fatal')) { + log('A reviewer called the specification fatal. Stopping before implementation.') + return { stopped: 'fatal specification', spec, reviews } +} + +phase('Implement') +const python = await agent(`${BASE} + +## YOUR STAGE: the Python environment. + +Build \`envs/${pkg}/\` — engine, protocol, reward (with \`# region: pig-demo/reward\` markers), a reference policy, and tests. Register your probe policies in \`envs/probe.py\` for taskset "${slug}". + +Implement the specification below, INCLUDING every fix the reviewers demanded. Where a reviewer found a gameable policy, add it to the probe ladder as a rung that must score below the good policies — a cheat you have named and measured is a cheat that cannot come back. + +**Your gate:** \`uv run python envs/probe.py\` passes every assertion, including that your two good policies do not dominate each other. Iterate until it does. If it cannot pass, the reward is wrong — fix the reward, not the assertion. + +=== SPECIFICATION === +${specText} + +=== WHAT THE REVIEWERS DEMANDED === +${reviewText}`, { label: 'python', phase: 'Implement', effort: 'high' }) + +phase('Port') +const ported = (await parallel([ + () => agent(`${BASE} + +## YOUR STAGE: the TypeScript engine. + +Port \`envs/${pkg}/\` 's scorer to \`src/demos/${slug}/engine.ts\`, and extend \`scripts/conformance.mjs\` so this environment's digest is gated the way wordle's is. Seed with FNV-1a over the decimal seed — never a built-in RNG, or the same seed picks a different task on each side and every permalink lies. + +**Your gate:** \`pnpm conformance\` — the two digests match. + +The Python that landed: +${python}`, { label: 'port:ts', phase: 'Port', effort: 'high' }), + + () => agent(`${BASE} + +## YOUR STAGE: the demo module. + +Build \`src/demos/${slug}/\` — \`meta.ts\`, \`narrative.ts\` (\`claims\` for all four tabs: play, watch, reward, evidence), \`surface.tsx\`, \`adapter.ts\`, \`reward.ts\`, and \`interactive\` if a human can play this task. Read \`src/demos/wordle/\` as the reference. + +Build against \`src/components/demo/mock.tsx\` — do NOT wait for real rollouts, that is the whole point of the wire format. + +Anything the demo needs that the shell does not offer is a contract bug to fix in \`demo-kit\` or expose as a \`SlotRegion\`, never a slug check in \`src/components/demo/\`. + +**Your gate:** \`pnpm check\` and \`./node_modules/.bin/tsc --noEmit\` clean for your files. + +The specification: +${specText}`, { label: 'port:demo', phase: 'Port', effort: 'high' }), +])).filter(Boolean) + +phase('Capture') +const captured = await agent(`${BASE} + +## YOUR STAGE: real rollouts. + +Extend \`envs/capture.py\` for taskset "${slug}" and capture seeds ${seeds} for at least three arms: a weak one, a strong one, and a generated reference policy. + +⚠️ **spark-1 serves one model and is single-stream — run the arms one after another, never concurrently.** Check it is up first: \`curl http://100.127.247.67:8001/v1/models\`. A thinking-enabled arm can take minutes per seed; run it in the background, redirect output to a FILE (piping to head/tail block-buffers Python's stdout and a working run looks hung), and poll the written fixtures rather than the log. + +Include at least one run the agent loses. Then \`uv run python envs/build_manifest.py\`. + +**Your gate:** \`uv run python envs/verify_fixtures.py\` — every fixture replays and reproduces its own recorded rewards at delta 0. + +If spark-1 is unreachable, do NOT invent fixtures. Say so and stop; a fabricated rollout on this site would be worse than a missing one.`, { label: 'capture', phase: 'Capture', effort: 'high' }) + +phase('Verify') +const CHECKS = [ + 'Run the full gate chain and make it green: uv run python envs/probe.py, uv run pytest, uv run python envs/verify_fixtures.py, pnpm check, pnpm conformance, pnpm build. Fix whatever is broken.', + 'Open the new demo in a real browser at 1280px and 390px (playwright is installed; serve dist). Confirm: it appears in the landing picker, all four tabs render, Play works if it has an interactive mode, ?tab= round-trips, zero console errors, no horizontal overflow at 390px.', + 'Audit honesty against the rules in the skill: is synthetic data labelled synthetic ON THE PAGE, are recorded runs labelled recorded, does any claim imply a customer or a result we do not have, and does the page state how this environment differs from the real job? Fix the copy where it does not hold.', +] + +const verified = (await parallel(CHECKS.map((c, i) => () => + agent(`${BASE}\n\n## YOUR STAGE: verification.\n\n${c}\n\nReport precisely what you ran, what failed, and what you changed. Do not report success you did not observe.`, + { label: `verify:${i}`, phase: 'Verify', effort: 'high' }) +))).filter(Boolean) + +return { + slug, vertical, + spec, reviews: reviewText, + python, ported, captured, + verified, +} diff --git a/AGENTS.md b/AGENTS.md index 46b0c8c..efff6b5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,5 +1,18 @@ # Working in this repository +## Adding an environment + +Read **`.claude/skills/new-environment/SKILL.md`**. It is the recipe and, more +importantly, the judgement: which gate means "iterate" and which means "this +environment does not work". The pipeline that runs it is +`.claude/workflows/new-environment.js`. + +The short version: the specification comes from `src/content/verticals.ts` +(`rank: 1` is the one to build next), it gets written up and attacked before any +code, the probe ladder is what proves the reward measures something, and the +conformance digest is what proves the browser and the environment are playing +the same game. + ## The three commands ```bash diff --git a/README.md b/README.md index e51e632..8d4d55d 100644 --- a/README.md +++ b/README.md @@ -137,6 +137,12 @@ canonical name is the first one and every page says so in its ` ``` +The full recipe — specification, environment, port, capture, page — is in +`.claude/skills/new-environment/SKILL.md`, and +`.claude/workflows/new-environment.js` runs it as a pipeline with adversarial +review between the stages. The specifications for the eleven environments not +yet built are already written, in `src/content/verticals.ts`. + Copies the templates and wires nothing — the registry finds demos by existence, so the header, the gallery, the router and the sitemap all pick it up with zero edits to shared files. `pnpm check` enforces the contract; see diff --git a/public/og/demos.png b/public/og/demos.png index f0f5a83..664de01 100644 Binary files a/public/og/demos.png and b/public/og/demos.png differ diff --git a/public/og/gallery.png b/public/og/gallery.png index f0f5a83..664de01 100644 Binary files a/public/og/gallery.png and b/public/og/gallery.png differ diff --git a/public/og/home.png b/public/og/home.png index ee8b6db..220261c 100644 Binary files a/public/og/home.png and b/public/og/home.png differ diff --git a/public/og/wordle.png b/public/og/wordle.png index 759d684..c4fc88b 100644 Binary files a/public/og/wordle.png and b/public/og/wordle.png differ diff --git a/scripts/check-demos.mjs b/scripts/check-demos.mjs index a961a92..9141a97 100644 --- a/scripts/check-demos.mjs +++ b/scripts/check-demos.mjs @@ -6,10 +6,11 @@ * shapes; this enforces everything a type cannot: that the directory name * matches the slug, that a `spec` demo is a real published specification rather * than a coming-soon card, that a `live` demo's traces exist on disk, that the - * shared shell has no idea any particular demo exists, and that the reward has - * something pulling against its objective. + * shared shell has no idea any particular demo exists, that every tab on the + * page asserts something a reader can check, and that the reward has something + * pulling against its objective. * - * Thirteen numbered rules, each reported with the file to open. Run it with + * Fourteen numbered rules, each reported with the file to open. Run it with * `pnpm check`. * * Most rules are checked by READING the TypeScript, not by running it — the @@ -34,12 +35,15 @@ import { die, exists, findInDemo, + evalLiteral, isUnresolved, + literalAfter, loadManifest, loadMeta, parseStringUnion, read, rel, + segment, traceFile, walk, } from './_lib.mjs'; @@ -56,9 +60,50 @@ const typesSrc = read(TYPES_FILE); // would drift, and it would drift silently in the direction of passing. const VERTICALS = parseStringUnion(typesSrc, 'Vertical'); const STATUSES = parseStringUnion(typesSrc, 'DemoStatus'); +const TAB_IDS = parseStringUnion(typesSrc, 'DemoTabId'); const ROLES = ['objective', 'counterweight', 'gate']; if (!VERTICALS) die(`could not parse the \`Vertical\` union out of ${rel(TYPES_FILE)}.`); if (!STATUSES) die(`could not parse the \`DemoStatus\` union out of ${rel(TYPES_FILE)}.`); +if (!TAB_IDS) { + die( + `could not parse the \`DemoTabId\` union out of ${rel(TYPES_FILE)}. ` + + 'Rule 3 requires one claim per tab, and the list of tabs lives there.', + ); +} + +/** + * `DEMO_TABS` is the runtime half of `DemoTabId`, and the shell orders the tab + * bar from it. If the two ever disagree, rule 3 would happily pass a demo that + * is missing a claim for a tab the page actually renders — so they are compared + * here, once, before any demo is looked at. + */ +(() => { + let text = null; + try { + text = literalAfter(typesSrc, /(?:export\s+)?const\s+DEMO_TABS\s*(?::\s*[^=]+)?=\s*/, '['); + } catch { + text = null; + } + if (text === null) { + report.fail( + rel(TYPES_FILE), + 'rule 3 (one claim per tab)', + 'no `DEMO_TABS` array literal found. The union says what a tab id is; the array says what order the ' + + 'tabs come in, and the shell reads the array. Both have to exist.', + ); + return; + } + const parsed = evalLiteral(text, 'DEMO_TABS'); + const listed = Array.isArray(parsed.value) ? parsed.value : null; + report.check( + listed !== null && listed.length === TAB_IDS.length && listed.every((id, i) => id === TAB_IDS[i]), + rel(TYPES_FILE), + 'rule 3 (one claim per tab)', + `DEMO_TABS is ${JSON.stringify(listed)} but the DemoTabId union is ${JSON.stringify(TAB_IDS)}. ` + + 'They are the same list written twice; a demo cannot be checked against a contract that disagrees ' + + 'with itself.', + ); +})(); /** * Every icon name lucide-react actually exports, from its own type @@ -97,12 +142,73 @@ const META_FIELDS = { /* --------------------------------------------------------- literal anchors */ const COMPONENT_ANCHORS = [/\bcomponents\s*:\s*/]; +const NARRATIVE_ANCHORS = [ + /(?:export\s+)?const\s+narrative\s*(?::\s*[^=]+)?=\s*/, + /(?:export\s+)?const\s+[A-Za-z_$][\w$]*Narrative\s*(?::\s*[^=]+)?=\s*/, + /\bnarrative\s*:\s*/, +]; const ANATOMY_ANCHORS = [/(?:export\s+)?const\s+anatomy\s*(?::\s*[^=]+)?=\s*/, /\banatomy\s*:\s*/]; const PROVENANCE_ANCHORS = [ /(?:export\s+)?const\s+provenance\s*(?::\s*[^=]+)?=\s*/, /\bprovenance\s*:\s*/, ]; +/* --------------------------------------------------------- rule 3 helpers */ + +/** + * The shortest thing that can still be a claim rather than a heading. + * + * Calibrated against the shortest real one on the site — "Move one slider and + * the winner changes." at 42 characters — with a wide margin, because the point + * is to catch `play: 'Play'`, not to police brevity. + */ +const MIN_CLAIM_CHARS = 20; +const MIN_CLAIM_WORDS = 4; + +/** Lower-cased, punctuation stripped, whitespace collapsed. */ +const normalise = (text) => + text + .toLowerCase() + .replace(/[^a-z0-9]+/g, ' ') + .trim(); + +/** + * Why `claims[tab]` is not a claim, or null if it is one. + * + * A claim is a sentence the tab then has to demonstrate. The three ways one + * arrives broken are: absent, a heading pasted in as prose, and the tab's own + * name — all of which type-check, and all of which render as a page that + * asserts nothing. + */ +function claimProblem(tab, value) { + const label = `narrative.claims.${tab}`; + if (isUnresolved(value)) { + return `${label} is the imported identifier \`${Object.values(value)[0] ?? '?'}\`, not a written-out ` + + 'string. The claims are read on the page beside the surface they describe, so they are written here.'; + } + if (typeof value !== 'string' || value.trim().length === 0) { + return `${label} is ${value === undefined ? 'missing' : JSON.stringify(value)}. Every tab needs one, ` + + 'including a tab this demo may not render — writing the claim is how you find out whether the tab has ' + + 'anything in it.'; + } + const text = value.trim(); + const norm = normalise(text); + // Stripped of the articles and the word "tab", is it just the tab's name? + const bare = norm.replace(/^(?:the|a|an)\s+/, '').replace(/\s+tab$/, ''); + if (bare === tab) { + return `${label} is ${JSON.stringify(text)}, which is the tab's own name. The tab bar already says ` + + 'that. This is the sentence the tab has to earn.'; + } + if (text.length < MIN_CLAIM_CHARS) { + return `${label} is ${text.length} characters (${JSON.stringify(text)}). Under ${MIN_CLAIM_CHARS} it is a ` + + 'heading, not a claim — say what the reader will be able to see for themselves on this tab.'; + } + if (norm.split(' ').filter(Boolean).length < MIN_CLAIM_WORDS) { + return `${label} is ${JSON.stringify(text)}. A claim is a sentence; ${MIN_CLAIM_WORDS} words is the floor.`; + } + return null; +} + /* ------------------------------------------------------------- the checks */ const slugs = demoSlugs(); @@ -142,6 +248,9 @@ for (const slug of slugs) { report.staticOnly(`${slug}: lucide-react is not installed, so \`icon\` was only checked for PascalCase shape.`); } + /* -- 3. the narrative: a thesis, an anxiety, four claims, a limit ------- */ + checkNarrative(slug, report); + /* -- 4. the social card exists ------------------------------------------ */ if (nonEmptyString(meta.ogImage)) { const card = abs('public', String(meta.ogImage).replace(/^\/+/, '')); @@ -347,6 +456,9 @@ for (const slug of slugs) { /* -- 13. every step announces itself ------------------------------------ */ checkAnnounce(slug, report, manifest); + + /* -- 14. the page still renders without the interactive half ------------ */ + checkEagerHalfStandsAlone(slug, report, metaFile); } /* ---------------------------------------- 8. the shell knows no demo names */ @@ -412,6 +524,211 @@ report.finish(); /* ------------------------------------------------------------- helpers */ +/** + * Rule 3: the narrative asserts something, tab by tab. + * + * The page is four tabs, not an essay, so the narrative is no longer a list of + * beats with their own surfaces — the shell decides which tabs exist and in + * what order. What the demo owes is the one sentence each tab has to earn, and + * a claim is the field most likely to be left as a heading, because a heading + * type-checks. + */ +function checkNarrative(slug, report) { + const rule = 'rule 3 (one claim per tab)'; + const dir = path.join(DEMOS_DIR, slug); + // Lenient, because a narrative may legitimately interpolate a shared + // constant; `claimProblem` then reports the stand-in by name rather than + // letting the whole literal fail to evaluate. + const found = findInDemo(slug, NARRATIVE_ANCHORS, '{', 'narrative', { lenient: true }); + + if (found.error) { + report.fail(found.file ?? rel(dir), rule, found.error); + return; + } + const narrative = found.value; + const file = found.file; + if (narrative === null || typeof narrative !== 'object') { + report.fail(file, rule, 'the narrative is not an object literal.'); + return; + } + + report.check( + nonEmptyString(narrative.thesis), + file, + rule, + 'narrative.thesis is empty. It is the paragraph that says why this environment is worth the reader\'s ' + + 'next five minutes, and no tab supplies it.', + ); + report.check( + nonEmptyString(narrative.anxiety), + file, + rule, + 'narrative.anxiety is empty. It is the question already in the reader\'s head when they land; the page ' + + 'is built to answer it and cannot if it is not written down.', + ); + + /* The claims. */ + if (Array.isArray(narrative.beats) || 'beats' in narrative) { + report.fail( + file, + rule, + 'narrative still has `beats`. The page is four tabs now, and which tabs exist is the shell\'s decision, ' + + 'not the demo\'s. Replace `beats` with `claims`: one sentence per tab id.', + ); + } + + const claims = narrative.claims; + if (claims === null || typeof claims !== 'object' || Array.isArray(claims)) { + report.fail( + file, + rule, + `narrative.claims is ${claims === undefined ? 'missing' : JSON.stringify(claims)}. It must be an object ` + + `keyed by tab id: ${TAB_IDS.join(', ')}.`, + ); + return; + } + + for (const tab of TAB_IDS) { + const problem = claimProblem(tab, claims[tab]); + report.check(problem === null, file, rule, problem ?? ''); + } + for (const key of Object.keys(claims)) { + report.check( + TAB_IDS.includes(key), + file, + rule, + `narrative.claims has a key "${key}", which is not a tab. The tabs are ${TAB_IDS.join(', ')}, and they ` + + 'are fixed by the contract — a demo that needs a fifth surface adds it through `tabs`, not here.', + ); + } + + /* The limits. */ + const limits = narrative.limits; + if (!Array.isArray(limits) || limits.length === 0) { + report.fail( + file, + rule, + 'narrative.limits is empty. Every environment teaches something narrower than the thing it is standing ' + + 'in for, and the page that will not say what is the one a reader is right to distrust.', + ); + return; + } + limits.forEach((limit, i) => { + report.check( + nonEmptyString(limit?.text), + file, + rule, + `narrative.limits[${i}].text is empty.`, + ); + }); +} + +/** + * Rule 14: the eager half of a demo does not depend on the interactive half. + * + * Play is a tab now, and it is the tab a demo may not have: a demo with no + * `interactive` opens on Watch, and one with no runs has no Watch tab at all. + * That only works if `meta`, `narrative` and `reward` can be rendered on their + * own. `meta.ts` is where it goes wrong first, because it is loaded EAGERLY for + * every demo on every page — an import of `./demo` there drags one demo's board, + * its controls and its word list into the entry chunk for all of them, and does + * it silently. + * + * Only imports that stay inside the demo's own directory are followed. Reaching + * out of it is rule 9's business, and following `@/lib/demo-kit` would walk into + * the shell's own React and report a failure that belongs to nobody. + */ +function checkEagerHalfStandsAlone(slug, report, metaFile) { + const rule = 'rule 14 (the eager half stands alone)'; + const dir = path.join(DEMOS_DIR, slug); + const entry = abs(metaFile); + if (!exists(entry)) return; + + // Where `interactive` is declared. In practice demo.tsx, but the contract is + // about the value, not the filename. + const interactiveFiles = demoFiles(slug).filter((f) => /\binteractive\s*:/.test(codeOnly(read(f)))); + + const closure = localClosure(entry, dir); + + for (const file of closure) { + if (interactiveFiles.includes(file)) { + report.fail( + rel(entry), + rule, + `reaches ${rel(file)}, which declares \`interactive\`. meta must be renderable with the interactive ` + + 'mode absent — it is what the gallery card, the header and the router are built from, and Play is ' + + 'the one tab a demo is allowed not to have. Move whatever meta needs into a plain data module.', + ); + } else if (/\.tsx$/.test(file)) { + report.fail( + rel(entry), + rule, + `imports ${rel(file)}, a component module. meta is eagerly loaded for EVERY demo on every page, so it ` + + 'stays plain serialisable data: no React, no board, no controls. That is also what keeps Play ' + + 'optional rather than load-bearing.', + ); + } else { + report.passed += 1; + } + } + if (closure.length === 0) report.passed += 1; + + // `interactive` present but nothing to play with is the other half of the + // same contract: the shell would render a Play tab over an empty board. + for (const file of interactiveFiles) { + const code = codeOnly(read(file)); + report.check( + /\binit\s*:/.test(code) && /\bControls\s*:/.test(code), + rel(file), + rule, + 'declares `interactive` without both `init` and `Controls`. The shell shows the Play tab because ' + + '`interactive` exists; a half-declared one is a default tab with nothing in it.', + ); + } +} + +/** Source with comments and string literals blanked out. */ +function codeOnly(src) { + return segment(src) + .map((span) => (span.code ? span.text : ' ')) + .join(''); +} + +/** + * Every file inside `dir` that `entry` imports, transitively. Excludes `entry`. + */ +function localClosure(entry, dir) { + const seen = new Set([entry]); + const queue = [entry]; + while (queue.length > 0) { + const file = queue.pop(); + for (const spec of importSpecifiers(read(file))) { + const resolved = resolveWithin(spec, file, dir); + if (resolved && !seen.has(resolved)) { + seen.add(resolved); + queue.push(resolved); + } + } + } + seen.delete(entry); + return [...seen].sort(); +} + +/** Resolve a specifier to a file under `dir`, or null if it leaves it. */ +function resolveWithin(spec, fromFile, dir) { + const bare = String(spec).split('?')[0]; + let base; + if (bare.startsWith('.')) base = path.resolve(path.dirname(fromFile), bare); + else if (bare.startsWith('@/')) base = abs('src', bare.slice(2)); + else return null; + if (path.relative(dir, base).startsWith('..')) return null; + for (const ext of ['', '.ts', '.tsx', '/index.ts', '/index.tsx']) { + const candidate = base + ext; + if (exists(candidate) && fs.statSync(candidate).isFile()) return candidate; + } + return null; +} + /** Every module specifier a file imports, static, dynamic or side-effect. */ function importSpecifiers(src) { const out = new Set(); diff --git a/src/components/demo/BeatSection.tsx b/src/components/demo/BeatSection.tsx deleted file mode 100644 index d22e10d..0000000 --- a/src/components/demo/BeatSection.tsx +++ /dev/null @@ -1,49 +0,0 @@ -import type { ReactNode } from 'react'; -import type { StoryBeat } from '@/lib/demo-kit/types'; -import { cn } from '@/lib/utils'; - -export interface BeatSectionProps { - beat: StoryBeat; - /** 1-based. The narrative is numbered so a reader can be told "see beat 3". */ - number: number; - children: ReactNode; - className?: string; -} - -/** - * One beat of the exec narrative: a number, a title, a claim, and the surface - * that makes the claim true. - * - * The claim is typeset as an assertion — large, high contrast, above the - * evidence — because the failure mode of a demo site is a visitor watching a - * pretty animation and never learning what it was supposed to prove. - */ -export function BeatSection({ beat, number, children, className }: BeatSectionProps) { - const headingId = `beat-${beat.id}-title`; - return ( -
-
-
- -

- {beat.title} -

-
-

- {beat.claim} -

-
- {children} -
- ); -} diff --git a/src/components/demo/DemoCard.tsx b/src/components/demo/DemoCard.tsx deleted file mode 100644 index 4f2ecc5..0000000 --- a/src/components/demo/DemoCard.tsx +++ /dev/null @@ -1,93 +0,0 @@ -import type { ComponentType } from 'react'; -import { Link } from 'react-router-dom'; -import { ArrowRight } from 'lucide-react'; -import type { DemoMeta } from '@/lib/demo-kit/types'; -import { VERTICAL_LABELS } from '@/lib/demo-kit/registry'; -import { Badge } from '@/components/ui/badge'; -import { DemoIcon } from '@/components/site/DemoIcon'; -import { cn } from '@/lib/utils'; - -export interface DemoCardProps { - meta: DemoMeta; - /** Defaults to the canonical demo route. */ - href?: string; - /** - * The demo's OWN board, drawn compact, as the thumbnail. A screenshot would - * go stale the first time the board changed and nobody would notice; this - * cannot, because it is the same component the demo page renders. - */ - Surface?: ComponentType<{ state: T; compact?: boolean }>; - /** A representative state for the thumbnail — usually a solved board. */ - thumbnailState?: T; - className?: string; -} - -export function DemoCard({ meta, href, Surface, thumbnailState, className }: DemoCardProps) { - const to = href ?? `/demos/${meta.slug}`; - const isSpec = meta.status === 'spec'; - const showSurface = Surface !== undefined && thumbnailState !== undefined; - - return ( -
-
- - - -
-

- {/* Stretched link: the whole card is the hit target, but there is - still exactly ONE link in the accessibility tree for it. */} - - {meta.title} - -

-

{meta.tagline}

-
- {isSpec ? ( - - Spec - - ) : null} -
- - {showSurface ? ( -
- {/* Decorative: the title and tagline already name the demo, and a - board with no run behind it is not information. */} - -
- ) : null} - -
-
-
For
-
{meta.persona}
-
-
-
Vertical
-
{VERTICAL_LABELS[meta.vertical]}
-
-
- -

- Reward: - {meta.rewardLine} -

- -

- {isSpec ? 'Read the specification' : 'Open the demo'} -

-
- ); -} diff --git a/src/components/demo/DemoShell.tsx b/src/components/demo/DemoShell.tsx index 32f5f84..af46c80 100644 --- a/src/components/demo/DemoShell.tsx +++ b/src/components/demo/DemoShell.tsx @@ -7,14 +7,14 @@ import { listRuns, loadEpisode, rewardTotal } from '@/lib/demo-kit/episode'; import { usePlayer } from '@/lib/demo-kit/player'; import { loadDemoModule } from '@/lib/demo-kit/registry'; import type { AnyDemoModule } from '@/lib/demo-kit/registry'; -import type { DemoEpisode, DemoStep, RunRef, StoryBeat } from '@/lib/demo-kit/types'; +import type { DemoEpisode, DemoStep, DemoTabId, RunRef } from '@/lib/demo-kit/types'; import { useRunParam, useSpeedParam, useStepParam, useTabParam } from '@/lib/url-state'; import * as st from '@/content/styles'; import { cn } from '@/lib/utils'; -import { BeatSection } from './BeatSection'; import { BlindCompare } from './BlindCompare'; import { CodeReceipt } from './CodeReceipt'; import { DemoErrorBoundary } from './DemoErrorBoundary'; +import { DemoTabBar, TabClaim, resolveTab, visibleTabs } from './DemoTabs'; import { EnvAnatomy } from './EnvAnatomy'; import { LimitsCallout } from './LimitsCallout'; import { MetricMover } from './MetricMover'; @@ -28,17 +28,24 @@ import { RewardEditor } from './RewardEditor'; import type { RewardArm } from './RewardEditor'; import { SegmentedControl } from './SegmentedControl'; import { SlotRegion } from './SlotRegion'; +import { StepTimeline } from './StepTimeline'; import { StatStrip } from './StatStrip'; import type { Stat } from './StatStrip'; -import { StepTimeline } from './StepTimeline'; import { RecordedBadge, TracePlayer } from './TracePlayer'; import { VerifyBadge } from './VerifyBadge'; import { formatOrDash, useIsDesktop } from './format'; const REPO_BLOB = 'https://git.karti.ai/PIG/PIG-Demo/src/branch/main/'; -/** The tab the step-detail strip opens on. Kept out of the URL when it is this. */ -const DEFAULT_DETAIL_TAB = 'reasoning'; +/** + * The panel the step-detail strip inside the Watch tab opens on. + * + * This control is deliberately NOT in the URL. `?tab=` now belongs to the page's + * four top-level tabs, and one param cannot address two nested controls without + * one of them silently winning; a permalink to `?tab=call` would land the reader + * on a page with no such top-level tab. + */ +const DEFAULT_DETAIL_PANEL = 'reasoning'; /** Reserved slug for the shell's own hand-written demo. Dev builds only. */ const MOCK_SLUG = '__mock'; @@ -96,7 +103,7 @@ export interface DemoShellProps { /** * The route component every demo is rendered through. * - * It owns four things and no more: loading, the narrative beats, the URL state, + * It owns four things and no more: loading, which tabs exist, the URL state, * and the page's single polite live region. Everything visual is delegated to * the surfaces in this directory, and the demo module is never reached into — * the shell only ever calls `adapt` and renders `Surface`. @@ -162,7 +169,6 @@ function DemoBody({ bundle }: { bundle: DemoBundle }) { const [runParam, setRunParam] = useRunParam(); const [stepParam, setStepParam] = useStepParam(); - const [tabParam, setTabParam] = useTabParam(DEFAULT_DETAIL_TAB); const [speedParam, setSpeedParam] = useSpeedParam(); const run = useMemo( @@ -192,11 +198,22 @@ function DemoBody({ bundle }: { bundle: DemoBundle }) { // re-run the effect on the player's own advance and fight it. }, [stepParam, seek]); - const hasBeat = (surface: StoryBeat['surface']) => - demo.narrative.beats.some((beat) => beat.surface === surface); - const timelineInSplit = !hasBeat('scrubber'); - const extras = demo.tabs ?? []; - const extrasInCustomBeat = hasBeat('custom'); + // Derived, never declared. A demo that ships no interactive mode has no Play + // tab and opens on Watch; one whose traces failed to load has no Watch tab + // and opens on Reward. + const hasRecording = Boolean(run && episode && steps.length > 0); + const tabs = useMemo( + () => visibleTabs({ play: Boolean(demo.interactive), watch: hasRecording }), + [demo.interactive, hasRecording], + ); + // `visibleTabs` always keeps `reward` and `evidence`, so index 0 exists; the + // fallback is here only so the type does not need an assertion. + const defaultTab: DemoTabId = tabs[0] ?? 'evidence'; + const [tabParam, setTabParam] = useTabParam(defaultTab); + const activeTab = resolveTab(tabParam, tabs, defaultTab); + + // React-only, not a URL param. See DEFAULT_DETAIL_PANEL. + const [detailPanel, setDetailPanel] = useState(DEFAULT_DETAIL_PANEL); const arms = useMemo( () => @@ -230,54 +247,39 @@ function DemoBody({ bundle }: { bundle: DemoBundle }) { return null; }, [runs, episodes]); - if (!run || !episode || steps.length === 0) { - return ( -
-

{demo.meta.title}

-

- No recorded run is available for this demo yet. The environment and its grader are in - the repository; the traces come from the eval command on the provenance card. -

- -
- ); - } - const Surface = demo.Surface as ComponentType<{ state: unknown; compact?: boolean }>; const current = steps[player.index]; - const lastStep = steps[steps.length - 1]; + const claims = demo.narrative.claims; + const extras = demo.tabs ?? []; - const heroStats: Stat[] = [ - { - label: 'Outcome', - value: episode.outcome, - tone: episode.outcome === 'solved' ? 'positive' : 'warning', - ...(episode.truncated ? { hint: 'Truncated before a terminal state' } : {}), - }, - { - label: 'Total reward', - value: formatOrDash(rewardTotal(episode.rewards, demo.reward.components)), - tone: 'brand', - hint: 'Shipped weights', - }, - { label: 'Steps', value: steps.length, hint: 'Model calls in this run' }, - { label: 'Seed', value: episode.seed, hint: 'The same seed reproduces this board' }, - ]; + // The seed is the shared coordinate between the visitor's board and the + // agent's. With no recording to match, the demo's own first board will do. + const playSeed = run?.seed ?? 0; - const timeline = ( - { - player.pause(); - player.seek(next); - }} - Surface={Surface} - onTogglePlay={player.toggle} - /> - ); + const headerStats: Stat[] = episode + ? [ + { + label: 'Outcome', + value: episode.outcome, + tone: episode.outcome === 'solved' ? 'positive' : 'warning', + ...(episode.truncated ? { title: 'Truncated before a terminal state' } : {}), + }, + { + label: 'Reward', + value: formatOrDash(rewardTotal(episode.rewards, demo.reward.components)), + tone: 'brand', + title: 'Total under the shipped weights', + }, + { label: 'Steps', value: steps.length, title: 'Model calls in this run' }, + { + label: 'Seed', + value: episode.seed, + title: 'The same seed reproduces this board', + }, + ] + : []; - const detailTabs: { id: string; label: string; content: ReactNode }[] = [ + const detailPanels: { id: string; label: string; content: ReactNode }[] = [ { id: 'reasoning', label: 'Reasoning', @@ -319,51 +321,100 @@ function DemoBody({ bundle }: { bundle: DemoBundle }) { ), }, - ...(extrasInCustomBeat - ? [] - : extras.map((tab) => ({ id: tab.id, label: tab.label, content: }))), + // A demo's own extra panels ride alongside the step detail, where they sit + // next to the step they are almost always about. With no recording there is + // no step detail, so the Evidence tab picks them up instead. + ...(hasRecording + ? extras.map((tab) => ({ id: tab.id, label: tab.label, content: })) + : []), ]; - const activeTab = detailTabs.some((tab) => tab.id === tabParam) ? tabParam : DEFAULT_DETAIL_TAB; + const activeDetail = detailPanels.some((panel) => panel.id === detailPanel) + ? detailPanel + : DEFAULT_DETAIL_PANEL; - const renderSurface = (beat: StoryBeat): ReactNode => { - switch (beat.surface) { - case 'hero': - return ( -
-
- {lastStep ? : null} -
-
- - -

{demo.narrative.thesis}

-
+ const timeline = ( + { + player.pause(); + player.seek(next); + }} + Surface={Surface} + onTogglePlay={player.toggle} + /> + ); + + return ( +
+ {/* + The page's ONE live region. Every step change lands here and nowhere + else: with reduced motion the board animation is gone, so this sentence + is the only thing that tells a screen-reader user what just happened. + */} +
+ {current?.announce ?? ''} +
+ +
+

For {demo.meta.persona}

+

{demo.meta.title}

+

{demo.meta.tagline}

+ {/* The buyer's question, kept quiet on purpose: it is the thing they + walked in with, not the thing this page is asserting. */} +

+ {demo.narrative.anxiety} +

+ {/* + The stats describe the RECORDED RUN, so they only belong on the tabs + whose subject is that run. On Play they sat above the visitor's own + empty board reading "Outcome: failed", which parses as *your* game + having already failed before you have touched a key. + */} + {run && headerStats.length > 0 && activeTab !== 'play' ? ( + // One line that scrolls itself rather than a block that wraps: every + // row this header spends is a row of the interactive board pushed + // below the fold on a 390px phone. +
+ +
- ); + ) : null} + +
- case 'anatomy': - return ; + + - case 'split-play': - return ( -
- {demo.interactive ? ( - <> - -
-

What the model did

-

- Same hidden answer, same rules, same budget — replayed at the - speed it actually happened. -

-
- - ) : null} + {tabs.includes('play') ? ( + // `forceMount` keeps the visitor's half-finished board alive while + // they read the other tabs, so a game in progress survives a trip to + // Reward and back. Radix leaves the hiding to the author under + // `forceMount`, which is what the `data-[state=inactive]` class does — + // it is load-bearing, not belt-and-braces. + + {claims.play} + + +
+

The machine you are inside

+ +
+ + ) : null} + + {tabs.includes('watch') && run && episode ? ( + + {claims.watch} {runs.length > 1 ? (
- + - {detailTabs.map((tab) => ( - - {tab.label} + {detailPanels.map((panel) => ( + + {panel.label} ))} - {detailTabs.map((tab) => ( - - {tab.content} + {detailPanels.map((panel) => ( + + {panel.content} ))}
- {timelineInSplit ? timeline : null} - -
- ); - - case 'scrubber': - return ( -
{timeline} {player.timingIsReal ? null : (

@@ -444,42 +487,7 @@ function DemoBody({ bundle }: { bundle: DemoBundle }) {

)} -
- ); - case 'reward-editor': - return ( -
- - - {arms.length > 1 ? : null} -
- ); - - case 'metric': { - const currentTotal = rewardTotal(episode.rewards, demo.reward.components); - const points = arms - .map((arm) => ({ x: arm.label, y: rewardTotal(arm.values, demo.reward.components) })) - .filter((point): point is { x: string; y: number } => point.y !== null); - const baselineArm = arms[0]; - const baselineValue = baselineArm - ? rewardTotal(baselineArm.values, demo.reward.components) - : null; - return ( -
- 1 - ? { baseline: { value: baselineValue, label: baselineArm.label } } - : {})} - series={points} - caption="Every point is a recorded run scored by the same grader. Nothing here is a projection." - /> {blindPair ? ( ) : null} -
- ); - } + + ) : null} + + + {claims.reward} + + {episode ? ( + + ) : ( + <> +

+ No recorded run has been scored for this environment yet, so every term below + reads as not scored rather than as zero. The weights are the ones the environment + ships. +

+ + + )} + + {/* The editor carries the ranking it re-orders in its own right-hand + column, so the two are never on screen apart. */} + {arms.length > 1 ? : null} + + {episode ? : null} + + {run && episode ? ( + + ) : null} + + + + + + {claims.evidence} + + {/* + `narrative.thesis` is a required field of the contract and the only + paragraph on a demo that argues for the environment as a whole + rather than for one tab. It has to be SOMEWHERE, and this is the + tab a visitor opens to read rather than to do — Play stays a board + above the fold, which is the one thing a paragraph here would cost. + */} +

{demo.narrative.thesis}

+ + - case 'receipt': - return (
- +
- ); - case 'limits': - return ; + {!hasRecording && extras.length > 0 ? ( +
+ {extras.map((tab) => ( + + ))} +
+ ) : null} - case 'custom': - return extras.length > 0 ? ( -
- {extras.map((tab) => ( - - ))} -
- ) : ( - ); + + + + + ); +} - default: - return null; - } - }; +/** + * The headline metric, with every recorded arm on the same line. + * + * Arms that were never scored are dropped rather than plotted at zero — the + * difference between "scored badly" and "not scored" is the site's whole + * argument, and a chart is the easiest place in the world to lose it. + */ +function HeadlineMetric({ + label, + arms, + components, + currentRewards, +}: { + label: string; + arms: RewardArm[]; + components: AnyDemoModule['reward']['components']; + currentRewards: DemoEpisode['rewards']; +}) { + const points = arms + .map((arm) => ({ x: arm.label, y: rewardTotal(arm.values, components) })) + .filter((point): point is { x: string; y: number } => point.y !== null); + const baselineArm = arms[0]; + const baselineValue = baselineArm ? rewardTotal(baselineArm.values, components) : null; return ( -
- {/* - The page's ONE live region. Every step change lands here and nowhere - else: with reduced motion the board animation is gone, so this sentence - is the only thing that tells a screen-reader user what just happened. - */} -
- {current?.announce ?? ''} -
- -
-

For {demo.meta.persona}

-

{demo.meta.title}

-

{demo.meta.tagline}

-

- {demo.narrative.anxiety} -

-
- -
- {demo.narrative.beats.map((beat, index) => ( - - {renderSurface(beat)} - - ))} -
-
+ 1 + ? { baseline: { value: baselineValue, label: baselineArm.label } } + : {})} + series={points} + caption="Every point is a recorded run scored by the same grader. Nothing here is a projection." + /> ); } @@ -647,7 +706,6 @@ function RunSwitcher({ ); } - /** * The loading state. Shaped like the page it becomes, and with no spinner: a * spinner here would imply a live model call, which is the one thing the whole @@ -659,12 +717,9 @@ function ShellSkeleton() {

Loading the recorded run.

-
- {[0, 1, 2, 3].map((index) => ( - - ))} -
- + + + ); } diff --git a/src/components/demo/DemoTabs.tsx b/src/components/demo/DemoTabs.tsx new file mode 100644 index 0000000..767633b --- /dev/null +++ b/src/components/demo/DemoTabs.tsx @@ -0,0 +1,126 @@ +/** + * The demo page's tab list, and the rules that decide which tabs exist. + * + * A demo never declares its tabs. The shell derives them from what the demo + * actually has, which is the only way the four labels can mean the same thing + * on every page: `Play` is always the visitor doing the task, `Watch` is always + * a recorded attempt, and the absence of one is a fact about the demo rather + * than an authoring choice someone forgot to make. + */ + +import type { ReactNode } from 'react'; + +import { TabsList, TabsTrigger } from '@/components/ui/tabs'; +import { DEMO_TABS } from '@/lib/demo-kit/types'; +import type { DemoTabId } from '@/lib/demo-kit/types'; +import { cn } from '@/lib/utils'; + +/** What a person reads. One word each, so four of them fit at 390px. */ +export const TAB_LABELS: Readonly> = { + play: 'Play', + watch: 'Watch', + reward: 'Reward', + evidence: 'Evidence', +}; + +/** Read to a screen reader, where a one-word label is not enough context. */ +const TAB_DESCRIPTIONS: Readonly> = { + play: 'Play the environment yourself', + watch: 'Watch a recorded agent attempt', + reward: 'The reward, and what happens when you change it', + evidence: 'The environment, its provenance and its limits', +}; + +export interface TabAvailability { + /** The demo ships an interactive mode. */ + play: boolean; + /** At least one recorded run adapted to at least one step. */ + watch: boolean; +} + +/** + * The tabs this demo has, in contract order. + * + * `reward` and `evidence` are unconditional: a demo with neither an interactive + * mode nor a recorded run is still an environment with a grader you can read, + * and that is the one thing the site refuses to leave out. + */ +export function visibleTabs(has: TabAvailability): DemoTabId[] { + return DEMO_TABS.filter((id) => { + if (id === 'play') return has.play; + if (id === 'watch') return has.watch; + return true; + }); +} + +/** + * Coerce whatever the URL says into a tab that exists. + * + * An unknown or unavailable `?tab=` lands on the default rather than rendering + * an empty page — a stale permalink to `?tab=watch` on a demo whose traces were + * pulled must still show something. + */ +export function resolveTab( + raw: string, + visible: readonly DemoTabId[], + fallback: DemoTabId, +): DemoTabId { + return visible.find((id) => id === raw) ?? fallback; +} + +/** + * The tab strip. + * + * Full width and equal columns below `sm` so four tabs land inside 390px + * without the list becoming a horizontal scroller — a tab bar you have to + * scroll hides the tabs, which is the one thing it exists to advertise. From + * `sm` up it shrinks back to its natural width and sits left. + */ +export function DemoTabBar({ + tabs, + className, +}: { + tabs: readonly DemoTabId[]; + className?: string; +}) { + return ( + + {tabs.map((id) => ( + + {TAB_LABELS[id]} + + ))} + + ); +} + +/** + * The sentence a tab has to earn, typeset as an assertion rather than a + * heading. + * + * The failure mode of a demo site is a visitor watching something pretty and + * never learning what it was supposed to prove. The claim is the first thing in + * every panel for that reason, and it is deliberately not an `

`: the tab + * trigger is already this panel's accessible name, and a heading here would + * make the claim navigable furniture instead of a thing someone reads. + */ +export function TabClaim({ children, className }: { children: ReactNode; className?: string }) { + return ( +

+ {children} +

+ ); +} diff --git a/src/components/demo/StatStrip.tsx b/src/components/demo/StatStrip.tsx index b56c285..2d8d800 100644 --- a/src/components/demo/StatStrip.tsx +++ b/src/components/demo/StatStrip.tsx @@ -5,22 +5,12 @@ import { cn } from '@/lib/utils'; export type StatTone = 'default' | 'positive' | 'warning' | 'danger' | 'info' | 'brand'; export interface Stat { - /** Short. Two or three words; it sits above the number. */ + /** Short. Two or three words; it sits beside the number. */ label: string; value: ReactNode; - /** One clarifying line, shown under the number at a smaller size. */ - hint?: string; tone?: StatTone; - /** Set when this number was derived under an edited reward, not recorded. */ - edited?: boolean; -} - -export interface StatStripProps { - stats: Stat[]; - className?: string; - /** Announce changes as they happen. Off by default — the shell owns the - * page's single live region and two competing ones talk over each other. */ - live?: boolean; + /** One clarifying line. A tooltip here, because the strip is one line high. */ + title?: string; } const TONE: Record = { @@ -33,40 +23,38 @@ const TONE: Record = { }; /** - * A row of headline numbers. Scrolls horizontally on a phone rather than - * wrapping into a ragged grid: four stats reflowing to 2x2 at 390px puts the - * least important number in the most prominent corner. + * The run's headline numbers, one line high. + * + * This used to be a row of cards, which is the right thing in the middle of a + * page and the wrong thing in a header: four stat cards push the interactive + * board below the fold on a 390px phone, and the board arriving above the fold + * is what the whole page is now organised around. So the cards are gone and + * this is the only stat strip — a second, compact variant living beside the + * card version is how the two drift into disagreeing about what a stat looks + * like. + * + * It scrolls itself rather than wrapping: every row this header spends is a row + * of the board pushed down. */ -export function StatStrip({ stats, className, live = false }: StatStripProps) { +export function StatStrip({ stats, className }: { stats: Stat[]; className?: string }) { if (stats.length === 0) return null; return (
{stats.map((stat) => (
-
- {stat.label} - {stat.edited ? : null} -
-
+
{stat.label}
+
{stat.value}
- {stat.hint ?
{stat.hint}
: null}
))}
diff --git a/src/components/demo/mock.tsx b/src/components/demo/mock.tsx index 8b4428d..97c3424 100644 --- a/src/components/demo/mock.tsx +++ b/src/components/demo/mock.tsx @@ -8,7 +8,13 @@ * has something complete to render — including the awkward cases a real trace * eventually produces: a step with no reasoning, a null model call, a * not-scored reward component, and a truncated run. + * + * It ships an `interactive` mode for the same reason. Play is the tab the page + * opens on, so a fixture without one would leave the shell's default tab the + * only surface here with nothing to render against. */ +import { useCallback, useId, useState } from 'react'; + import type { DemoEpisode, DemoModule, @@ -16,6 +22,7 @@ import type { RewardValues, RunRef, } from '@/lib/demo-kit/types'; +import { Button } from '@/components/ui/button'; import { cn } from '@/lib/utils'; export type MarkKind = 'exact' | 'present' | 'absent'; @@ -104,6 +111,112 @@ function MockSurface({ state, compact = false }: { state: MockState; compact?: b ); } +/* ------------------------------------------------------------- interactive */ + +/** + * Answers the playable board can draw. Small on purpose: this is a fixture. + * + * Ordered so that seed 7 — the seed the recorded runs below use — draws the + * same answer they were recorded against, which keeps Play and Watch showing + * the same puzzle while the shell is being built. + */ +const PLAYABLE = ['SLATE', 'PRIZE', 'MOUND', ANSWER] as const; + +/** Deterministic in the seed, as the contract requires. */ +function initMock(seed: number): MockState { + const answer = PLAYABLE[Math.abs(Math.trunc(seed)) % PLAYABLE.length] as string; + return { answer, guesses: [], solved: false }; +} + +const LETTERS = /^[A-Za-z]*$/; + +/** + * The playable controls: type a five-letter word, submit, see it marked. + * + * Deliberately an input rather than an on-screen keyboard. The shell only ever + * sees `onChange(next)`, so the fixture's job is to produce every state a real + * demo's controls can — mid-word, illegal, solved, out of guesses — with as + * little of its own machinery as possible. + */ +function MockControls({ + state, + onChange, + seed, +}: { + state: MockState; + onChange: (next: MockState) => void; + seed: number; +}) { + const [draft, setDraft] = useState(''); + const fieldId = useId(); + const over = state.solved || state.guesses.length >= MAX_GUESSES; + const ready = draft.length === 5 && !over; + + const submit = useCallback(() => { + if (!ready) return; + const word = draft.toUpperCase(); + const guesses = [...state.guesses, { word, marks: mark(word, state.answer) }]; + onChange({ ...state, guesses, solved: word === state.answer }); + setDraft(''); + }, [draft, onChange, ready, state]); + + const status = state.solved + ? `Solved in ${state.guesses.length} ${state.guesses.length === 1 ? 'guess' : 'guesses'}.` + : over + ? `Out of guesses. The word was ${state.answer}.` + : `${MAX_GUESSES - state.guesses.length} guesses left.`; + + return ( +
+
{ + event.preventDefault(); + submit(); + }} + > + + { + const next = event.target.value.slice(0, 5); + if (LETTERS.test(next)) setDraft(next.toUpperCase()); + }} + autoComplete="off" + autoCapitalize="characters" + spellCheck={false} + inputMode="text" + placeholder="GUESS" + className="tap min-w-0 flex-1 rounded-md border border-border bg-surface px-3 font-mono text-base uppercase tracking-[0.3em] text-fg placeholder:tracking-normal placeholder:text-muted focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-brand disabled:opacity-50" + /> + + +
+ {/* The board is an image to a screen reader, so the outcome has to be + said in words somewhere that announces itself. */} +

+ {status} +

+
+ ); +} + const REWARD_SOURCE = `import verifiers as vf @@ -192,15 +305,12 @@ export const mockDemo: DemoModule = { thesis: 'An environment is an eval you can take the gradient of. The task, the legal moves and the grader are all code — so you can change what "good" means and watch the number move.', anxiety: 'Is this a benchmark I read, or a thing I can actually change?', - beats: [ - { id: 'hero', title: 'The run', claim: 'This is a recorded rollout, not a live request.', surface: 'hero' }, - { id: 'anatomy', title: 'The machine', claim: 'Four boxes: task, legal actions, grader, score.', surface: 'anatomy' }, - { id: 'play', title: 'Watch it think', claim: 'Every move has a reason and a cost, both recorded.', surface: 'split-play' }, - { id: 'reward', title: 'Change what good means', claim: 'Move a weight and the ranking moves with it.', surface: 'reward-editor' }, - { id: 'metric', title: 'The number that moves', claim: 'The score is a measurement, not a claim.', surface: 'metric' }, - { id: 'receipt', title: 'The receipts', claim: 'Every number here has a command that reproduces it.', surface: 'receipt' }, - { id: 'limits', title: 'What this does not teach', claim: 'A word game is not your business process.', surface: 'limits' }, - ], + claims: { + play: 'Play a round yourself, because the rest of this page is about what happened when a model played the same one.', + watch: 'This is a recorded attempt, replayed one turn at a time, with the reasoning and the token cost of every turn attached.', + reward: 'Move a single weight and the ranking of the runs re-orders underneath it. That is the whole product.', + evidence: 'The grader is thirty lines of Python, printed here beside the command that ran it and the runs it produced.', + }, limits: [ { text: 'A five-letter word has one right answer. Most business decisions do not, and a grader that pretends otherwise scores confidence rather than correctness.', @@ -239,7 +349,7 @@ export const mockDemo: DemoModule = { { key: 'guesses_used', label: 'Guesses used', description: 'How many of the six were spent.' }, { key: 'unique_letters', label: 'Unique letters tried', description: 'Breadth of the search.' }, ], - source: { path: 'envs/wordle_five/wordle_five/rewards.py', code: REWARD_SOURCE, marker: '--8<-- efficiency' }, + source: { path: 'envs/mock_five/mock_five/rewards.py', code: REWARD_SOURCE, marker: '--8<-- efficiency' }, }, provenance: { envPackage: 'wordle_five', @@ -258,6 +368,7 @@ export const mockDemo: DemoModule = { }, adapt, Surface: MockSurface, + interactive: { init: initMock, Controls: MockControls }, verify, }; diff --git a/src/components/site/EnvironmentPicker.tsx b/src/components/site/EnvironmentPicker.tsx new file mode 100644 index 0000000..ff3ef8b --- /dev/null +++ b/src/components/site/EnvironmentPicker.tsx @@ -0,0 +1,441 @@ +/** + * The environment picker: one card shape, two pages. + * + * Home leads with it and Gallery filters it, so the card lives here rather than + * twice. The list is built at module scope from the registry and the vertical + * lineup — creating `src/demos//` adds a card with no edit to this file, + * which is the same discovery-by-existence property the registry itself has. + * + * The asymmetry is the honest part. Almost everything in the lineup is written + * rather than built, and a wall of identical tiles would imply a wall of + * working demos on a site whose entire argument is that its claims are + * checkable. So a built environment gets its own board, its own block and the + * only filled call to action, and the written ones read as a published roadmap + * underneath it: dimmed, badged, and each linking to a real specification + * rather than to nothing. Both pages count them from the data, never from a + * number typed into the copy. + */ + +import { useEffect, useState, type ReactNode } from 'react'; +import { Link } from 'react-router-dom'; +import { ArrowRight, FileText, Play } from 'lucide-react'; + +import { DemoIcon } from '@/components/site/DemoIcon'; +import { Badge } from '@/components/ui/badge'; +import { Skeleton } from '@/components/ui/skeleton'; +import { lineup, routes } from '@/content/lineup'; +import * as s from '@/content/styles'; +import { + listDemos, + listVerticals, + loadDemoModule, + VERTICAL_LABELS, + VERTICAL_ORDER, +} from '@/lib/demo-kit/registry'; +import type { DemoStatus, Vertical } from '@/lib/demo-kit/types'; +import { cn } from '@/lib/utils'; + +/** One card's worth of environment, whether it is built or only written. */ +export interface EnvironmentEntry { + /** React key. Namespaced, because a demo and a vertical can share a slug. */ + id: string; + title: string; + /** One sentence on what the agent DOES. */ + tagline: string; + /** The job title that owns the budget for it. */ + persona: string; + /** What the reward pays for, and what it takes away. */ + rewardLine: string; + /** A lucide icon name, resolved by `DemoIcon`. */ + icon: string; + status: DemoStatus; + href: string; + /** Exec-facing vertical name. */ + verticalLabel: string | null; + /** Filter identity. `null` matches no filter and shows only under "All". */ + verticalKey: Vertical | null; + /** Set when the entry is a built demo; drives the board thumbnail. */ + slug: string | null; + /** A standing qualifier printed on the card, e.g. "Not in the first set". */ + note: string | null; +} + +/** + * The card wants one sentence. `verticals.ts` writes two or three, because the + * vertical page needs the whole argument. Trimming here rather than adding a + * `tagline` field to the data is what stops a specification and its summary + * drifting apart — there is only ever one copy of the sentence. + */ +function firstSentence(text: string): string { + const trimmed = text.trim(); + const match = /^[\s\S]*?[.!?](?=\s|$)/.exec(trimmed); + const first = match?.[0]?.trim(); + // A very short first fragment means the split found an abbreviation rather + // than a sentence end, and half a clause on a card is worse than three lines. + return first !== undefined && first.length >= 24 ? first : trimmed; +} + +/** Verticals a demo already exists for. Those are shown as the demo, not as a proposal. */ +const builtVerticals = new Set(listVerticals().map((group) => group.vertical)); + +const demoEntries: readonly EnvironmentEntry[] = listDemos().map((meta) => ({ + id: `demo:${meta.slug}`, + title: meta.title, + tagline: meta.tagline, + persona: meta.persona, + rewardLine: meta.rewardLine, + icon: meta.icon, + status: meta.status, + /* + * `?tab=play` is written out even though `DEFAULT_TAB` in + * `src/lib/url-state.ts` is already `play` and the param would be stripped + * from a permalink. The promise this card makes is that one click puts you on + * the board, and that promise should not quietly depend on which tab the + * shell happens to default to next quarter. + */ + href: meta.status === 'live' ? `${routes.demo(meta.slug)}?tab=play` : routes.demo(meta.slug), + verticalLabel: VERTICAL_LABELS[meta.vertical], + verticalKey: meta.vertical, + slug: meta.slug, + note: null, +})); + +const proposalEntries: readonly EnvironmentEntry[] = lineup + .filter((vertical) => vertical.key === null || !builtVerticals.has(vertical.key)) + .map((vertical) => ({ + id: `vertical:${vertical.slug}`, + title: vertical.title, + tagline: firstSentence(vertical.task), + persona: vertical.persona, + rewardLine: firstSentence(vertical.reward), + icon: vertical.icon, + status: 'spec' as const, + // Its own page, with the task, the reward, the counterweight and the + // caveat written out. A dimmed card is only honest if it goes somewhere. + href: routes.vertical(vertical.slug), + verticalLabel: vertical.key === null ? null : VERTICAL_LABELS[vertical.key], + verticalKey: vertical.key, + slug: null, + note: vertical.plannedForV1 ? null : 'Not in the first set', + })); + +/** Everything, built first. `lineup` is already ranked, so proposals keep that order. */ +export const environments: readonly EnvironmentEntry[] = [ + ...demoEntries.filter((entry) => entry.status === 'live'), + ...demoEntries.filter((entry) => entry.status === 'spec'), + ...proposalEntries, +]; + +/** Playable in this tab. */ +export const builtEnvironments: readonly EnvironmentEntry[] = environments.filter( + (entry) => entry.status === 'live', +); + +/** Published as a specification, with no interactive surface yet. */ +export const writtenEnvironments: readonly EnvironmentEntry[] = environments.filter( + (entry) => entry.status === 'spec', +); + +/** + * Filter options, in the contract's own order. + * + * An entry whose `verticalKey` is null — a proposal the contract has no key for + * — is reachable only under "All". That is deliberate: such a proposal is on + * the site as the strongest form of the argument, not as something we would + * build, so it belongs to no filterable industry. + */ +export const filterKeys: readonly Vertical[] = VERTICAL_ORDER.filter((key) => + environments.some((entry) => entry.verticalKey === key), +); + +/* ------------------------------------------------------------------ cards */ + +function StatusBadge({ status }: { status: DemoStatus }) { + if (status === 'live') { + return ( + + Live + + ); + } + return ( + + + ); +} + +/** + * The two lines an executive actually scans for: whose budget this is, and + * what the number pays for. The vertical is not repeated here — both card + * layouts already print it beside the icon. + */ +function MetaRows({ entry }: { entry: EnvironmentEntry }) { + return ( +
+
+
For
+
{entry.persona}
+
+
+
Reward
+
{entry.rewardLine}
+
+
+ ); +} + +/** + * One environment, as a card. + * + * The whole rectangle is the link — one target, one focus ring, no nested + * interactive elements to trap a keyboard in. + */ +export function EnvironmentCard({ + entry, + className, +}: { + entry: EnvironmentEntry; + className?: string; +}) { + const isSpec = entry.status === 'spec'; + return ( + + + + + + + {entry.verticalLabel === null ? null : ( + {entry.verticalLabel} + )} + + + + +

+ {entry.title} +

+

{entry.tagline}

+ + + + {/* `mt-auto` so the call to action sits on the same line in every card of + a row, however long the tagline above it ran. */} + + + {isSpec ? 'Read the specification' : 'Play it'} + + {entry.note === null ? null : {entry.note}} + + + ); +} + +/** + * The board, drawn by the demo's own `Surface`. + * + * A screenshot would go stale the first time the board changed and nobody would + * notice. This cannot: it is the same component the demo page renders, in the + * state `interactive.init` hands a new player. + */ +const THUMBNAIL_SEED = 0; + +function BoardThumbnail({ slug }: { slug: string }) { + const [board, setBoard] = useState(null); + + useEffect(() => { + let live = true; + /* + * Deliberately after paint, and deliberately not budgeted as a picture. + * This pulls the demo's own lazy chunk — the one the visitor is a click + * away from — so the cost is a prefetch of the destination, and the board + * arriving is proof the destination is already loaded. + */ + void loadDemoModule(slug) + .then((module) => { + if (!live || !module.interactive) return; + const { Surface } = module; + setBoard(); + }) + .catch(() => { + // A card without a board is still a complete card. A demo whose chunk + // fails to load has its own error page to say so; this is not it. + }); + return () => { + live = false; + }; + }, [slug]); + + return ( + // Decorative: the title, the tagline and the reward already say everything + // this board says, and an unplayed board is a shape, not information. + + ); +} + +/** + * The built environment, given the weight it has earned. + * + * Wider than a card, with the real board in it and the only filled call to + * action on the page. If a second demo ever ships, this renders for that one + * too and the picker stays honest without an edit. + */ +export function FeaturedEnvironment({ + entry, + className, +}: { + entry: EnvironmentEntry; + className?: string; +}) { + return ( + +
+ {/* + The copy is FIRST in the source and the board is pulled left only from + `lg`. On a phone a full-width board above the title pushes the name of + the environment and its call to action below the fold, which is the + one thing this card exists to avoid. + */} +
+ + + {entry.verticalLabel === null ? null : ( + + + {entry.verticalLabel} + + )} + + +

+ {entry.title} +

+

{entry.tagline}

+ + + +

+ Four tabs. Play the board, watch a recorded model play the same board, move the reward + weights and see the ranking change, then read the Python that scored it. +

+ + + {/* Styled as the button it behaves as. It is not a
+ + {entry.slug === null ? null : ( +
+ +
+ )} +
+ + ); +} + +/* ----------------------------------------------------------------- filter */ + +/** The "no vertical chosen" value. Not a `Vertical`, so it cannot collide with one. */ +export const ALL_VERTICALS = 'all'; + +export type VerticalFilterValue = Vertical | typeof ALL_VERTICALS; + +/** + * Vertical chips. Wraps on a phone rather than scrolling sideways — a chip + * hidden off the right edge is a filter nobody knows exists. + */ +export function VerticalFilter({ + active, + onSelect, + className, +}: { + active: VerticalFilterValue; + onSelect: (next: VerticalFilterValue) => void; + className?: string; +}) { + const options: readonly VerticalFilterValue[] = [ALL_VERTICALS, ...filterKeys]; + return ( +
    + {options.map((key) => { + const selected = key === active; + return ( +
  • + +
    • + ); + })} +
+ ); +} + +/** Cards in a responsive grid. One list, so a screen reader gets the count. */ +export function EnvironmentGrid({ + entries, + className, +}: { + entries: readonly EnvironmentEntry[]; + className?: string; +}) { + return ( +
    + {entries.map((entry) => ( +
  • + +
    • + ))} +
+ ); +} diff --git a/src/demos/_template/demo.tsx b/src/demos/_template/demo.tsx index 52cbdf0..aeb2d5d 100644 --- a/src/demos/_template/demo.tsx +++ b/src/demos/_template/demo.tsx @@ -15,6 +15,10 @@ * · Every step sets a non-empty `announce`. Reduced motion clamps the * animation to nothing, so for a screen-reader user the announcement IS * the result, not a courtesy (rule 13). + * · `meta.ts` never imports this file. The demo page is four tabs and Play + * is the one a demo is allowed not to have, so the eager half — meta, + * narrative, reward — has to render with the interactive half absent + * (rule 14). */ import { defineDemo, type DemoEpisode, type DemoStep, type RewardValues } from '@/lib/demo-kit'; @@ -128,14 +132,53 @@ export default defineDemo<__Pascal__State>({ 'The queue is not the problem. Deciding which three of four hundred items are worth a person is the ' + 'problem, and that decision is exactly the kind of judgement a reward can be written down for.', anxiety: 'What stops it escalating everything so it never misses one?', - beats: [ - { id: 'hero', title: 'One queue, one decision', claim: 'Every item is either worth a person or it is not.', surface: 'hero' }, - { id: 'anatomy', title: 'What the environment is', claim: 'A task, a fixed action set, a grader, and a score that moves.', surface: 'anatomy' }, - { id: 'play', title: 'Watch a recorded run', claim: 'These are recorded turns, not a scripted animation.', surface: 'split-play' }, - { id: 'reward', title: 'Move the weights yourself', claim: 'Pay only for catches and the queue gets escalated whole.', surface: 'reward-editor' }, - { id: 'receipt', title: 'The code that scored it', claim: 'The number on this page came out of the function below it.', surface: 'receipt' }, - { id: 'limits', title: 'What this does not show', claim: 'One queue, one grader, and no cost of being wrong.', surface: 'limits' }, - ], + + /** + * ── THE FOUR CLAIMS ────────────────────────────────────────────────── + * + * The demo page is four tabs — Play, Watch, Reward, Evidence — and this is + * the one sentence each of them has to earn. You do not choose the tabs or + * their order; the shell does, and it drops Play when there is no + * `interactive` and Watch when there are no recorded runs. You only write + * what each one asserts. + * + * Write all four even if this demo will not render all four. A tab whose + * claim you cannot write is a tab with nothing in it, and finding that out + * here is cheaper than finding it out in review. + * + * A good claim is: + * + * · a SENTENCE, not a label. "The reward" is a heading. "Move one weight + * and the ranking of the runs re-orders under it" is a claim. + * · falsifiable BY THE TAB IT SITS ON. The reader should be able to look + * at the surface below it and agree or disagree within a few seconds. + * · about this environment, not about reinforcement learning. The thesis + * above is where the general argument goes. + * · addressed to the person in `meta.persona`, in their words. + * + * The four below are real claims for this worked example. Replace them — + * do not delete the shape. + */ + claims: { + // Play: what the visitor learns by doing the task themselves, and why + // doing it first makes the other three tabs mean something. + play: + 'Work the queue yourself for thirty seconds and you will feel the trade the agent is being scored on: ' + + 'every item you escalate has to be worth someone opening it.', + // Watch: what the recorded run proves that a claim about the run cannot. + watch: + 'This is a real recorded run, replayed one turn at a time — the reasoning, the model call and the cost ' + + 'of each decision are exactly what came off the wire.', + // Reward: what changes on screen when the reader changes what "good" + // means. Name the thing that moves. + reward: + 'Take the weight off restraint and the run that escalated everything climbs to the top of the ranking. ' + + 'That is not a bug in the score; it is the score doing what you asked.', + // Evidence: what the reader can go and check, and what this does not show. + evidence: + 'The grader is a short Python function, printed here with the command that ran it, so you can disagree ' + + 'with the number by reading the code rather than by trusting us.', + }, limits: [ { text: 'The grader knows which items needed a person because the dataset says so. A real queue has no such column, and building one is most of the work.', @@ -213,6 +256,29 @@ export default defineDemo<__Pascal__State>({ adapt, Surface: Board, + + /** + * ── ADDING PLAY ────────────────────────────────────────────────────────── + * + * There is deliberately no `interactive` here, because a scaffolded demo + * starts as `spec`. Add one and the shell grows a Play tab and OPENS ON IT — + * that is the whole shape of the page, so it is worth doing: + * + * interactive: { + * init: (seed: number) => empty__Pascal__(seed), // pure in the seed + * Controls: __Pascal__Controls, // its own module + * }, + * + * `init` must be deterministic in the seed: the shell re-inits on reset and + * on a shared link, and a board that comes back different has quietly told + * the visitor the environment is not reproducible. + * + * `Controls` gets `{ state, onChange, seed }` and nothing else. It owns no + * state the board does not — hand the next board to `onChange` and let the + * shell re-render, or Play and the replay will drift apart. Put it in its own + * file and import it here, never from `meta.ts` (rule 14). + */ + verify: recompute, }); diff --git a/src/demos/wordle/narrative.ts b/src/demos/wordle/narrative.ts index 5af3925..fed87ac 100644 --- a/src/demos/wordle/narrative.ts +++ b/src/demos/wordle/narrative.ts @@ -1,75 +1,30 @@ import type { Narrative } from '@/lib/demo-kit'; /** - * The six beats, in order. The shell renders them; this file decides what the - * page argues and in what sequence. + * What this environment argues, tab by tab. + * + * The page is a set of tabs rather than an essay because an executive who has + * chosen an environment wants to be doing the task, not reading a case for it. + * Each claim is the one sentence that tab has to earn. */ export const narrative: Narrative = { thesis: 'This is the smallest complete reinforcement-learning environment we could find that needs no ' + 'domain knowledge at all. It has everything the ones that matter to your business have: a task, ' + 'a fixed set of legal moves, a grader that cannot be argued with, and a score that moves when ' + - 'the model gets better. Learn the machine here, and every demo after this is the same machine ' + - 'with a different grader.', + 'the model gets better. Learn the machine here, and every environment after this is the same ' + + 'machine with a different grader.', anxiety: 'How would we know it was actually working?', - beats: [ - { - id: 'hero', - title: 'Their hello-world, not ours', - claim: - 'Prime Intellect ship this exact game as a starter environment in three of their public repositories. We did not pick a game. We picked theirs.', - surface: 'hero', - }, - { - id: 'anatomy', - title: 'What an environment actually is', - claim: - 'Four parts: a task, the moves that are legal, a grader that computes rather than opines, and a number that moves.', - surface: 'anatomy', - }, - { - id: 'play', - title: 'You and the model get the same word', - claim: - 'Same hidden word, same six guesses, same rules. Play it, then watch what the model did with it.', - surface: 'split-play', - }, - { - id: 'watch', - title: 'Watch it think', - claim: - 'This is not a video. It is a recorded attempt replayed at the speed it actually happened, and you can step through it one guess at a time.', - surface: 'scrubber', - }, - { - id: 'reward', - title: 'You decide what good means', - claim: - 'Move one slider and the winner changes. That is not a trick — it is the product.', - surface: 'reward-editor', - }, - { - id: 'metric', - title: 'The number that moves', - claim: - 'Out of the box, this model solved none of eight. Letting it think first is the cheapest intervention there is, and you can measure exactly what it bought.', - surface: 'metric', - }, - { - id: 'receipt', - title: 'The whole environment, in one screen', - claim: - 'The grader is thirty lines of Python. Here it is, and here is the command that runs it.', - surface: 'receipt', - }, - { - id: 'limits', - title: 'What this does not teach', - claim: - 'A word game is missing four things your business has. Each one is why the next demo exists.', - surface: 'limits', - }, - ], + claims: { + play: + 'Play it yourself first. Everything else on this page is about what happened when a model tried the same thing.', + watch: + 'This is not a video. It is a recorded attempt replayed at the speed it actually happened, and you can step through it one guess at a time.', + reward: + 'Move one slider and the winner changes. That is not a trick — it is the product.', + evidence: + 'The grader is thirty lines of Python. Here it is, here is the command that runs it, and here is what this environment does not teach you.', + }, limits: [ { text: diff --git a/src/lib/demo-kit/index.ts b/src/lib/demo-kit/index.ts index d204503..6cf4243 100644 --- a/src/lib/demo-kit/index.ts +++ b/src/lib/demo-kit/index.ts @@ -33,10 +33,11 @@ export type { RewardSpec, RewardValues, RunRef, - StoryBeat, + DemoTabId, Vertical, } from './types'; +export { DEMO_TABS } from './types'; export { defineDemo, defineMeta } from './define'; /** `null` is "not scored", never 0.0. Every absence goes through these two. */ diff --git a/src/lib/demo-kit/types.ts b/src/lib/demo-kit/types.ts index 9d239ee..e23c29b 100644 --- a/src/lib/demo-kit/types.ts +++ b/src/lib/demo-kit/types.ts @@ -170,24 +170,19 @@ export interface DemoEpisode { }[]; } -/** One beat of the exec narrative. The shell renders these in order. */ -export interface StoryBeat { - id: string; - title: string; - /** One sentence, asserted as a claim the page then demonstrates. */ - claim: string; - /** Which shared surface renders it. */ - surface: - | 'hero' - | 'anatomy' - | 'split-play' - | 'scrubber' - | 'reward-editor' - | 'metric' - | 'receipt' - | 'limits' - | 'custom'; -} +/** + * The four tabs every demo page has, in order. + * + * `play` is the landing tab and the reason the page exists: a visitor should be + * doing the task within one click of choosing an environment, not reading about + * it. The other three are what they reach for once they have felt it. + * + * A demo with no `interactive` mode has no `play` tab and opens on `watch`; + * the shell works that out, not the demo. + */ +export type DemoTabId = 'play' | 'watch' | 'reward' | 'evidence'; + +export const DEMO_TABS: readonly DemoTabId[] = ['play', 'watch', 'reward', 'evidence']; /** What this demo deliberately does not teach, and which demo answers it. */ export interface Limit { @@ -201,7 +196,14 @@ export interface Narrative { thesis: string; /** The question in the buyer's head when they land. */ anxiety: string; - beats: StoryBeat[]; + /** + * One sentence per tab, asserted as a claim the tab then demonstrates. + * + * Required for every tab, including ones this demo may not render — writing + * the claim is how an author works out whether the tab has anything to say. + * A tab whose claim is hard to write is usually a tab with nothing in it. + */ + claims: Record; limits: Limit[]; } @@ -254,6 +256,12 @@ export interface DemoModule { * means 'unverifiable' — a truncated trace — and must never render as zero. */ verify?: (episode: DemoEpisode) => RewardValues | null; - /** Extra tabs beside the default ones. */ + /** + * Extra panels this demo adds. NOT extra top-level tabs: the page's four + * tabs are fixed and derived, so these ride in the step-detail strip inside + * `watch`, beside Reasoning and Model call, where they sit next to the step + * they are almost always about. A demo with no recording has no step-detail + * strip, and they fall to the bottom of `evidence` instead. + */ tabs?: { id: string; label: string; Component: React.ComponentType }[]; } diff --git a/src/pages/DemoPage.tsx b/src/pages/DemoPage.tsx index a09129f..7ba49ff 100644 --- a/src/pages/DemoPage.tsx +++ b/src/pages/DemoPage.tsx @@ -3,7 +3,7 @@ * * It owns almost nothing on purpose. The router's loader has already validated * the slug against the registry and started the demo's chunk, and `DemoShell` - * owns the loading, the beats and the URL state, so all that is left here is + * owns the loading, the tabs and the URL state, so all that is left here is * the document head — the half of SEO that `scripts/prerender.mjs` cannot do, * because a visitor who lands on `/` and clicks through never fetches a new * document and would otherwise keep the home page's title and canonical link. diff --git a/src/pages/Gallery.tsx b/src/pages/Gallery.tsx index 03e54fb..9299e8f 100644 --- a/src/pages/Gallery.tsx +++ b/src/pages/Gallery.tsx @@ -1,30 +1,41 @@ import { useMemo } from 'react'; import { Link, useSearchParams } from 'react-router-dom'; -import { ArrowRight, FileText } from 'lucide-react'; -import { iconFor } from '@/content/icons'; import { - allDemos, - demosForVertical, - lineup, - routes, - verticalForDemo, - verticalKeysInUse, -} from '@/content/lineup'; + ALL_VERTICALS, + EnvironmentGrid, + FeaturedEnvironment, + environments, + filterKeys, + VerticalFilter, + type VerticalFilterValue, +} from '@/components/site/EnvironmentPicker'; +import { routes } from '@/content/lineup'; import { PROPOSAL_NOTICE } from '@/content/verticals'; -// The registry owns the taxonomy's exec-facing names. The lineup's own titles -// are longer marketing headings ("Customer Support Resolution") and would wrap -// two lines inside a filter chip on a phone, so chips use the registry label. import { VERTICAL_LABELS } from '@/lib/demo-kit/registry'; import type { Vertical } from '@/lib/demo-kit/types'; import * as s from '@/content/styles'; import { pageTitle, useSeo } from '@/lib/seo'; -const ALL = 'all'; +/** + * The fuller list. + * + * Home leads with the picker; this is where you go to see everything and cut it + * by industry. The cards are the same component, so the two pages cannot drift + * into describing the same environment two different ways — which is how an + * honest site quietly becomes a dishonest one. + */ -function verticalLabel(key: Vertical): string { - return VERTICAL_LABELS[key]; -} +/** What a `spec` has to contain before it is allowed on this page. Contract rule 12. */ +const SPEC_PARTS: readonly { label: string; body: string }[] = [ + { label: 'A task', body: 'One unit of work with a beginning and an end.' }, + { label: 'Legal moves', body: 'What the agent is allowed to do, and what is refused.' }, + { label: 'A grader', body: 'Deterministic code that marks the attempt. No judge, no rubric.' }, + { + label: 'A counterweight', + body: 'The term that stops the objective being maximised the crude way.', + }, +]; export default function Gallery() { useSeo({ @@ -42,58 +53,41 @@ export default function Gallery() { */ const [params, setParams] = useSearchParams(); const raw = params.get('vertical'); - const active: Vertical | typeof ALL = - raw && verticalKeysInUse.includes(raw as Vertical) ? (raw as Vertical) : ALL; + const active: VerticalFilterValue = + raw && filterKeys.includes(raw as Vertical) ? (raw as Vertical) : ALL_VERTICALS; const shown = useMemo( - () => (active === ALL ? allDemos : allDemos.filter((d) => d.vertical === active)), + () => + active === ALL_VERTICALS + ? environments + : environments.filter((entry) => entry.verticalKey === active), [active], ); - function select(next: Vertical | typeof ALL) { + const built = shown.filter((entry) => entry.status === 'live'); + const written = shown.filter((entry) => entry.status === 'spec'); + + function select(next: VerticalFilterValue) { // `replace` so a run of filter taps leaves one entry in history, not eight. - if (next === ALL) setParams({}, { replace: true }); + if (next === ALL_VERTICALS) setParams({}, { replace: true }); else setParams({ vertical: next }, { replace: true }); } - const filters: readonly (Vertical | typeof ALL)[] = [ALL, ...verticalKeysInUse]; - - const unbuilt = lineup.filter((v) => demosForVertical(v.key).length === 0).length; - return (

Gallery

Every environment we have built or specified.

- A live demo is playable in this tab. A spec is a written environment — task, action set, - grader, counterweight and the command that evaluates it — published in full, with no - interactive surface yet. There are no coming-soon cards here. + A built environment is playable in this tab, on the same code the repository ships. A spec is + a written environment — task, action set, grader, counterweight and the command that + evaluates it — published in full, with no interactive surface yet. There are no coming-soon + cards here.

- {filters.length > 2 ? ( + {filterKeys.length > 1 ? (

Filter by vertical

-
    - {filters.map((key) => { - const selected = key === active; - return ( -
  • - -
    • - ); - })} -
+
) : null} @@ -103,7 +97,8 @@ export default function Gallery() { */}

{shown.length === 1 ? '1 environment' : `${shown.length} environments`} - {active === ALL ? '' : ` in ${verticalLabel(active)}`} + {active === ALL_VERTICALS ? '' : ` in ${VERTICAL_LABELS[active]}`} + {shown.length === 0 ? '' : ` · ${built.length} built · ${written.length} written`}

{shown.length === 0 ? ( @@ -111,102 +106,71 @@ export default function Gallery() { {/* Two different nothings. Telling a visitor "no environment is filed under this vertical" when they have not filtered anything reads as a broken page rather than an empty one. */} -

- {active === ALL ? 'No environments are registered.' : 'Nothing under this vertical.'} -

+

+ {active === ALL_VERTICALS + ? 'No environments are registered.' + : 'Nothing under this vertical.'} +

- {active === ALL - ? 'The registry is empty, which means the site is mid-build rather than hiding something. The lineup below is written either way.' - : 'No environment is filed here yet. The proposal for it is still on its own page, written out in full.'} + {active === ALL_VERTICALS + ? 'The registry is empty and the lineup is empty with it, which means the site is mid-build rather than hiding something.' + : 'No environment is filed here yet, built or written.'}

- {active === ALL ? null : ( - )} - ) : ( -
    - {shown.map((demo) => { - const Icon = iconFor(demo.icon); - const isSpec = demo.status === 'spec'; - const vertical = verticalForDemo(demo); - return ( -
  • -
    - - - + ) : null} -

    {demo.title}

    -

    {demo.tagline}

    + {built.length > 0 ? ( +
    +

    + {built.length === 1 ? 'The one you can play' : 'The ones you can play'} +

    +
    + {built.map((entry) => ( + + ))} +
    +
    + ) : null} -
    -
    -
    For
    -
    {demo.persona}
    -
    -
    -
    Reward
    -
    {demo.rewardLine}
    -
    -
    - - - {isSpec ? 'Read the specification' : 'Play it'} - - - - {vertical ? ( -

    - - {vertical.title} - {' '} - · {PROPOSAL_NOTICE} -

    - ) : null} -
    -
    • - ); - })} -
- )} + {written.length > 0 ? ( +
+
+

+ {written.length === 1 ? 'The one that is written' : `The ${written.length} that are written`} +

+ {PROPOSAL_NOTICE} +
+ +
+ ) : null}
- {/* Counted, not typed. A hard-coded "eleven" on a site about checkable - numbers goes stale the first time a demo ships. */} -

- The {unbuilt} we have not built -

+

What a spec has to contain

- The lineup is a set of proposals, written to the same four-part shape as the live one. - Reading one takes a minute and tells you whether the idea survives contact with your own - numbers. + A card that says a demo is coming is not a specification, and the build refuses one. Every + written environment above states all four of these, and the reward weights that go with + them, before it is allowed on this page.

- - See the lineup +
    + {SPEC_PARTS.map((part, i) => ( +
  1. + 0{i + 1} +

    {part.label}

    +

    {part.body}

    +
    2. + ))} +
+ + What we measured, and what we didn’t
diff --git a/src/pages/Home.tsx b/src/pages/Home.tsx index d946fbf..4137c14 100644 --- a/src/pages/Home.tsx +++ b/src/pages/Home.tsx @@ -1,38 +1,31 @@ import { Link } from 'react-router-dom'; -import { ArrowRight, ArrowUpRight, Play } from 'lucide-react'; +import { ArrowRight, ArrowUpRight } from 'lucide-react'; -import { helloWorldCitations, reproduce, trainingResult } from '@/content/evidence'; -import { iconFor } from '@/content/icons'; -import { featuredDemo, lineup, routes } from '@/content/lineup'; +import { + EnvironmentGrid, + FeaturedEnvironment, + builtEnvironments, + writtenEnvironments, +} from '@/components/site/EnvironmentPicker'; +import { helloWorldCitations, trainingResult } from '@/content/evidence'; +import { routes } from '@/content/lineup'; import { PROPOSAL_NOTICE } from '@/content/verticals'; import * as s from '@/content/styles'; import { pageTitle, useSeo } from '@/lib/seo'; /** - * The four boxes. This is the definition the whole site rests on, so it is - * written once, here, in the order a person who has never heard the words - * "reinforcement learning" can read it: what the job is, what you are allowed - * to do, who marks it, what the mark is. + * The landing page: a line of thesis, then the picker. + * + * The page used to open with the argument and bury the demos below it. That is + * an essay, and an executive who has read the first sentence has already + * decided whether to click something. So the picker is the primary object here + * and everything that used to lead — the credential, the measured number — sits + * underneath it as support for a visitor who wants it before they click. + * + * Counts in the copy are derived from the registry and the lineup, never typed. + * A hard-coded "eleven" on a site whose whole argument is that its claims are + * checkable goes stale the first time a demo ships. */ -const ANATOMY: readonly { label: string; body: string }[] = [ - { - label: 'A task', - body: 'One unit of work with a beginning and an end. Guess a five-letter word in six tries.', - }, - { - label: 'Legal moves', - body: 'What the player is allowed to do. Any word on the list, once, five letters.', - }, - { - label: 'A grader', - body: 'Code that marks the attempt. It runs the same way every time and there is nobody to appeal to.', - }, - { - label: 'A score that moves', - body: 'One number per attempt. Train against it and it goes up, or it does not and you found that out cheaply.', - }, -]; - export default function Home() { // The head is set here as well as baked by `scripts/prerender.mjs`, and the // two are not redundant: prerender covers the crawler that fetches the @@ -42,265 +35,183 @@ export default function Home() { useSeo({ title: pageTitle(), description: - 'Interactive demos of reinforcement-learning environments for executives. Real verifiers environments, real recorded rollouts, and a reward you can change to see the ranking flip.', + 'Interactive demos of reinforcement-learning environments for executives. Play the environment, watch a recorded model play the same board, then move the reward and watch the ranking change.', canonical: routes.home, ogImage: '/og/home.png', }); - const Featured = featuredDemo; + const [featured, ...otherBuilt] = builtEnvironments; + const builtCount = builtEnvironments.length; + const writtenCount = writtenEnvironments.length; + + // Written out rather than interpolated inline so the zero and the singular + // both read as English. Both numbers come from the data — `VERTICALS` has + // twelve entries and none of them is keyed `reference`, so the one live demo + // suppresses no proposal and the lineup reads one and twelve today. It will + // not stay that way, and the sentence has to survive the day it changes. + const builtSentence = + builtCount === 1 + ? 'One environment is built and playable in this tab.' + : `${builtCount} environments are built and playable in this tab.`; + const writtenSentence = + writtenCount === 1 + ? 'One more is a written specification:' + : `The other ${writtenCount} are written specifications:`; return (
- {/* ── The thesis ─────────────────────────────────────────────────── */} + {/* ── The thesis, in two sentences ───────────────────────────────── */}

Environments, demonstrated

An environment is an eval you can take the gradient of.

- You write down what good means, in code. A model attempts the work. The grader scores it - and cannot be argued with. Then you train against that score and watch the number move — - or watch it not move, which you found out in an afternoon instead of a quarter. + You write down what good means, in code, and then you train against that number and watch + it move — or watch it sit still, which you found out in an afternoon instead of a quarter. +

+
+ + {/* ── The picker: the built one ──────────────────────────────────── */} +
+

+ Pick one. You are playing it in a click. +

+

+ {builtSentence}{' '} + {writtenCount === 0 ? null : ( + <> + {writtenSentence} the task, the grader, the counterweight that stops the grader being + farmed, and the command that evaluates it. Nothing here is a coming-soon card. + + )}

-
- {Featured ? ( - -
+ + {/* ── The picker: the written ones ───────────────────────────────── */} + {/* Dropped entirely rather than rendered as "The 0 we have written": the + day every proposal becomes a demo, this section should disappear. */} + {writtenCount === 0 ? null : ( +
+
+

+ The {writtenCount} we have written but not built. +

+ {PROPOSAL_NOTICE}
-
+

+ Each one is a task an environment could run, a reward stated in a number your board already + reads, and the counterweight that stops that reward being maximised the crude way. Every + card opens the whole argument, including where it stops being honest. +

- {/* ── The credential, before anything else we say ────────────────── */} -
-
-

Why a word game

-

We didn’t pick a game. We picked theirs.

-

- Wordle is Prime Intellect’s own hello-world. It is one of five basic end-to-end examples - in their trainer, a shipped environment in their library, and the environment their - official tutorial optimises prompts against. A demo of their idea should start where - they start. -

- -
-
+ - {/* ── What an environment is, in four boxes ──────────────────────── */} -
-

Four parts. That is the whole of it.

-
    - {ANATOMY.map((box, i) => ( -
  1. - 0{i + 1} -

    {box.label}

    -

    {box.body}

    -
    2. - ))} -
+ + Filter the lineup by industry +
+ )} - {/* ── The one measured number ────────────────────────────────────── */} -
-
-
-
-

Published by Prime Intellect

-

- - {trainingResult.before} - -

-

- {trainingResult.model} {trainingResult.metric} on this task, before and after - training. -

-
-
-

- Out of the box, a 1.7-billion-parameter model never once guesses the word. After an{' '} - {trainingResult.method}, it wins about six games in ten. Measured on{' '} - {trainingResult.evalDescription}. Both checkpoints are public, so the claim is - checkable rather than quotable. -

-

- - The write-up - - {trainingResult.checkpoints.map((c) => ( + {/* ── The support: why this game, and the one measured number ────── */} +

+

+ Why this environment, and what has been measured on it +

+
+
+

Why a word game

+

+ We didn’t pick a game. We picked theirs. +

+

+ Wordle is Prime Intellect’s own hello-world: a basic example in their trainer, a + shipped environment in their library, and the environment their official tutorial + optimises prompts against. +

+
+ + ))} +
-
-
- {/* ── The live demo ──────────────────────────────────────────────── */} - {Featured ? ( -
-

The live one

-

Play it, then change what counts as good.

-

- The demo runs the same environment the repository ships. You can play a board yourself, - watch a recorded model play the same board, read the Python that scored it, and then - move the reward weights and watch the ranking of two recorded runs change under you. -

- - - - Live +
+

Published by Prime Intellect

+

+ + {trainingResult.before} - For the {Featured.persona} - - - {Featured.title} - - {Featured.tagline} - - - Reward: {Featured.rewardLine} + - -

-
-

- Or skip the browser -

- 
-                
-                  {reproduce.clone}
-                  {'\n'}
-                  {reproduce.install}
-                  {'\n'}
-                  {reproduce.evaluate}
-                
-
-
-

- Three commands and you have the environment on your own machine, scoring your own - model. Nothing on this page needs our servers to be up. +

+

+ {trainingResult.model} {trainingResult.metric} on this task, before and after training. + Out of the box it never once guesses the word; after an {trainingResult.method}, it + wins about six games in ten. Measured on {trainingResult.evalDescription}, and both + checkpoints are public, so the claim is checkable rather than quotable. +

+

+ + The write-up + + {trainingResult.checkpoints.map((c) => ( + + {c.label.replace('PrimeIntellect/', '')} + + ))} +

+ {/* + The same write-up publishes average-reward figures for these runs. + They are deliberately not on this page — see /honesty. + */} +

+ We quote the win rate only. The reward numbers in that write-up span versions of the + environment and were never re-measured together.

-
- ) : null} - - {/* ── The lineup ─────────────────────────────────────────────────── */} -
-
-
-

The lineup

-

Twelve of these, ranked.

-
- {PROPOSAL_NOTICE}
-

- Each one is a task an environment could run, a reward in a number your board already - reads, and the counterweight that stops that reward being farmed the crude way. They are - our proposals. Nobody’s roadmap, nobody’s customer list. -

-
    - {lineup.map((v) => { - const Icon = iconFor(v.icon); - return ( -
  • - - - - {v.title} - {v.reward} - {!v.plannedForV1 ? ( - Not in the first set - ) : null} - -
    • - ); - })} -
- -
- - See what is built - +
- Read the honesty page first + What we measured, and what we didn’t +

+ Every number on this site, with the one that is ours and the ones that are not. +