karti-ai
5bbf913664
The environment page was a linear scroll of eight narrative beats. That is an essay, and it is the wrong shape for somebody who has just chosen an environment and wants to use it. It is now four tabs — Play, Watch, Reward, Evidence — opening on Play, with the board above the fold at 390x844 and the anatomy strip directly beneath it. The landing page leads with the picker instead of burying it under the thesis. The contract changed rather than layering tabs over beats. `Narrative.beats` is gone; `claims: Record<DemoTabId, string>` replaces it, one required sentence per tab. Writing the claim is how an author discovers whether a tab has anything to say — a tab whose claim is hard to write is usually a tab with nothing in it. Doing this now costs one migration; doing it after eleven more environments costs twelve. Tabs are derived, never declared: Play iff the demo ships an `interactive` mode, Watch iff it has recorded runs. A demo that could name its own tabs would mean environment seven inventing a fifth one and the site ceasing to be one product. One thing the browser caught that no gate would have. The header stat strip describes the RECORDED RUN, and on Play it sat above the visitor's own empty board reading "Outcome: failed" — which parses as your game having already failed before you touch a key. It now renders only on the tabs whose subject is that run, which also moved the board 54px up the page. The picker is honest about the shape of the lineup by construction: one built environment gets its own block and the demo's real board as its thumbnail, twelve written specifications render dimmed with a Spec badge, and every count on the page is derived from the data rather than typed. 206 contract checks pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019mt6sHQHEnEYrJZvoMCJSB
6.6 KiB
name, description
| name | description |
|---|---|
| new-environment | Build a new RL environment demo for this repo, end to end — specification, Python environment, TypeScript port, captured rollouts, and the demo page. Use when adding an environment to demo.primeintellectgrowth.com, promoting a vertical from spec to live, or when asked to "build environment N" / "add the claims demo" / "make the support one real". |
Building an environment
The scarce thing here is not code. It is a defensible environment: a task with a grader that computes rather than opines, and a reward that cannot be maximised by doing the crude thing. Everything in this repository — the probe ladder, the conformance digest, the contract validator — exists to stop a plausible-looking fake from shipping.
Work in this order. Each stage has a gate, and a stage that cannot pass its gate is telling you the environment is wrong, not that the gate is too strict.
0. The specification already exists
src/content/verticals.ts carries a reviewed entry for every planned
environment: task, reward, counterweight, persona, anxiety, and a
rank (1 is the one to build next). Start from that entry. If you are
building something not in that file, add the entry first and have it reviewed
before writing a line of Python — the spec is the cheapest place to discover
that an environment does not work.
1. Write envs/<pkg>/SPEC.md before any code
Concrete answers, not restatements of the vertical entry:
- The episode. Exactly what one instance is, and what ends it.
- The data. Where it comes from. If synthetic, say so on the page — never imply real customer data. If real, name the licence.
- The action set. Legal moves, and what happens to a malformed reply. (It costs a turn and is recorded. It never crashes the rollout.)
- The grader. How the score is computed from the transcript, deterministically.
- The reward components, with weights summing to 1.0 and a
counterweight. - The probe ladder: the policies, and what each must score.
- The held-out slice.
⚠️ If the grader needs an LLM judge, stop. A judge is the exact thing a
verifiable reward is positioned against, and shipping one here would undercut
every other environment on the site. Either find the computable signal, or ship
the environment as a spec with the caveat stated — legal-playbook-redline
is in the lineup for precisely this reason and is honest about it.
Gate: the spec is reviewed adversarially before implementation. Ask of it: could a competent sceptic call this grader subjective? Could the reward be maxed by doing something the buyer would hate?
2. The Python environment
envs/<pkg>/ — engine.py (pure, seeded, never raises on bad input),
protocol.py (parse/render; a parse failure is a rejected move, not an
exception), reward.py (with # region: pig-demo/reward markers around the
part the page quotes), a reference policy, and tests/.
Add the policies to envs/probe.py and register them for your taskset.
Gate: uv run python envs/probe.py passes every assertion. Specifically:
doing nothing scores exactly 0.000; no weighted component is flat across the
ladder; and your two good policies do not dominate each other — if one wins
on every component, your counterweight is decorative and the reward editor on
the page would be theatre.
Two mistakes already made here, both worth avoiding twice:
- Score the counterweight over turns spent, not moves accepted. Otherwise a policy that acts once and then jams the parser scores a perfect 1.0 on it.
- The efficiency component's denominator is the depth your shipped reference policy reaches, not a true optimum. Grade against an optimum and your own oracle rung fails its assertion on some seeds.
3. The TypeScript port
src/demos/<slug>/engine.ts mirrors the Python. Both sides hash the full
cross-product of the task set and the digest is committed and gated in CI.
⚠️ Never use a language's built-in RNG for seeding. random.Random(seed) and
any JS PRNG disagree, so the same seed picks a different instance on each side
and every ?seed= permalink silently shows a different task than the run it
claims to replay. Use FNV-1a over the decimal seed, as engine.py does.
Gate: pnpm conformance — the two digests match.
4. Capture
uv run python envs/capture.py --arm <arm> --seeds 0-7
⚠️ spark-1 serves one model and is single-stream. Capture is a serialised
queue, never parallel. A thinking-enabled arm can take minutes per seed.
Front-end work does not wait on it — build against mock.tsx instead.
Capture at least: a weak arm, a strong arm, and a generated reference policy.
Include at least one run the agent loses; a demo where the agent always wins
teaches nothing about the reward. A run that had anything done to it is
kind: 'intervened' and must name the intervention, so a sampling change can
never be read as a training result.
Gate: uv run python envs/verify_fixtures.py — every fixture replays through
the engine and reproduces its own recorded rewards.
5. The demo module
pnpm demo:new <slug> scaffolds it. Fill in meta.ts, narrative.ts
(claims for all four tabs), surface.tsx, adapter.ts, reward.ts, and
interactive if the task can be played by a human. Read src/demos/wordle/ as
the reference implementation.
Where the demo needs something the shell does not offer, that is a contract bug
to fix in demo-kit or expose as a SlotRegion — never an if (slug === ...)
in src/components/demo/. check-demos fails on it.
Gate: pnpm check && pnpm build.
6. Ship
Add the OG card (node scripts/og.mjs — x86 only), rebuild, and
bash deploy/deploy.sh. Flip the vertical's status so the picker stops
advertising it as a specification.
The honesty rules, which are not negotiable
- Rollouts are recorded, not live, and the page says so.
- The reward editor re-scores recorded attempts; it does not retrain.
- Synthetic data is labelled synthetic, on the page, not in a footnote.
- No customer logos and no named companies as customers, ever.
- Cite someone else's number as theirs, with a link, and never re-quote a figure across a version change without re-measuring it.
- If the environment is materially easier or harder than the real task it stands for, say which and by how much.
Running it as a pipeline
.claude/workflows/new-environment.js runs stages 1–5 as a workflow with
adversarial review between them. Invoke with
Workflow({ name: 'new-environment', args: { slug, vertical, seeds } }).
Read this file first regardless — the workflow encodes the sequence, this
encodes the judgement.