karti-ai
5bbf913664
The environment page was a linear scroll of eight narrative beats. That is an essay, and it is the wrong shape for somebody who has just chosen an environment and wants to use it. It is now four tabs — Play, Watch, Reward, Evidence — opening on Play, with the board above the fold at 390x844 and the anatomy strip directly beneath it. The landing page leads with the picker instead of burying it under the thesis. The contract changed rather than layering tabs over beats. `Narrative.beats` is gone; `claims: Record<DemoTabId, string>` replaces it, one required sentence per tab. Writing the claim is how an author discovers whether a tab has anything to say — a tab whose claim is hard to write is usually a tab with nothing in it. Doing this now costs one migration; doing it after eleven more environments costs twelve. Tabs are derived, never declared: Play iff the demo ships an `interactive` mode, Watch iff it has recorded runs. A demo that could name its own tabs would mean environment seven inventing a fifth one and the site ceasing to be one product. One thing the browser caught that no gate would have. The header stat strip describes the RECORDED RUN, and on Play it sat above the visitor's own empty board reading "Outcome: failed" — which parses as your game having already failed before you touch a key. It now renders only on the tabs whose subject is that run, which also moved the board 54px up the page. The picker is honest about the shape of the lineup by construction: one built environment gets its own block and the demo's real board as its thumbnail, twelve written specifications render dimmed with a Spec badge, and every count on the page is derived from the data rather than typed. 206 contract checks pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019mt6sHQHEnEYrJZvoMCJSB
138 lines
6.6 KiB
Markdown
138 lines
6.6 KiB
Markdown
---
|
||
name: new-environment
|
||
description: Build a new RL environment demo for this repo, end to end — specification, Python environment, TypeScript port, captured rollouts, and the demo page. Use when adding an environment to demo.primeintellectgrowth.com, promoting a vertical from spec to live, or when asked to "build environment N" / "add the claims demo" / "make the support one real".
|
||
---
|
||
|
||
# Building an environment
|
||
|
||
The scarce thing here is not code. It is a **defensible environment**: a task
|
||
with a grader that computes rather than opines, and a reward that cannot be
|
||
maximised by doing the crude thing. Everything in this repository — the probe
|
||
ladder, the conformance digest, the contract validator — exists to stop a
|
||
plausible-looking fake from shipping.
|
||
|
||
Work in this order. Each stage has a gate, and a stage that cannot pass its gate
|
||
is telling you the environment is wrong, not that the gate is too strict.
|
||
|
||
## 0. The specification already exists
|
||
|
||
`src/content/verticals.ts` carries a reviewed entry for every planned
|
||
environment: `task`, `reward`, `counterweight`, `persona`, `anxiety`, and a
|
||
`rank` (1 is the one to build next). **Start from that entry.** If you are
|
||
building something not in that file, add the entry first and have it reviewed
|
||
before writing a line of Python — the spec is the cheapest place to discover
|
||
that an environment does not work.
|
||
|
||
## 1. Write `envs/<pkg>/SPEC.md` before any code
|
||
|
||
Concrete answers, not restatements of the vertical entry:
|
||
|
||
- **The episode.** Exactly what one instance is, and what ends it.
|
||
- **The data.** Where it comes from. If synthetic, say so on the page — never
|
||
imply real customer data. If real, name the licence.
|
||
- **The action set.** Legal moves, and what happens to a malformed reply.
|
||
(It costs a turn and is recorded. It never crashes the rollout.)
|
||
- **The grader.** How the score is computed from the transcript, deterministically.
|
||
- **The reward components**, with weights summing to 1.0 and a `counterweight`.
|
||
- **The probe ladder**: the policies, and what each must score.
|
||
- **The held-out slice.**
|
||
|
||
⚠️ **If the grader needs an LLM judge, stop.** A judge is the exact thing a
|
||
verifiable reward is positioned against, and shipping one here would undercut
|
||
every other environment on the site. Either find the computable signal, or ship
|
||
the environment as a `spec` with the caveat stated — `legal-playbook-redline`
|
||
is in the lineup for precisely this reason and is honest about it.
|
||
|
||
**Gate:** the spec is reviewed adversarially before implementation. Ask of it:
|
||
could a competent sceptic call this grader subjective? Could the reward be
|
||
maxed by doing something the buyer would hate?
|
||
|
||
## 2. The Python environment
|
||
|
||
`envs/<pkg>/` — `engine.py` (pure, seeded, never raises on bad input),
|
||
`protocol.py` (parse/render; a parse failure is a rejected move, not an
|
||
exception), `reward.py` (with `# region: pig-demo/reward` markers around the
|
||
part the page quotes), a reference policy, and `tests/`.
|
||
|
||
Add the policies to `envs/probe.py` and register them for your taskset.
|
||
|
||
**Gate:** `uv run python envs/probe.py` passes every assertion. Specifically:
|
||
doing nothing scores exactly `0.000`; no weighted component is flat across the
|
||
ladder; and **your two good policies do not dominate each other** — if one wins
|
||
on every component, your counterweight is decorative and the reward editor on
|
||
the page would be theatre.
|
||
|
||
Two mistakes already made here, both worth avoiding twice:
|
||
- Score the counterweight over turns **spent**, not moves accepted. Otherwise a
|
||
policy that acts once and then jams the parser scores a perfect 1.0 on it.
|
||
- The efficiency component's denominator is the depth **your shipped reference
|
||
policy** reaches, not a true optimum. Grade against an optimum and your own
|
||
oracle rung fails its assertion on some seeds.
|
||
|
||
## 3. The TypeScript port
|
||
|
||
`src/demos/<slug>/engine.ts` mirrors the Python. Both sides hash the full
|
||
cross-product of the task set and the digest is committed and gated in CI.
|
||
|
||
⚠️ Never use a language's built-in RNG for seeding. `random.Random(seed)` and
|
||
any JS PRNG disagree, so the same seed picks a different instance on each side
|
||
and every `?seed=` permalink silently shows a different task than the run it
|
||
claims to replay. Use FNV-1a over the decimal seed, as `engine.py` does.
|
||
|
||
**Gate:** `pnpm conformance` — the two digests match.
|
||
|
||
## 4. Capture
|
||
|
||
`uv run python envs/capture.py --arm <arm> --seeds 0-7`
|
||
|
||
⚠️ **spark-1 serves one model and is single-stream. Capture is a serialised
|
||
queue, never parallel.** A thinking-enabled arm can take minutes per seed.
|
||
Front-end work does not wait on it — build against `mock.tsx` instead.
|
||
|
||
Capture at least: a weak arm, a strong arm, and a generated reference policy.
|
||
Include at least one run the agent **loses**; a demo where the agent always wins
|
||
teaches nothing about the reward. A run that had anything done to it is
|
||
`kind: 'intervened'` and must name the `intervention`, so a sampling change can
|
||
never be read as a training result.
|
||
|
||
**Gate:** `uv run python envs/verify_fixtures.py` — every fixture replays through
|
||
the engine and reproduces its own recorded rewards.
|
||
|
||
## 5. The demo module
|
||
|
||
`pnpm demo:new <slug>` scaffolds it. Fill in `meta.ts`, `narrative.ts`
|
||
(`claims` for all four tabs), `surface.tsx`, `adapter.ts`, `reward.ts`, and
|
||
`interactive` if the task can be played by a human. Read `src/demos/wordle/` as
|
||
the reference implementation.
|
||
|
||
Where the demo needs something the shell does not offer, that is a contract bug
|
||
to fix in `demo-kit` or expose as a `SlotRegion` — never an `if (slug === ...)`
|
||
in `src/components/demo/`. `check-demos` fails on it.
|
||
|
||
**Gate:** `pnpm check && pnpm build`.
|
||
|
||
## 6. Ship
|
||
|
||
Add the OG card (`node scripts/og.mjs` — x86 only), rebuild, and
|
||
`bash deploy/deploy.sh`. Flip the vertical's status so the picker stops
|
||
advertising it as a specification.
|
||
|
||
## The honesty rules, which are not negotiable
|
||
|
||
- Rollouts are **recorded, not live**, and the page says so.
|
||
- The reward editor **re-scores recorded attempts**; it does not retrain.
|
||
- Synthetic data is labelled synthetic, on the page, not in a footnote.
|
||
- No customer logos and no named companies as customers, ever.
|
||
- Cite someone else's number as theirs, with a link, and never re-quote a figure
|
||
across a version change without re-measuring it.
|
||
- If the environment is materially easier or harder than the real task it
|
||
stands for, say which and by how much.
|
||
|
||
## Running it as a pipeline
|
||
|
||
`.claude/workflows/new-environment.js` runs stages 1–5 as a workflow with
|
||
adversarial review between them. Invoke with
|
||
`Workflow({ name: 'new-environment', args: { slug, vertical, seeds } })`.
|
||
Read this file first regardless — the workflow encodes the sequence, this
|
||
encodes the judgement.
|