karti-ai
185 lines
11 KiB
JavaScript
185 lines
11 KiB
JavaScript
export const meta = {
|
|
name: 'new-environment',
|
|
description: 'Promote a vertical from published spec to a live environment: specification, Python env, TypeScript port, rollouts, demo page',
|
|
whenToUse: 'Adding an environment to demo.primeintellectgrowth.com. Pass { slug, vertical, seeds }.',
|
|
phases: [
|
|
{ title: 'Specify', detail: 'design the environment, then attack the design' },
|
|
{ title: 'Implement', detail: 'Python environment and its probe ladder' },
|
|
{ title: 'Port', detail: 'TypeScript engine and the demo module, in parallel' },
|
|
{ title: 'Capture', detail: 'real rollouts — serialised, spark-1 is single-stream' },
|
|
{ title: 'Verify', detail: 'adversarial review across four lenses' },
|
|
],
|
|
}
|
|
|
|
const REPO = '/home/kartios/repos/gitea/PIG-Demo'
|
|
|
|
const slug = (args && args.slug) || 'support-resolution'
|
|
const vertical = (args && args.vertical) || 'customer-support-resolution'
|
|
const pkg = slug.replace(/-/g, '_')
|
|
const seeds = (args && args.seeds) || '0-7'
|
|
const reference = (args && args.reference) || ''
|
|
|
|
const BASE = `You are building environment "${slug}" for the repo at ${REPO}, promoting the vertical "${vertical}" from a published specification to a live, playable demo at demo.primeintellectgrowth.com.
|
|
|
|
**Read \`.claude/skills/new-environment/SKILL.md\` first, in full.** It carries the judgement; this prompt carries only your stage. Also read \`CONTRACT.md\`, \`AGENTS.md\`, and the reference implementation in \`envs/wordle_five/\` and \`src/demos/wordle/\`.
|
|
|
|
The specification for this vertical — task, reward, counterweight, persona, anxiety — is already written and reviewed in \`src/content/verticals.ts\`. Start from that entry; do not invent a different task.
|
|
|
|
Package: \`envs/${pkg}/\`. Demo: \`src/demos/${slug}/\`. Traces: \`public/traces/${slug}/\`.
|
|
${reference ? `\n**Prior art to read before designing anything:** ${reference}\n` : ''}
|
|
|
|
Do NOT run git. Do NOT edit another stage's files.`
|
|
|
|
const SPEC_SCHEMA = {
|
|
type: 'object', additionalProperties: false,
|
|
required: ['episode', 'data', 'actions', 'grader', 'components', 'ladder', 'heldOut', 'risks'],
|
|
properties: {
|
|
episode: { type: 'string' },
|
|
data: { type: 'string', description: 'Source, synthetic-or-real, licence' },
|
|
actions: { type: 'string' },
|
|
grader: { type: 'string', description: 'How the score is computed, deterministically' },
|
|
components: { type: 'array', items: { type: 'object', additionalProperties: false,
|
|
required: ['key','label','weight','role','definition'],
|
|
properties: { key:{type:'string'}, label:{type:'string'}, weight:{type:'number'},
|
|
role:{type:'string', enum:['objective','counterweight','gate']}, definition:{type:'string'} } } },
|
|
ladder: { type: 'array', items: { type: 'object', additionalProperties: false,
|
|
required: ['policy','behaviour','expected'],
|
|
properties: { policy:{type:'string'}, behaviour:{type:'string'}, expected:{type:'string'} } } },
|
|
heldOut: { type: 'string' },
|
|
risks: { type: 'array', items: { type: 'string' } },
|
|
},
|
|
}
|
|
|
|
phase('Specify')
|
|
const spec = await agent(`${BASE}
|
|
|
|
## YOUR STAGE: the specification. No code.
|
|
|
|
Design the environment concretely and write \`envs/${pkg}/SPEC.md\`. Answer the seven questions in the skill's stage 1 with specifics, not restatements of the vertical entry.
|
|
|
|
The two decisions that make or break this:
|
|
|
|
**The grader must compute, not opine.** If you find yourself reaching for an LLM judge, you have not found the environment yet. Look for the signal that already exists in the business: a downstream system's response, a settled number, a later observed outcome. If there genuinely is not one, say so plainly in \`risks\` — shipping it as a spec with the caveat stated is a legitimate outcome and better than faking it.
|
|
|
|
**The counterweight must be in genuine tension with the objective.** Name the crude policy that maxes the objective, and the component that stops it. Then name a good policy that pays the counterweight's price — if you cannot, the counterweight is a gate wearing the wrong label.
|
|
|
|
Also decide the data honestly. Synthetic is fine and often the only option; it must be labelled synthetic on the page, and it must not imply real customer records.
|
|
|
|
Return the structured spec AND write SPEC.md.`, { label: 'spec', phase: 'Specify', schema: SPEC_SCHEMA, effort: 'high' })
|
|
|
|
const specText = spec ? JSON.stringify(spec, null, 2) : '(the spec stage returned nothing)'
|
|
|
|
const LENSES = [
|
|
'DETERMINISM: is this grader really computable from the transcript, every time, with no human or model judgement anywhere in it? Hunt for a step that quietly needs an opinion.',
|
|
'GAMEABILITY: find the policy that maximises this reward while doing something the buyer would be appalled by. If you find one, the counterweight is insufficient. Try at least four distinct cheats.',
|
|
'HONESTY: would an executive be misled by this? Check the data provenance, whether the task is materially easier than the real job, and whether anything here implies a customer, a result, or a capability we have not got.',
|
|
]
|
|
|
|
const REVIEW_SCHEMA = {
|
|
type: 'object', additionalProperties: false,
|
|
required: ['verdict', 'findings'],
|
|
properties: {
|
|
verdict: { type: 'string', enum: ['sound', 'fixable', 'fatal'] },
|
|
findings: { type: 'array', items: { type: 'object', additionalProperties: false,
|
|
required: ['problem','fix'], properties: { problem:{type:'string'}, fix:{type:'string'} } } },
|
|
},
|
|
}
|
|
|
|
const reviews = (await parallel(LENSES.map((lens, i) => () =>
|
|
agent(`Attack this environment specification through ONE lens. Be adversarial: your job is to find the reason it should not ship, not to approve it.
|
|
|
|
LENS: ${lens}
|
|
|
|
${BASE}
|
|
|
|
=== THE SPECIFICATION ===
|
|
${specText}`, { label: `attack:${i}`, phase: 'Specify', schema: REVIEW_SCHEMA, effort: 'high' })
|
|
))).filter(Boolean)
|
|
|
|
const reviewText = reviews.map((r, i) => `--- LENS ${i} — verdict: ${r.verdict}\n${r.findings.map(f => ` ! ${f.problem}\n -> ${f.fix}`).join('\n')}`).join('\n\n')
|
|
|
|
if (reviews.some(r => r.verdict === 'fatal')) {
|
|
log('A reviewer called the specification fatal. Stopping before implementation.')
|
|
return { stopped: 'fatal specification', spec, reviews }
|
|
}
|
|
|
|
phase('Implement')
|
|
const python = await agent(`${BASE}
|
|
|
|
## YOUR STAGE: the Python environment.
|
|
|
|
Build \`envs/${pkg}/\` — engine, protocol, reward (with \`# region: pig-demo/reward\` markers), a reference policy, and tests. Register your probe policies in \`envs/probe.py\` for taskset "${slug}".
|
|
|
|
Implement the specification below, INCLUDING every fix the reviewers demanded. Where a reviewer found a gameable policy, add it to the probe ladder as a rung that must score below the good policies — a cheat you have named and measured is a cheat that cannot come back.
|
|
|
|
**Your gate:** \`uv run python envs/probe.py\` passes every assertion, including that your two good policies do not dominate each other. Iterate until it does. If it cannot pass, the reward is wrong — fix the reward, not the assertion.
|
|
|
|
=== SPECIFICATION ===
|
|
${specText}
|
|
|
|
=== WHAT THE REVIEWERS DEMANDED ===
|
|
${reviewText}`, { label: 'python', phase: 'Implement', effort: 'high' })
|
|
|
|
phase('Port')
|
|
const ported = (await parallel([
|
|
() => agent(`${BASE}
|
|
|
|
## YOUR STAGE: the TypeScript engine.
|
|
|
|
Port \`envs/${pkg}/\` 's scorer to \`src/demos/${slug}/engine.ts\`, and extend \`scripts/conformance.mjs\` so this environment's digest is gated the way wordle's is. Seed with FNV-1a over the decimal seed — never a built-in RNG, or the same seed picks a different task on each side and every permalink lies.
|
|
|
|
**Your gate:** \`pnpm conformance\` — the two digests match.
|
|
|
|
The Python that landed:
|
|
${python}`, { label: 'port:ts', phase: 'Port', effort: 'high' }),
|
|
|
|
() => agent(`${BASE}
|
|
|
|
## YOUR STAGE: the demo module.
|
|
|
|
Build \`src/demos/${slug}/\` — \`meta.ts\`, \`narrative.ts\` (\`claims\` for all four tabs: play, watch, reward, evidence), \`surface.tsx\`, \`adapter.ts\`, \`reward.ts\`, and \`interactive\` if a human can play this task. Read \`src/demos/wordle/\` as the reference.
|
|
|
|
Build against \`src/components/demo/mock.tsx\` — do NOT wait for real rollouts, that is the whole point of the wire format.
|
|
|
|
Anything the demo needs that the shell does not offer is a contract bug to fix in \`demo-kit\` or expose as a \`SlotRegion\`, never a slug check in \`src/components/demo/\`.
|
|
|
|
**Your gate:** \`pnpm check\` and \`./node_modules/.bin/tsc --noEmit\` clean for your files.
|
|
|
|
The specification:
|
|
${specText}`, { label: 'port:demo', phase: 'Port', effort: 'high' }),
|
|
])).filter(Boolean)
|
|
|
|
phase('Capture')
|
|
const captured = await agent(`${BASE}
|
|
|
|
## YOUR STAGE: real rollouts.
|
|
|
|
Extend \`envs/capture.py\` for taskset "${slug}" and capture seeds ${seeds} for at least three arms: a weak one, a strong one, and a generated reference policy.
|
|
|
|
⚠️ **spark-1 serves one model and is single-stream — run the arms one after another, never concurrently.** Check it is up first: \`curl http://100.127.247.67:8001/v1/models\`. A thinking-enabled arm can take minutes per seed; run it in the background, redirect output to a FILE (piping to head/tail block-buffers Python's stdout and a working run looks hung), and poll the written fixtures rather than the log.
|
|
|
|
Include at least one run the agent loses. Then \`uv run python envs/build_manifest.py\`.
|
|
|
|
**Your gate:** \`uv run python envs/verify_fixtures.py\` — every fixture replays and reproduces its own recorded rewards at delta 0.
|
|
|
|
If spark-1 is unreachable, do NOT invent fixtures. Say so and stop; a fabricated rollout on this site would be worse than a missing one.`, { label: 'capture', phase: 'Capture', effort: 'high' })
|
|
|
|
phase('Verify')
|
|
const CHECKS = [
|
|
'Run the full gate chain and make it green: uv run python envs/probe.py, uv run pytest, uv run python envs/verify_fixtures.py, pnpm check, pnpm conformance, pnpm build. Fix whatever is broken.',
|
|
'Open the new demo in a real browser at 1280px and 390px (playwright is installed; serve dist). Confirm: it appears in the landing picker, all four tabs render, Play works if it has an interactive mode, ?tab= round-trips, zero console errors, no horizontal overflow at 390px.',
|
|
'Audit honesty against the rules in the skill: is synthetic data labelled synthetic ON THE PAGE, are recorded runs labelled recorded, does any claim imply a customer or a result we do not have, and does the page state how this environment differs from the real job? Fix the copy where it does not hold.',
|
|
]
|
|
|
|
const verified = (await parallel(CHECKS.map((c, i) => () =>
|
|
agent(`${BASE}\n\n## YOUR STAGE: verification.\n\n${c}\n\nReport precisely what you ran, what failed, and what you changed. Do not report success you did not observe.`,
|
|
{ label: `verify:${i}`, phase: 'Verify', effort: 'high' })
|
|
))).filter(Boolean)
|
|
|
|
return {
|
|
slug, vertical,
|
|
spec, reviews: reviewText,
|
|
python, ported, captured,
|
|
verified,
|
|
}
|