Capture harness, fixture verification, CI, and the public README
The site does no live inference. Rollouts are captured once against spark-1 and replayed at their recorded wall-clock — a public demo with no auth cannot hold an API key, and a recorded run can be scrubbed, permalinked, blind-compared and verified in ways a live one cannot. What stops it being a video is that the browser re-derives every number from the recorded moves. verify_fixtures.py is the Python half of that: it replays every committed fixture through the engine and reproduces its own rewards. All 16 land at delta 0.0. A fixture that cannot be regenerated is a claim with no receipt. First real measurement, thinking off, 8 seeds: solved 0/8. The model repeats guesses it has already played, invents words (trape, slith, postt, boomy), and contradicts its own feedback — consistency 0.09 to 0.17. That is the published failure taxonomy showing up in our own data on the first run, and it is why `consistency` is a reward component rather than a footnote. A capture failure is recorded as a turn with a null reply, never dropped. A capture that silently discarded failed turns would be reporting a better model than the one that ran. CI gates both halves and four things that fail silently in production: the word lists must rebuild byte-identically, the prerendered routes must carry their own baked og tags (crawlers do not run JS, so without them every shared link previews as the homepage), no blob: URL may reach the bundle (the site's CSP has no worker-src, so it falls back to default-src 'self' and a blob worker is blocked with no error), and the conformance digest must match across languages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019mt6sHQHEnEYrJZvoMCJSB
This commit is contained in:
karti-ai
@@ -0,0 +1,192 @@
|
||||
import { useState } from 'react';
|
||||
import type { ComponentType } from 'react';
|
||||
import { Eye, Trophy } from 'lucide-react';
|
||||
import type { DemoStep } from '@/lib/demo-kit/types';
|
||||
import { cn } from '@/lib/utils';
|
||||
import { formatOrDash } from './format';
|
||||
|
||||
export interface BlindCompareRun<T> {
|
||||
runId: string;
|
||||
/** The identity, revealed only after the visitor commits. */
|
||||
label: string;
|
||||
model: string;
|
||||
/** Required by the contract on an `intervened` run; shown at reveal. */
|
||||
intervention?: string;
|
||||
steps: DemoStep<T>[];
|
||||
total: number | null;
|
||||
}
|
||||
|
||||
export type BlindVote = 'A' | 'B' | 'tie';
|
||||
|
||||
export interface BlindCompareProps<T> {
|
||||
/** Both runs must be the same seed or the comparison is meaningless. */
|
||||
seed: number;
|
||||
a: BlindCompareRun<T>;
|
||||
b: BlindCompareRun<T>;
|
||||
Surface: ComponentType<{ state: T; compact?: boolean }>;
|
||||
question?: string;
|
||||
onVote?: (vote: BlindVote) => void;
|
||||
className?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Two runs on the same seed, unlabelled, until you commit.
|
||||
*
|
||||
* The point is not the vote. The point is that the visitor forms an opinion
|
||||
* from the behaviour BEFORE they learn which one had the better prompt, the
|
||||
* bigger model or the training run — because once they know, they cannot
|
||||
* unknow it, and every "obviously the trained one looks better" is worthless
|
||||
* after the fact.
|
||||
*
|
||||
* There is a "reveal without voting" escape on purpose: a visitor who does not
|
||||
* want to play should not be held hostage by a modal-shaped page.
|
||||
*/
|
||||
export function BlindCompare<T>({
|
||||
seed,
|
||||
a,
|
||||
b,
|
||||
Surface,
|
||||
question = 'Which agent would you rather have running this?',
|
||||
onVote,
|
||||
className,
|
||||
}: BlindCompareProps<T>) {
|
||||
const [vote, setVote] = useState<BlindVote | null>(null);
|
||||
const [revealed, setRevealed] = useState(false);
|
||||
|
||||
const commit = (choice: BlindVote) => {
|
||||
setVote(choice);
|
||||
setRevealed(true);
|
||||
onVote?.(choice);
|
||||
};
|
||||
|
||||
const winner: 'A' | 'B' | 'tie' =
|
||||
a.total === null || b.total === null
|
||||
? 'tie'
|
||||
: a.total > b.total
|
||||
? 'A'
|
||||
: b.total > a.total
|
||||
? 'B'
|
||||
: 'tie';
|
||||
|
||||
const sides: { id: 'A' | 'B'; run: BlindCompareRun<T> }[] = [
|
||||
{ id: 'A', run: a },
|
||||
{ id: 'B', run: b },
|
||||
];
|
||||
|
||||
return (
|
||||
<section aria-label="Blind comparison" className={cn('space-y-3', className)}>
|
||||
<div className="flex flex-wrap items-baseline gap-x-3 gap-y-1">
|
||||
<h3 className="text-sm font-semibold">{question}</h3>
|
||||
<p className="nums text-xs text-muted">Same puzzle, same seed ({seed}).</p>
|
||||
</div>
|
||||
|
||||
<div className="grid gap-3 sm:grid-cols-2">
|
||||
{sides.map(({ id, run }) => {
|
||||
const last = run.steps[run.steps.length - 1];
|
||||
const picked = vote === id;
|
||||
return (
|
||||
<div
|
||||
key={id}
|
||||
className={cn(
|
||||
'card flex flex-col gap-3 p-3 transition-colors duration-2 ease-enter',
|
||||
picked && 'border-brand',
|
||||
revealed && winner === id && 'border-positive',
|
||||
)}
|
||||
>
|
||||
<div className="flex items-center justify-between gap-2">
|
||||
<h4 className="text-sm font-semibold">Agent {id}</h4>
|
||||
<span className="nums text-xs text-muted">{run.steps.length} steps</span>
|
||||
</div>
|
||||
|
||||
<div className="rounded-lg bg-surface-2 p-3">
|
||||
{last ? (
|
||||
<Surface state={last.state} />
|
||||
) : (
|
||||
<p className="text-sm text-muted">This run recorded no steps.</p>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{!revealed ? (
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => commit(id)}
|
||||
className="tap w-full rounded-lg bg-primary px-3 text-sm font-medium text-primary-foreground transition-colors duration-2 ease-enter hover:bg-primary/90"
|
||||
>
|
||||
Agent {id} is better
|
||||
</button>
|
||||
) : (
|
||||
<dl className="space-y-1 text-sm">
|
||||
<div className="flex items-baseline justify-between gap-2">
|
||||
<dt className="text-muted">Identity</dt>
|
||||
<dd className="text-right font-medium">{run.label}</dd>
|
||||
</div>
|
||||
<div className="flex items-baseline justify-between gap-2">
|
||||
<dt className="text-muted">Model</dt>
|
||||
<dd className="nums text-right font-mono text-xs">{run.model}</dd>
|
||||
</div>
|
||||
<div className="flex items-baseline justify-between gap-2">
|
||||
<dt className="text-muted">Intervention</dt>
|
||||
<dd className="text-right text-xs">
|
||||
{run.intervention ?? (
|
||||
<span className="text-muted">none — plain rollout</span>
|
||||
)}
|
||||
</dd>
|
||||
</div>
|
||||
<div className="flex items-baseline justify-between gap-2 border-t border-border pt-1">
|
||||
<dt className="text-muted">Reward</dt>
|
||||
<dd className="nums flex items-center gap-1 text-right font-mono font-semibold">
|
||||
{revealed && winner === id ? (
|
||||
<Trophy className="h-3.5 w-3.5 text-positive" aria-hidden="true" />
|
||||
) : null}
|
||||
{formatOrDash(run.total)}
|
||||
</dd>
|
||||
</div>
|
||||
</dl>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
|
||||
{!revealed ? (
|
||||
<div className="flex flex-wrap gap-2">
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => commit('tie')}
|
||||
className="tap rounded-lg border border-border px-3 text-sm font-medium transition-colors duration-2 ease-enter hover:bg-surface-2"
|
||||
>
|
||||
Too close to call
|
||||
</button>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => setRevealed(true)}
|
||||
className="tap inline-flex items-center gap-1.5 rounded-lg px-3 text-sm font-medium text-muted transition-colors duration-2 ease-enter hover:text-fg"
|
||||
>
|
||||
<Eye className="h-4 w-4" aria-hidden="true" />
|
||||
Just show me
|
||||
</button>
|
||||
</div>
|
||||
) : (
|
||||
<p role="status" className="card bg-surface-2 p-3 text-sm leading-relaxed">
|
||||
{vote === null ? (
|
||||
<>Revealed without a vote. </>
|
||||
) : vote === winner ? (
|
||||
<>
|
||||
<span className="font-semibold text-positive">You picked the higher-scoring run.</span>{' '}
|
||||
</>
|
||||
) : (
|
||||
<>
|
||||
<span className="font-semibold text-warning">
|
||||
You picked the lower-scoring run.
|
||||
</span>{' '}
|
||||
</>
|
||||
)}
|
||||
The environment scored these two with the same grader, on the same seed. The difference
|
||||
between them is stated above — and if it is a prompt change rather than a training run,
|
||||
it says so, because a prompt change presented as a training result is the oldest trick
|
||||
in this business.
|
||||
</p>
|
||||
)}
|
||||
</section>
|
||||
);
|
||||
}
|
||||
Reference in New Issue
Block a user