claude
f0173440e4
Piggy was a hand-rolled OpenAI tool loop. It is now a Prime Agent session — Prime Intellect's own harness, embedded as a Node library — answering from PIG's tools and, for the first time, able to put information into the CRM rather than only read it out. The harness is a coding agent, so the first job was taking the coding agent away from it. `noTools: 'all'` plus an explicit allowlist leaves the model with PIG's ten `pig_*` tools and no bash, no filesystem, no IPython. That holds under attack: a hostile extension, a skill and a settings file planted in the agent's own directory, then `setActiveToolsByName` called with every built-in, still leaves ten tools, all ours. Both lines are load-bearing — `noTools` alone registers nothing, and the allowlist is what admits our own. Writing is gated rather than assumed. A change is proposed, not made: the tool returns a description, the transcript renders a diff card, and nothing reaches the database until someone presses Apply. Contracts, commitments, allocations and compliance always stop for a human whatever the mode. Every write runs through `executeMutation` as the calling user, so their capabilities and the audit trail apply exactly as they would to a human's. Four things about the SDK are wrong in its own documentation and cost a debugging cycle each: models.json does not resolve an env var name for `apiKey`, it sends the literal string; there is no built-in prime-inference provider in 0.84.1; a ResourceLoader you pass in is never reloaded for you; and the stock system prompt is a coding-assistant prompt that must be replaced — but replacing it also silently removes the tool list, because the harness only renders that section when it owns the prompt. AGENTS.md records all four. The expensive one was thinking level. The harness defaults to `medium`, and nemotron spent an entire 4,096-token budget reasoning and returned an empty answer. `low` was worse; `off` omits the parameter so the endpoint's default wins. An explicit `reasoning_effort: none` via `thinkingLevelMap` took a turn from 6,195 output tokens to 149. And a turn is now bounded. The harness loop is `while (true)` with no iteration cap; a runaway on a frontier model would have eaten the credit it is supposed to report on. Ceilings on model calls and tokens, enforced both through the harness hook and independently from the event stream, plus a per-user daily spend limit — and the ledger now records spend on turns that fail, which it previously discarded. Signing in lands on /piggy, which is a workspace: conversations down one side, the agent in the middle, what it did and what it cost beside it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
84 lines
3.6 KiB
TypeScript
84 lines
3.6 KiB
TypeScript
import assert from 'node:assert/strict';
|
|
import { readFileSync } from 'node:fs';
|
|
import { fileURLToPath } from 'node:url';
|
|
import test from 'node:test';
|
|
import {
|
|
isPiggyModelId,
|
|
piggyDefaultModelId,
|
|
piggyInferenceBaseUrl,
|
|
piggyModelCatalogue,
|
|
} from '../src/agent/models';
|
|
|
|
const modelsJson = JSON.parse(
|
|
readFileSync(fileURLToPath(new URL('../src/agent/models.json', import.meta.url)), 'utf8'),
|
|
) as {
|
|
providers: Record<string, { models: { id: string }[] }>;
|
|
};
|
|
|
|
test('every id in the picker is one the provider actually registers', () => {
|
|
// The whole point of a curated shortlist is that nothing in it 404s. The
|
|
// catalogue and models.json are the same five models by construction, and
|
|
// this is what keeps them that way when someone adds a sixth to one file.
|
|
const registered = (modelsJson.providers['prime-inference']?.models ?? []).map(
|
|
(model) => model.id,
|
|
);
|
|
const offered = piggyModelCatalogue().map((option) => option.id);
|
|
|
|
assert.deepEqual(offered, registered);
|
|
assert.ok(offered.length >= 4, 'the picker should offer a real choice, not just the default');
|
|
for (const id of offered) {
|
|
// Prime Inference ids are always provider-qualified. A bare model name is
|
|
// the classic copy-and-paste error and it fails as a 404 at the endpoint.
|
|
assert.match(id, /^[a-zA-Z0-9._-]+\/[a-zA-Z0-9._-]+$/, `${id} is not provider-qualified`);
|
|
assert.ok(isPiggyModelId(id));
|
|
}
|
|
});
|
|
|
|
test('the default is in the catalogue and there is exactly one of it', () => {
|
|
const catalogue = piggyModelCatalogue();
|
|
const defaults = catalogue.filter((option) => option.isDefault);
|
|
|
|
assert.equal(defaults.length, 1);
|
|
assert.equal(defaults[0]?.id, piggyDefaultModelId());
|
|
assert.equal(piggyDefaultModelId(), 'nvidia/nemotron-3-nano-30b-a3b');
|
|
assert.equal(isPiggyModelId('nvidia/nemotron-3-nano-30b-a3b'), true);
|
|
assert.equal(isPiggyModelId('nvidia/nemotron-9000'), false);
|
|
});
|
|
|
|
test('the picker can price and size every choice', () => {
|
|
for (const option of piggyModelCatalogue()) {
|
|
// Dollars per million tokens, NOT cents: the field names say so, and this
|
|
// is the one money field in PIG that is not an integer of cents. A price
|
|
// of 0 here would render as "free" in the picker, which no model is.
|
|
assert.ok(option.costPerMTokIn > 0, `${option.id} has no input price`);
|
|
assert.ok(option.costPerMTokOut > 0, `${option.id} has no output price`);
|
|
assert.ok(option.costPerMTokOut >= option.costPerMTokIn, `${option.id} prices output too low`);
|
|
assert.ok(option.contextWindow >= 100_000, `${option.id} is too small for a CRM transcript`);
|
|
assert.ok(option.label.length > 0);
|
|
assert.ok((option.hint ?? '').length > 0, `${option.id} would render as a blank picker row`);
|
|
}
|
|
});
|
|
|
|
test('the default is the cheapest thing on offer', () => {
|
|
// The panel is docked on every page, so the default is the price of a typo.
|
|
// If a costlier model ever becomes the default it should be a deliberate act
|
|
// that fails this test first.
|
|
const catalogue = piggyModelCatalogue();
|
|
const cheapest = [...catalogue].sort((a, b) => a.costPerMTokIn - b.costPerMTokIn)[0];
|
|
|
|
assert.equal(cheapest?.id, piggyDefaultModelId());
|
|
});
|
|
|
|
test('the catalogue cannot be reordered by a caller', () => {
|
|
// It is serialised to the browser on every session; one sort() at a call
|
|
// site would reorder the picker for every other session in the process.
|
|
const first = piggyModelCatalogue();
|
|
first.reverse();
|
|
|
|
assert.equal(piggyModelCatalogue()[0]?.id, piggyDefaultModelId());
|
|
});
|
|
|
|
test('the provider points at Prime Inference', () => {
|
|
assert.equal(piggyInferenceBaseUrl(), 'https://api.pinference.ai/api/v1');
|
|
});
|