Files
pig/apps/piggy/e2e/write-tools.test.ts
claude f0173440e4
CI / verify (push) Successful in 7m6s
CI / publish (push) Has been skipped
Put Piggy on Prime Agent, and let it write to the book
Piggy was a hand-rolled OpenAI tool loop. It is now a Prime Agent session —
Prime Intellect's own harness, embedded as a Node library — answering from
PIG's tools and, for the first time, able to put information into the CRM
rather than only read it out.

The harness is a coding agent, so the first job was taking the coding agent
away from it. `noTools: 'all'` plus an explicit allowlist leaves the model
with PIG's ten `pig_*` tools and no bash, no filesystem, no IPython. That
holds under attack: a hostile extension, a skill and a settings file planted
in the agent's own directory, then `setActiveToolsByName` called with every
built-in, still leaves ten tools, all ours. Both lines are load-bearing —
`noTools` alone registers nothing, and the allowlist is what admits our own.

Writing is gated rather than assumed. A change is proposed, not made: the
tool returns a description, the transcript renders a diff card, and nothing
reaches the database until someone presses Apply. Contracts, commitments,
allocations and compliance always stop for a human whatever the mode. Every
write runs through `executeMutation` as the calling user, so their
capabilities and the audit trail apply exactly as they would to a human's.

Four things about the SDK are wrong in its own documentation and cost a
debugging cycle each: models.json does not resolve an env var name for
`apiKey`, it sends the literal string; there is no built-in prime-inference
provider in 0.84.1; a ResourceLoader you pass in is never reloaded for you;
and the stock system prompt is a coding-assistant prompt that must be
replaced — but replacing it also silently removes the tool list, because the
harness only renders that section when it owns the prompt. AGENTS.md records
all four.

The expensive one was thinking level. The harness defaults to `medium`, and
nemotron spent an entire 4,096-token budget reasoning and returned an empty
answer. `low` was worse; `off` omits the parameter so the endpoint's default
wins. An explicit `reasoning_effort: none` via `thinkingLevelMap` took a turn
from 6,195 output tokens to 149.

And a turn is now bounded. The harness loop is `while (true)` with no
iteration cap; a runaway on a frontier model would have eaten the credit it
is supposed to report on. Ceilings on model calls and tokens, enforced both
through the harness hook and independently from the event stream, plus a
per-user daily spend limit — and the ledger now records spend on turns that
fail, which it previously discarded.

Signing in lands on /piggy, which is a workspace: conversations down one
side, the agent in the middle, what it did and what it cost beside it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-14 05:26:28 -07:00

237 lines
8.3 KiB
TypeScript
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
/**
* The write tools, taken all the way through a real transaction.
*
* `test/write-tools.test.ts` proves the negative — that a change nobody agreed
* to never opens a transaction — against a fake handle. It cannot prove the
* positive, because the interesting part of an applied write is what the
* database ends up holding: whether the row is really there, and whether the
* audit trail says Piggy wrote it. That needs Postgres.
*
* It needs its own Postgres, too. These cases INSERT, and the development
* database is a book people are looking at — an activity that appears in
* somebody's feed because a test ran is exactly the kind of thing a CRM must
* never do. So the URL is supplied separately and `pig_combined` is refused by
* name.
*
* docker exec pig-ux-db psql -U pig -d postgres -c "CREATE DATABASE pig_a2_scratch"
* DATABASE_URL=postgres://pig:pig@localhost:54330/pig_a2_scratch pnpm -F @pig/db run migrate
* PIGGY_WRITE_DATABASE_URL=postgres://pig:pig@localhost:54330/pig_a2_scratch \
* pnpm -F @pig/piggy run test:e2e
*/
import assert from 'node:assert/strict';
import { randomUUID } from 'node:crypto';
import test, { after, before } from 'node:test';
import type { ExtensionContext } from '@earendil-works/pi-coding-agent';
import type { Principal } from '@pig/api/src/lib/auth';
import {
accounts,
activities,
createDatabase,
demandDeals,
users,
type Database,
} from '@pig/db';
import { and, eq, like } from 'drizzle-orm';
import { createPigWriteTools, type PigWriteDetails } from '../src/write-tools';
const databaseUrl = process.env.PIGGY_WRITE_DATABASE_URL;
// A skipped suite that says why beats one that silently passes: these are the
// only cases in the repo that watch a write land.
if (!databaseUrl) {
test.skip('the write-tool E2E needs PIGGY_WRITE_DATABASE_URL pointing at a scratch database');
}
if (databaseUrl?.includes('pig_combined')) {
throw new Error('The write-tool E2E must never run against the development book.');
}
const db: Database = createDatabase({ url: databaseUrl ?? 'postgres://unused', max: 2 });
const ctx = {} as ExtensionContext;
const marker = `PIGGY-A2-${randomUUID()}`;
const fixture = { userId: '', accountId: '', dealId: '' };
function seller(): Principal {
return {
userId: fixture.userId,
email: `${marker}@example.test`,
name: 'Dana Okonjo',
isPlatformAdmin: false,
teams: [
{ team: 'demand', role: 'member' },
{ team: 'supply', role: 'member' },
],
via: 'jwt',
scopes: ['read', 'write'],
};
}
function tools(mode: 'confirm' | 'auto', decision: 'apply' | 'reject') {
return createPigWriteTools({
db,
principal: seller(),
mode,
propose: async () => decision,
});
}
function named(list: ReturnType<typeof tools>, name: string) {
const found = list.find((candidate) => candidate.name === name);
assert.ok(found, `${name} is missing`);
return found;
}
function detailsOf(result: { details: unknown }): PigWriteDetails {
return result.details as PigWriteDetails;
}
before(async () => {
if (!databaseUrl) return;
const [user] = await db
.insert(users)
.values({ email: `${marker}@example.test`, name: 'Dana Okonjo', authSubject: randomUUID() })
.returning({ id: users.id });
assert.ok(user);
fixture.userId = user.id;
const [account] = await db
.insert(accounts)
.values({ name: `${marker} Northwind Robotics`, side: 'demand' })
.returning({ id: accounts.id });
assert.ok(account);
fixture.accountId = account.id;
const [deal] = await db
.insert(demandDeals)
.values({ accountId: account.id, name: `${marker} H200 reserved`, stage: 'proposal' })
.returning({ id: demandDeals.id });
assert.ok(deal);
fixture.dealId = deal.id;
});
after(async () => {
if (!databaseUrl) return;
// Activities and deals cascade from the account; the user does not.
if (fixture.accountId) await db.delete(accounts).where(eq(accounts.id, fixture.accountId));
if (fixture.userId) await db.delete(users).where(eq(users.id, fixture.userId));
// Closed explicitly: an open pool keeps the event loop alive, and a suite
// that passes but never exits looks exactly like one that hangs.
await db.$client.end({ timeout: 5 });
});
test('an approved activity is written, and marked as Piggys', { skip: !databaseUrl }, async () => {
const result = await named(tools('confirm', 'apply'), 'pig_log_activity').execute(
'call-1',
{
type: 'call',
subject: 'Pricing call with procurement',
body: 'They want H200 pricing before the board meets.',
accountId: fixture.accountId,
},
undefined,
undefined,
ctx,
);
assert.equal(detailsOf(result).status, 'applied');
const written = await db
.select()
.from(activities)
.where(eq(activities.accountId, fixture.accountId));
assert.equal(written.length, 1);
const [row] = written;
assert.ok(row);
assert.equal(row.subject, 'Pricing call with procurement');
assert.equal(row.actorUserId, fixture.userId, 'the write is attributed to the caller');
// The row IS its own audit event, so the provenance rides on the external id.
assert.match(row.externalId ?? '', /^piggy:/);
const piggyRows = await db
.select()
.from(activities)
.where(and(eq(activities.accountId, fixture.accountId), like(activities.externalId, 'piggy:%')));
assert.equal(piggyRows.length, 1, 'every write Piggy made is selectable by that prefix');
});
test('a rejected change leaves the book exactly as it was', { skip: !databaseUrl }, async () => {
const before = await db.select().from(demandDeals).where(eq(demandDeals.id, fixture.dealId));
const activitiesBefore = await db
.select()
.from(activities)
.where(eq(activities.demandDealId, fixture.dealId));
const result = await named(tools('confirm', 'reject'), 'pig_update_deal_stage').execute(
'call-2',
{
dealType: 'demand',
dealId: fixture.dealId,
stage: 'procurement',
reason: 'Legal cleared the MSA this morning.',
},
undefined,
undefined,
ctx,
);
assert.equal(detailsOf(result).status, 'declined');
const after = await db.select().from(demandDeals).where(eq(demandDeals.id, fixture.dealId));
assert.equal(after[0]?.stage, before[0]?.stage, 'the stage did not move');
const activitiesAfter = await db
.select()
.from(activities)
.where(eq(activities.demandDealId, fixture.dealId));
assert.equal(activitiesAfter.length, activitiesBefore.length, 'no audit row was written');
});
test('an approved stage change carries Piggy in its audit row', { skip: !databaseUrl }, async () => {
const result = await named(tools('confirm', 'apply'), 'pig_update_deal_stage').execute(
'call-3',
{
dealType: 'demand',
dealId: fixture.dealId,
stage: 'procurement',
reason: 'Legal cleared the MSA this morning.',
},
undefined,
undefined,
ctx,
);
assert.equal(detailsOf(result).status, 'applied');
const [deal] = await db.select().from(demandDeals).where(eq(demandDeals.id, fixture.dealId));
assert.equal(deal?.stage, 'procurement');
const [audit] = await db
.select()
.from(activities)
.where(and(eq(activities.demandDealId, fixture.dealId), eq(activities.type, 'stage_change')));
assert.ok(audit, 'the mutation convention wrote its audit row');
assert.equal(audit.subject, 'proposal → procurement');
assert.equal(audit.actorUserId, fixture.userId, 'still the caller, never an elevated principal');
// `actorAgent` on the column stays null because the request really did
// authenticate as a person; the provenance goes where the caller legitimately
// controls the content.
assert.equal(audit.meta?.actorAgent, 'piggy');
assert.equal(audit.meta?.piggyTool, 'pig_update_deal_stage');
assert.equal(audit.meta?.piggyReason, 'Legal cleared the MSA this morning.');
assert.match(audit.body ?? '', /Recorded by Piggy \(pig_update_deal_stage\) on behalf of Dana/);
});
test('a task becomes a calendar entry the user owns', { skip: !databaseUrl }, async () => {
const result = await named(tools('auto', 'apply'), 'pig_create_task').execute(
'call-4',
{
title: 'Send the H200 quote',
startsAt: '2026-09-01',
accountId: fixture.accountId,
},
undefined,
undefined,
ctx,
);
const details = detailsOf(result);
assert.equal(details.status, 'applied');
assert.ok(details.recordId);
});