PIG/pig
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Make every tool result say what it counted
CI / verify (push) Successful in 7m44s
Details
CI / publish (push) Has been skipped
Details

Browse Source 
Asked how many capacity commitments were on the book, Piggy answered "3".
Production holds 5. It had called the idle-capacity tool, which filters to
blocks above an idle threshold, and read the length of that list as the size
of the book.

The system prompt already forbade this in terms — "never report a filtered
count as a total; pig_get_idle_capacity returns the blocks with idle hours,
not the book" — and the model did it anyway. That is the second time this
argument has been lost in the prompt, so it is settled in the payload
instead: a result that cannot describe its own scope will be misread
eventually, however firmly the prompt objects.

Every tool that returns a count or a collection now carries one shape:
what it covers, how many matched, out of how many, under which filters, and
whether the list was truncated. The denominators are read from the database
rather than inferred. The pre-formatted headline states the scope too, since
that is the sentence a small model quotes most readily — the idle tool now
opens "3 of 5 live capacity commitments on the book", which is the sentence
that makes the original mistake impossible to phrase.

Two details worth keeping. Record reads enumerate rather than filter, so
their scope states a boundary instead of a ratio: these are that record's own
figures, never book-wide totals. And the workspace summary's idle threshold
is deliberately recorded as 0, distinct from the idle tool's 0.25 — that
mismatch is why three different idle figures appeared across the UI, and
naming it in the data is how it stops being invisible.

Verified against the live model: the failing question now answers 5, demand
deals 13 and contracts 20 — each drawn from a payload whose filtered figure
was smaller — while "which blocks are sitting idle" still names exactly the
blocks that are.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
claude
2026-08-14 19:03:40 -07:00
parent 18d5f5bfc0
commit f2ef403ee9
6 changed files with 1117 additions and 113 deletions
Download Patch File Download Diff File Expand all files Collapse all files
apps/piggy/src/chat-tools.ts
+279 -62
View File
@@ -41,15 +41,52 @@ import {
} from '@pig/db';
import { CapacityService } from '@pig/api/src/services/capacity';
import { renewalAlarm } from '@pig/api/src/services/contracts';
import { and, asc, eq, gt, ilike, inArray, isNotNull, isNull } from 'drizzle-orm';
import { and, asc, count, eq, gt, ilike, inArray, isNotNull, isNull } from 'drizzle-orm';
import { z } from 'zod';
import type { PiggyChatContext } from './chat';
import { createAccountLifecycleTool } from './lifecycle-tools';
import { createPagePigTools } from './page-tools';
import { atLeast, createPagePigTools, resultScope, type ResultScope } from './page-tools';
import { defineTool, type AgentTool } from './provider';
const noInput = z.object({}).strict();
/**
* The per-collection cap on a record read.
*
* Named rather than repeated as a literal because the scope below reports it:
* a related list that came back exactly full is a list that was probably cut,
* and a cut list the reader cannot see is how "this account has 100 deals"
* gets said about an account with three hundred.
*/
const RELATED_LIMIT = 100;
/**
* The scope of a record read.
*
* Unlike the page tools, a record read filters nothing — it enumerates what
* belongs to one row — so `matched` equals `total` and the sentence is not
* hedged. What it must still say is the boundary, because the failure here is
* the same shape as the /capacity one measured in production: asked how many
* deals are on the book while an account is in focus, a model with only this
* payload counts the four in front of it. `totalLabel` therefore names the
* record and says the figures stop there.
*/
function recordScope(subject: string, collections: Record<string, readonly unknown[]>): ResultScope {
const entries = Object.entries(collections);
const rows = entries.reduce((sum, [, list]) => sum + list.length, 0);
const capped = entries.some(([, list]) => list.length >= RELATED_LIMIT);
const breakdown = entries.map(([label, list]) => `${list.length} ${label}`).join(', ');
return resultScope({
covers: `belong to ${subject}`,
matched: rows,
total: rows,
totalLabel: `record(s) belonging to ${subject} and to no other — ${breakdown}; these are that record's own figures, never book-wide totals`,
listed: rows,
filters: capped ? { rowCapPerCollection: RELATED_LIMIT } : {},
truncated: capped,
});
}
/**
* Interactive chat gets one scoped read tool for where it is, plus the lookup
* layer, and no ambient access.
@@ -102,12 +139,24 @@ async function readFocusedRecord(db: Database, context: PiggyRecordContext): Pro
const [account] = await db.select().from(accounts).where(eq(accounts.id, context.id)).limit(1);
if (!account) throw missingRecord(context.type, context.id);
const [people, demand, supply, paperwork] = await Promise.all([
db.select().from(contacts).where(eq(contacts.accountId, context.id)).limit(100),
db.select().from(demandDeals).where(eq(demandDeals.accountId, context.id)).limit(100),
db.select().from(supplyDeals).where(eq(supplyDeals.accountId, context.id)).limit(100),
db.select().from(contracts).where(eq(contracts.accountId, context.id)).limit(100),
db.select().from(contacts).where(eq(contacts.accountId, context.id)).limit(RELATED_LIMIT),
db.select().from(demandDeals).where(eq(demandDeals.accountId, context.id)).limit(RELATED_LIMIT),
db.select().from(supplyDeals).where(eq(supplyDeals.accountId, context.id)).limit(RELATED_LIMIT),
db.select().from(contracts).where(eq(contracts.accountId, context.id)).limit(RELATED_LIMIT),
]);
return { account, contacts: people, demandDeals: demand, supplyDeals: supply, contracts: paperwork };
return {
scope: recordScope(`the account ${account.name}`, {
'contact(s)': people,
'demand deal(s)': demand,
'supply deal(s)': supply,
'contract(s)': paperwork,
}),
account,
contacts: people,
demandDeals: demand,
supplyDeals: supply,
contracts: paperwork,
};
}
if (context.type === 'contact') {
@@ -116,7 +165,13 @@ async function readFocusedRecord(db: Database, context: PiggyRecordContext): Pro
const [account] = contact.accountId
? await db.select().from(accounts).where(eq(accounts.id, contact.accountId)).limit(1)
: [];
return { contact, account: account ?? null };
return {
scope: recordScope(`the contact ${contact.fullName}`, {
'account(s)': account ? [account] : [],
}),
contact,
account: account ?? null,
};
}
if (context.type === 'demand_deal') {
@@ -127,8 +182,16 @@ async function readFocusedRecord(db: Database, context: PiggyRecordContext): Pro
.select()
.from(allocations)
.where(eq(allocations.demandDealId, deal.id))
.limit(100);
return { deal, account: account ?? null, allocations: reservations };
.limit(RELATED_LIMIT);
return {
scope: recordScope(`the demand deal ${deal.name}`, {
'allocation(s)': reservations,
'account(s)': account ? [account] : [],
}),
deal,
account: account ?? null,
allocations: reservations,
};
}
if (context.type === 'supply_deal') {
@@ -139,8 +202,16 @@ async function readFocusedRecord(db: Database, context: PiggyRecordContext): Pro
.select()
.from(capacityCommitments)
.where(eq(capacityCommitments.supplyDealId, deal.id))
.limit(100);
return { deal, account: account ?? null, commitments };
.limit(RELATED_LIMIT);
return {
scope: recordScope(`the supply deal ${deal.name}`, {
'capacity commitment(s)': commitments,
'account(s)': account ? [account] : [],
}),
deal,
account: account ?? null,
commitments,
};
}
if (context.type === 'commitment') {
@@ -154,8 +225,14 @@ async function readFocusedRecord(db: Database, context: PiggyRecordContext): Pro
.select()
.from(allocations)
.where(eq(allocations.capacityCommitmentId, commitment.id))
.limit(100);
return { commitment, allocations: reservations };
.limit(RELATED_LIMIT);
return {
scope: recordScope(`the capacity commitment ${commitment.name}`, {
'allocation(s)': reservations,
}),
commitment,
allocations: reservations,
};
}
const [contract] = await db.select().from(contracts).where(eq(contracts.id, context.id)).limit(1);
@@ -166,16 +243,26 @@ async function readFocusedRecord(db: Database, context: PiggyRecordContext): Pro
.select()
.from(contractObligations)
.where(eq(contractObligations.contractId, contract.id))
.limit(100),
.limit(RELATED_LIMIT),
]);
const metrics = serviceLevels[0]
? await db
.select()
.from(slaMetricTargets)
.where(eq(slaMetricTargets.slaTermId, serviceLevels[0].id))
.limit(100)
.limit(RELATED_LIMIT)
: [];
return { contract, slaTerms: serviceLevels, slaMetricTargets: metrics, obligations };
return {
scope: recordScope(`the contract ${contract.title}`, {
'SLA term(s)': serviceLevels,
'SLA metric target(s)': metrics,
'obligation(s)': obligations,
}),
contract,
slaTerms: serviceLevels,
slaMetricTargets: metrics,
obligations,
};
}
// ---------------------------------------------------------------------------
@@ -416,7 +503,18 @@ async function searchRecords(db: Database, query: string): Promise<unknown> {
const fragment = likeFragment(query);
const take = SEARCH_PER_TYPE + 1;
const [accountRows, demandRows, supplyRows, contractRows, commitmentRows] = await Promise.all([
const [
accountRows,
demandRows,
supplyRows,
contractRows,
commitmentRows,
accountsAll,
demandAll,
supplyAll,
contractsAll,
commitmentsAll,
] = await Promise.all([
db
.select({
id: accounts.id,
@@ -482,6 +580,14 @@ async function searchRecords(db: Database, query: string): Promise<unknown> {
.from(capacityCommitments)
.where(ilike(capacityCommitments.name, fragment))
.limit(take),
// The five denominators. A search that reports only its hits invites
// "there are 3 accounts" from a book of twenty-three, and these counts also
// make this tool able to answer how many of a thing exist at all.
db.select({ value: count() }).from(accounts).where(isNull(accounts.archivedAt)),
db.select({ value: count() }).from(demandDeals),
db.select({ value: count() }).from(supplyDeals),
db.select({ value: count() }).from(contracts),
db.select({ value: count() }).from(capacityCommitments),
]);
const names = await accountNames(db, [
@@ -491,6 +597,8 @@ async function searchRecords(db: Database, query: string): Promise<unknown> {
...commitmentRows.map((row) => row.accountId),
]);
const rows = (result: readonly { value: number }[]): number => result[0]?.value ?? 0;
return assembleSearchResult(query, {
accounts: accountRows,
demandDeals: demandRows,
@@ -498,6 +606,13 @@ async function searchRecords(db: Database, query: string): Promise<unknown> {
contracts: contractRows,
commitments: commitmentRows,
accountNames: names,
totals: {
account: rows(accountsAll),
demand_deal: rows(demandAll),
supply_deal: rows(supplyAll),
contract: rows(contractsAll),
commitment: rows(commitmentsAll),
},
});
}
@@ -549,8 +664,18 @@ export interface SearchRowSets {
costPerGpuHourCents: number;
}[];
accountNames: ReadonlyMap<string, string>;
/**
* How many rows each searched table holds in total — the denominators the
* per-type match counts are drawn from. Keyed by the same names the results
* carry, so a model reading `counts.account: 1` beside `totals.account: 23`
* cannot mistake a name match for a census.
*/
totals: Record<SearchedRecordType, number>;
}
/** The five types a name search covers. People are deliberately not indexed. */
export type SearchedRecordType = Exclude<PiggyRecordType, 'contact'>;
/**
* Ranking, capping and counting, with no database in sight.
*
@@ -664,20 +789,35 @@ export function assembleSearchResult(query: string, sets: SearchRowSets): unknow
commitmentsCut.truncated;
const results = ranked.slice(0, SEARCH_RESULTS);
const truncated = perTypeTruncated || ranked.length > results.length;
const searched = Object.values(sets.totals).reduce((sum, rows) => sum + rows, 0);
const breakdown = Object.entries(counts)
.filter(([, matches]) => matches > 0)
.map(([type, matches]) => `${matches} of ${sets.totals[type as SearchedRecordType]} ${type}(s)`)
.join(', ');
return {
headline:
results.length === 0
? `No account, deal, contract or capacity commitment has a name containing "${query}".`
: `${truncated ? 'at least ' : ''}${ranked.length} record(s) match "${query}": ` +
Object.entries(counts)
.filter(([, count]) => count > 0)
.map(([type, count]) => `${count} ${type}(s)`)
.join(', ') +
'.',
? `None of the ${searched} account(s), deal(s), contract(s) and capacity commitment(s) ` +
`on the book has a name containing "${query}".`
: `${truncated ? 'At least ' : ''}${ranked.length} of ${searched} searchable record(s) ` +
`match "${query}": ${breakdown}. Those are name matches, not totals; the counts they ` +
'were drawn from are beside them.',
scope: resultScope({
covers: `have a name containing "${query}"`,
matched: ranked.length,
total: searched,
totalLabel:
'record(s) searchable by name: accounts, demand deals, supply deals, contracts and capacity commitments',
listed: results.length,
filters: { query },
truncated,
}),
query,
truncated,
counts,
/** The denominator for each entry in `counts`, keyed identically. */
totals: sets.totals,
results: results.map((entry) => entry.hit),
};
}
@@ -702,26 +842,36 @@ export function assembleSearchResult(query: string, sets: SearchRowSets): unknow
*/
async function listRenewals(db: Database, side: 'demand' | 'supply' | undefined): Promise<unknown> {
const now = new Date();
const rows = await db
.select({ contract: contracts, accountName: accounts.name })
.from(contracts)
.leftJoin(accounts, eq(accounts.id, contracts.accountId))
.where(
and(
eq(contracts.status, 'executed'),
isNull(contracts.terminatedAt),
isNotNull(contracts.expiresAt),
gt(contracts.expiresAt, now),
side ? eq(contracts.side, side) : undefined,
),
)
.orderBy(asc(contracts.expiresAt))
.limit(SCAN_LIMIT + 1);
// The denominator is counted on the same side filter the list uses, so
// "4 of 20" and "4 of 11 on the demand side" are both answers to the
// question that was actually asked.
const [rows, all] = await Promise.all([
db
.select({ contract: contracts, accountName: accounts.name })
.from(contracts)
.leftJoin(accounts, eq(accounts.id, contracts.accountId))
.where(
and(
eq(contracts.status, 'executed'),
isNull(contracts.terminatedAt),
isNotNull(contracts.expiresAt),
gt(contracts.expiresAt, now),
side ? eq(contracts.side, side) : undefined,
),
)
.orderBy(asc(contracts.expiresAt))
.limit(SCAN_LIMIT + 1),
db
.select({ value: count() })
.from(contracts)
.where(side ? eq(contracts.side, side) : undefined),
]);
return assembleRenewals(rows.slice(0, SCAN_LIMIT), {
now,
side,
truncated: rows.length > SCAN_LIMIT,
totalContracts: all[0]?.value ?? 0,
});
}
@@ -745,9 +895,15 @@ export interface RenewalRow {
*/
export function assembleRenewals(
rows: readonly RenewalRow[],
options: { now: Date; side?: 'demand' | 'supply'; truncated: boolean },
options: {
now: Date;
side?: 'demand' | 'supply';
truncated: boolean;
/** Every contract on this side, whatever its status. The denominator. */
totalContracts: number;
},
): unknown {
const { now, side, truncated } = options;
const { now, side, truncated, totalContracts } = options;
const renewals = rows
.flatMap(({ contract, accountName }) => {
// The query already requires an expiry; narrowing here rather than
@@ -788,26 +944,51 @@ export function assembleRenewals(
const statedValueCents = noticeOpen.reduce((sum, row) => sum + (row.valueCents ?? 0), 0);
const anyStatedValue = noticeOpen.some((row) => row.valueCents != null);
const sideLabel = side ? `${side}-side contract(s) on the book` : 'contract(s) on the book';
const listed = Math.min(renewals.length, EXEMPLARS);
return {
headline:
(nearest
? `${truncated ? 'At least ' : ''}${renewals.length} executed contract(s) still live` +
`${side ? ` on the ${side} side` : ''}. Nearest deadline: the ` +
? `${truncated ? 'At least ' : ''}${renewals.length} of ${totalContracts} ${sideLabel} ` +
'are executed and not yet expired' +
`${renewals.length > listed ? `; the nearest ${listed} are listed` : ''}. ` +
'Nearest deadline: the ' +
`${nearest.deadlineKind === 'renewal_notice' ? 'renewal notice' : 'expiry'} for ` +
`${nearest.title}${nearest.accountName ? ` (${nearest.accountName})` : ''} on ` +
`${nearest.deadlineAt.slice(0, 10)}` +
`${nearest.daysUntilDeadline < 0 ? ', which has already passed' : ''}.`
: `No executed contract${side ? ` on the ${side} side` : ''} has an expiry date ahead of it.`) +
: `None of the ${totalContracts} ${sideLabel} is executed with an expiry date ahead of it.`) +
(noticeOpen.length > 0
? ` ${noticeOpen.length} notice window(s) already open` +
? ` ${noticeOpen.length} of those ${renewals.length} have a notice window already open` +
(anyStatedValue
? `, covering ${formatCents(statedValueCents)} of stated contract value.`
: '; none of those contracts states a value of its own.')
: ''),
scope: resultScope({
covers: 'are executed, not terminated and not yet expired',
matched: renewals.length,
total: totalContracts,
totalLabel: sideLabel,
listed,
filters: { side: side ?? 'both', status: 'executed', expired: 'excluded' },
truncated,
}),
side: side ?? 'both',
truncated,
count: renewals.length,
totalContracts,
noticeWindowOpenCount: noticeOpen.length,
/** A filter over a filter, so it states its own denominator too. */
noticeWindowOpenScope: resultScope({
covers: 'have a renewal-notice window that is already open',
matched: noticeOpen.length,
total: renewals.length,
totalLabel: `executed, unexpired ${sideLabel}`,
listed: 0,
filters: { renewalState: 'due' },
truncated,
}),
renewals: renewals.slice(0, EXEMPLARS),
};
}
@@ -836,11 +1017,19 @@ interface InventoryQuery {
* bounded read. The width never leaves this process; only EXEMPLARS rows do.
*/
async function listInventory(db: Database, query: InventoryQuery): Promise<unknown> {
const listings = await new CapacityService(db).searchInventory({
minGpuCount: query.minGpuCount,
requiresHighSpeedInterconnect: query.requiresFastInterconnect,
limit: SCAN_LIMIT,
});
const capacity = new CapacityService(db);
// The unfiltered read is the denominator, and it is taken through the same
// service rather than counted here: the service decides what "purchasable"
// means (it drops Unavailable stock), and a denominator computed from a
// second definition of that word would disagree with its own numerator.
const [listings, market] = await Promise.all([
capacity.searchInventory({
minGpuCount: query.minGpuCount,
requiresHighSpeedInterconnect: query.requiresFastInterconnect,
limit: SCAN_LIMIT,
}),
capacity.searchInventory({ limit: SCAN_LIMIT }),
]);
const providerNames = await accountNames(
db,
listings.flatMap((listing) => (listing.accountId ? [listing.accountId] : [])),
@@ -850,6 +1039,8 @@ async function listInventory(db: Database, query: InventoryQuery): Promise<unkno
// available that there was more behind it.
truncated: listings.length >= SCAN_LIMIT,
providerNames,
totalListings: market.length,
totalTruncated: market.length >= SCAN_LIMIT,
});
}
@@ -879,9 +1070,15 @@ export type InventoryOffer = Pick<
export function assembleInventoryResult(
query: InventoryQuery,
listings: readonly InventoryOffer[],
options: { truncated: boolean; providerNames: ReadonlyMap<string, string> },
options: {
truncated: boolean;
providerNames: ReadonlyMap<string, string>;
/** Purchasable listings on the market with no filter applied at all. */
totalListings: number;
totalTruncated: boolean;
},
): unknown {
const { truncated, providerNames: providers } = options;
const { truncated, providerNames: providers, totalListings, totalTruncated } = options;
const needle = query.gpuType?.toLowerCase();
const matched = needle
? listings.filter((listing) => listing.gpuType.toLowerCase().includes(needle))
@@ -894,23 +1091,43 @@ export function assembleInventoryResult(
);
const cheapest = ranked.find((listing) => listing.onDemandPriceCents != null);
const filters = {
gpuType: query.gpuType ?? null,
minGpuCount: query.minGpuCount ?? null,
requiresFastInterconnect: query.requiresFastInterconnect ?? false,
};
const anyFilter = Object.values(filters).some((value) => value !== null && value !== false);
const listed = Math.min(ranked.length, EXEMPLARS);
const market = 'purchasable listing(s) on the market';
return {
headline:
ranked.length === 0
? `No provider is currently listing capacity matching that request${query.gpuType ? ` for ${query.gpuType}` : ''}.`
: `${truncated ? 'At least ' : ''}${ranked.length} purchasable listing(s)` +
`${query.gpuType ? ` matching ${query.gpuType}` : ''}` +
? `None of the ${atLeast(totalListings, totalTruncated)} ${market} matches that ` +
`request${query.gpuType ? ` for ${query.gpuType}` : ''}.`
: `${ranked.length} of ${atLeast(totalListings, totalTruncated)} ${market} match` +
`${anyFilter ? ' the filters given' : ' (no filter was applied)'}` +
`${query.gpuType ? `, including ${query.gpuType}` : ''}` +
(cheapest
? `; cheapest on-demand is ${formatCents(cheapest.onDemandPriceCents ?? 0)} per ` +
`GPU-hour for ${cheapest.gpuType}.`
: '; none of them carry a published on-demand price.'),
: '; none of them carry a published on-demand price.') +
` ${listed} listed here.`,
scope: resultScope({
covers: anyFilter ? 'match the filters given' : 'are purchasable',
matched: ranked.length,
total: totalListings,
totalLabel: market,
listed,
// An unasked-for filter is not a filter: passing the three nulls through
// would have an unfiltered result describe itself as a slice.
filters: anyFilter ? filters : {},
truncated: truncated || totalTruncated,
}),
truncated,
count: ranked.length,
filters: {
gpuType: query.gpuType ?? null,
minGpuCount: query.minGpuCount ?? null,
requiresFastInterconnect: query.requiresFastInterconnect ?? false,
},
totalListings,
filters,
listings: ranked.slice(0, EXEMPLARS).map((listing) => shapeListing(listing, providers)),
};
}