claude
99d165b5e5
Piggy answered in raw markdown, threw away every tool result it streamed, and fought the reader's scroll on every token. The three surfaces that made it worth having — what it read, how it reasoned, what it cost — were all on the wire and none of them reached the screen. The transcript is now composed of five parts under components/piggy: answers render through streamdown, the container sticks to the bottom without pinning the reader there, tool steps say what they read and link to the record, and each turn carries its model and token count. Three lifecycle bugs went with them: Stop left a permanent spinner, a truncated stream was indistinguishable from thinking, and a failed send destroyed the message it failed to send. Underneath, the inference path grew timeouts, jittered retries on 429 and 5xx, tolerance of the malformed frames a 30B model emits, and an agent_runs row per turn so chat spend is observable. The system prompt now states that a field ending in Cents is cents — without it nemotron renders costPerGpuHourCents: 189 as "$189 per GPU-hour", which is a 100x error on the most scrutinised number in the room. The demo book was arithmetically incoherent: every deal's value contradicted its own allocation revenue by up to 3.6x, nothing had ever closed, no customer had any paper, and the marketplace was empty. Deal value is now derived from the allocation, the book clears 5.3% across five blocks with one deliberately underwater, and the renewal, compliance and agent-provenance machinery finally has rows to act on. A --clear that deleted every obligation, SLA term and capacity request in the database regardless of origin is scoped to the demo's own ids. Around that: accounts have a detail page, ⌘K searches the book, Settings can mint the API keys it always claimed to, and deploy.sh actually ships the agent instead of silently skipping its compose profile. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
232 lines
10 KiB
Bash
Executable File
232 lines
10 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
#
|
|
# Poll the registry for a new release and deploy it. Runs on the production
|
|
# host, on a timer. Installation: see deploy/README.md.
|
|
#
|
|
# This is the half of "push to deploy" that does not require handing the CI
|
|
# runner a key to this machine. CI publishes an image when a human tags
|
|
# `release-*`; this pulls. The only credential involved lives HERE, is
|
|
# read-only, and can do nothing but fetch a manifest.
|
|
#
|
|
# What it does, in order:
|
|
#
|
|
# 1. Ask the registry for the newest release-* tag.
|
|
# 2. Compare that tag's manifest digest with the digest of the image the
|
|
# running app container was started from — and the piggy container's too,
|
|
# when Piggy is enabled, because it is the same image and a Piggy left
|
|
# behind runs old code against a migrated schema. All equal -> exit 0,
|
|
# silently. This is the normal case and it happens every five minutes.
|
|
# 3. Otherwise check the working tree out at that tag — the compose file and
|
|
# the migrations must come from the same commit as the image — and hand
|
|
# over to scripts/deploy.sh with PIG_IMAGE set.
|
|
#
|
|
# A failed deploy is recorded and NOT retried. A poller that reattempts the
|
|
# same broken release every five minutes turns one bad tag into a restart loop;
|
|
# deleting the record (or tagging a new release) is the human's signal to try
|
|
# again.
|
|
|
|
set -euo pipefail
|
|
|
|
# git checkout rewrites this file while bash is reading it — bash reads a
|
|
# script incrementally, by byte offset, so a checkout mid-run resumes at an
|
|
# offset into different content. Re-exec from a private copy before touching
|
|
# git. This has a long history of producing genuinely baffling syntax errors.
|
|
if [ "${PIG_AUTODEPLOY_REEXEC:-}" != '1' ]; then
|
|
_copy=$(mktemp -t pig-autodeploy.XXXXXX)
|
|
cat "$0" > "$_copy"
|
|
PIG_AUTODEPLOY_REEXEC=1 exec bash "$_copy" "$@"
|
|
fi
|
|
trap 'rm -f "$0"' EXIT
|
|
|
|
REPO_DIR="${PIG_REPO_DIR:-/opt/pig}"
|
|
REGISTRY="${PIG_REGISTRY:-git.karti.ai}"
|
|
# Gitea namespaces packages under the lowercased owner: PIG/pig -> pig/pig.
|
|
IMAGE_REPO="${PIG_IMAGE_REPO:-pig/pig}"
|
|
TOKEN_FILE="${PIG_REGISTRY_TOKEN_FILE:-/etc/pig/registry-token}"
|
|
REGISTRY_USER="${PIG_REGISTRY_USER:-pig-deploy}"
|
|
STATE_DIR="${PIG_STATE_DIR:-/var/lib/pig}"
|
|
TAG_PREFIX="${PIG_RELEASE_TAG_PREFIX:-release-}"
|
|
|
|
IMAGE_NAME="$REGISTRY/$IMAGE_REPO"
|
|
FAILED_MARKER="$STATE_DIR/failed-release"
|
|
|
|
log() { printf '%s autodeploy: %s\n' "$(date -Is)" "$*"; }
|
|
die() { log "ERROR: $*"; exit 1; }
|
|
|
|
# Only one deploy at a time. The systemd timer will not overlap a run with
|
|
# itself, but a human running this by hand during a scheduled run would.
|
|
LOCK="${PIG_LOCK_FILE:-/var/lock/pig-autodeploy.lock}"
|
|
exec 9>"$LOCK"
|
|
if ! flock -n 9; then
|
|
log "another run holds $LOCK; nothing to do"
|
|
exit 0
|
|
fi
|
|
|
|
[ -r "$TOKEN_FILE" ] || die "no readable registry token at $TOKEN_FILE"
|
|
# The token is the only secret here and it is pull-only, but a world-readable
|
|
# credential on a production host is still worth refusing.
|
|
PERMS=$(stat -c '%a' "$TOKEN_FILE")
|
|
case "$PERMS" in
|
|
600|400) ;;
|
|
*) die "$TOKEN_FILE is mode $PERMS; must be 0600" ;;
|
|
esac
|
|
REGISTRY_TOKEN=$(tr -d '\r\n' < "$TOKEN_FILE")
|
|
[ -n "$REGISTRY_TOKEN" ] || die "$TOKEN_FILE is empty"
|
|
|
|
cd "$REPO_DIR" || die "no checkout at $REPO_DIR"
|
|
|
|
# ---------------------------------------------------------------- registry
|
|
#
|
|
# Gitea's registry speaks the standard v2 token dance: an unauthenticated
|
|
# request gets a 401 pointing at /v2/token. Exchange the pull-only credential
|
|
# for a bearer token scoped to this one repository and fall back to basic auth
|
|
# if the exchange gives nothing, because Gitea accepts both and which one you
|
|
# get has changed across versions.
|
|
AUTH_ARGS=()
|
|
BEARER=$(curl -sS --max-time 20 -u "$REGISTRY_USER:$REGISTRY_TOKEN" \
|
|
"https://$REGISTRY/v2/token?service=$REGISTRY&scope=repository:$IMAGE_REPO:pull" \
|
|
2>/dev/null | sed -n 's/.*"token":"\([^"]*\)".*/\1/p' | head -n1 || true)
|
|
if [ -n "$BEARER" ]; then
|
|
AUTH_ARGS=(-H "Authorization: Bearer $BEARER")
|
|
else
|
|
AUTH_ARGS=(-u "$REGISTRY_USER:$REGISTRY_TOKEN")
|
|
fi
|
|
|
|
MANIFEST_ACCEPT=(
|
|
-H 'Accept: application/vnd.oci.image.index.v1+json'
|
|
-H 'Accept: application/vnd.oci.image.manifest.v1+json'
|
|
-H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json'
|
|
-H 'Accept: application/vnd.docker.distribution.manifest.v2+json'
|
|
)
|
|
|
|
TAGS_JSON=$(curl -sS --max-time 30 "${AUTH_ARGS[@]}" \
|
|
"https://$REGISTRY/v2/$IMAGE_REPO/tags/list?n=1000") \
|
|
|| die "could not reach https://$REGISTRY/v2/$IMAGE_REPO/tags/list"
|
|
|
|
# sort -V rather than sort: release-10 must come after release-9. This assumes
|
|
# release tags sort sensibly — date-stamped or semver. A tag scheme that does
|
|
# not is a tag scheme this poller will deploy in the wrong order.
|
|
LATEST_TAG=$(printf '%s' "$TAGS_JSON" | tr ',' '\n' \
|
|
| sed -n "s/.*\"\(${TAG_PREFIX}[^\"]*\)\".*/\1/p" | sort -V | tail -n1)
|
|
[ -n "$LATEST_TAG" ] || { log "no ${TAG_PREFIX}* tags published yet"; exit 0; }
|
|
|
|
REMOTE_DIGEST=$(curl -sSI --max-time 30 "${AUTH_ARGS[@]}" "${MANIFEST_ACCEPT[@]}" \
|
|
"https://$REGISTRY/v2/$IMAGE_REPO/manifests/$LATEST_TAG" \
|
|
| tr -d '\r' | sed -n 's/^[Dd]ocker-[Cc]ontent-[Dd]igest:[[:space:]]*//p' | tail -n1)
|
|
[ -n "$REMOTE_DIGEST" ] || die "no digest for $IMAGE_NAME:$LATEST_TAG"
|
|
|
|
# ---------------------------------------------------------------- compare
|
|
#
|
|
# Compare digests, not tags. A tag can be moved; a digest is the content. This
|
|
# also means a re-pushed tag redeploys, which is what you want the one time it
|
|
# matters.
|
|
|
|
# Read one key out of the deployed .env WITHOUT sourcing it: sourcing an
|
|
# environment file executes it, and this one holds every secret the deployment
|
|
# has. The same helper as scripts/deploy.sh, for the same reason.
|
|
env_value() {
|
|
[ -f .env ] || return 0
|
|
sed -n "s/^[[:space:]]*$1=//p" .env | tail -n1 | sed -e 's/^"\(.*\)"$/\1/' -e "s/^'\(.*\)'\$/\1/"
|
|
}
|
|
|
|
# The words the API accepts, from `envBoolean` in apps/api/src/lib/config.ts.
|
|
is_true() {
|
|
# Whitespace and a trailing inline comment are both dropped by compose before
|
|
# a container sees the value; read it the same way deploy.sh does.
|
|
local value=${1%%[[:space:]]#*}
|
|
case "$(printf '%s' "$value" | tr '[:upper:]' '[:lower:]' | tr -d '[:space:]')" in
|
|
1 | true | yes | on) return 0 ;;
|
|
*) return 1 ;;
|
|
esac
|
|
}
|
|
|
|
# The piggy service is profile-gated, and compose skips a profile-gated service
|
|
# silently — with no profile, `ps -q piggy` prints nothing at all unless that
|
|
# container happens to be running already. Without this, a Piggy left on an
|
|
# older image is invisible here: the app matches the newest tag, this reports
|
|
# "up to date" every five minutes, and the agent runs last month's code.
|
|
COMPOSE_PROFILES=''
|
|
if is_true "$(env_value PIGGY_ENABLED)"; then
|
|
COMPOSE_PROFILES='piggy'
|
|
fi
|
|
export COMPOSE_PROFILES
|
|
|
|
# The registry digest of the image a running container was started from, or the
|
|
# empty string when there is no such container.
|
|
running_digest() {
|
|
local cid image
|
|
cid=$(docker compose -p pig ps -q "$1" 2>/dev/null | head -n1 || true)
|
|
[ -n "$cid" ] || return 0
|
|
image=$(docker inspect -f '{{.Image}}' "$cid" 2>/dev/null || true)
|
|
[ -n "$image" ] || return 0
|
|
docker image inspect "$image" --format '{{range .RepoDigests}}{{println .}}{{end}}' 2>/dev/null \
|
|
| sed -n "s|^$IMAGE_NAME@||p" | head -n1 || true
|
|
}
|
|
|
|
RUNNING_DIGEST=$(running_digest app)
|
|
PIGGY_DIGEST=''
|
|
[ -z "$COMPOSE_PROFILES" ] || PIGGY_DIGEST=$(running_digest piggy)
|
|
|
|
# Both halves of the release, or neither. app and piggy are the same image
|
|
# running two commands, so a piggy behind the app is a deploy that only half
|
|
# happened — and it is the half that writes to the database.
|
|
if [ "$RUNNING_DIGEST" = "$REMOTE_DIGEST" ] \
|
|
&& { [ -z "$COMPOSE_PROFILES" ] || [ "$PIGGY_DIGEST" = "$REMOTE_DIGEST" ]; }; then
|
|
log "up to date at $LATEST_TAG ($REMOTE_DIGEST)"
|
|
exit 0
|
|
fi
|
|
|
|
if [ -f "$FAILED_MARKER" ] && [ "$(cat "$FAILED_MARKER")" = "$REMOTE_DIGEST" ]; then
|
|
log "$LATEST_TAG ($REMOTE_DIGEST) already failed to deploy; not retrying."
|
|
log "Remove $FAILED_MARKER to try again, or publish a new release tag."
|
|
exit 0
|
|
fi
|
|
|
|
log "deploying $LATEST_TAG"
|
|
log " running: ${RUNNING_DIGEST:-<none>}"
|
|
[ -z "$COMPOSE_PROFILES" ] || log " piggy: ${PIGGY_DIGEST:-<none>}"
|
|
log " wanted: $REMOTE_DIGEST"
|
|
|
|
# ---------------------------------------------------------------- deploy
|
|
#
|
|
# Log in as root, which is the identity `sudo docker` in deploy.sh runs as, so
|
|
# the pull there finds this credential. Deliberately not logged out afterwards:
|
|
# a human re-running deploy.sh by hand needs the same pull to work, and the
|
|
# token is read-only.
|
|
printf '%s' "$REGISTRY_TOKEN" | docker login "$REGISTRY" -u "$REGISTRY_USER" --password-stdin > /dev/null \
|
|
|| die "docker login to $REGISTRY failed"
|
|
|
|
git fetch --tags --force --prune origin
|
|
# Detached, at the exact tag. The image, the compose file and the migration set
|
|
# must all come from one commit; a tree left on main while running a tagged
|
|
# image is how a migration gets applied that the image knows nothing about.
|
|
git checkout --detach --force "refs/tags/$LATEST_TAG" \
|
|
|| die "cannot check out refs/tags/$LATEST_TAG"
|
|
|
|
mkdir -p "$STATE_DIR"
|
|
if PIG_IMAGE="$IMAGE_NAME:$LATEST_TAG" bash scripts/deploy.sh; then
|
|
rm -f "$FAILED_MARKER"
|
|
printf '%s %s\n' "$LATEST_TAG" "$REMOTE_DIGEST" > "$STATE_DIR/current-release"
|
|
log "deployed $LATEST_TAG"
|
|
else
|
|
STATUS=$?
|
|
printf '%s' "$REMOTE_DIGEST" > "$FAILED_MARKER"
|
|
log "deploy of $LATEST_TAG FAILED (exit $STATUS)."
|
|
# What is serving right now depends on WHICH gate failed, and this line is
|
|
# what the on-call reads at 04:00. It used to claim a rollback unconditionally
|
|
# — including in the one case where the broken release is still up, which is
|
|
# precisely the case that needs hands. deploy.sh exits 3 for that.
|
|
if [ "$STATUS" -eq 3 ]; then
|
|
log "THE FAILING RELEASE IS STILL LIVE — deploy.sh did not roll back (it"
|
|
log "either judged the fault external to the release, had no previous image,"
|
|
log "or the restored image did not come up). Check the site NOW."
|
|
log "One exit-3 case leaves the CRM serving normally: Piggy enabled but not"
|
|
log "coming up, usually a missing PIGGY_INFERENCE_API_KEY. The log above says"
|
|
log "which it was."
|
|
else
|
|
log "deploy.sh rolls the app back on a failed health, auth or public-marker"
|
|
log "gate, so the previous release should still be serving — verify that first."
|
|
fi
|
|
exit "$STATUS"
|
|
fi
|