compute/docs/operations.md

Lumbridge Compute node operations

Persistent Scene state

Compute stores state at <root>/.compute/state.yaml. State writes use an atomic temporary-file replacement. The persisted lifecycle fields are:

• desired_scene: the Scene the node should resume after an agent or node restart;
• active_scene: the Scene whose exact model health was last verified;
• last_known_good_scene: the previous proven Scene used for recovery;
• transition_scene: an in-progress transaction, cleared on success or failure;
• last_error: the most recent activation or watchdog failure.

Process ownership is bound to (boot_id, pid, process_start_ticks). Compute will never signal a legacy PID-only record or a PID that Linux has reused.

One-time adoption on an existing node

Before the first transactional switch, bind the already-running exact Scene to process identities. Adoption neither starts nor stops a model.

lumbridge-compute --root $LUMBRIDGE_COMPUTE_ROOT scene adopt voice-qwen
lumbridge-compute --root $LUMBRIDGE_COMPUTE_ROOT status
lumbridge-compute --root $LUMBRIDGE_COMPUTE_ROOT scene activate voice-qwen --dry-run

Adoption fails if a Scene model is unhealthy, its health marker identifies the wrong model, another registered model is serving, a legacy process record is missing, or its process group does not equal its PID.

Transactional switching and recovery

lumbridge-compute scene activate voice-laguna --dry-run
lumbridge-compute scene activate voice-laguna
lumbridge-compute scene resume

Activation validates the complete Scene and all process ownership before any stop. A shared port occupied by the wrong model is a hard failure. Every started model must report its exact health marker. If a transition fails after mutation, Compute stops the partial target and restores the previously active Scene.

scene resume tries the persisted desired Scene. If it cannot become exactly healthy, it tries the previous last-known-good Scene.

Resident agent and stable gateway

The resident agent resumes desired state, serves a transparent TCP gateway, enforces the unified-memory floor, and reconciles only registry models that explicitly opt into supervision. Models without that block are never restarted by the steady-state agent.

lumbridge-compute agent --listen 127.0.0.1:8011 --upstream 127.0.0.1:8001 --floor 3

The gateway is deliberately protocol-transparent, preserving OpenAI-compatible HTTP streaming and SSE. Apps use http://<compute-node>:8011; model runtimes may continue to move behind the local upstream port.

The same agent samples the local runtime's Prometheus endpoint into a durable, privacy-safe SQLite history. SGLang models must opt in with --enable-metrics; telemetry failure never takes down the gateway, watchdog, or model supervisor. See telemetry.md for the stored fields and label contract.

An opted-in model is restarted only after its configured number of consecutive exact-health failures. Recovery takes the same transition lock as Scene activation, refuses to signal an unowned or stale process identity, repeats both memory admission checks, and uses exponential backoff after a failed attempt. This is intentionally per-model: one failed voice runtime does not stop or roll back the healthy brain, ASR, or embedding processes in the same Scene.

Installing the agent

Install the unit only after adoption and a no-op resume test on that node. The resident agent is the only unit — it subsumes the earlier default-Scene oneshot and the standalone watchdog, both retired.

sudo install -m 0644 systemd/lumbridge-compute-agent.service \
  /etc/systemd/system/lumbridge-compute-agent.service
sudo systemctl daemon-reload
sudo systemctl enable --now lumbridge-compute-agent.service

The unit sets LUMBRIDGE_COMPUTE_ROOT and the agent takes its root from there, so the node's root path is declared in exactly one place.

Verify exact model identity through both the backend and the gateway before pointing any consumer at the node:

curl -fsS http://127.0.0.1:8001/v1/models
curl -fsS http://127.0.0.1:8011/v1/models
lumbridge-compute usage collect
lumbridge-compute usage summary --since 24h
systemctl is-active lumbridge-compute-agent.service

For an unprivileged appliance deployment, install the supplied user unit instead:

install -Dm0644 systemd/lumbridge-compute-agent-user.service \
  ~/.config/systemd/user/lumbridge-compute-agent.service
systemctl --user daemon-reload
systemctl --user enable --now lumbridge-compute-agent.service

The reference user unit expects a clean, reviewed checkout at ~/lumbridge-prod. Change both paths together if the production checkout lives elsewhere. Do not point it at a dirty development checkout.

Node repository access

A node pulls with a read-only deploy key, not an account token. Each node gets its own keypair generated on that node (the private half never transits the network), and only the public half is registered against this one repository. A compromised node can therefore read this repo and nothing else — it cannot push, cannot reach other repos, and cannot use the gitea API. Revoke a single node by deleting its deploy key.

Setting one up:

# on the node — private key stays here
ssh-keygen -t ed25519 -f ~/.ssh/gitea_lumbridge_compute -N "" -C "<node> deploy key (read-only)"

Register the public half as a read-only deploy key on lumbridge-public/compute, then point the node's remote at it via an ~/.ssh/config host alias (gitea SSH listens on 2223):

Host gitea-lumbridge
    HostName <gitea-host>
    Port 2223
    User git
    IdentityFile ~/.ssh/gitea_lumbridge_compute
    IdentitiesOnly yes

git remote set-url origin gitea:lumbridge-public/compute.git
git branch --set-upstream-to=origin/main main

Verify it is genuinely read-only before trusting it — a push must be rejected:

git fetch origin                                        # succeeds
git push origin HEAD:refs/heads/write-test              # must fail

Upgrading the agent binary

KillMode=process means model process groups outlive an agent restart, and the agent revalidates every (boot_id, pid, start_time_ticks) identity on resume. A binary upgrade therefore does not disturb a healthy Scene:

git pull && cargo build --release
sudo systemctl restart lumbridge-compute-agent.service

Confirm the model PIDs are unchanged afterwards. If the agent cannot revalidate an identity it refuses to signal that process rather than guessing — investigate before forcing anything. To roll back, check out the previous commit, rebuild, and restart the same unit; the model processes are untouched either way.

Do not reboot solely to test an upgrade. Prove resume and gateway parity in the live boot first.