Lumbridge Compute — Apache-2.0

docs/operations.md

@@ -0,0 +1,141 @@
+# Lumbridge Compute node operations
+
+## Persistent Scene state
+
+Compute stores compatibility state at `<root>/.kuda/state.yaml`. State writes use
+an atomic temporary-file replacement. The persisted lifecycle fields are:
+
+- `desired_scene`: the Scene the node should resume after an agent or node restart;
+- `active_scene`: the Scene whose exact model health was last verified;
+- `last_known_good_scene`: the previous proven Scene used for recovery;
+- `transition_scene`: an in-progress transaction, cleared on success or failure;
+- `last_error`: the most recent activation or watchdog failure.
+
+Process ownership is bound to `(boot_id, pid, process_start_ticks)`. Compute will
+never signal a legacy PID-only record or a PID that Linux has reused.
+
+## One-time adoption on an existing node
+
+Before the first transactional switch, bind the already-running exact Scene to
+process identities. Adoption neither starts nor stops a model.
+
+```bash
+lumbridge-compute --root $LUMBRIDGE_COMPUTE_ROOT scene adopt voice-qwen
+lumbridge-compute --root $LUMBRIDGE_COMPUTE_ROOT status
+lumbridge-compute --root $LUMBRIDGE_COMPUTE_ROOT scene activate voice-qwen --dry-run
+```
+
+Adoption fails if a Scene model is unhealthy, its health marker identifies the
+wrong model, another registered model is serving, a legacy process record is
+missing, or its process group does not equal its PID.
+
+## Transactional switching and recovery
+
+```bash
+lumbridge-compute scene activate voice-laguna --dry-run
+lumbridge-compute scene activate voice-laguna
+lumbridge-compute scene resume
+```
+
+Activation validates the complete Scene and all process ownership before any
+stop. A shared port occupied by the wrong model is a hard failure. Every started
+model must report its exact health marker. If a transition fails after mutation,
+Compute stops the partial target and restores the previously active Scene.
+
+`scene resume` tries the persisted desired Scene. If it cannot become exactly
+healthy, it tries the previous last-known-good Scene.
+
+## Resident agent and stable gateway
+
+The resident agent resumes desired state once, serves a transparent TCP gateway,
+and enforces the unified-memory floor:
+
+```bash
+lumbridge-compute agent --listen 127.0.0.1:8011 --upstream 127.0.0.1:8001 --floor 3
+```
+
+The gateway is deliberately protocol-transparent, preserving OpenAI-compatible
+HTTP streaming and SSE. Apps use `http://<compute-node>:8011`; model runtimes may
+continue to move behind the local upstream port.
+
+## Installing the agent
+
+Install the unit only after adoption and a no-op resume test on that node. The resident
+agent is the only unit — it subsumes the earlier default-Scene oneshot and the standalone
+watchdog, both retired.
+
+```bash
+sudo install -m 0644 systemd/lumbridge-compute-agent.service \
+  /etc/systemd/system/lumbridge-compute-agent.service
+sudo systemctl daemon-reload
+sudo systemctl enable --now lumbridge-compute-agent.service
+```
+
+The unit sets `LUMBRIDGE_COMPUTE_ROOT` and the agent takes its root from there, so the
+node's root path is declared in exactly one place.
+
+Verify exact model identity through both the backend and the gateway before pointing any
+consumer at the node:
+
+```bash
+curl -fsS http://127.0.0.1:8001/v1/models
+curl -fsS http://127.0.0.1:8011/v1/models
+systemctl is-active lumbridge-compute-agent.service
+```
+
+## Node repository access
+
+A node pulls with a **read-only deploy key**, not an account token. Each node gets its own
+keypair generated *on that node* (the private half never transits the network), and only the
+public half is registered against this one repository. A compromised node can therefore read
+this repo and nothing else — it cannot push, cannot reach other repos, and cannot use the
+gitea API. Revoke a single node by deleting its deploy key.
+
+Setting one up:
+
+```bash
+# on the node — private key stays here
+ssh-keygen -t ed25519 -f ~/.ssh/gitea_lumbridge_compute -N "" -C "<node> deploy key (read-only)"
+```
+
+Register the public half as a **read-only** deploy key on `karti/lumbridge-compute`, then point
+the node's remote at it via an `~/.ssh/config` host alias (gitea SSH listens on **2223**):
+
+```
+Host gitea-lumbridge
+    HostName <gitea-host>
+    Port 2223
+    User git
+    IdentityFile ~/.ssh/gitea_lumbridge_compute
+    IdentitiesOnly yes
+```
+
+```bash
+git remote set-url origin gitea-lumbridge:karti/lumbridge-compute.git
+git branch --set-upstream-to=origin/main main
+```
+
+Verify it is genuinely read-only before trusting it — a push must be rejected:
+
+```bash
+git fetch origin                                        # succeeds
+git push origin HEAD:refs/heads/write-test              # must fail
+```
+
+## Upgrading the agent binary
+
+`KillMode=process` means model process groups outlive an agent restart, and the agent
+revalidates every `(boot_id, pid, start_time_ticks)` identity on resume. A binary upgrade
+therefore does not disturb a healthy Scene:
+
+```bash
+git pull && cargo build --release
+sudo systemctl restart lumbridge-compute-agent.service
+```
+
+Confirm the model PIDs are unchanged afterwards. If the agent cannot revalidate an identity
+it refuses to signal that process rather than guessing — investigate before forcing anything.
+To roll back, check out the previous commit, rebuild, and restart the same unit; the model
+processes are untouched either way.
+
+Do not reboot solely to test an upgrade. Prove resume and gateway parity in the live boot first.