This commit is contained in:
Karti Tripathi
@@ -0,0 +1,170 @@
|
||||
# Lumbridge Compute node operations
|
||||
|
||||
## Persistent Scene state
|
||||
|
||||
Compute stores state at `<root>/.compute/state.yaml`. State writes use
|
||||
an atomic temporary-file replacement. The persisted lifecycle fields are:
|
||||
|
||||
- `desired_scene`: the Scene the node should resume after an agent or node restart;
|
||||
- `active_scene`: the Scene whose exact model health was last verified;
|
||||
- `last_known_good_scene`: the previous proven Scene used for recovery;
|
||||
- `transition_scene`: an in-progress transaction, cleared on success or failure;
|
||||
- `last_error`: the most recent activation or watchdog failure.
|
||||
|
||||
Process ownership is bound to `(boot_id, pid, process_start_ticks)`. Compute will
|
||||
never signal a legacy PID-only record or a PID that Linux has reused.
|
||||
|
||||
## One-time adoption on an existing node
|
||||
|
||||
Before the first transactional switch, bind the already-running exact Scene to
|
||||
process identities. Adoption neither starts nor stops a model.
|
||||
|
||||
```bash
|
||||
lumbridge-compute --root $LUMBRIDGE_COMPUTE_ROOT scene adopt voice-qwen
|
||||
lumbridge-compute --root $LUMBRIDGE_COMPUTE_ROOT status
|
||||
lumbridge-compute --root $LUMBRIDGE_COMPUTE_ROOT scene activate voice-qwen --dry-run
|
||||
```
|
||||
|
||||
Adoption fails if a Scene model is unhealthy, its health marker identifies the
|
||||
wrong model, another registered model is serving, a legacy process record is
|
||||
missing, or its process group does not equal its PID.
|
||||
|
||||
## Transactional switching and recovery
|
||||
|
||||
```bash
|
||||
lumbridge-compute scene activate voice-laguna --dry-run
|
||||
lumbridge-compute scene activate voice-laguna
|
||||
lumbridge-compute scene resume
|
||||
```
|
||||
|
||||
Activation validates the complete Scene and all process ownership before any
|
||||
stop. A shared port occupied by the wrong model is a hard failure. Every started
|
||||
model must report its exact health marker. If a transition fails after mutation,
|
||||
Compute stops the partial target and restores the previously active Scene.
|
||||
|
||||
`scene resume` tries the persisted desired Scene. If it cannot become exactly
|
||||
healthy, it tries the previous last-known-good Scene.
|
||||
|
||||
## Resident agent and stable gateway
|
||||
|
||||
The resident agent resumes desired state, serves a transparent TCP gateway,
|
||||
enforces the unified-memory floor, and reconciles only registry models that
|
||||
explicitly opt into `supervision`. Models without that block are never restarted
|
||||
by the steady-state agent.
|
||||
|
||||
```bash
|
||||
lumbridge-compute agent --listen 127.0.0.1:8011 --upstream 127.0.0.1:8001 --floor 3
|
||||
```
|
||||
|
||||
The gateway is deliberately protocol-transparent, preserving OpenAI-compatible
|
||||
HTTP streaming and SSE. Apps use `http://<compute-node>:8011`; model runtimes may
|
||||
continue to move behind the local upstream port.
|
||||
|
||||
The same agent samples the local runtime's Prometheus endpoint into a durable,
|
||||
privacy-safe SQLite history. SGLang models must opt in with `--enable-metrics`;
|
||||
telemetry failure never takes down the gateway, watchdog, or model supervisor.
|
||||
See [telemetry.md](telemetry.md) for the stored fields and label contract.
|
||||
|
||||
An opted-in model is restarted only after its configured number of consecutive
|
||||
exact-health failures. Recovery takes the same transition lock as Scene
|
||||
activation, refuses to signal an unowned or stale process identity, repeats both
|
||||
memory admission checks, and uses exponential backoff after a failed attempt.
|
||||
This is intentionally per-model: one failed voice runtime does not stop or roll
|
||||
back the healthy brain, ASR, or embedding processes in the same Scene.
|
||||
|
||||
## Installing the agent
|
||||
|
||||
Install the unit only after adoption and a no-op resume test on that node. The resident
|
||||
agent is the only unit — it subsumes the earlier default-Scene oneshot and the standalone
|
||||
watchdog, both retired.
|
||||
|
||||
```bash
|
||||
sudo install -m 0644 systemd/lumbridge-compute-agent.service \
|
||||
/etc/systemd/system/lumbridge-compute-agent.service
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now lumbridge-compute-agent.service
|
||||
```
|
||||
|
||||
The unit sets `LUMBRIDGE_COMPUTE_ROOT` and the agent takes its root from there, so the
|
||||
node's root path is declared in exactly one place.
|
||||
|
||||
Verify exact model identity through both the backend and the gateway before pointing any
|
||||
consumer at the node:
|
||||
|
||||
```bash
|
||||
curl -fsS http://127.0.0.1:8001/v1/models
|
||||
curl -fsS http://127.0.0.1:8011/v1/models
|
||||
lumbridge-compute usage collect
|
||||
lumbridge-compute usage summary --since 24h
|
||||
systemctl is-active lumbridge-compute-agent.service
|
||||
```
|
||||
|
||||
For an unprivileged appliance deployment, install the supplied user unit instead:
|
||||
|
||||
```bash
|
||||
install -Dm0644 systemd/lumbridge-compute-agent-user.service \
|
||||
~/.config/systemd/user/lumbridge-compute-agent.service
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user enable --now lumbridge-compute-agent.service
|
||||
```
|
||||
|
||||
The reference user unit expects a clean, reviewed checkout at `~/lumbridge-prod`.
|
||||
Change both paths together if the production checkout lives elsewhere. Do not
|
||||
point it at a dirty development checkout.
|
||||
|
||||
## Node repository access
|
||||
|
||||
A node pulls with a **read-only deploy key**, not an account token. Each node gets its own
|
||||
keypair generated *on that node* (the private half never transits the network), and only the
|
||||
public half is registered against this one repository. A compromised node can therefore read
|
||||
this repo and nothing else — it cannot push, cannot reach other repos, and cannot use the
|
||||
gitea API. Revoke a single node by deleting its deploy key.
|
||||
|
||||
Setting one up:
|
||||
|
||||
```bash
|
||||
# on the node — private key stays here
|
||||
ssh-keygen -t ed25519 -f ~/.ssh/gitea_lumbridge_compute -N "" -C "<node> deploy key (read-only)"
|
||||
```
|
||||
|
||||
Register the public half as a **read-only** deploy key on `lumbridge-public/compute`, then point
|
||||
the node's remote at it via an `~/.ssh/config` host alias (gitea SSH listens on **2223**):
|
||||
|
||||
```
|
||||
Host gitea-lumbridge
|
||||
HostName <gitea-host>
|
||||
Port 2223
|
||||
User git
|
||||
IdentityFile ~/.ssh/gitea_lumbridge_compute
|
||||
IdentitiesOnly yes
|
||||
```
|
||||
|
||||
```bash
|
||||
git remote set-url origin gitea:lumbridge-public/compute.git
|
||||
git branch --set-upstream-to=origin/main main
|
||||
```
|
||||
|
||||
Verify it is genuinely read-only before trusting it — a push must be rejected:
|
||||
|
||||
```bash
|
||||
git fetch origin # succeeds
|
||||
git push origin HEAD:refs/heads/write-test # must fail
|
||||
```
|
||||
|
||||
## Upgrading the agent binary
|
||||
|
||||
`KillMode=process` means model process groups outlive an agent restart, and the agent
|
||||
revalidates every `(boot_id, pid, start_time_ticks)` identity on resume. A binary upgrade
|
||||
therefore does not disturb a healthy Scene:
|
||||
|
||||
```bash
|
||||
git pull && cargo build --release
|
||||
sudo systemctl restart lumbridge-compute-agent.service
|
||||
```
|
||||
|
||||
Confirm the model PIDs are unchanged afterwards. If the agent cannot revalidate an identity
|
||||
it refuses to signal that process rather than guessing — investigate before forcing anything.
|
||||
To roll back, check out the previous commit, rebuild, and restart the same unit; the model
|
||||
processes are untouched either way.
|
||||
|
||||
Do not reboot solely to test an upgrade. Prove resume and gateway parity in the live boot first.
|
||||
Reference in New Issue
Block a user