Karti Tripathi
142 lines
5.3 KiB
Markdown
142 lines
5.3 KiB
Markdown
# Lumbridge Compute node operations
|
|
|
|
## Persistent Scene state
|
|
|
|
Compute stores compatibility state at `<root>/.kuda/state.yaml`. State writes use
|
|
an atomic temporary-file replacement. The persisted lifecycle fields are:
|
|
|
|
- `desired_scene`: the Scene the node should resume after an agent or node restart;
|
|
- `active_scene`: the Scene whose exact model health was last verified;
|
|
- `last_known_good_scene`: the previous proven Scene used for recovery;
|
|
- `transition_scene`: an in-progress transaction, cleared on success or failure;
|
|
- `last_error`: the most recent activation or watchdog failure.
|
|
|
|
Process ownership is bound to `(boot_id, pid, process_start_ticks)`. Compute will
|
|
never signal a legacy PID-only record or a PID that Linux has reused.
|
|
|
|
## One-time adoption on an existing node
|
|
|
|
Before the first transactional switch, bind the already-running exact Scene to
|
|
process identities. Adoption neither starts nor stops a model.
|
|
|
|
```bash
|
|
lumbridge-compute --root $LUMBRIDGE_COMPUTE_ROOT scene adopt voice-qwen
|
|
lumbridge-compute --root $LUMBRIDGE_COMPUTE_ROOT status
|
|
lumbridge-compute --root $LUMBRIDGE_COMPUTE_ROOT scene activate voice-qwen --dry-run
|
|
```
|
|
|
|
Adoption fails if a Scene model is unhealthy, its health marker identifies the
|
|
wrong model, another registered model is serving, a legacy process record is
|
|
missing, or its process group does not equal its PID.
|
|
|
|
## Transactional switching and recovery
|
|
|
|
```bash
|
|
lumbridge-compute scene activate voice-laguna --dry-run
|
|
lumbridge-compute scene activate voice-laguna
|
|
lumbridge-compute scene resume
|
|
```
|
|
|
|
Activation validates the complete Scene and all process ownership before any
|
|
stop. A shared port occupied by the wrong model is a hard failure. Every started
|
|
model must report its exact health marker. If a transition fails after mutation,
|
|
Compute stops the partial target and restores the previously active Scene.
|
|
|
|
`scene resume` tries the persisted desired Scene. If it cannot become exactly
|
|
healthy, it tries the previous last-known-good Scene.
|
|
|
|
## Resident agent and stable gateway
|
|
|
|
The resident agent resumes desired state once, serves a transparent TCP gateway,
|
|
and enforces the unified-memory floor:
|
|
|
|
```bash
|
|
lumbridge-compute agent --listen 127.0.0.1:8011 --upstream 127.0.0.1:8001 --floor 3
|
|
```
|
|
|
|
The gateway is deliberately protocol-transparent, preserving OpenAI-compatible
|
|
HTTP streaming and SSE. Apps use `http://<compute-node>:8011`; model runtimes may
|
|
continue to move behind the local upstream port.
|
|
|
|
## Installing the agent
|
|
|
|
Install the unit only after adoption and a no-op resume test on that node. The resident
|
|
agent is the only unit — it subsumes the earlier default-Scene oneshot and the standalone
|
|
watchdog, both retired.
|
|
|
|
```bash
|
|
sudo install -m 0644 systemd/lumbridge-compute-agent.service \
|
|
/etc/systemd/system/lumbridge-compute-agent.service
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl enable --now lumbridge-compute-agent.service
|
|
```
|
|
|
|
The unit sets `LUMBRIDGE_COMPUTE_ROOT` and the agent takes its root from there, so the
|
|
node's root path is declared in exactly one place.
|
|
|
|
Verify exact model identity through both the backend and the gateway before pointing any
|
|
consumer at the node:
|
|
|
|
```bash
|
|
curl -fsS http://127.0.0.1:8001/v1/models
|
|
curl -fsS http://127.0.0.1:8011/v1/models
|
|
systemctl is-active lumbridge-compute-agent.service
|
|
```
|
|
|
|
## Node repository access
|
|
|
|
A node pulls with a **read-only deploy key**, not an account token. Each node gets its own
|
|
keypair generated *on that node* (the private half never transits the network), and only the
|
|
public half is registered against this one repository. A compromised node can therefore read
|
|
this repo and nothing else — it cannot push, cannot reach other repos, and cannot use the
|
|
gitea API. Revoke a single node by deleting its deploy key.
|
|
|
|
Setting one up:
|
|
|
|
```bash
|
|
# on the node — private key stays here
|
|
ssh-keygen -t ed25519 -f ~/.ssh/gitea_lumbridge_compute -N "" -C "<node> deploy key (read-only)"
|
|
```
|
|
|
|
Register the public half as a **read-only** deploy key on `karti/lumbridge-compute`, then point
|
|
the node's remote at it via an `~/.ssh/config` host alias (gitea SSH listens on **2223**):
|
|
|
|
```
|
|
Host gitea-lumbridge
|
|
HostName <gitea-host>
|
|
Port 2223
|
|
User git
|
|
IdentityFile ~/.ssh/gitea_lumbridge_compute
|
|
IdentitiesOnly yes
|
|
```
|
|
|
|
```bash
|
|
git remote set-url origin gitea-lumbridge:karti/lumbridge-compute.git
|
|
git branch --set-upstream-to=origin/main main
|
|
```
|
|
|
|
Verify it is genuinely read-only before trusting it — a push must be rejected:
|
|
|
|
```bash
|
|
git fetch origin # succeeds
|
|
git push origin HEAD:refs/heads/write-test # must fail
|
|
```
|
|
|
|
## Upgrading the agent binary
|
|
|
|
`KillMode=process` means model process groups outlive an agent restart, and the agent
|
|
revalidates every `(boot_id, pid, start_time_ticks)` identity on resume. A binary upgrade
|
|
therefore does not disturb a healthy Scene:
|
|
|
|
```bash
|
|
git pull && cargo build --release
|
|
sudo systemctl restart lumbridge-compute-agent.service
|
|
```
|
|
|
|
Confirm the model PIDs are unchanged afterwards. If the agent cannot revalidate an identity
|
|
it refuses to signal that process rather than guessing — investigate before forcing anything.
|
|
To roll back, check out the previous commit, rebuild, and restart the same unit; the model
|
|
processes are untouched either way.
|
|
|
|
Do not reboot solely to test an upgrade. Prove resume and gateway parity in the live boot first.
|