# Lumbridge Compute node operations

## Persistent Scene state

Compute stores compatibility state at `<root>/.kuda/state.yaml`. State writes use
an atomic temporary-file replacement. The persisted lifecycle fields are:

- `desired_scene`: the Scene the node should resume after an agent or node restart;
- `active_scene`: the Scene whose exact model health was last verified;
- `last_known_good_scene`: the previous proven Scene used for recovery;
- `transition_scene`: an in-progress transaction, cleared on success or failure;
- `last_error`: the most recent activation or watchdog failure.

Process ownership is bound to `(boot_id, pid, process_start_ticks)`. Compute will
never signal a legacy PID-only record or a PID that Linux has reused.

## One-time adoption on an existing node

Before the first transactional switch, bind the already-running exact Scene to
process identities. Adoption neither starts nor stops a model.

```bash
lumbridge-compute --root $LUMBRIDGE_COMPUTE_ROOT scene adopt voice-qwen
lumbridge-compute --root $LUMBRIDGE_COMPUTE_ROOT status
lumbridge-compute --root $LUMBRIDGE_COMPUTE_ROOT scene activate voice-qwen --dry-run
```

Adoption fails if a Scene model is unhealthy, its health marker identifies the
wrong model, another registered model is serving, a legacy process record is
missing, or its process group does not equal its PID.

## Transactional switching and recovery

```bash
lumbridge-compute scene activate voice-laguna --dry-run
lumbridge-compute scene activate voice-laguna
lumbridge-compute scene resume
```

Activation validates the complete Scene and all process ownership before any
stop. A shared port occupied by the wrong model is a hard failure. Every started
model must report its exact health marker. If a transition fails after mutation,
Compute stops the partial target and restores the previously active Scene.

`scene resume` tries the persisted desired Scene. If it cannot become exactly
healthy, it tries the previous last-known-good Scene.

## Resident agent and stable gateway

The resident agent resumes desired state once, serves a transparent TCP gateway,
and enforces the unified-memory floor:

```bash
lumbridge-compute agent --listen 127.0.0.1:8011 --upstream 127.0.0.1:8001 --floor 3
```

The gateway is deliberately protocol-transparent, preserving OpenAI-compatible
HTTP streaming and SSE. Apps use `http://<compute-node>:8011`; model runtimes may
continue to move behind the local upstream port.

## Installing the agent

Install the unit only after adoption and a no-op resume test on that node. The resident
agent is the only unit — it subsumes the earlier default-Scene oneshot and the standalone
watchdog, both retired.

```bash
sudo install -m 0644 systemd/lumbridge-compute-agent.service \
  /etc/systemd/system/lumbridge-compute-agent.service
sudo systemctl daemon-reload
sudo systemctl enable --now lumbridge-compute-agent.service
```

The unit sets `LUMBRIDGE_COMPUTE_ROOT` and the agent takes its root from there, so the
node's root path is declared in exactly one place.

Verify exact model identity through both the backend and the gateway before pointing any
consumer at the node:

```bash
curl -fsS http://127.0.0.1:8001/v1/models
curl -fsS http://127.0.0.1:8011/v1/models
systemctl is-active lumbridge-compute-agent.service
```

## Node repository access

A node pulls with a **read-only deploy key**, not an account token. Each node gets its own
keypair generated *on that node* (the private half never transits the network), and only the
public half is registered against this one repository. A compromised node can therefore read
this repo and nothing else — it cannot push, cannot reach other repos, and cannot use the
gitea API. Revoke a single node by deleting its deploy key.

Setting one up:

```bash
# on the node — private key stays here
ssh-keygen -t ed25519 -f ~/.ssh/gitea_lumbridge_compute -N "" -C "<node> deploy key (read-only)"
```

Register the public half as a **read-only** deploy key on `karti/lumbridge-compute`, then point
the node's remote at it via an `~/.ssh/config` host alias (gitea SSH listens on **2223**):

```
Host gitea-lumbridge
    HostName <gitea-host>
    Port 2223
    User git
    IdentityFile ~/.ssh/gitea_lumbridge_compute
    IdentitiesOnly yes
```

```bash
git remote set-url origin gitea-lumbridge:karti/lumbridge-compute.git
git branch --set-upstream-to=origin/main main
```

Verify it is genuinely read-only before trusting it — a push must be rejected:

```bash
git fetch origin                                        # succeeds
git push origin HEAD:refs/heads/write-test              # must fail
```

## Upgrading the agent binary

`KillMode=process` means model process groups outlive an agent restart, and the agent
revalidates every `(boot_id, pid, start_time_ticks)` identity on resume. A binary upgrade
therefore does not disturb a healthy Scene:

```bash
git pull && cargo build --release
sudo systemctl restart lumbridge-compute-agent.service
```

Confirm the model PIDs are unchanged afterwards. If the agent cannot revalidate an identity
it refuses to signal that process rather than guessing — investigate before forcing anything.
To roll back, check out the previous commit, rebuild, and restart the same unit; the model
processes are untouched either way.

Do not reboot solely to test an upgrade. Prove resume and gateway parity in the live boot first.