lumbridge-public/compute
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Lumbridge Compute — Apache-2.0
ci / rust (push) Failing after 18s
Details

Browse Source
This commit is contained in:
Karti Tripathi
2026-08-03 23:47:51 -07:00
commit a8c8532105
40 changed files with 6101 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/evals.md
+37
View File
@@ -0,0 +1,37 @@
# Lumbridge Compute evaluations (`lumbridge/v1`)
Lumbridge Compute evaluates the model configuration that is actually serving: weights,
quantization, context, runtime, parsers, and speculative decoder. A model name
without its serving configuration is not a reproducible benchmark target.
```bash
lumbridge-compute eval ls
lumbridge-compute eval run smoke
lumbridge-compute eval run performance --model brain --repeat 5
lumbridge-compute eval run finance-core --base-url http://your-node:8001/v1
```
Suites live in `evals/*.eval.yaml`. Each case has a stable id, prompt, category,
generation limit, and deterministic assertions. Runs produce append-only JSON in
`eval-results/` with raw outputs and per-sample metrics.
## Metrics
- **TTFT**: wall time until the first streamed content or reasoning token.
- **Prefill tok/s (approximate)**: API-reported prompt tokens divided by TTFT.
This is client-observed and includes queueing/scheduling; server-native prefill
metrics should be added as a separate source rather than conflated with it.
- **Decode tok/s**: completion tokens divided by time after the first token.
- **Score**: share of samples satisfying every declared assertion.
Performance runs should include warmups in automation and record hardware, Lumbridge Compute
scene, runtime version, model revision, and cold/warm cache state. The v1 artifact
is deliberately local and portable; a future registry can ingest the same JSON.
## Eval-to-training bridge
Capability suites should graduate into environment packages containing a dataset,
harness, and reward function. That common contract can be adapted to Prime
Intellect `verifiers` for evaluation, synthetic-data generation, SFT, or RL with
`prime-rl`. Lumbridge Compute owns scene scheduling, memory admission, checkpoints, and process
lifecycle; the training framework owns optimization and distributed execution.
docs/operations.md
+141
View File
@@ -0,0 +1,141 @@
# Lumbridge Compute node operations
## Persistent Scene state
Compute stores compatibility state at `<root>/.kuda/state.yaml`. State writes use
an atomic temporary-file replacement. The persisted lifecycle fields are:
- `desired_scene`: the Scene the node should resume after an agent or node restart;
- `active_scene`: the Scene whose exact model health was last verified;
- `last_known_good_scene`: the previous proven Scene used for recovery;
- `transition_scene`: an in-progress transaction, cleared on success or failure;
- `last_error`: the most recent activation or watchdog failure.
Process ownership is bound to `(boot_id, pid, process_start_ticks)`. Compute will
never signal a legacy PID-only record or a PID that Linux has reused.
## One-time adoption on an existing node
Before the first transactional switch, bind the already-running exact Scene to
process identities. Adoption neither starts nor stops a model.
```bash
lumbridge-compute --root $LUMBRIDGE_COMPUTE_ROOT scene adopt voice-qwen
lumbridge-compute --root $LUMBRIDGE_COMPUTE_ROOT status
lumbridge-compute --root $LUMBRIDGE_COMPUTE_ROOT scene activate voice-qwen --dry-run
```
Adoption fails if a Scene model is unhealthy, its health marker identifies the
wrong model, another registered model is serving, a legacy process record is
missing, or its process group does not equal its PID.
## Transactional switching and recovery
```bash
lumbridge-compute scene activate voice-laguna --dry-run
lumbridge-compute scene activate voice-laguna
lumbridge-compute scene resume
```
Activation validates the complete Scene and all process ownership before any
stop. A shared port occupied by the wrong model is a hard failure. Every started
model must report its exact health marker. If a transition fails after mutation,
Compute stops the partial target and restores the previously active Scene.
`scene resume` tries the persisted desired Scene. If it cannot become exactly
healthy, it tries the previous last-known-good Scene.
## Resident agent and stable gateway
The resident agent resumes desired state once, serves a transparent TCP gateway,
and enforces the unified-memory floor:
```bash
lumbridge-compute agent --listen 127.0.0.1:8011 --upstream 127.0.0.1:8001 --floor 3
```
The gateway is deliberately protocol-transparent, preserving OpenAI-compatible
HTTP streaming and SSE. Apps use `http://<compute-node>:8011`; model runtimes may
continue to move behind the local upstream port.
## Installing the agent
Install the unit only after adoption and a no-op resume test on that node. The resident
agent is the only unit — it subsumes the earlier default-Scene oneshot and the standalone
watchdog, both retired.
```bash
sudo install -m 0644 systemd/lumbridge-compute-agent.service \
/etc/systemd/system/lumbridge-compute-agent.service
sudo systemctl daemon-reload
sudo systemctl enable --now lumbridge-compute-agent.service
```
The unit sets `LUMBRIDGE_COMPUTE_ROOT` and the agent takes its root from there, so the
node's root path is declared in exactly one place.
Verify exact model identity through both the backend and the gateway before pointing any
consumer at the node:
```bash
curl -fsS http://127.0.0.1:8001/v1/models
curl -fsS http://127.0.0.1:8011/v1/models
systemctl is-active lumbridge-compute-agent.service
```
## Node repository access
A node pulls with a **read-only deploy key**, not an account token. Each node gets its own
keypair generated *on that node* (the private half never transits the network), and only the
public half is registered against this one repository. A compromised node can therefore read
this repo and nothing else — it cannot push, cannot reach other repos, and cannot use the
gitea API. Revoke a single node by deleting its deploy key.
Setting one up:
```bash
# on the node — private key stays here
ssh-keygen -t ed25519 -f ~/.ssh/gitea_lumbridge_compute -N "" -C "<node> deploy key (read-only)"
```
Register the public half as a **read-only** deploy key on `karti/lumbridge-compute`, then point
the node's remote at it via an `~/.ssh/config` host alias (gitea SSH listens on **2223**):
```
Host gitea-lumbridge
HostName <gitea-host>
Port 2223
User git
IdentityFile ~/.ssh/gitea_lumbridge_compute
IdentitiesOnly yes
```
```bash
git remote set-url origin gitea-lumbridge:karti/lumbridge-compute.git
git branch --set-upstream-to=origin/main main
```
Verify it is genuinely read-only before trusting it — a push must be rejected:
```bash
git fetch origin # succeeds
git push origin HEAD:refs/heads/write-test # must fail
```
## Upgrading the agent binary
`KillMode=process` means model process groups outlive an agent restart, and the agent
revalidates every `(boot_id, pid, start_time_ticks)` identity on resume. A binary upgrade
therefore does not disturb a healthy Scene:
```bash
git pull && cargo build --release
sudo systemctl restart lumbridge-compute-agent.service
```
Confirm the model PIDs are unchanged afterwards. If the agent cannot revalidate an identity
it refuses to signal that process rather than guessing — investigate before forcing anything.
To roll back, check out the previous commit, rebuild, and restart the same unit; the model
processes are untouched either way.
Do not reboot solely to test an upgrade. Prove resume and gateway parity in the live boot first.
docs/positioning.md
+51
View File
@@ -0,0 +1,51 @@
# Positioning
Where Lumbridge Compute sits relative to the tools people already know. The short version: the
neighbors solve *different* problems, and the closest one has exactly the gap Lumbridge Compute fills.
## The landscape
| Tool | Axis | What it does | Gap for a unified-memory box |
|---|---|---|---|
| **exo** | scale **out** | Shards *one* big model *across* many boxes (Thunderbolt/RDMA, MLX, Apple-first) | Doesn't run many models on one box; not GB10/GB300-native |
| **Ollama** | single model | Dead-simple pull/run one LLM, huge library | No multi-service orchestration, **no memory governor** |
| **llama-swap** | swap **in/out** | On-demand load/unload, TTL idle-unload, co-run matrix, any OpenAI server | **"No memory budgeting or OOM prevention"** (their words), LLM-swapping not capability-stacks, no scheduling/sharing |
| **LiteLLM** | routing | One OpenAI endpoint in front of many backends | A proxy — doesn't manage lifecycle or memory |
## The one-liner
> **exo runs one model across many boxes. Lumbridge Compute runs many models on one box — safely.**
Orthogonal, not competing — you can run Lumbridge Compute on each node of an exo cluster.
## Where Lumbridge Compute is unique
1. **The Governor.** No one else does unified-memory admission control + an OOM-wedge
watchdog. It's the pain *specific* to Grace-Blackwell shared memory, where over-commit
wedges the whole box instead of failing a single allocation. This is the moat.
2. **Scenes are capability stacks, not model swaps.** llama-swap picks *which LLM* answers.
Scenes compose *heterogeneous services together* — ASR + LLM + TTS + image + music — as
one activatable unit.
3. **Scheduled scenes** — time-multiplex a box by hour (live assistant by day, image farm
overnight). Nobody does this.
4. **Shareable scene registry** — publish and pull scenes. The community flywheel.
5. **GB10 / GB300 / RTX-native** — NVFP4, `flashinfer` MoE backend, `sm_121a`. exo is
Apple-first; Lumbridge Compute is Grace-Blackwell-first.
## Ideas worth borrowing (don't reinvent)
- **From llama-swap:** TTL idle-unload; an OpenAI-compatible gateway in front so apps hit
one stable endpoint regardless of which scene is live; a web UI; Prometheus metrics;
install via brew/binary/docker.
- **From Ollama:** one-command onboarding and a friendly `model pull/ls/rm` UX + a library.
- **From LiteLLM:** the router-in-front pattern — pairs with scenes (the `served_name`
aliases already point this way).
- **From exo:** OpenAI + Anthropic + Ollama-compatible APIs; later, auto-discovery so Lumbridge Compute
can coordinate scenes across two Sparks.
## The model manager angle
Lumbridge Compute's `model` commands aren't just another Ollama. They're **governor-aware and
scene-aware**: Lumbridge Compute knows each model's footprint, warns *before* you pull something no
scene could ever hold, and shows which scenes a model belongs to. Ollama manages files;
Lumbridge Compute manages *what can actually run together*.
docs/roadmap.md
+29
View File
@@ -0,0 +1,29 @@
# Lumbridge Compute roadmap
Lumbridge Compute is the resident AI workload control plane for unified-memory machines.
## Core contracts
1. **Models** — local vetted launch recipes with immutable revisions and measured footprints.
2. **Scenes** — named workload sets activated through admission control.
3. **Evals** — versioned datasets, prompts, harnesses, assertions, and portable results.
4. **Artifacts** — resumable weights, adapters, checkpoints, datasets, and result bundles.
5. **Jobs** — inference, evaluation, download, conversion, SFT, and RL lifecycles.
## Near term
- Resumable, checksum-verified `lumbridge-compute model pull` with leases and progress state.
- Eval comparison, regression thresholds, warmup policy, concurrency/load tests,
server-native Prometheus metrics, and hardware/config provenance.
- Scene hooks for preflight, post-activation eval gates, rollback, and schedules.
- Gateway aliases so clients follow the active scene without configuration changes.
- Training scenes and a Prime Intellect adapter for Qwen 0.6B/1.7B experiments.
- Checkpoint discovery, pause/resume, retention, and promotion into the model registry.
## Open-source readiness
- Keep public scene/eval manifests command-free; commands remain in the trusted local registry.
- Add CI across x86_64 and aarch64, unit/integration tests, security policy,
contribution guide, code of conduct, changelog, and versioned JSON schemas.
- Remove private hostnames, voices, paths, and finance datasets from public fixtures;
ship generic examples and keep personal overlays outside the repository.
docs/scene-spec.md
+178
View File
@@ -0,0 +1,178 @@
# Lumbridge Compute Scene Spec (`lumbridge/v1`)
This is the public contract. Everything else — the Governor, the CLI, the
scheduler — can be refactored freely. This format cannot, once people publish
scenes against it. So it is deliberately small.
## The core idea: scenes reference ids, not weights
A **scene** is a manifest listing **model ids**. A **model registry** resolves each
id to actual weights + a launch command. The registry churns as models evolve (new
quants, new backends, bigger context); the scene stays stable.
```
scene (stable, shareable) registry (local, evolves)
───────────────────────── ──────────────────────────────
models: [ears, brain, voice] ──▶ brain → ~/models/Qwen3.6-35B-A3B-NVFP4-Fast
→ vllm serve, gmu 0.55, flashinfer...
```
This is the same decoupling the original harness used: it kept the id `brain` and a
stable `served_name` while the underlying weights swapped dense-27B → 35B-A3B MoE,
and downstream agents never noticed. The spec formalizes that as the mechanism that
lets scenes "update over time as models evolve" without breaking anyone.
### Why this also solves scene-sharing security
A published scene contains **only ids and parameters — never shell commands.** The
launch commands live in your *local, vetted* registry. So activating a downloaded
scene can only ever start models your own registry already trusts. If a scene
references an id you don't have, Lumbridge Compute asks you to add it to your registry, showing
the launch command for review — an explicit opt-in, not silent remote code
execution. Declarative-by-construction; there is no field in which a scene can smuggle
a command.
---
## Scene manifest
`scenes/studio.scene.yaml`
```yaml
apiVersion: lumbridge/v1
kind: Scene
metadata:
name: studio
version: 3 # bump on any change; a published scene is reproducible
description: "Live voice assistant — ears, brain, mouth, and music."
author: karti
tags: [assistant, voice, always-on]
models: # stable ids; the registry resolves each
- ears # ASR
- brain # MoE LLM
- voice # TTS
- music # ACE-Step
budget_gb: 100 # optional; overrides the Governor's global budget
activation:
order: footprint-asc # small models first so the big load spike lands last
wait_healthy: true # block until each model's health check passes
```
`scenes/darkroom.scene.yaml`
```yaml
apiVersion: lumbridge/v1
kind: Scene
metadata:
name: darkroom
version: 1
description: "Overnight image farm — drops the brain to make room for FLUX.2-dev."
tags: [image, overnight, unattended]
models:
- ears
- voice
- music
- image # FLUX.2-dev — only fits because `brain` is not in this scene
activation:
order: footprint-asc
wait_healthy: true
```
### Fields
| Field | Required | Meaning |
|---|---|---|
| `apiVersion` | ✔ | `lumbridge/v1`. |
| `kind` | ✔ | `Scene`. |
| `metadata.name` | ✔ | Unique scene name; the CLI handle. |
| `metadata.version` | ✔ | Integer, bumped on any change. Reproducibility. |
| `metadata.description` | ✔ | One line, shown in `lumbridge-compute scene ls`. |
| `metadata.tags` | – | For the (future) registry search. |
| `models` | ✔ | Ordered list of model ids resolved via the registry. |
| `budget_gb` | – | Per-scene budget override; defaults to the global Governor budget. |
| `activation.order` | – | `footprint-asc` (default) \| `listed`. |
| `activation.wait_healthy` | – | Default `true`. Block until health checks pass. |
A scene **never** contains: weight paths, shell commands, or GPU flags. Those live in
the registry. This is load-bearing for both stability and security.
---
## Model registry
`registry/models.yaml` — local to each box, evolves freely. Ids are the stable
contract; everything under `serve` can change.
```yaml
apiVersion: lumbridge/v1
kind: Registry
version: 1
models:
brain:
name: "Qwen3.6 35B-A3B MoE (NVFP4)"
footprint_gb: 66 # worst-case unified memory once serving (weights + KV + encoder)
channel: stable # stable | latest — how aggressively to track new weights
health: "http://localhost:8001/v1/models"
serve:
kind: vllm
port: 8001
weights: "~/models/Qwen3.6-35B-A3B-NVFP4-Fast"
served_name: # stable aliases so downstream clients survive a weight swap
- brain
- "local-moe"
- "unsloth/Qwen3.6-35B-A3B-NVFP4-Fast"
args:
max-model-len: 65536
kv-cache-dtype: fp8
gpu-memory-utilization: 0.55
enforce-eager: true
moe-backend: flashinfer_b12x # Unsloth DGX Spark recipe; critical for speed
limit-mm-per-prompt: '{"image": 0, "video": 0}'
env:
CUTE_DSL_ARCH: sm_121a
image:
name: "FLUX.2-dev (FP8)"
footprint_gb: 32
channel: stable
health: "http://localhost:8007/health"
serve:
kind: diffusers # not vllm — a separate runtime (ComfyUI/diffusers)
port: 8007
weights: "~/models/FLUX.2-dev"
args: { dtype: fp8 }
# ears / voice / music elaborated the same way (ASR, Chatterbox, ACE-Step).
```
### `channel`: how scenes track evolving weights
- `stable` — pin the exact `weights` path. Reproducible; you update deliberately.
- `latest` — the registry may resolve to a newer quant of the same model family
(e.g. a fresh NVFP4 build) on activation. Bleeding edge; use for your own box, not
for scenes you publish for others.
The scene picks the *id*; the registry's `channel` decides how much the weights are
allowed to drift underneath it. That's the whole "scenes evolve as models evolve"
story, made explicit and controllable.
---
## Governor interaction
On `activate`, Lumbridge Compute computes the diff between the running set and the target
scene's `models`, then:
1. **Stops** running models not in the scene (frees their footprint first).
2. **Starts** the scene's models in `activation.order`, each passing **admission
control** against `budget_gb` before launch.
3. Waits for health if `wait_healthy`.
The watchdog runs throughout, unchanged — the safety net if any `footprint_gb` is
wrong. A scene can never talk the Governor into over-committing; admission control is
not bypassable by a scene.