Karti Tripathi
a4490ec80e
Governed compute for unified-memory AI hardware — machines where CPU and GPU share one pool and there is no separate VRAM allocation to bounce off. Over-commit that pool and the box thrashes and wedges, SSH and ping included, before the OOM killer gets a turn. Compute does not run inference. It supervises the servers that do: admission control against both a declared budget and what the machine actually has free, a 1 Hz watchdog that stops the newest model before thrash, Scenes activated as one transactional unit with rollback, process ownership bound to (boot_id, pid, start_time_ticks, pgid) so a reused PID can never be group-killed, a protocol-transparent gateway, an MCP server, and a read-only HTTP API for dashboards. Registry footprints in this release are measured on a live node rather than estimated. One binary, six direct dependencies. Apache-2.0. Generated by scripts/publish-compute.sh, which refuses to publish a tree it cannot prove clean.
5.4 KiB
Lumbridge Compute
The AI compute layer for Lumbridge clusters. Safe models, scenes, and evals on your own hardware.
Run many models on one box — safely — on NVIDIA DGX Spark (GB10), DGX Station (GB300), and RTX.
Lumbridge Compute is the tool you install the minute you open your DGX Spark. Boxes like the DGX Spark (GB10), DGX Station (GB300), and RTX workstations share one memory pool between CPU and GPU with little swap. On that hardware, over-committing memory doesn't fail gracefully — the whole machine thrashes and wedges (SSH and ping included) before the OOM killer acts. Lumbridge Compute makes that impossible, and turns the box into something you can load with different workloads on a schedule.
exo runs one model across many boxes. Lumbridge Compute runs many models on one box — safely. They're orthogonal; you can run Lumbridge Compute on each node of an exo cluster.
Three layers
| Layer | Analogous to | Job |
|---|---|---|
| Governor | kernel / memory cgroup + OOM policy | Nothing starts unless it fits a hard budget; a watchdog kills the newest model before the box wedges. |
| Scenes | systemd targets | Named, shareable, activatable bundles of models (studio, darkroom). |
| lumbridge-compute (the shell) | the CLI you install | Onboard a fresh box, manage models, activate/schedule scenes. |
The Governor (why Lumbridge Compute exists)
Unified memory means no separate VRAM pool to bounce off. vLLM, diffusers, and friends will happily reserve past 100% of the shared pool, and the box wedges. The Governor prevents this with two mechanisms:
- Admission control — a model starts only if its declared footprint fits the budget given what's already running, plus a safety margin.
- Watchdog — a 1 Hz thread on
MemAvailable; if it dips below a critical floor, it kills the most-recently-started model before thrash. A backstop for a wrong estimate.
Scenes
A scene is a named set of models brought up together — the unit you activate, publish, and schedule. A single unified-memory box can't hold every model at once, so scenes let you time-multiplex it: run a live voice assistant by day, then switch to an overnight image farm at 3am. One box, the utilization of several.
Scenes reference model ids, never weight paths — so requantizing or upgrading a model
never breaks a published scene. See docs/scene-spec.md, and
docs/positioning.md for how Lumbridge Compute relates to Ollama / llama-swap / exo.
Evals
Lumbridge Compute ships a native, model-server-agnostic evaluation runner. Versioned YAML suites measure streamed TTFT, client-observed prefill throughput, decode throughput, and deterministic capability assertions against any OpenAI-compatible endpoint.
lumbridge-compute eval ls
lumbridge-compute eval run smoke
lumbridge-compute eval run performance --repeat 5
lumbridge-compute eval run finance-core
Every run writes a portable JSON artifact for regression tracking and future eval
registries. See docs/evals.md.
Install
# from source (single static binary, no runtime deps)
cargo install --path .
Usage
lumbridge-compute status # Governor: memory, budget, running set, headroom
lumbridge-compute model ls # registry: footprints + live state + which port
lumbridge-compute scene ls # scenes with total footprint
lumbridge-compute scene show darkroom # models + footprints + admission verdict
lumbridge-compute scene adopt voice-qwen # one-time identity capture for a legacy live node
lumbridge-compute scene resume # desired Scene, then last-known-good fallback
lumbridge-compute gateway # stable streaming endpoint -> local model server
lumbridge-compute agent # resident resume + gateway + memory supervisor
# planned:
lumbridge-compute model pull <hf-id> # footprint-aware; warns if no scene can hold it
lumbridge-compute scene schedule darkroom 03:00 04:00 # time-multiplex the box
By default Lumbridge Compute reads registry/ and scenes/ from the current directory (override with
--root <dir>, $LUMBRIDGE_COMPUTE_ROOT, or the compatibility variable $KUDA_ROOT).
Status
The Governor, identity-bound process ownership, transactional Scene switching,
persistent desired state, last-known-good recovery, resident agent, streaming gateway,
eval runner, and watchdog work today. Next: the model artifact manager (pull), fleet API,
and scheduler. See node operations and the stable
scene spec.
Compatibility
Lumbridge Compute grew out of an earlier prototype named Kuda. The rename is complete: the
executable is lumbridge-compute, manifests declare apiVersion: lumbridge/v1, and the root
is $LUMBRIDGE_COMPUTE_ROOT.
Two compatibility surfaces are kept deliberately, so a node that predates the rename keeps running without a flag day:
$KUDA_ROOTis still read if$LUMBRIDGE_COMPUTE_ROOTis unset..kuda/remains the on-disk state directory name, andapiVersion: kuda/v1is still accepted on eval suites.
Both are inert aliases — nothing new should use them.
Lumbridge Compute runs on NVIDIA hardware but is independent and is not affiliated with or endorsed by NVIDIA.
License
MIT — see LICENSE.