Karti Tripathi
124 lines
5.4 KiB
Markdown
124 lines
5.4 KiB
Markdown
<div align="center">
|
|
|
|
# Lumbridge Compute
|
|
|
|
**The AI compute layer for Lumbridge clusters. Safe models, scenes, and evals on your own hardware.**
|
|
|
|
*Run many models on one box — safely — on NVIDIA DGX Spark (GB10), DGX Station (GB300), and RTX.*
|
|
|
|
</div>
|
|
|
|
---
|
|
|
|
Lumbridge Compute is the tool you install the minute you open your DGX Spark. Boxes like the
|
|
DGX Spark (GB10), DGX Station (GB300), and RTX workstations share **one memory pool**
|
|
between CPU and GPU with little swap. On that hardware, over-committing memory doesn't
|
|
fail gracefully — the whole machine thrashes and wedges (SSH and ping included) before
|
|
the OOM killer acts. Lumbridge Compute makes that impossible, and turns the box into something you
|
|
can load with different workloads on a schedule.
|
|
|
|
> **exo runs one model across many boxes. Lumbridge Compute runs many models on one box — safely.**
|
|
> They're orthogonal; you can run Lumbridge Compute on each node of an exo cluster.
|
|
|
|
## Three layers
|
|
|
|
| Layer | Analogous to | Job |
|
|
|---|---|---|
|
|
| **Governor** | kernel / memory cgroup + OOM policy | Nothing starts unless it fits a hard budget; a watchdog kills the newest model *before* the box wedges. |
|
|
| **Scenes** | systemd targets | Named, shareable, activatable bundles of models (`studio`, `darkroom`). |
|
|
| **lumbridge-compute** (the shell) | the CLI you install | Onboard a fresh box, manage models, activate/schedule scenes. |
|
|
|
|
## The Governor (why Lumbridge Compute exists)
|
|
|
|
Unified memory means no separate VRAM pool to bounce off. vLLM, diffusers, and friends
|
|
will happily reserve past 100% of the shared pool, and the box wedges. The Governor
|
|
prevents this with two mechanisms:
|
|
|
|
1. **Admission control** — a model starts only if its declared footprint fits the budget
|
|
given what's already running, plus a safety margin.
|
|
2. **Watchdog** — a 1 Hz thread on `MemAvailable`; if it dips below a critical floor, it
|
|
kills the most-recently-started model before thrash. A backstop for a wrong estimate.
|
|
|
|
## Scenes
|
|
|
|
A **scene** is a named set of models brought up together — the unit you activate, publish,
|
|
and schedule. A single unified-memory box can't hold every model at once, so scenes let
|
|
you **time-multiplex** it: run a live voice assistant by day, then switch to an overnight
|
|
image farm at 3am. One box, the utilization of several.
|
|
|
|
Scenes reference **model ids**, never weight paths — so requantizing or upgrading a model
|
|
never breaks a published scene. See [`docs/scene-spec.md`](docs/scene-spec.md), and
|
|
[`docs/positioning.md`](docs/positioning.md) for how Lumbridge Compute relates to Ollama / llama-swap / exo.
|
|
|
|
## Evals
|
|
|
|
Lumbridge Compute ships a native, model-server-agnostic evaluation runner. Versioned YAML suites
|
|
measure streamed TTFT, client-observed prefill throughput, decode throughput, and
|
|
deterministic capability assertions against any OpenAI-compatible endpoint.
|
|
|
|
```bash
|
|
lumbridge-compute eval ls
|
|
lumbridge-compute eval run smoke
|
|
lumbridge-compute eval run performance --repeat 5
|
|
lumbridge-compute eval run finance-core
|
|
```
|
|
|
|
Every run writes a portable JSON artifact for regression tracking and future eval
|
|
registries. See [`docs/evals.md`](docs/evals.md).
|
|
|
|
## Install
|
|
|
|
```bash
|
|
# from source (single static binary, no runtime deps)
|
|
cargo install --path .
|
|
```
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
lumbridge-compute status # Governor: memory, budget, running set, headroom
|
|
lumbridge-compute model ls # registry: footprints + live state + which port
|
|
lumbridge-compute scene ls # scenes with total footprint
|
|
lumbridge-compute scene show darkroom # models + footprints + admission verdict
|
|
lumbridge-compute scene adopt voice-qwen # one-time identity capture for a legacy live node
|
|
lumbridge-compute scene resume # desired Scene, then last-known-good fallback
|
|
lumbridge-compute gateway # stable streaming endpoint -> local model server
|
|
lumbridge-compute agent # resident resume + gateway + memory supervisor
|
|
|
|
# planned:
|
|
lumbridge-compute model pull <hf-id> # footprint-aware; warns if no scene can hold it
|
|
lumbridge-compute scene schedule darkroom 03:00 04:00 # time-multiplex the box
|
|
```
|
|
|
|
By default Lumbridge Compute reads `registry/` and `scenes/` from the current directory (override with
|
|
`--root <dir>`, `$LUMBRIDGE_COMPUTE_ROOT`, or the compatibility variable `$KUDA_ROOT`).
|
|
|
|
## Status
|
|
|
|
The Governor, identity-bound process ownership, transactional Scene switching,
|
|
persistent desired state, last-known-good recovery, resident agent, streaming gateway,
|
|
eval runner, and watchdog work today. Next: the model artifact manager (`pull`), fleet API,
|
|
and scheduler. See [node operations](docs/operations.md) and the stable
|
|
[scene spec](docs/scene-spec.md).
|
|
|
|
## Compatibility
|
|
|
|
Lumbridge Compute grew out of an earlier prototype named Kuda. The rename is complete: the
|
|
executable is `lumbridge-compute`, manifests declare `apiVersion: lumbridge/v1`, and the root
|
|
is `$LUMBRIDGE_COMPUTE_ROOT`.
|
|
|
|
Two compatibility surfaces are kept deliberately, so a node that predates the rename keeps
|
|
running without a flag day:
|
|
|
|
- `$KUDA_ROOT` is still read if `$LUMBRIDGE_COMPUTE_ROOT` is unset.
|
|
- `.kuda/` remains the on-disk state directory name, and `apiVersion: kuda/v1` is still
|
|
accepted on eval suites.
|
|
|
|
Both are inert aliases — nothing new should use them.
|
|
|
|
Lumbridge Compute runs on NVIDIA hardware but is independent and is not affiliated with or endorsed by NVIDIA.
|
|
|
|
## License
|
|
|
|
Apache-2.0 — see [LICENSE](LICENSE) and [NOTICE](NOTICE).
|