113 lines
5.1 KiB
Markdown
113 lines
5.1 KiB
Markdown
<div align="center">
|
|
|
|
# Lumbridge Compute
|
|
|
|
**The AI compute layer for Lumbridge clusters. Safe models, scenes, and evals on your own hardware.**
|
|
|
|
*Run many models on one box — safely — on NVIDIA DGX Spark (GB10), DGX Station (GB300), and RTX.*
|
|
|
|
</div>
|
|
|
|
---
|
|
|
|
Lumbridge Compute is the tool you install the minute you open your DGX Spark. Boxes like the
|
|
DGX Spark (GB10), DGX Station (GB300), and RTX workstations share **one memory pool**
|
|
between CPU and GPU with little swap. On that hardware, over-committing memory doesn't
|
|
fail gracefully — the whole machine thrashes and wedges (SSH and ping included) before
|
|
the OOM killer acts. Lumbridge Compute makes that impossible, and turns the box into something you
|
|
can load with different workloads on a schedule.
|
|
|
|
> **exo runs one model across many boxes. Lumbridge Compute runs many models on one box — safely.**
|
|
> They're orthogonal; you can run Lumbridge Compute on each node of an exo cluster.
|
|
|
|
## Three layers
|
|
|
|
| Layer | Analogous to | Job |
|
|
|---|---|---|
|
|
| **Governor** | kernel / memory cgroup + OOM policy | Nothing starts unless it fits a hard budget; a watchdog kills the newest model *before* the box wedges. |
|
|
| **Scenes** | systemd targets | Named, shareable, activatable bundles of models (`studio`, `darkroom`). |
|
|
| **lumbridge-compute** (the shell) | the CLI you install | Onboard a fresh box, manage models, activate/schedule scenes. |
|
|
|
|
## The Governor (why Lumbridge Compute exists)
|
|
|
|
Unified memory means no separate VRAM pool to bounce off. vLLM, diffusers, and friends
|
|
will happily reserve past 100% of the shared pool, and the box wedges. The Governor
|
|
prevents this with two mechanisms:
|
|
|
|
1. **Admission control** — a model starts only if its declared footprint fits the budget
|
|
given what's already running, plus a safety margin.
|
|
2. **Watchdog** — a 1 Hz thread on `MemAvailable`; if it dips below a critical floor, it
|
|
kills the most-recently-started model before thrash. A backstop for a wrong estimate.
|
|
|
|
## Scenes
|
|
|
|
A **scene** is a named set of models brought up together — the unit you activate, publish,
|
|
and schedule. A single unified-memory box can't hold every model at once, so scenes let
|
|
you **time-multiplex** it: run a live voice assistant by day, then switch to an overnight
|
|
image farm at 3am. One box, the utilization of several.
|
|
|
|
Scenes reference **model ids**, never weight paths — so requantizing or upgrading a model
|
|
never breaks a published scene. See [`docs/scene-spec.md`](docs/scene-spec.md), and
|
|
[`docs/positioning.md`](docs/positioning.md) for how Lumbridge Compute relates to Ollama / llama-swap / exo.
|
|
|
|
## Evals
|
|
|
|
Lumbridge Compute ships a native, model-server-agnostic evaluation runner. Versioned YAML suites
|
|
measure streamed TTFT, client-observed prefill throughput, decode throughput, and
|
|
deterministic capability assertions against any OpenAI-compatible endpoint.
|
|
|
|
```bash
|
|
lumbridge-compute eval ls
|
|
lumbridge-compute eval run smoke
|
|
lumbridge-compute eval run performance --repeat 5
|
|
lumbridge-compute eval run finance-core
|
|
lumbridge-compute usage summary --since 24h # calls, tokens, vision, C0-C4, queueing
|
|
lumbridge-compute usage agents --since 7d # bounded client/agent/workload labels
|
|
lumbridge-compute usage concurrency --since 30d
|
|
```
|
|
|
|
Every run writes a portable JSON artifact for regression tracking and future eval
|
|
registries. See [`docs/evals.md`](docs/evals.md).
|
|
|
|
## Install
|
|
|
|
```bash
|
|
# from source (single static binary, no runtime deps)
|
|
cargo install --path .
|
|
```
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
lumbridge-compute status # Governor: memory, budget, running set, headroom
|
|
lumbridge-compute model ls # registry: footprints + live state + which port
|
|
lumbridge-compute scene ls # scenes with total footprint
|
|
lumbridge-compute scene show darkroom # models + footprints + admission verdict
|
|
lumbridge-compute scene adopt voice-qwen # one-time identity capture for a legacy live node
|
|
lumbridge-compute scene resume # desired Scene, then last-known-good fallback
|
|
lumbridge-compute gateway # stable streaming endpoint -> local model server
|
|
lumbridge-compute agent # resume + gateway + memory + opt-in model supervision
|
|
|
|
# planned:
|
|
lumbridge-compute model pull <hf-id> # footprint-aware; warns if no scene can hold it
|
|
lumbridge-compute scene schedule darkroom 03:00 04:00 # time-multiplex the box
|
|
```
|
|
|
|
By default Lumbridge Compute reads `registry/` and `scenes/` from the current directory (override with
|
|
`--root <dir>` or `$LUMBRIDGE_COMPUTE_ROOT`).
|
|
|
|
## Status
|
|
|
|
The Governor, identity-bound process ownership, transactional Scene switching,
|
|
persistent desired state, last-known-good recovery, opt-in model supervision,
|
|
streaming gateway, eval runner, watchdog, and privacy-safe usage telemetry work
|
|
today. Next: the model artifact manager (`pull`), fleet API, and telemetry-driven
|
|
Jobs scheduler. See [usage telemetry](docs/telemetry.md), [node operations](docs/operations.md),
|
|
and the stable [scene spec](docs/scene-spec.md).
|
|
|
|
Lumbridge Compute runs on NVIDIA hardware but is independent and is not affiliated with or endorsed by NVIDIA.
|
|
|
|
## License
|
|
|
|
Apache-2.0 — see [LICENSE](LICENSE).
|