Karti Tripathi
75aa63d737
ci / rust (push) Successful in 2m26s
Governed compute for unified-memory AI hardware — the machines where CPU and GPU share one pool and there is no separate VRAM allocation to bounce off. Over-commit that pool and the box thrashes and wedges, SSH and ping included, before the OOM killer gets a turn. Compute does not run inference. It supervises the servers that do: - Admission control against two ceilings: a declared budget, and what the machine actually has free. The refusal is the feature. - A 1 Hz watchdog on MemAvailable that stops the newest model before thrash, and defers to a Scene transition rather than racing it. - Scenes: named sets of models activated as one transactional unit, with pre-flight validation and rollback to the previously active Scene on failure. Scenes reference model ids, never weight paths or commands, so a Scene obtained from elsewhere cannot introduce code. - Process ownership bound to (boot_id, pid, start_time_ticks, pgid == pid), so a reused PID can never be group-killed. - A protocol-transparent TCP gateway, so clients keep one address while model runtimes move behind it. - An MCP server, so agents drive the node as tools rather than as a CLI. One binary, no async runtime outside the MCP surface. Apache-2.0. Generated from the internal monorepo by scripts/publish-compute.sh, which refuses to publish a tree it cannot prove clean.
52 lines
3.0 KiB
Markdown
52 lines
3.0 KiB
Markdown
# Positioning
|
|
|
|
Where Lumbridge Compute sits relative to the tools people already know. The short version: the
|
|
neighbors solve *different* problems, and the closest one has exactly the gap Lumbridge Compute fills.
|
|
|
|
## The landscape
|
|
|
|
| Tool | Axis | What it does | Gap for a unified-memory box |
|
|
|---|---|---|---|
|
|
| **exo** | scale **out** | Shards *one* big model *across* many boxes (Thunderbolt/RDMA, MLX, Apple-first) | Doesn't run many models on one box; not GB10/GB300-native |
|
|
| **Ollama** | single model | Dead-simple pull/run one LLM, huge library | No multi-service orchestration, **no memory governor** |
|
|
| **llama-swap** | swap **in/out** | On-demand load/unload, TTL idle-unload, co-run matrix, any OpenAI server | **"No memory budgeting or OOM prevention"** (their words), LLM-swapping not capability-stacks, no scheduling/sharing |
|
|
| **LiteLLM** | routing | One OpenAI endpoint in front of many backends | A proxy — doesn't manage lifecycle or memory |
|
|
|
|
## The one-liner
|
|
|
|
> **exo runs one model across many boxes. Lumbridge Compute runs many models on one box — safely.**
|
|
|
|
Orthogonal, not competing — you can run Lumbridge Compute on each node of an exo cluster.
|
|
|
|
## Where Lumbridge Compute is unique
|
|
|
|
1. **The Governor.** No one else does unified-memory admission control + an OOM-wedge
|
|
watchdog. It's the pain *specific* to Grace-Blackwell shared memory, where over-commit
|
|
wedges the whole box instead of failing a single allocation. This is the moat.
|
|
2. **Scenes are capability stacks, not model swaps.** llama-swap picks *which LLM* answers.
|
|
Scenes compose *heterogeneous services together* — ASR + LLM + TTS + image + music — as
|
|
one activatable unit.
|
|
3. **Scheduled scenes** — time-multiplex a box by hour (live assistant by day, image farm
|
|
overnight). Nobody does this.
|
|
4. **Shareable scene registry** — publish and pull scenes. The community flywheel.
|
|
5. **GB10 / GB300 / RTX-native** — NVFP4, `flashinfer` MoE backend, `sm_121a`. exo is
|
|
Apple-first; Lumbridge Compute is Grace-Blackwell-first.
|
|
|
|
## Ideas worth borrowing (don't reinvent)
|
|
|
|
- **From llama-swap:** TTL idle-unload; an OpenAI-compatible gateway in front so apps hit
|
|
one stable endpoint regardless of which scene is live; a web UI; Prometheus metrics;
|
|
install via brew/binary/docker.
|
|
- **From Ollama:** one-command onboarding and a friendly `model pull/ls/rm` UX + a library.
|
|
- **From LiteLLM:** the router-in-front pattern — pairs with scenes (the `served_name`
|
|
aliases already point this way).
|
|
- **From exo:** OpenAI + Anthropic + Ollama-compatible APIs; later, auto-discovery so Lumbridge Compute
|
|
can coordinate scenes across two Sparks.
|
|
|
|
## The model manager angle
|
|
|
|
Lumbridge Compute's `model` commands aren't just another Ollama. They're **governor-aware and
|
|
scene-aware**: Lumbridge Compute knows each model's footprint, warns *before* you pull something no
|
|
scene could ever hold, and shows which scenes a model belongs to. Ollama manages files;
|
|
Lumbridge Compute manages *what can actually run together*.
|