97 lines
3.6 KiB
Markdown
97 lines
3.6 KiB
Markdown
# Usage telemetry and concurrency history
|
|
|
|
Lumbridge Compute records operational model usage without recording model input
|
|
or output. The runtime remains the source of truth: Compute samples its local
|
|
Prometheus endpoint and makes counter resets durable across model and Scene
|
|
restarts.
|
|
|
|
## SGLang launch contract
|
|
|
|
The managed SGLang recipe enables:
|
|
|
|
```text
|
|
--enable-metrics
|
|
--enable-cache-report
|
|
--tokenizer-metrics-allowed-custom-labels client agent workload
|
|
```
|
|
|
|
Callers may attach a bounded label dictionary in SGLang's `x-custom-labels`
|
|
header, for example:
|
|
|
|
```text
|
|
x-custom-labels: {"client":"cloud-agent","agent":"research-1","workload":"interactive"}
|
|
```
|
|
|
|
Vision callers use `"workload":"vision"`. Compute prefers native encoder
|
|
counters when the serving build exports them and otherwise uses this label for
|
|
vision call share and vision prompt-token volume. The report includes
|
|
`vision_request_source` so this fallback is never ambiguous.
|
|
|
|
`client`, `agent`, and `workload` must be low-cardinality stable categories.
|
|
Never put a request id, conversation id, user id, file name, URL, or job id in a
|
|
Prometheus label. Job ids belong in the future Jobs ledger.
|
|
|
|
## What is stored
|
|
|
|
The database is `<root>/.compute/usage/telemetry.sqlite3` in WAL mode. It contains:
|
|
|
|
- time-sampled running and queued requests, configured concurrency, generation
|
|
throughput, and cache hit rate;
|
|
- reset-safe request, prompt-token, generation-token, cached-token, and abort
|
|
counters;
|
|
- vision request share and vision prompt-token volume, plus multimodal image
|
|
items, encoder tokens, and image-cache hits when the serving build exports
|
|
those native counters;
|
|
- the active Scene, model id, and a fingerprint of the launch configuration; and
|
|
- the three bounded caller labels above.
|
|
|
|
It never stores prompts, input token ids, model output, image/audio/video data,
|
|
image URLs, arbitrary headers, or sampling parameters. Counter snapshots are
|
|
written once per minute. Scheduler gauges are sampled every five seconds by
|
|
default, which makes the C0-C4 distribution a time-weighted approximation rather
|
|
than a count of request admissions.
|
|
|
|
## Commands
|
|
|
|
```bash
|
|
lumbridge-compute usage collect
|
|
lumbridge-compute usage summary --since 24h
|
|
lumbridge-compute usage agents --since 7d
|
|
lumbridge-compute usage concurrency --since 30d
|
|
|
|
# Every report also has machine-readable output.
|
|
lumbridge-compute usage summary --since 7d --json
|
|
```
|
|
|
|
The resident agent collects automatically. Use `--no-telemetry` only when the
|
|
runtime cannot expose local metrics; failures degrade telemetry and do not stop
|
|
the gateway, watchdog, or model supervisor.
|
|
|
|
The loopback control API exposes the same read-only records:
|
|
|
|
```text
|
|
GET /v1/usage?since=24h
|
|
GET /v1/usage/agents?since=7d
|
|
GET /v1/usage/concurrency?since=30d
|
|
```
|
|
|
|
## Four-lane scheduling policy
|
|
|
|
Telemetry is the gate for Jobs scheduling, not a reason to keep all lanes busy
|
|
unconditionally. The measured DFlash2 curve on the reference GB10 node is
|
|
31.54, 57.09, 81.43, and 85.96 aggregate token/s at C1-C4. C4 adds only 5.6%
|
|
over C3.
|
|
|
|
The initial scheduler policy therefore is:
|
|
|
|
1. three fair-share background lanes;
|
|
2. one latency-reserve lane for interactive work;
|
|
3. background borrowing of lane four only while queueing and TTFT remain healthy;
|
|
4. weighted round-robin across agents, with job ids in a durable ledger rather
|
|
than metric labels; and
|
|
5. one simultaneous vision-prefill job until measured encoder queueing supports
|
|
relaxing the limit.
|
|
|
|
Compute should own admission and job lifecycle. Bench may ingest aggregate
|
|
artifacts, but it should not receive raw operational traffic or request content.
|