83 lines
4.2 KiB
YAML
83 lines
4.2 KiB
YAML
apiVersion: lumbridge/v1
|
|
kind: Scene
|
|
metadata:
|
|
name: qwen38
|
|
version: 2
|
|
description: "Qwen3.8-27B on SGLang as a seeing brain, plus ears and both narrators."
|
|
author: karti
|
|
tags: [brain, vision, voice, long-context, sglang]
|
|
|
|
# Created 2026-08-17. A copy of `stock` with two changes:
|
|
#
|
|
# 1. brain-nemotron-128k (Lightning 30B-A3B, vLLM) -> brain-qwen38
|
|
# (Qwen3.8-27B NVFP4+MTP, SGLang EAGLE, in Docker)
|
|
# 2. `eye` (Cosmos 3 Edge, 14 GB) is DROPPED — the new brain is a native VLM
|
|
# and SGLang serves its vision tower in-process, so the separate eye is
|
|
# redundant. This also kills the enable_thinking:false hack the eye needed
|
|
# because vLLM 0.27.1 has no cosmos3_edge reasoning parser.
|
|
#
|
|
# brain-qwen38 56 GB Qwen3.8-27B @ 262k, sees images and video :8001
|
|
# voice-vox 20 GB VoxCPM 2, quality narrator, capped 3600 chars :8096
|
|
# ears 10 GB streaming ASR, self-limiting at 6m39s of audio :8006
|
|
# voice 5 GB Chatterbox TURBO, request-path narrator :8095
|
|
# -----
|
|
# 91 GB committed +8 GB Governor safety margin = 99 of 100.
|
|
#
|
|
# ⚠️ ORDER IS `listed`, NOT footprint-asc, AND THE BRAIN IS FIRST. SGLang_s
|
|
# --mem-fraction-static is a fraction of TOTAL memory and is reserved as a
|
|
# static pool at boot — if ears/voice/vox are already resident it computes its
|
|
# pool against a box that looks full and dies with a mamba-state-cache error.
|
|
# This is the same failure mode the primary scene documents for `brain`.
|
|
#
|
|
# ⚠️ THE SPEED TRADE vs `stock`. Lightning is a 3B-active MoE and decodes ~46
|
|
# tok/s class on this box; Qwen3.8-27B is DENSE and decodes ~25.6 tok/s single
|
|
# stream. You are buying a large quality jump (Terminal-Bench 2.1 63.4 -> 73.0,
|
|
# OSWorld-Verified 63.9 -> 84.3 over Qwen3.6-27B) with roughly half the
|
|
# single-stream token rate. Under concurrency the gap closes and reverses:
|
|
# SGLang EAGLE measured 123.90 tok/s at c8.
|
|
#
|
|
# ⚠️ THINKING IS OFF BY DEFAULT (added v2, 2026-08-21). Qwen3.8 thinks by
|
|
# default and SGLang's auto-detect confirms it
|
|
# (reasoning_config=ReasoningToggleConfig(toggle_param='enable_thinking',
|
|
# default_enabled=True)). That silently broke the assistant stack on first activation:
|
|
# V9's normalizeSpeech (apps/server/src/lib/llm.ts:77) sends max_tokens: 80, the
|
|
# reasoning block consumed all 80, finish_reason came back "length" and content
|
|
# was EMPTY — so news and squawk narration produced nothing, with no error.
|
|
# The registry entry now passes --default-chat-template-kwargs
|
|
# '{"enable_thinking": false}', matching what every brain-nemotron* entry
|
|
# already did. Per-request chat_template_kwargs still overrides, so an agent
|
|
# that wants thinking just asks for it.
|
|
#
|
|
# CANARIES after any change to this scene (all three must pass):
|
|
# 1. 19 x 23 -> 437 (417 means FP8 KV regressed)
|
|
# 2. max_tokens:80 request -> NON-EMPTY content (guards the above)
|
|
# 3. chat_template_kwargs {"enable_thinking":true} -> reasoning_content present
|
|
#
|
|
# ⚠️ ACTIVATION IS MEMORY-RACY. `docker stop brain-qwen38` followed immediately
|
|
# by `scene activate` fails: --mem-fraction-static is computed against TOTAL
|
|
# memory but the stopped container's pages are not released yet, so the static
|
|
# pool lands short and the container dies at boot with no useful error. The
|
|
# activate process then hangs on "waiting for exact health" holding the
|
|
# transition lock, and the next activate returns "another Scene transition is
|
|
# already running". Wait for `free -g` to show the memory back (~90 GB avail)
|
|
# before re-activating. Boot then takes ~4 min (FlashInfer autotune + CUDA graph).
|
|
#
|
|
# VERIFIED 2026-08-21 on this scene: all three canaries pass; vision reads a
|
|
# generated test image exactly (shapes, colours and the literal string);
|
|
# tool calling picks the right tool with a correct command under a 21-tool
|
|
# payload; the larger local model answered a synthetic container-count prompt correctly with
|
|
# the command cited — the same question the previous Nemotron brain got wrong
|
|
# five times running.
|
|
#
|
|
# Restore the previous default with `lumbridge-compute scene activate stock`.
|
|
models:
|
|
- brain-qwen38
|
|
- ears
|
|
- voice-vox
|
|
- voice
|
|
|
|
budget_gb: 100
|
|
activation:
|
|
order: listed # brain FIRST — it reserves a static pool, see above
|
|
wait_healthy: true
|