77 lines
3.4 KiB
YAML
77 lines
3.4 KiB
YAML
apiVersion: lumbridge/v1
|
|
kind: Scene
|
|
metadata:
|
|
name: qwen38-dflash
|
|
version: 1
|
|
description: "qwen38 with the brain on DFlash2 speculative decoding instead of EAGLE."
|
|
author: karti
|
|
tags: [brain, vision, voice, long-context, sglang, dflash, experimental]
|
|
|
|
# ⛔ BLOCKED 2026-08-21 — DO NOT ACTIVATE ON THE CURRENT IMAGE.
|
|
#
|
|
# Our pinned image (lmsysorg/sglang@sha256:3c0abdf4…) ADVERTISES DFLASH as a
|
|
# builtin speculative algorithm — `sglang serve --help` lists
|
|
# "Builtins: EAGLE, EAGLE3, NEXTN, STANDALONE, NGRAM, DFLASH, DSPARK"
|
|
# and accepts every flag (--speculative-draft-model-path,
|
|
# --mamba-radix-cache-strategy). It boots, loads all 4 shards, then dies at the
|
|
# draft-worker step with:
|
|
#
|
|
# File ".../speculative/dflash_worker_v2.py", line 195, in __init__
|
|
# File ".../speculative/draft_worker_common.py", line 83, in build_draft_tp_worker
|
|
# ValueError: Cannot find model module. 'DFlash2DraftModel' is not a
|
|
# registered model in the Transformers library ... and 'AutoModel' is not
|
|
# present in the model config's 'auto_map'
|
|
#
|
|
# i.e. the ALGORITHM is builtin but the DRAFT MODEL ARCHITECTURE is not
|
|
# registered. That is precisely why MiaAI-Lab's recipe builds a DERIVED image
|
|
# from its `patch/` directory for DFlash2 while DSpark runs on the stock image.
|
|
# The flag surface being present is NOT evidence the path works — it boots for
|
|
# ~3 minutes before failing, and with `--rm` the logs vanish with the container.
|
|
#
|
|
# TO UNBLOCK: build MiaAI-Lab/Qwen3.8-27B-SGLang-DGX-Spark's patched image
|
|
# (./start-dflash.sh builds it), then repin the digest in the registry entry.
|
|
#
|
|
# WORTH IT? MiaAI-Lab's measured table for this box class:
|
|
# probe EAGLE/MTP DSpark DFlash2
|
|
# code 34.5 51.5 50.9
|
|
# long essay 24.1 18.3 25.4
|
|
# short chat 21.0 23.2 66.6
|
|
# Our EAGLE baseline measured HERE on 2026-08-21: decode 22.8 tok/s mean
|
|
# (TTFT 223ms), tool-calling 15/15, with live squawk traffic on the box —
|
|
# consistent with the 21.0 short-chat column. DSpark is NOT worth the swap for
|
|
# us (23.2 vs 22.8 short chat; its win is coding, which is Claude's job here,
|
|
# not spark's). DFlash2's 66.6 short-chat number is the only prize worth the
|
|
# image build — and it is the number most in need of independent confirmation,
|
|
# since MiaAI-Lab itself footnotes it as "once token-counted correctly".
|
|
#
|
|
# The drafter is already downloaded: /srv/models/Qwen3.8-27B-DFlash2
|
|
# (z-lab/Qwen3.8-27B-DFlash2@50307d4, 3.6 GB).
|
|
#
|
|
# A/B TWIN OF `qwen38`. Identical model set and identical brain flags except the
|
|
# speculative algorithm (EAGLE -> DFLASH) and the drafter mount, so a benchmark
|
|
# delta is attributable to the algorithm alone.
|
|
#
|
|
# Same ordering rule as `qwen38`: brain FIRST. --mem-fraction-static is a
|
|
# fraction of TOTAL memory and is reserved as a static pool at boot.
|
|
#
|
|
# Same activation race too: do not `docker stop` then immediately activate —
|
|
# wait for `free -g` to show the memory back first.
|
|
#
|
|
# Compare with:
|
|
# lumbridge-compute eval run performance --model brain-qwen38-dflash
|
|
# python3 /tmp/toolbench.py http://127.0.0.1:8001/v1 brain-qwen38-dflash 3
|
|
# Baseline to beat (EAGLE, measured 2026-08-21, with live squawk traffic on the
|
|
# box): decode 22.8 tok/s mean, TTFT 223ms, tool-calling 15/15.
|
|
#
|
|
# Revert with `lumbridge-compute scene activate qwen38`.
|
|
models:
|
|
- brain-qwen38-dflash
|
|
- ears
|
|
- voice-vox
|
|
- voice
|
|
|
|
budget_gb: 100
|
|
activation:
|
|
order: listed
|
|
wait_healthy: true
|