Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Configuration

Tested vLLM serving configuration for Qwen3.6-27B-FP8 on the LAB workstation (dual RTX 5090, TP=2). Last validated 2026-05-12 against vLLM 0.20.2.

Environment

set -x CUDA_DEVICE_ORDER PCI_BUS_ID
set -x CUDA_VISIBLE_DEVICES 0,1
set -x CUDA_HOME /opt/cuda
set -x NCCL_P2P_DISABLE 1
set -x NCCL_IB_DISABLE 1
set -x PYTORCH_CUDA_ALLOC_CONF expandable_segments:True
set -x NVCC_CCBIN /usr/bin/gcc-15

Launch command

vllm serve Qwen/Qwen3.6-27B-FP8 \
    --tensor-parallel-size 2 \
    --dtype bfloat16 \
    --gpu-memory-utilization 0.94 \
    --max-model-len auto \
    --max-num-seqs 1 \
    --max-num-batched-tokens 2096 \
    --language-model-only \
    --disable-custom-all-reduce \
    --chat-template ~/Labonsky/Project_Freedom/qwen3.6_template/qwen3.6-enhanced.jinja \
    --default-chat-template-kwargs '{"preserve_thinking": true}' \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_xml \
    --enable-prefix-caching \
    --mamba-cache-mode align

Flags

FlagValuePurpose
--tensor-parallel-size2Shard across both RTX 5090s
--dtypebfloat16Compute dtype (FP8 weights stay FP8)
--gpu-memory-utilization0.94KV-cache headroom
--max-model-lenautoResolves to model’s 262 k context cap
--max-num-seqs1Single-stream agentic workload
--max-num-batched-tokens2096Must be ≥ attention block size (1568 for this model)
--language-model-only—Skip vision tower
--disable-custom-all-reduce—Use NCCL all-reduce (avoids custom kernel issues on Blackwell)
--chat-templateqwen3.6-enhanced.jinjaLocal template with thinking preservation
--default-chat-template-kwargs{"preserve_thinking": true}Keep <think> blocks across turns
--reasoning-parserqwen3Parse <think> reasoning
--enable-auto-tool-choice—OpenAI-compatible tool-choice routing
--tool-call-parserqwen3_xmlParse Qwen3’s XML tool-call format
--enable-prefix-caching—Required — Qwen3.6 is hybrid GDN; vLLM defaults off for hybrid attention
--mamba-cache-modealignMamba prefix caching (vLLM PR #30877, merged 2026-01-23)

Prefix-caching note

vLLM’s is_prefix_caching_supported returns False for hybrid-attention models, so the default-on prefix cache silently disables itself unless --enable-prefix-caching is set explicitly. Qwen3.6’s architecture is 48× linear attention + 16× full attention — hybrid GDN — so both flags above are required.

vLLM auto-resolves the attention block size to 1568 tokens for this model (logged at engine start). --max-num-batched-tokens must be ≥ 1568; 2096 satisfies that with headroom.

Validation

10 identical-prefix probes (6,434-token deterministic system prompt) against a fresh engine:

MetricResult
Cache queries0 → 64,340
Cache hits0 → 56,448
Overall hit rate87.7%
Per-request hit rate (req 2–10)97.5%
End-to-end req 1 (cold)1.26 s
End-to-end req 2–10 (warm)0.14 s each
Speedup~9×

Hit rate increased monotonically across all 10 requests; no decay under single-session traffic at --max-num-seqs 1.

Tool-parser patch

vLLM 0.20.2 ships a Qwen3XMLToolParser that fires logger.info("Successfully imported tool parser …") from inside __init__. The class is instantiated twice per chat completion, so this prints twice per request. Patch: move the logger.info to module scope so it fires once at engine boot.

File: vllm/tool_parsers/qwen3xml_tool_parser.py

The patch is fragile — any pip install --upgrade vllm overwrites it. Keep a backup alongside (qwen3xml_tool_parser.py.bak-pre-patch).

References