Configuration
Tested vLLM serving configuration for Qwen3.6-27B-FP8 on the LAB workstation (dual RTX 5090, TP=2). Last validated 2026-05-12 against vLLM 0.20.2.
Environment
set -x CUDA_DEVICE_ORDER PCI_BUS_ID
set -x CUDA_VISIBLE_DEVICES 0,1
set -x CUDA_HOME /opt/cuda
set -x NCCL_P2P_DISABLE 1
set -x NCCL_IB_DISABLE 1
set -x PYTORCH_CUDA_ALLOC_CONF expandable_segments:True
set -x NVCC_CCBIN /usr/bin/gcc-15
Launch command
vllm serve Qwen/Qwen3.6-27B-FP8 \
--tensor-parallel-size 2 \
--dtype bfloat16 \
--gpu-memory-utilization 0.94 \
--max-model-len auto \
--max-num-seqs 1 \
--max-num-batched-tokens 2096 \
--language-model-only \
--disable-custom-all-reduce \
--chat-template ~/Labonsky/Project_Freedom/qwen3.6_template/qwen3.6-enhanced.jinja \
--default-chat-template-kwargs '{"preserve_thinking": true}' \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_xml \
--enable-prefix-caching \
--mamba-cache-mode align
Flags
| Flag | Value | Purpose |
|---|---|---|
--tensor-parallel-size | 2 | Shard across both RTX 5090s |
--dtype | bfloat16 | Compute dtype (FP8 weights stay FP8) |
--gpu-memory-utilization | 0.94 | KV-cache headroom |
--max-model-len | auto | Resolves to model’s 262 k context cap |
--max-num-seqs | 1 | Single-stream agentic workload |
--max-num-batched-tokens | 2096 | Must be ≥ attention block size (1568 for this model) |
--language-model-only | — | Skip vision tower |
--disable-custom-all-reduce | — | Use NCCL all-reduce (avoids custom kernel issues on Blackwell) |
--chat-template | qwen3.6-enhanced.jinja | Local template with thinking preservation |
--default-chat-template-kwargs | {"preserve_thinking": true} | Keep <think> blocks across turns |
--reasoning-parser | qwen3 | Parse <think> reasoning |
--enable-auto-tool-choice | — | OpenAI-compatible tool-choice routing |
--tool-call-parser | qwen3_xml | Parse Qwen3’s XML tool-call format |
--enable-prefix-caching | — | Required — Qwen3.6 is hybrid GDN; vLLM defaults off for hybrid attention |
--mamba-cache-mode | align | Mamba prefix caching (vLLM PR #30877, merged 2026-01-23) |
Prefix-caching note
vLLM’s is_prefix_caching_supported returns False for hybrid-attention models, so the default-on prefix cache silently disables itself unless --enable-prefix-caching is set explicitly. Qwen3.6’s architecture is 48× linear attention + 16× full attention — hybrid GDN — so both flags above are required.
vLLM auto-resolves the attention block size to 1568 tokens for this model (logged at engine start). --max-num-batched-tokens must be ≥ 1568; 2096 satisfies that with headroom.
Validation
10 identical-prefix probes (6,434-token deterministic system prompt) against a fresh engine:
| Metric | Result |
|---|---|
| Cache queries | 0 → 64,340 |
| Cache hits | 0 → 56,448 |
| Overall hit rate | 87.7% |
| Per-request hit rate (req 2–10) | 97.5% |
| End-to-end req 1 (cold) | 1.26 s |
| End-to-end req 2–10 (warm) | 0.14 s each |
| Speedup | ~9× |
Hit rate increased monotonically across all 10 requests; no decay under single-session traffic at --max-num-seqs 1.
Tool-parser patch
vLLM 0.20.2 ships a Qwen3XMLToolParser that fires logger.info("Successfully imported tool parser …") from inside __init__. The class is instantiated twice per chat completion, so this prints twice per request. Patch: move the logger.info to module scope so it fires once at engine boot.
File: vllm/tool_parsers/qwen3xml_tool_parser.py
The patch is fragile — any pip install --upgrade vllm overwrites it. Keep a backup alongside (qwen3xml_tool_parser.py.bak-pre-patch).
References
- vLLM PR #30877 — Mamba prefix caching with
alignmode - vLLM Tracking #26201 — Hybrid-model prefix caching status
- vLLM Bug #36697 —
block_size > max_num_batched_tokensvalidation - vLLM Bug #40630 — Qwen3.6 + Claude Code slowness / hit-rate decay
- Qwen3.5/3.6 vLLM Recipes