Qwen3.5-27B Phase A Benchmark DGX Spark GB10 · SM 12.1
Framework × Quantization baseline sweep · 2×NVIDIA DGX Spark GB10 · 128GB unified memory · 2026-04-16/17
Phase A Summary
Peak Throughput
104 tok/s
vLLM GPTQ-Int4 · ISL1024/OSL1024 c=32
Best Latency (ITL p50)
130 ms
vLLM GPTQ-Int4 · ISL1024/OSL1024 c=8
FP8 Speedup over BF16
1.68×
vLLM FP8 vs vLLM BF16 at c=8
Configs Completed
5 / 8
All 5 ran to completion · SGLang/TRT-LLM GPTQ-Int4 blocked
ISL/OSL Benchmark Results
ISL = input sequence length, OSL = output sequence length, c = concurrency. Throughput in tok/s, latency in ms.
| Framework | Quantization | Progress | ISL1024/OSL1024 c=8 tput (tok/s) |
TTFT p50 (ms) | ITL p50 (ms) | ISL1024/OSL1024 c=32 tput (tok/s) |
ISL4096/OSL1024 c=8 tput (tok/s) |
|---|---|---|---|---|---|---|---|
| vLLM | GPTQ-Int4 fastest | 9/9 | 58.3 | 6490 | 130 | 104.1 | 52.3 |
| vLLM | FP8 | 9/9 | 54.2 | 12022 | 135 | 96.9 | 44.5 |
| SGLang | FP8 | 9/9 | 48.5 | 6132 | 159 | 81.5 | 43.8 |
| vLLM | BF16 | 9/9 | 32.2 | 5785 | 243 | 61.2 | 30.5 |
| SGLang | BF16 | 9/9 | 31.1 | 6325 | 250 | 59.8 | 29.0 |
| SGLang | GPTQ-Int4 | blocked | No image with both Qwen3.5 support + SM 12.1 Marlin kernels | ||||
| TRT-LLM | BF16 / FP8 / GPTQ-Int4 | blocked | qwen3_5 architecture unsupported in TRT-LLM 1.3.0rc11 | ||||
Blocked Configurations
SGLang GPTQ-Int4
•
gptq_marlin: Marlin kernels compiled for SM80 only — no SM 12.1 kernel image•
gptq: requires --dtype float16 which breaks Qwen3.5's Mamba causal_conv1d layers (conv_states dtype mismatch)•
compressed-tensors: rejected — model config declares gptq, must match•
lmsysorg/sglang:spark (SM 12.1 branch): qwen3_5 architecture unknown (transformers version too old)Fix: requires a custom SGLang build with both SM 12.1 Marlin kernels and Qwen3.5 support
TRT-LLM (all variants)
•
1.3.0rc10: no SM 12.1 kernels at all (PR #7937 not yet merged)•
1.3.0rc11: SM 12.1 kernels work (19.6GB weights loaded, GPU kernels launched) but load_single_module fails on model.layers.N.mlp.mlp.gate_up_proj — qwen3_5 weight mapping missingFix: wait for TRT-LLM to add qwen3_5 model support
Hardware
Node
DGX Spark GB10
Grace Blackwell · SM 12.1
Memory
128 GB
Unified CPU+GPU memory
Nodes Used
2
spark-01 (vLLM) · spark-02 (SGLang)
Namespace
K8s
Bare pods · Longhorn PVC · nsys profiling
Next Steps — Phase B
Phase B applies optimization techniques one-by-one to the Phase A winner (vLLM GPTQ-Int4) and measures the delta:
chunked-prefill
cuda-graphs
spec-decoding
fp8-kv-cache
prefix-caching