Qwen3.5-27B Phase A — Token Labs Benchmark

Peak Throughput

104 tok/s

vLLM GPTQ-Int4 · ISL1024/OSL1024 c=32

Best Latency (ITL p50)

130 ms

vLLM GPTQ-Int4 · ISL1024/OSL1024 c=8

FP8 Speedup over BF16

1.68×

vLLM FP8 vs vLLM BF16 at c=8

Configs Completed

5 / 8

All 5 ran to completion · SGLang/TRT-LLM GPTQ-Int4 blocked

ISL = input sequence length, OSL = output sequence length, c = concurrency. Throughput in tok/s, latency in ms.

Framework	Quantization	Progress	ISL1024/OSL1024 c=8 tput (tok/s)	TTFT p50 (ms)	ITL p50 (ms)	ISL1024/OSL1024 c=32 tput (tok/s)	ISL4096/OSL1024 c=8 tput (tok/s)
vLLM	GPTQ-Int4 fastest	9/9 ✓	58.3	6490	130	104.1	52.3
vLLM	FP8	9/9 ✓	54.2	12022	135	96.9	44.5
SGLang	FP8	9/9 ✓	48.5	6132	159	81.5	43.8
vLLM	BF16	9/9 ✓	32.2	5785	243	61.2	30.5
SGLang	BF16	9/9 ✓	31.1	6325	250	59.8	29.0
SGLang	GPTQ-Int4	blocked	No image with both Qwen3.5 support + SM 12.1 Marlin kernels
TRT-LLM	BF16 / FP8 / GPTQ-Int4	blocked	qwen3_5 architecture unsupported in TRT-LLM 1.3.0rc11

SGLang GPTQ-Int4

• gptq_marlin: Marlin kernels compiled for SM80 only — no SM 12.1 kernel image

• gptq: requires --dtype float16 which breaks Qwen3.5's Mamba causal_conv1d layers (conv_states dtype mismatch)

• compressed-tensors: rejected — model config declares gptq, must match

• lmsysorg/sglang:spark (SM 12.1 branch): qwen3_5 architecture unknown (transformers version too old)

Fix: requires a custom SGLang build with both SM 12.1 Marlin kernels and Qwen3.5 support

TRT-LLM (all variants)

• 1.3.0rc10: no SM 12.1 kernels at all (PR #7937 not yet merged)

• 1.3.0rc11: SM 12.1 kernels work (19.6GB weights loaded, GPU kernels launched) but load_single_module fails on model.layers.N.mlp.mlp.gate_up_proj — qwen3_5 weight mapping missing

Fix: wait for TRT-LLM to add qwen3_5 model support

Phase B applies optimization techniques one-by-one to the Phase A winner (vLLM GPTQ-Int4) and measures the delta:

Qwen3.5-27B Phase A Benchmark DGX Spark GB10 · SM 12.1