Token Labs
Benchmark Dashboard

Published findings appear first. These counters describe the open lab notebook, including planned work and documented failures; they are not a publication score. Browse study notes.

Matched disaggregated runs with prefill pinned to spark-01 and decode to spark-02. NVIDIA AIPerf 0.12.0, streaming chat, seed 42, exact-length synthetic prompts, one concurrency wave per point (two requests at c=1), fp8 KV, and DSpark speculative decoding. Prefix caching was disabled on both Dynamo and llm-d workers. Both paths performed remote prefill and transferred KV with NIXL; Dynamo used its frontend and llm-d used the decode-sidecar path.

Machine-readable 18-row matrices: Dynamo, llm-d, and combined comparison. This prefix-cache-off rerun supersedes the earlier cached Dynamo dataset. All 36 current comparison cells completed without request errors. Actual output lengths can differ slightly from requested lengths because of tokenizer and stop-token behavior. Request metrics are authoritative; sidecar server-counter resets produced telemetry warnings without benchmark request failures.

Single-replica control. Aggregated vLLM serving on separate 1× AMD Instinct MI300X instances. NVIDIA AIPerf 0.12.0, streaming chat, concurrency 4, four measured requests per point, fixed token lengths, seed 42. Dynamo path: frontend → vLLM worker. llm-d path: Envoy → EPP → vLLM. Use these results for per-replica capacity and wrapper overhead only; they do not test multi-worker routing, load balancing, horizontal scaling, or failover.

Versions: Dynamo 1.4.2 with vLLM 0.28.1rc1.dev43; llm-d with vLLM 0.28.0. Both used --max-model-len 32768, GPU memory utilization 0.90, exact output lengths, and the same model. Four requests per point make this a controlled snapshot rather than a statistically robust capacity study.

Raw summary: mi300x-qwen38-aiperf.json

Partial run · capacity interrupted. Two MI300X workers per system, one prefill and one decode. Dynamo has 18/18 validated points; 5/18 llm-d points were recovered before the RunPod host became unavailable. Missing cells are pending and will be replaced when MI300X capacity returns.

Validation: producer/consumer roles, matching NIXL compatibility hash, a successful 195.812 MiB remote KV transfer, and 0.0% local prefix-cache hits. Four additional jobs completed but their numeric artifacts were not recovered, so they remain pending.

Machine-readable data: mi300x-qwen38-disagg-partial.json

Single-replica control. Same model, same weights, same KV dtype, same speculative decoder — only the serving framework differs. Measured with NVIDIA AIPerf from the controller node, hitting each backend directly (no gateway in the path). Model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 · modelopt_fp4 weights · fp8 KV · dspark spec-decode (3 tokens) · TP=1 · DGX Spark GB10 · ISL 1024 / OSL 256 · concurrency 1–32 · one model replica per arm. This curve measures per-replica capacity and wrapper overhead, not multi-worker routing or horizontal scaling.

Up and to the right is better: y is what the box produces in total, x is what a single user feels. Raising concurrency walks each curve up and to the left — you buy aggregate throughput with per-user speed.

All planned and completed experiments. Click any row to expand hypothesis and notes.

Lab notebook · not a published finding. Estimated deltas remain visible as hypotheses until the matched runs complete.

Progressive optimization stack on Qwen/Qwen2.5-7B-Instruct · vLLM standalone → llm-d · DGX Spark GB10 · spark-01 · ~150 tok input / 256 tok output · all concurrency levels (c=1, c=8, c=32)

Marginal throughput and TTFT improvement each optimization adds vs the previous step. Model: Qwen/Qwen2.5-7B-Instruct · spark-01 GB10 · c=32 · ~150 tok input / 256 tok output

Throughput vs latency curves across concurrency levels (c=1,4,8,16,32) and input/output length combinations. Model: Qwen/Qwen2.5-7B-Instruct · Full-stack config (exp9) · DGX Spark GB10 · spark-02

Throughput vs latency across concurrency levels (c=1,2,4,8) and input/output length combinations. Model: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 · NVFP4 (Marlin backend, fp8 KV cache) · vllm-cu130-nightly · DGX Spark GB10 · spark-01 · single B200 · 2026-04-15

Throughput vs latency across concurrency levels (c=1,8,32) and input/output length combinations. Model: Qwen/Qwen3.5-27B-GPTQ-Int4 · GPTQ-Int4 (Marlin backend, enforce-eager) · vllm-cu130-nightly · DGX Spark GB10 · spark-01 · single B200 · 2026-04-17

SGLang vs TensorRT-LLM (PyTorch backend) vs llm-d (disaggregated vLLM) on identical workloads. Model: Qwen/Qwen2.5-7B-Instruct · TP=1 · DGX Spark GB10 · 5 ISL/OSL combos × 5 concurrency levels

Lab notebook · data pending. No deployment recommendation is made from this section.

Throughput delta relative to baseline for each optimization. Green = improvement, red = regression. Model: Qwen/Qwen2.5-7B-Instruct · DGX Spark GB10 · enforce_eager → CUDA graphs