NVIDIA H200 prefill → AMD MI300X decode
A 21-point Dynamo + vLLM production evaluation of heterogeneous prefill/decode using Mooncake TCP for KV-cache transfer. Qwen3-30B-A3B FP8 ran on one GPU per node with prefix caching disabled.
Throughput scaling
Output token throughput across the requested concurrency ladder. Both long-output shapes peak at c=32; c=64 exceeds the efficient operating point for this public cross-cloud TCP topology.
Topology and runtime
KV transfer verification
Exact token counts establish 98,304 KV bytes per input token. Independent socket-counter probes validate both input sizes; Mooncake reported zero transfer failures, failed receives, or expired KV requests.
100,744,384 TCP payload bytes observed, including 0.0806% protocol overhead.
805,896,192 TCP payload bytes observed, including 0.0732% protocol overhead.
Exact derived application KV payload across the complete 381-request matrix.
Complete matrix
Every row used exact input and output token lengths. TTFT and TPOT are milliseconds; throughput is output tokens per second.
| ISL / OSL | Concurrency | Output tok/s | Median TTFT | P99 TTFT | Median TPOT | Validation |
|---|---|---|---|---|---|---|
| 1k / 8k | 1 | 125.2 | 214.1 | 214.1 | 7.96 | pass |
| 1k / 8k | 2 | 244.5 | 355.1 | 355.2 | 8.14 | pass |
| 1k / 8k | 4 | 449.7 | 678.8 | 679.2 | 8.81 | pass |
| 1k / 8k | 8 | 782.7 | 845.0 | 2,838.6 | 9.86 | pass |
| 1k / 8k | 16 | 1,306.6 | 1,766.5 | 5,687.0 | 11.51 | pass |
| 1k / 8k | 32 | 1,971.1 | 5,905.7 | 11,831.1 | 14.69 | pass |
| 1k / 8k | 64 | 1,788.6 | 18,775.6 | 34,779.6 | 31.54 | pass |
| 8k / 8k | 1 | 109.6 | 1,279.7 | 1,279.7 | 8.97 | pass |
| 8k / 8k | 2 | 209.3 | 1,911.1 | 2,442.4 | 9.25 | pass |
| 8k / 8k | 4 | 347.6 | 3,259.6 | 4,822.7 | 10.91 | pass |
| 8k / 8k | 8 | 569.5 | 6,661.6 | 14,824.9 | 12.25 | pass |
| 8k / 8k | 16 | 755.7 | 27,818.7 | 44,247.4 | 15.82 | pass |
| 8k / 8k | 32 | 878.9 | 77,376.2 | 146,828.5 | 20.22 | pass |
| 8k / 8k | 64 | 809.1 | 136,243.0 | 324,594.1 | 45.67 | pass |
| 8k / 1k | 1 | 102.5 | 1,314.1 | 1,314.1 | 8.48 | pass |
| 8k / 1k | 2 | 180.6 | 1,904.6 | 2,431.8 | 8.70 | pass |
| 8k / 1k | 4 | 279.8 | 3,759.8 | 4,802.4 | 9.61 | pass |
| 8k / 1k | 8 | 304.6 | 8,848.9 | 16,782.0 | 10.02 | pass |
| 8k / 1k | 16 | 334.8 | 20,309.1 | 38,987.6 | 10.20 | pass |
| 8k / 1k | 32 | 351.0 | 44,365.5 | 83,639.1 | 10.18 | pass |
| 8k / 1k | 64 | 393.9 | 83,285.7 | 156,061.6 | 10.78 | pass |
Download evidence
Machine-readable performance data and the strict transfer-byte validation summary.