Token Labs · Inference benchmark campaign · October 21–22, 2026 UTC
Qwen3.8-27B: MTP3, EAGLE4, DFlash2 and OpenJet
Retain baseline MTP3. It led the measured NVFP4 AgentX c32 series at 171.19 output tokens/s. No tested configuration demonstrated a better overall AgentX result. The experiment has ended; the tuning Pod was deleted and production was unchanged.
One 96GB RTX PRO 6000 Blackwell Server Edition, one model worker. NVFP4 target: nvidia/Qwen3.8-27B-NVFP4, revision 482ca0f3832238542f8f5295dde86b5f22711d80. These are single-worker serving measurements, not distributed scheduling or a public API capacity guarantee.
AgentX trace replay, NVFP4
AgentX 1.0.7, streaming /v1/chat/completions, Weka062126 corpus, whole-session 64K filter, seed 42, temperature 0, ignore_eos, 900-second profile and 300-second grace. Client requested 32 sessions. Raw request-span output throughput includes agent waits and excludes final idle drain accounting. Native drain-inclusive aggregates are not interchangeable with this metric.
| Configuration | Sessions | Requests | Output tok/s | Decode tok/s/user | p95 TTFT, ms | p95 completion, ms |
|---|---|---|---|---|---|---|
| baseline-mtp3 | 32 | 376 | 171.19 | 52.74 | 67,573.28 | 103,120.02 |
| eagle4-depth6-prefix-full | 32 | 253 | 112.90 | 34.80 | 110,381.36 | 174,217.41 |
| maxseq16 | 32 | 360 | 160.84 | 46.52 | 57,795.44 | 128,162.49 |
| sglang-target-api-full | 32 | 229 | 93.25 | 21.97 | 138,104.31 | 214,085.14 |
| sglang-dflash2-api-full | 32 | 240 | 104.59 | 35.15 | 109,289.81 | 167,083.65 |
These are single runs without a calibrated noise floor. Cache history, runtime changes and API repairs limit attribution to individual knobs. This is the best measured configuration for this trace, not a global optimum or production release qualification.
OpenJet and Unsloth GGUF, separate comparison
Unsloth UD-Q4_K_XL uses a different quantization and engine. OpenJet executes one request at a time even when AgentX requests 32 sessions. Its high per-stream decode speed does not establish high concurrent aggregate throughput. The trial passed ten quality checks and four long-context canaries; complete API conformance was not established.
| Configuration | Sessions | Requests | Output tok/s | Decode tok/s/user | p95 TTFT, ms | p95 completion, ms |
|---|---|---|---|---|---|---|
| openjet-unsloth-udq4xl-mtp4-retry-20261022 | 32 | 84 | 36.51 | 147.32 | 483,911.54 | 486,075.14 |
Synthetic capacity, separate workload
Streaming 1,024-token input and 1,024-token output, 120 seconds per point, seed 42, fresh prefix. maxseq16 improved c16/c32 short-workload output throughput by about 58%, while its AgentX throughput declined 6.04% and p95 completion latency increased. Select using the workload actually served. Synthetic decode and AgentX decode use different definitions.
| Configuration | Concurrency | Runs | Output tok/s | Decode tok/s/user |
|---|---|---|---|---|
| baseline-mtp3 | 1 | 1 | 146.09 | 154.40 |
| baseline-mtp3 | 8 | 1 | 1,008.53 | 135.95 |
| baseline-mtp3 | 16 | 1 | 1,053.11 | 138.50 |
| baseline-mtp3 | 32 | 1 | 1,016.51 | 135.05 |
| baseline-mtp3-recheck | 1 | 1 | 144.64 | 155.02 |
| baseline-mtp3-recheck | 8 | 1 | 1,001.16 | 134.65 |
| baseline-mtp3-recheck | 16 | 1 | 997.67 | 133.26 |
| baseline-mtp3-recheck | 32 | 1 | 1,001.91 | 133.40 |
| eagle4-depth6-full | 1 | 1 | 220.08 | 244.38 |
| eagle4-depth6-full | 8 | 1 | 1,072.00 | 154.14 |
| eagle4-depth6-full | 16 | 1 | 1,124.01 | 154.73 |
| eagle4-depth6-full | 32 | 1 | 1,104.06 | 153.82 |
| eagle4-depth6-prefix-full | 1 | 1 | 199.55 | 226.91 |
| eagle4-depth6-prefix-full | 8 | 1 | 1,042.09 | 150.94 |
| eagle4-depth6-reasoning-full | 1 | 1 | 220.80 | 244.86 |
| eagle4-depth6-structured-full | 1 | 1 | 218.70 | 248.05 |
| eagle4-isolated-frontend-full | 1 | 1 | 147.52 | 153.94 |
| eagle4-isolated-frontend-full | 8 | 1 | 726.15 | 97.10 |
| eagle4-isolated-frontend-full | 16 | 1 | 733.70 | 95.70 |
| eagle4-isolated-frontend-full | 32 | 1 | 733.49 | 95.18 |
| maxseq16 | 1 | 1 | 145.79 | 154.28 |
| maxseq16 | 8 | 1 | 1,012.53 | 136.24 |
| maxseq16 | 16 | 2 | 1,576.33 | 107.64 |
| maxseq16 | 32 | 2 | 1,587.63 | 106.58 |
| sglang-target-api-full | 1 | 1 | 67.11 | 68.57 |
| sglang-target-api-full | 8 | 1 | 449.56 | 58.36 |
| sglang-target-api-full | 16 | 1 | 463.32 | 58.76 |
| sglang-target-api-full | 32 | 1 | 457.02 | 58.71 |
Throughput versus interactivity, every measured variation
Each figure preserves workload and transport boundaries. Sparse results show measured points; they do not establish a complete Pareto frontier. Repeated synthetic points use observed ranges, not confidence intervals.
AgentX NVFP4, candidate loopback
OpenJet / Unsloth GGUF, separate model family
AgentX NVFP4, candidate origin
AgentX NVFP4, direct origin
Synthetic 1K input / 1K output
Exact applied settings and coverage
Failed and unmeasured attempts are retained in the coverage records below. Pilot and API status are separate from measured throughput; a failed follow-up does not erase an earlier valid result. Framework patches are identified. Runtime digests, source hashes and serving flags are retained; credentials and raw request content are excluded.
baseline-mtp3-recheck
{
"config_id": "baseline-mtp3-recheck",
"config_sha": "cba63bfdce522de45d5fef39151b784613a143106a869918dae8c11699232715",
"runtime_sha": "7bda4493758cc3bd5d441f866a194751881d7b29dff91467187ea828d909626c",
"setup_sha": null,
"serve_sha": "6064a686c080a4e8690be66c6185d191497c807f53ef67e2cae61791b626685a",
"model": "nvidia/Qwen3.8-27B-NVFP4",
"model_revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"image": "nvcr.io/nvidia/ai-dynamo/vllm-runtime-nightly:20261018-3763cee@sha256:197284c3fd3037d4301ffa8282ed330446a5a371c0223fe1da448529930f064b",
"framework_source_patch": false,
"pilot_status": "no full pilot",
"api_passed": true,
"valid_capacity_points": 4,
"serving": {
"model": "nvidia/Qwen3.8-27B-NVFP4",
"revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"served_model": "Qwen/Qwen3.8-27B-NVFP4",
"frontend_args": [
"--http-host",
"127.0.0.1",
"--http-port",
"8000",
"--discovery-backend",
"file",
"--request-plane",
"tcp",
"--event-plane",
"zmq",
"--dyn-chat-processor",
"vllm",
"--reasoning-parser",
"qwen3",
"--enable-auto-tool-choice",
"--tool-call-parser",
"qwen3_coder"
],
"worker_args": [
"--discovery-backend",
"file",
"--request-plane",
"tcp",
"--event-plane",
"zmq",
"--tensor-parallel-size",
"1",
"--max-model-len",
"131072",
"--max-num-seqs",
"8",
"--gpu-memory-utilization",
"0.85",
"--language-model-only",
"--dtype",
"bfloat16",
"--kv-cache-dtype",
"fp8",
"--max-num-batched-tokens",
"8192",
"--mamba-cache-mode",
"align",
"--speculative-config",
"{\"method\":\"mtp\",\"num_speculative_tokens\":3}",
"--reasoning-parser",
"qwen3"
],
"structured_output_default_thinking": false
}
}eagle4-depth6-full
{
"config_id": "eagle4-depth6-full",
"config_sha": "34c015b081a4ca2d89435952ae5c8908c05b66d3e0e75b26ec7c9e271403278d",
"runtime_sha": "905e792039e2b2e3a13796532af7184dc85f14da009e9dc496f4823db563b2af",
"setup_sha": "4ea300ef9bbc69d13005889a5321d015f070d3662c275e9288c6eee6dafc4e26",
"serve_sha": "ca8f40eeeb662a46a1a7e7438d5f5a4af278e5babf065e11263a8c9f6174a51e",
"model": "nvidia/Qwen3.8-27B-NVFP4",
"model_revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"image": "nvcr.io/nvidia/ai-dynamo/vllm-runtime-nightly:20261018-3763cee@sha256:197284c3fd3037d4301ffa8282ed330446a5a371c0223fe1da448529930f064b",
"framework_source_patch": false,
"pilot_status": "complete",
"api_passed": false,
"valid_capacity_points": 4,
"serving": {
"model": "nvidia/Qwen3.8-27B-NVFP4",
"revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"served_model": "Qwen/Qwen3.8-27B-NVFP4",
"frontend_args": [
"--http-host",
"127.0.0.1",
"--http-port",
"8000",
"--discovery-backend",
"file",
"--request-plane",
"tcp",
"--event-plane",
"zmq",
"--dyn-chat-processor",
"vllm",
"--reasoning-parser",
"qwen3",
"--enable-auto-tool-choice",
"--tool-call-parser",
"qwen3_coder"
],
"worker_args": [
"--discovery-backend",
"file",
"--request-plane",
"tcp",
"--event-plane",
"zmq",
"--tensor-parallel-size",
"1",
"--max-model-len",
"131072",
"--max-num-seqs",
"8",
"--gpu-memory-utilization",
"0.85",
"--language-model-only",
"--dtype",
"bfloat16",
"--kv-cache-dtype",
"auto",
"--max-num-batched-tokens",
"8192",
"--mamba-cache-mode",
"align",
"--speculative-config",
"{\"method\": \"eagle_harness\", \"model\": \"/workspace/eagle-draft\", \"method_config\": {\"mode\": \"static\", \"node_top_k\": 4, \"tree_depth\": 6, \"max_verify_width\": 8, \"depth_stop_gain\": 0.0}}",
"--attention-backend",
"CUSTOM",
"--no-enable-prefix-caching"
]
}
}eagle4-depth6-prefix-full
{
"config_id": "eagle4-depth6-prefix-full",
"config_sha": "532f6a5f9463eb78a08c2aca77c564ad5837ba77960bea4333a3e57940aad018",
"runtime_sha": "905e792039e2b2e3a13796532af7184dc85f14da009e9dc496f4823db563b2af",
"setup_sha": "4ea300ef9bbc69d13005889a5321d015f070d3662c275e9288c6eee6dafc4e26",
"serve_sha": "6064a686c080a4e8690be66c6185d191497c807f53ef67e2cae61791b626685a",
"model": "nvidia/Qwen3.8-27B-NVFP4",
"model_revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"image": "nvcr.io/nvidia/ai-dynamo/vllm-runtime-nightly:20261018-3763cee@sha256:197284c3fd3037d4301ffa8282ed330446a5a371c0223fe1da448529930f064b",
"framework_source_patch": false,
"pilot_status": "complete",
"api_passed": true,
"valid_capacity_points": 2,
"serving": {
"model": "nvidia/Qwen3.8-27B-NVFP4",
"revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"served_model": "Qwen/Qwen3.8-27B-NVFP4",
"frontend_args": [
"--http-host",
"127.0.0.1",
"--http-port",
"8000",
"--discovery-backend",
"file",
"--request-plane",
"tcp",
"--event-plane",
"zmq",
"--dyn-chat-processor",
"vllm",
"--reasoning-parser",
"qwen3",
"--enable-auto-tool-choice",
"--tool-call-parser",
"qwen3_coder"
],
"worker_args": [
"--discovery-backend",
"file",
"--request-plane",
"tcp",
"--event-plane",
"zmq",
"--tensor-parallel-size",
"1",
"--max-model-len",
"131072",
"--max-num-seqs",
"8",
"--gpu-memory-utilization",
"0.85",
"--language-model-only",
"--dtype",
"bfloat16",
"--kv-cache-dtype",
"auto",
"--max-num-batched-tokens",
"8192",
"--mamba-cache-mode",
"align",
"--speculative-config",
"{\"method\": \"eagle_harness\", \"model\": \"/workspace/eagle-draft\", \"method_config\": {\"mode\": \"static\", \"node_top_k\": 4, \"tree_depth\": 6, \"max_verify_width\": 8, \"depth_stop_gain\": 0.0}}",
"--attention-backend",
"CUSTOM",
"--enable-prefix-caching",
"--reasoning-parser",
"qwen3"
],
"structured_output_default_thinking": false
}
}eagle4-depth6-reasoning-full
{
"config_id": "eagle4-depth6-reasoning-full",
"config_sha": "53c474de851b9511204e922d27355ba101614ea6192d0362b4793568dbe398f8",
"runtime_sha": "905e792039e2b2e3a13796532af7184dc85f14da009e9dc496f4823db563b2af",
"setup_sha": "4ea300ef9bbc69d13005889a5321d015f070d3662c275e9288c6eee6dafc4e26",
"serve_sha": "ca8f40eeeb662a46a1a7e7438d5f5a4af278e5babf065e11263a8c9f6174a51e",
"model": "nvidia/Qwen3.8-27B-NVFP4",
"model_revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"image": "nvcr.io/nvidia/ai-dynamo/vllm-runtime-nightly:20261018-3763cee@sha256:197284c3fd3037d4301ffa8282ed330446a5a371c0223fe1da448529930f064b",
"framework_source_patch": false,
"pilot_status": "complete",
"api_passed": false,
"valid_capacity_points": 1,
"serving": {
"model": "nvidia/Qwen3.8-27B-NVFP4",
"revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"served_model": "Qwen/Qwen3.8-27B-NVFP4",
"frontend_args": [
"--http-host",
"127.0.0.1",
"--http-port",
"8000",
"--discovery-backend",
"file",
"--request-plane",
"tcp",
"--event-plane",
"zmq",
"--dyn-chat-processor",
"vllm",
"--reasoning-parser",
"qwen3",
"--enable-auto-tool-choice",
"--tool-call-parser",
"qwen3_coder"
],
"worker_args": [
"--discovery-backend",
"file",
"--request-plane",
"tcp",
"--event-plane",
"zmq",
"--tensor-parallel-size",
"1",
"--max-model-len",
"131072",
"--max-num-seqs",
"8",
"--gpu-memory-utilization",
"0.85",
"--language-model-only",
"--dtype",
"bfloat16",
"--kv-cache-dtype",
"auto",
"--max-num-batched-tokens",
"8192",
"--mamba-cache-mode",
"align",
"--speculative-config",
"{\"method\": \"eagle_harness\", \"model\": \"/workspace/eagle-draft\", \"method_config\": {\"mode\": \"static\", \"node_top_k\": 4, \"tree_depth\": 6, \"max_verify_width\": 8, \"depth_stop_gain\": 0.0}}",
"--attention-backend",
"CUSTOM",
"--no-enable-prefix-caching",
"--reasoning-parser",
"qwen3"
]
}
}eagle4-depth6-structured-full
{
"config_id": "eagle4-depth6-structured-full",
"config_sha": "5ebd8eb1ed701ee9abff71b1e5a5d96a95e54d9d475aa0d3e4f3c4d64a1cc4b3",
"runtime_sha": "905e792039e2b2e3a13796532af7184dc85f14da009e9dc496f4823db563b2af",
"setup_sha": "4ea300ef9bbc69d13005889a5321d015f070d3662c275e9288c6eee6dafc4e26",
"serve_sha": "6064a686c080a4e8690be66c6185d191497c807f53ef67e2cae61791b626685a",
"model": "nvidia/Qwen3.8-27B-NVFP4",
"model_revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"image": "nvcr.io/nvidia/ai-dynamo/vllm-runtime-nightly:20261018-3763cee@sha256:197284c3fd3037d4301ffa8282ed330446a5a371c0223fe1da448529930f064b",
"framework_source_patch": false,
"pilot_status": "complete",
"api_passed": true,
"valid_capacity_points": 1,
"serving": {
"model": "nvidia/Qwen3.8-27B-NVFP4",
"revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"served_model": "Qwen/Qwen3.8-27B-NVFP4",
"frontend_args": [
"--http-host",
"127.0.0.1",
"--http-port",
"8000",
"--discovery-backend",
"file",
"--request-plane",
"tcp",
"--event-plane",
"zmq",
"--dyn-chat-processor",
"vllm",
"--reasoning-parser",
"qwen3",
"--enable-auto-tool-choice",
"--tool-call-parser",
"qwen3_coder"
],
"worker_args": [
"--discovery-backend",
"file",
"--request-plane",
"tcp",
"--event-plane",
"zmq",
"--tensor-parallel-size",
"1",
"--max-model-len",
"131072",
"--max-num-seqs",
"8",
"--gpu-memory-utilization",
"0.85",
"--language-model-only",
"--dtype",
"bfloat16",
"--kv-cache-dtype",
"auto",
"--max-num-batched-tokens",
"8192",
"--mamba-cache-mode",
"align",
"--speculative-config",
"{\"method\": \"eagle_harness\", \"model\": \"/workspace/eagle-draft\", \"method_config\": {\"mode\": \"static\", \"node_top_k\": 4, \"tree_depth\": 6, \"max_verify_width\": 8, \"depth_stop_gain\": 0.0}}",
"--attention-backend",
"CUSTOM",
"--no-enable-prefix-caching",
"--reasoning-parser",
"qwen3"
],
"structured_output_default_thinking": false
}
}eagle4-integration-4k
{
"config_id": "eagle4-integration-4k",
"config_sha": "e2446877a797235004889adbc4e61a207eb3c847c42cf2eb87286b33b62b1c42",
"runtime_sha": "905e792039e2b2e3a13796532af7184dc85f14da009e9dc496f4823db563b2af",
"setup_sha": "e17de8c931c26effd4ac8f3d737e6b7d92c483a7f5d04d2e6aed3d273af0d5ef",
"model": "nvidia/Qwen3.8-27B-NVFP4",
"model_revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"image": "nvcr.io/nvidia/ai-dynamo/vllm-runtime-nightly:20261018-3763cee@sha256:197284c3fd3037d4301ffa8282ed330446a5a371c0223fe1da448529930f064b",
"framework_source_patch": false,
"pilot_status": "startup_failed",
"api_passed": null,
"valid_capacity_points": 0,
"serving": {
"model": "nvidia/Qwen3.8-27B-NVFP4",
"revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"served_model": "Qwen/Qwen3.8-27B-NVFP4",
"frontend_args": [
"--http-host",
"127.0.0.1",
"--http-port",
"8000",
"--discovery-backend",
"file",
"--request-plane",
"tcp",
"--event-plane",
"zmq",
"--dyn-chat-processor",
"vllm",
"--reasoning-parser",
"qwen3",
"--enable-auto-tool-choice",
"--tool-call-parser",
"qwen3_coder"
],
"worker_args": [
"--discovery-backend",
"file",
"--request-plane",
"tcp",
"--event-plane",
"zmq",
"--tensor-parallel-size",
"1",
"--max-model-len",
"4096",
"--max-num-seqs",
"1",
"--gpu-memory-utilization",
"0.85",
"--language-model-only",
"--dtype",
"bfloat16",
"--kv-cache-dtype",
"auto",
"--max-num-batched-tokens",
"8192",
"--mamba-cache-mode",
"align",
"--speculative-config",
"{\"method\": \"eagle_harness\", \"model\": \"/workspace/eagle-draft\", \"method_config\": {\"mode\": \"static\", \"node_top_k\": 4, \"tree_depth\": 2, \"max_verify_width\": 8, \"depth_stop_gain\": 0.0}}",
"--attention-backend",
"CUSTOM",
"--no-enable-prefix-caching"
]
}
}eagle4-isolated-frontend-4k
{
"config_id": "eagle4-isolated-frontend-4k",
"config_sha": "e2446877a797235004889adbc4e61a207eb3c847c42cf2eb87286b33b62b1c42",
"runtime_sha": "905e792039e2b2e3a13796532af7184dc85f14da009e9dc496f4823db563b2af",
"setup_sha": "e17de8c931c26effd4ac8f3d737e6b7d92c483a7f5d04d2e6aed3d273af0d5ef",
"serve_sha": "ca8f40eeeb662a46a1a7e7438d5f5a4af278e5babf065e11263a8c9f6174a51e",
"model": "nvidia/Qwen3.8-27B-NVFP4",
"model_revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"image": "nvcr.io/nvidia/ai-dynamo/vllm-runtime-nightly:20261018-3763cee@sha256:197284c3fd3037d4301ffa8282ed330446a5a371c0223fe1da448529930f064b",
"framework_source_patch": false,
"pilot_status": "no full pilot",
"api_passed": null,
"valid_capacity_points": 0,
"serving": {
"model": "nvidia/Qwen3.8-27B-NVFP4",
"revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"served_model": "Qwen/Qwen3.8-27B-NVFP4",
"frontend_args": [
"--http-host",
"127.0.0.1",
"--http-port",
"8000",
"--discovery-backend",
"file",
"--request-plane",
"tcp",
"--event-plane",
"zmq",
"--dyn-chat-processor",
"vllm",
"--reasoning-parser",
"qwen3",
"--enable-auto-tool-choice",
"--tool-call-parser",
"qwen3_coder"
],
"worker_args": [
"--discovery-backend",
"file",
"--request-plane",
"tcp",
"--event-plane",
"zmq",
"--tensor-parallel-size",
"1",
"--max-model-len",
"4096",
"--max-num-seqs",
"1",
"--gpu-memory-utilization",
"0.85",
"--language-model-only",
"--dtype",
"bfloat16",
"--kv-cache-dtype",
"auto",
"--max-num-batched-tokens",
"8192",
"--mamba-cache-mode",
"align",
"--speculative-config",
"{\"method\": \"eagle_harness\", \"model\": \"/workspace/eagle-draft\", \"method_config\": {\"mode\": \"static\", \"node_top_k\": 4, \"tree_depth\": 2, \"max_verify_width\": 8, \"depth_stop_gain\": 0.0}}",
"--attention-backend",
"CUSTOM",
"--no-enable-prefix-caching"
]
}
}eagle4-isolated-frontend-full
{
"config_id": "eagle4-isolated-frontend-full",
"config_sha": "0c1d02d1d167852046a5a1e0ca7c9ba38b829438b2378b1854252f85639a4004",
"runtime_sha": "905e792039e2b2e3a13796532af7184dc85f14da009e9dc496f4823db563b2af",
"setup_sha": "e17de8c931c26effd4ac8f3d737e6b7d92c483a7f5d04d2e6aed3d273af0d5ef",
"serve_sha": "ca8f40eeeb662a46a1a7e7438d5f5a4af278e5babf065e11263a8c9f6174a51e",
"model": "nvidia/Qwen3.8-27B-NVFP4",
"model_revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"image": "nvcr.io/nvidia/ai-dynamo/vllm-runtime-nightly:20261018-3763cee@sha256:197284c3fd3037d4301ffa8282ed330446a5a371c0223fe1da448529930f064b",
"framework_source_patch": false,
"pilot_status": "complete",
"api_passed": null,
"valid_capacity_points": 4,
"serving": {
"model": "nvidia/Qwen3.8-27B-NVFP4",
"revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"served_model": "Qwen/Qwen3.8-27B-NVFP4",
"frontend_args": [
"--http-host",
"127.0.0.1",
"--http-port",
"8000",
"--discovery-backend",
"file",
"--request-plane",
"tcp",
"--event-plane",
"zmq",
"--dyn-chat-processor",
"vllm",
"--reasoning-parser",
"qwen3",
"--enable-auto-tool-choice",
"--tool-call-parser",
"qwen3_coder"
],
"worker_args": [
"--discovery-backend",
"file",
"--request-plane",
"tcp",
"--event-plane",
"zmq",
"--tensor-parallel-size",
"1",
"--max-model-len",
"131072",
"--max-num-seqs",
"8",
"--gpu-memory-utilization",
"0.85",
"--language-model-only",
"--dtype",
"bfloat16",
"--kv-cache-dtype",
"auto",
"--max-num-batched-tokens",
"8192",
"--mamba-cache-mode",
"align",
"--speculative-config",
"{\"method\": \"eagle_harness\", \"model\": \"/workspace/eagle-draft\", \"method_config\": {\"mode\": \"static\", \"node_top_k\": 4, \"tree_depth\": 2, \"max_verify_width\": 8, \"depth_stop_gain\": 0.0}}",
"--attention-backend",
"CUSTOM",
"--no-enable-prefix-caching"
]
}
}maxseq16
{
"config_id": "maxseq16",
"config_sha": "1aceda12bac158c5c475416bbd0acd08b458cc0e25ac230acb60b4c04224c9ca",
"runtime_sha": "7bda4493758cc3bd5d441f866a194751881d7b29dff91467187ea828d909626c",
"setup_sha": null,
"serve_sha": "6064a686c080a4e8690be66c6185d191497c807f53ef67e2cae61791b626685a",
"model": "nvidia/Qwen3.8-27B-NVFP4",
"model_revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"image": "nvcr.io/nvidia/ai-dynamo/vllm-runtime-nightly:20261018-3763cee@sha256:197284c3fd3037d4301ffa8282ed330446a5a371c0223fe1da448529930f064b",
"framework_source_patch": false,
"pilot_status": "complete",
"api_passed": true,
"valid_capacity_points": 6,
"serving": {
"model": "nvidia/Qwen3.8-27B-NVFP4",
"revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"served_model": "Qwen/Qwen3.8-27B-NVFP4",
"frontend_args": [
"--http-host",
"127.0.0.1",
"--http-port",
"8000",
"--discovery-backend",
"file",
"--request-plane",
"tcp",
"--event-plane",
"zmq",
"--dyn-chat-processor",
"vllm",
"--reasoning-parser",
"qwen3",
"--enable-auto-tool-choice",
"--tool-call-parser",
"qwen3_coder"
],
"worker_args": [
"--discovery-backend",
"file",
"--request-plane",
"tcp",
"--event-plane",
"zmq",
"--tensor-parallel-size",
"1",
"--max-model-len",
"131072",
"--max-num-seqs",
"16",
"--gpu-memory-utilization",
"0.85",
"--language-model-only",
"--dtype",
"bfloat16",
"--kv-cache-dtype",
"fp8",
"--max-num-batched-tokens",
"8192",
"--mamba-cache-mode",
"align",
"--speculative-config",
"{\"method\":\"mtp\",\"num_speculative_tokens\":3}",
"--reasoning-parser",
"qwen3"
],
"structured_output_default_thinking": false
}
}openjet-unsloth-udq4xl-mtp4
{
"config_id": "openjet-unsloth-udq4xl-mtp4",
"config_sha": "85a075eef6c81e884ce296c01cf9ca7c08e4907b3ce632e243e9d6884501b766",
"runtime_sha": "50e9759a278e135352111d7ed56f36378a3d91a415a77581b45bce84f8252a25",
"setup_sha": "29d338fc16042f27d8b44dacbe73c186211821b56fd5a1a8ede6d105f2780d5c",
"serve_sha": "9dab416b44291bde2b3e0f00a5a3d696b13a643dccf95e88fdda793d107f8ba2",
"model": "unsloth/Qwen3.8-27B-GGUF",
"model_revision": "4ca720788d1e01f1bff70c033e0d0028fd02e502",
"image": "lmsysorg/sglang@sha256:5fc09edb6a0bb1d5b6992b543723c66d08f86d7a507e951e87e0d05db4ac766e",
"framework_source_patch": false,
"pilot_status": "no full pilot",
"api_passed": null,
"valid_capacity_points": 0,
"serving": {
"model": "unsloth/Qwen3.8-27B-GGUF",
"revision": "4ca720788d1e01f1bff70c033e0d0028fd02e502",
"served_model": "Qwen3.8-27B-UD-Q4_K_XL.gguf",
"framework": "openjet",
"frontend_args": [],
"worker_args": [
"--host",
"127.0.0.1",
"--port",
"8000",
"--ctx",
"131072",
"--draft",
"4",
"--kv",
"f16"
],
"model_file": "Qwen3.8-27B-UD-Q4_K_XL.gguf",
"model_sha256": "3f227079003add2511437e5b1e94812e363385225bf6a9b47b0054a72bc8b01e"
}
}openjet-unsloth-udq4xl-mtp4-retry-20261022
{
"config_id": "openjet-unsloth-udq4xl-mtp4-retry-20261022",
"config_sha": "85a075eef6c81e884ce296c01cf9ca7c08e4907b3ce632e243e9d6884501b766",
"runtime_sha": "03dddcb90bf56df929758d34116b127141caf193afe83ef001642ba35c442fca",
"setup_sha": "29d338fc16042f27d8b44dacbe73c186211821b56fd5a1a8ede6d105f2780d5c",
"serve_sha": "9dab416b44291bde2b3e0f00a5a3d696b13a643dccf95e88fdda793d107f8ba2",
"model": "unsloth/Qwen3.8-27B-GGUF",
"model_revision": "4ca720788d1e01f1bff70c033e0d0028fd02e502",
"image": "lmsysorg/sglang@sha256:5fc09edb6a0bb1d5b6992b543723c66d08f86d7a507e951e87e0d05db4ac766e",
"framework_source_patch": false,
"pilot_status": "standalone-trial-complete",
"api_passed": null,
"valid_capacity_points": 0,
"serving": {
"model": "unsloth/Qwen3.8-27B-GGUF",
"revision": "4ca720788d1e01f1bff70c033e0d0028fd02e502",
"served_model": "Qwen3.8-27B-UD-Q4_K_XL.gguf",
"framework": "openjet",
"frontend_args": [],
"worker_args": [
"--host",
"127.0.0.1",
"--port",
"8000",
"--ctx",
"131072",
"--draft",
"4",
"--kv",
"f16"
],
"model_file": "Qwen3.8-27B-UD-Q4_K_XL.gguf",
"model_sha256": "3f227079003add2511437e5b1e94812e363385225bf6a9b47b0054a72bc8b01e"
}
}sglang-dflash2-api-4k
{
"config_id": "sglang-dflash2-api-4k",
"config_sha": "9bb068f917fbd396b9d0b741bf9f8f59ecb093ad04ace2ec6f602b4aa8a3c968",
"runtime_sha": "59289d2af0278031437dec2713719f2062476445b8ceb8e16af75fbd5ec6b905",
"setup_sha": "32405077d9194e34795197a2ade3d12b64133e5c0fabe780a5f31ff7b610203a",
"serve_sha": "75f9dad11c535bed7889e57f098a99589616655ed274f2932310b5e77db000da",
"model": "nvidia/Qwen3.8-27B-NVFP4",
"model_revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"image": "lmsysorg/sglang@sha256:5fc09edb6a0bb1d5b6992b543723c66d08f86d7a507e951e87e0d05db4ac766e",
"framework_source_patch": true,
"pilot_status": "complete",
"api_passed": null,
"valid_capacity_points": 0,
"serving": {
"model": "nvidia/Qwen3.8-27B-NVFP4",
"revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"served_model": "Qwen/Qwen3.8-27B-NVFP4",
"framework": "sglang",
"frontend_args": [],
"worker_args": [
"--host",
"127.0.0.1",
"--port",
"8000",
"--tp-size",
"1",
"--context-length",
"4096",
"--max-running-requests",
"1",
"--mem-fraction-static",
"0.75",
"--chunked-prefill-size",
"8192",
"--dtype",
"bfloat16",
"--kv-cache-dtype",
"bf16",
"--speculative-algorithm",
"DFLASH",
"--speculative-draft-model-path",
"/workspace/dflash2-draft",
"--speculative-draft-model-revision",
"50307d4c4cde6860d4eee73e2547cd786fe8e8a4",
"--speculative-num-draft-tokens",
"8",
"--speculative-draft-kv-cache-dtype",
"bf16",
"--enable-torch-compile",
"--reasoning-parser",
"qwen3",
"--tool-call-parser",
"qwen3_coder",
"--enable-metrics",
"--speculative-draft-model-quantization",
"unquant"
],
"structured_output_default_thinking": false
}
}sglang-dflash2-api-full
{
"config_id": "sglang-dflash2-api-full",
"config_sha": "7de6eb9823288f39fc48a486596ba6c32f5c9d61273146072575736e9f9221e0",
"runtime_sha": "59289d2af0278031437dec2713719f2062476445b8ceb8e16af75fbd5ec6b905",
"setup_sha": "32405077d9194e34795197a2ade3d12b64133e5c0fabe780a5f31ff7b610203a",
"serve_sha": "75f9dad11c535bed7889e57f098a99589616655ed274f2932310b5e77db000da",
"model": "nvidia/Qwen3.8-27B-NVFP4",
"model_revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"image": "lmsysorg/sglang@sha256:5fc09edb6a0bb1d5b6992b543723c66d08f86d7a507e951e87e0d05db4ac766e",
"framework_source_patch": true,
"pilot_status": "failed",
"api_passed": true,
"valid_capacity_points": 0,
"serving": {
"model": "nvidia/Qwen3.8-27B-NVFP4",
"revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"served_model": "Qwen/Qwen3.8-27B-NVFP4",
"framework": "sglang",
"frontend_args": [],
"worker_args": [
"--host",
"127.0.0.1",
"--port",
"8000",
"--tp-size",
"1",
"--context-length",
"131072",
"--max-running-requests",
"8",
"--mem-fraction-static",
"0.75",
"--chunked-prefill-size",
"8192",
"--dtype",
"bfloat16",
"--kv-cache-dtype",
"bf16",
"--speculative-algorithm",
"DFLASH",
"--speculative-draft-model-path",
"/workspace/dflash2-draft",
"--speculative-draft-model-revision",
"50307d4c4cde6860d4eee73e2547cd786fe8e8a4",
"--speculative-num-draft-tokens",
"8",
"--speculative-draft-kv-cache-dtype",
"bf16",
"--enable-torch-compile",
"--reasoning-parser",
"qwen3",
"--tool-call-parser",
"qwen3_coder",
"--enable-metrics",
"--speculative-draft-model-quantization",
"unquant"
],
"structured_output_default_thinking": false
}
}sglang-target-api-full
{
"config_id": "sglang-target-api-full",
"config_sha": "5948fc9fce19cfdfeeac37a7bf7b8a984962e66b2dd94c0c49ddffb19077370e",
"runtime_sha": "59289d2af0278031437dec2713719f2062476445b8ceb8e16af75fbd5ec6b905",
"setup_sha": "af13f7fe357e9c4384721375fba9b747e7d16c3941259b4a40ea2a8c419fe89f",
"serve_sha": "75f9dad11c535bed7889e57f098a99589616655ed274f2932310b5e77db000da",
"model": "nvidia/Qwen3.8-27B-NVFP4",
"model_revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"image": "lmsysorg/sglang@sha256:5fc09edb6a0bb1d5b6992b543723c66d08f86d7a507e951e87e0d05db4ac766e",
"framework_source_patch": true,
"pilot_status": "complete",
"api_passed": true,
"valid_capacity_points": 4,
"serving": {
"model": "nvidia/Qwen3.8-27B-NVFP4",
"revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"served_model": "Qwen/Qwen3.8-27B-NVFP4",
"framework": "sglang",
"frontend_args": [],
"worker_args": [
"--host",
"127.0.0.1",
"--port",
"8000",
"--tp-size",
"1",
"--context-length",
"131072",
"--max-running-requests",
"8",
"--mem-fraction-static",
"0.75",
"--chunked-prefill-size",
"8192",
"--dtype",
"bfloat16",
"--kv-cache-dtype",
"bf16",
"--enable-torch-compile",
"--reasoning-parser",
"qwen3",
"--tool-call-parser",
"qwen3_coder",
"--enable-metrics"
],
"structured_output_default_thinking": false
}
}sglang-target-linear-eager-full
{
"config_id": "sglang-target-linear-eager-full",
"config_sha": "5948fc9fce19cfdfeeac37a7bf7b8a984962e66b2dd94c0c49ddffb19077370e",
"runtime_sha": "59289d2af0278031437dec2713719f2062476445b8ceb8e16af75fbd5ec6b905",
"setup_sha": "af13f7fe357e9c4384721375fba9b747e7d16c3941259b4a40ea2a8c419fe89f",
"serve_sha": "09f4449cb314b339fb7762a02c534a14190e3bc2e69011cc8a6a8f85ab2fe983",
"model": "nvidia/Qwen3.8-27B-NVFP4",
"model_revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"image": "lmsysorg/sglang@sha256:5fc09edb6a0bb1d5b6992b543723c66d08f86d7a507e951e87e0d05db4ac766e",
"framework_source_patch": true,
"pilot_status": "failed",
"api_passed": false,
"valid_capacity_points": 0,
"serving": {
"model": "nvidia/Qwen3.8-27B-NVFP4",
"revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"served_model": "Qwen/Qwen3.8-27B-NVFP4",
"framework": "sglang",
"frontend_args": [],
"worker_args": [
"--host",
"127.0.0.1",
"--port",
"8000",
"--tp-size",
"1",
"--context-length",
"131072",
"--max-running-requests",
"8",
"--mem-fraction-static",
"0.75",
"--chunked-prefill-size",
"8192",
"--dtype",
"bfloat16",
"--kv-cache-dtype",
"bf16",
"--enable-torch-compile",
"--reasoning-parser",
"qwen3",
"--tool-call-parser",
"qwen3_coder",
"--enable-metrics"
],
"structured_output_default_thinking": false
}
}sglang-target-native-full
{
"config_id": "sglang-target-native-full",
"config_sha": "37f2327d97c7952c5bec5e64fb7235d7ab100e1097081ba4881182866d0e239b",
"runtime_sha": "59289d2af0278031437dec2713719f2062476445b8ceb8e16af75fbd5ec6b905",
"setup_sha": "50ef8f6e745e13a27c04bd0fa947f7245ba5dde8566a8b294738c89cba772517",
"serve_sha": "09f4449cb314b339fb7762a02c534a14190e3bc2e69011cc8a6a8f85ab2fe983",
"model": "nvidia/Qwen3.8-27B-NVFP4",
"model_revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"image": "lmsysorg/sglang@sha256:5fc09edb6a0bb1d5b6992b543723c66d08f86d7a507e951e87e0d05db4ac766e",
"framework_source_patch": false,
"pilot_status": "failed",
"api_passed": null,
"valid_capacity_points": 0,
"serving": {
"model": "nvidia/Qwen3.8-27B-NVFP4",
"revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"served_model": "Qwen/Qwen3.8-27B-NVFP4",
"framework": "sglang",
"frontend_args": [],
"worker_args": [
"--host",
"127.0.0.1",
"--port",
"8000",
"--tp-size",
"1",
"--context-length",
"131072",
"--max-running-requests",
"8",
"--mem-fraction-static",
"0.85",
"--chunked-prefill-size",
"8192",
"--dtype",
"bfloat16",
"--kv-cache-dtype",
"bf16",
"--enable-torch-compile",
"--reasoning-parser",
"qwen3",
"--tool-call-parser",
"qwen3_coder",
"--enable-metrics"
],
"structured_output_default_thinking": false
}
}sglang-target-only-full
{
"config_id": "sglang-target-only-full",
"config_sha": "3e9881c2456f03acd75d249e6002144198d13fcbd797ed702a6924dfef1ad1db",
"runtime_sha": "59289d2af0278031437dec2713719f2062476445b8ceb8e16af75fbd5ec6b905",
"setup_sha": "50ef8f6e745e13a27c04bd0fa947f7245ba5dde8566a8b294738c89cba772517",
"serve_sha": "09f4449cb314b339fb7762a02c534a14190e3bc2e69011cc8a6a8f85ab2fe983",
"model": "nvidia/Qwen3.8-27B-NVFP4",
"model_revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"image": "lmsysorg/sglang@sha256:5fc09edb6a0bb1d5b6992b543723c66d08f86d7a507e951e87e0d05db4ac766e",
"framework_source_patch": false,
"pilot_status": "failed",
"api_passed": null,
"valid_capacity_points": 0,
"serving": {
"model": "nvidia/Qwen3.8-27B-NVFP4",
"revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"served_model": "Qwen/Qwen3.8-27B-NVFP4",
"framework": "sglang",
"frontend_args": [],
"worker_args": [
"--host",
"127.0.0.1",
"--port",
"8000",
"--tp-size",
"1",
"--context-length",
"131072",
"--max-running-requests",
"8",
"--mem-fraction-static",
"0.85",
"--chunked-prefill-size",
"8192",
"--dtype",
"bfloat16",
"--kv-cache-dtype",
"bf16",
"--enable-torch-compile",
"--language-model-only",
"--reasoning-parser",
"qwen3",
"--tool-call-parser",
"qwen3_coder",
"--enable-metrics"
],
"structured_output_default_thinking": false
}
}sglang-target-room75-full
{
"config_id": "sglang-target-room75-full",
"config_sha": "5948fc9fce19cfdfeeac37a7bf7b8a984962e66b2dd94c0c49ddffb19077370e",
"runtime_sha": "59289d2af0278031437dec2713719f2062476445b8ceb8e16af75fbd5ec6b905",
"setup_sha": "50ef8f6e745e13a27c04bd0fa947f7245ba5dde8566a8b294738c89cba772517",
"serve_sha": "09f4449cb314b339fb7762a02c534a14190e3bc2e69011cc8a6a8f85ab2fe983",
"model": "nvidia/Qwen3.8-27B-NVFP4",
"model_revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"image": "lmsysorg/sglang@sha256:5fc09edb6a0bb1d5b6992b543723c66d08f86d7a507e951e87e0d05db4ac766e",
"framework_source_patch": false,
"pilot_status": "operator-aborted-memory-retries",
"api_passed": null,
"valid_capacity_points": 0,
"serving": {
"model": "nvidia/Qwen3.8-27B-NVFP4",
"revision": "482ca0f3832238542f8f5295dde86b5f22711d80",
"served_model": "Qwen/Qwen3.8-27B-NVFP4",
"framework": "sglang",
"frontend_args": [],
"worker_args": [
"--host",
"127.0.0.1",
"--port",
"8000",
"--tp-size",
"1",
"--context-length",
"131072",
"--max-running-requests",
"8",
"--mem-fraction-static",
"0.75",
"--chunked-prefill-size",
"8192",
"--dtype",
"bfloat16",
"--kv-cache-dtype",
"bf16",
"--enable-torch-compile",
"--reasoning-parser",
"qwen3",
"--tool-call-parser",
"qwen3_coder",
"--enable-metrics"
],
"structured_output_default_thinking": false
}
}Data and reproduction
Aggregate data and exact settings, JSON · File checksums
The complete local campaign, raw exports, original commands, client lock and detailed setup are retained in the inference-platform repository workspace under results/2026-10-21-qwen38-nvfp4-optimization. Raw artifacts are not included in this public aggregate bundle. No new experiment or GPU is needed to publish this report.