Open LLM inference systems lab

Measure the workload.
Make the system decision.

Reproducible experiments in serving performance, disaggregated inference, routing, observability, and production SLOs across heterogeneous GPU infrastructure.

Read the latest studyReview methodologyReproduce on GitHub

Published findings

Reviewed results with operational recommendations and machine-readable evidence.

H200 → MI300X · 2026-08-30
1,971 tok/s

Cross-vendor prefill/decode works—but the network defines the operating point.

A 21-point Qwen3-30B-A3B FP8 matrix completed with 381 requests, zero failures, and independently verified KV transfer bytes.

Recommendation: treat host-staged public TCP as a compatibility baseline, not a production KV fabric; c=32 was the efficient peak for long-output traffic.
Study, matrix, and raw evidence →
DGX Spark P/D · 2026-08-30
+22.5%

Framework choice changes throughput under a matched disaggregated workload.

Dynamo outperformed llm-d at c=32 across 1K/8K, 8K/8K, and 8K/1K shapes in a 36-point, prefix-cache-off comparison.

Recommendation: use Dynamo for maximum throughput in this tested two-node configuration; revalidate against the latency SLO before generalizing.
Comparison and reproducibility details →
Long context · DGX Spark
1.1s → 12.0s

Long-context prefill is a tail-latency capacity problem.

Nemotron-120B TTFT p99 rose from 1.127s to 11.990s as input length increased from 128 to 2,048 tokens at concurrency 8.

Recommendation: isolate long-context traffic, define a separate TTFT SLO, and evaluate chunked prefill and admission control.
Explore the workload sweep →

Question → telemetry → controlled run → recommendation

Each published study names the decision, fixes the workload and environment, reports tail latency and throughput, explains the mechanism, and states where the conclusion does not generalize.

Published work and notebook are separate

Reviewed findings live here. Partial runs, capacity interruptions, failed configurations, and active hypotheses remain visible in the lab notebook.

Live research endpoints

OpenAI-compatible models used to exercise the serving platform. Availability is live; displayed prices are research estimates.

🔍
Loading models...
-
Live models
—
Research estimate / 1M input

Prices are demo/research estimates derived from benchmark data, not a commercial offer.

Selected experiments

Published studies, controls, and partial work are labeled by evidence maturity.

View all runs →
2026-08-30 · NVIDIA H200 → AMD MI300X
Qwen3-30B-A3B FP8 heterogeneous P/D
Dynamo + vLLM on both vendors, Mooncake TCP KV transfer, 21/21 exact-token points, prefix caching disabled, and zero request errors.
Open cross-vendor study →
2026-08-30 · NVIDIA DGX Spark
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4: Dynamo vs llm-d P/D
Matched 36-point two-node comparison with prefill on spark-01, decode on spark-02, prefix caching disabled for both frameworks, and zero request errors.
Open P/D comparison →
Partial · AMD MI300X
Qwen3.8-27B P/D recovery set
Partial disaggregated run: Dynamo 18/18 points; llm-d 5/18 recovered before MI300X capacity interruption. Measured 1K/8K throughput is within 1% through c=16.
Open partial P/D results →
Single-replica control · DGX Spark
Nemotron 3.5 aggregated serving controls
One model replica per arm across c=1–32. Use for per-replica capacity and wrapper overhead, not multi-worker routing or scaling.
Inspect control curve →
2026-04-17 · NVIDIA DGX Spark
Qwen3.5-27B framework sweep
vLLM and SGLang across BF16, FP8, and GPTQ-Int4, selecting vLLM GPTQ-Int4 for the next optimization phase.
Read Phase A report →
01 · Understand

Built for production-shaped inference

Review the model, serving footprint, and measured cost profile before sending a request.

Model Specifications
Demo cost estimate · per token
Input tokens
per 1M tokens
$0.00
Cached input tokens
per 1M tokens
$0.0000
Output tokens
per 1M tokens
$0.00

Research estimate derived from CI benchmarks; not a commercial offer.

Quick links
📈Grafana Monitoring→ 🔗GitHub Repository→
API base URL

Compatible with openai, litellm, langchain.

All Available Models
02 · Serve

Send a request to the live endpoint

Add your API key, tune only what you need, and see streaming output in context.

Model
Primary
System Prompt
Parameters
Temperature0.70
Max Tokens2048
Top P1.00
Options
Streaming
Mode: Select a model to compare
💬
Start a conversation
Send a message to chat with the selected model.
Quantization benefits
LLM haiku
FP8 vs NVFP4
About DGX Spark
OpenAI-compatible API
03 · Integrate

Move from test to application

Copy an OpenAI-compatible example and connect the same endpoint from your stack.

Endpoint
POST

OpenAI-compatible chat completions. Streaming via stream: true.

Model identifier
Code Examples
cURL
Python
Node.js
LiteLLM

          

          

          

        

      

      

        Advanced request parameters
        

          

            

              Parameter
              Type
              Description
            

            
              
modelstringFull model identifier

              
messagesarrayArray of {role, content} objects

              
max_tokensintegerMax tokens to generate (default 512)

              
temperaturefloatSampling temperature 0–2 (default 0.7)

              
top_pfloatNucleus sampling (default 1.0)

              
streambooleanStream via SSE (default false)