CUDA Graphs, reasoning models and disaggregated serving

Modern AI infrastructure pushes performance to the physical limits of silicon and clusters through CPU-overhead elimination, inference-time reasoning loops, and phase-specialized GPU cluster routing. These three techniques power today’s fastest, smartest production AI systems.

1. CUDA Graphs and Kernel Fusion: Eliminating CPU & Memory Overhead

As modern GPUs (like H100 and B200) have grown exponentially faster, executing a single lightweight GPU kernel can take as little as 2 to 5 microseconds. However, the CPU often takes 10 to 20 microseconds just to prepare, validate, and launch that kernel over the PCIe driver—leaving the multi-million-dollar GPU sitting completely idle waiting for CPU instructions.

• Kernel Fusion (Combining Operations to Save Memory & Launches).

• The Unfused Bottleneck. A standard Transformer layer executes dozens of tiny operations sequentially (e.g., LayerNorm → Linear → Add Residual → GeLU). Without fusion, the GPU launches separate kernels for each step, writing intermediate tensors out to slow HBM (VRAM) and reading them right back in for the next step.
• Fused Execution. Compilers like torch.compile (TorchInductor) and TensorRT-LLM fuse multiple element-wise and reduction operations into a single custom CUDA/Triton kernel. Values are loaded from HBM once, transformed inside ultra-fast on-chip registers/SRAM, and written back to HBM only at the very end—cutting both memory bandwidth traffic and kernel launch counts dramatically.

• CUDA Graphs (Replaying Entire Execution Pipelines in One Shot).

• Record and Replay. Even after kernel fusion, an LLM forward pass still requires hundreds of kernel launches per generated token. CUDA Graphs solve this by capturing an entire sequence of GPU operations—kernels, memory copies, and dependencies—during a warmup run and compiling them into a static execution graph stored directly in GPU memory.
• Zero CPU Intervention. During real-time inference, instead of the CPU launching 300 separate kernels one by one, the CPU issues one single API call (cudaGraphLaunch) that triggers the entire forward pass on the GPU autonomously.
• Handling Dynamic Shapes. Because CUDA Graphs require static memory addresses and tensor shapes, inference engines pad incoming batches to fixed bucket sizes (e.g., batch sizes 1, 2, 4, 8, 16) and update input pointers in-place before replaying the pre-captured graph.

2. Test-Time Compute & Reasoning Models (o1/o3 & DeepSeek-R1)

Historically, AI labs scaled intelligence strictly by throwing more compute at pre-training (bigger models trained on more internet text). Reasoning models like OpenAI’s o1/o3 and DeepSeek-R1 unlocked a second scaling axis: Test-Time Compute (Inference-Time Scaling), allowing models to "think" longer before answering.

• Reinforcement Learning for Chain-of-Thought (CoT).

• Instead of relying purely on human-written examples of step-by-step reasoning, these models are trained via Large-Scale Reinforcement Learning (RL) (such as GRPO – Group Relative Policy Optimization in DeepSeek-R1) on verifiable domains like math, coding, and formal logic.
• The model generates multiple candidate reasoning paths, receives objective rewards only when its final answer passes unit tests or mathematical verification, and autonomously discovers emergent cognitive behaviors—including self-verification, backtracking when a dead end is reached, and breaking complex problems into sub-goals.

• How Inference Compute Scales.

• When given a hard prompt, the model generates thousands of internal "thinking tokens" inside a hidden scratchpad before outputting a final response. Accuracy improves logarithmically as the model spends more inference tokens exploring hypotheses.
• Compute Trade-Off Shift. In traditional LLMs, inference is cheap and fast (generating ~50 output tokens). In reasoning models, a single query may generate 10,000 to 50,000 thinking tokens, turning inference into a massive, sustained batch-generation workload dominated by KV-cache growth and long-context attention efficiency.

3. Disaggregated Prefill and Decode

Serving an LLM consists of two fundamentally opposite computational phases. Running both phases on the exact same GPU creates severe scheduling conflicts and hardware underutilization.

• The Co-Location Interference Problem.

• In traditional unified serving, if one user submits a huge 32,000-token document for Prefill while 50 other users are in the middle of streaming Decode tokens, the massive prefill matrix math hogs the GPU's Tensor Cores—causing noticeable freezes and latency spikes ("stuttering") for all 50 streaming users.
• Furthermore, Prefill needs high compute FLOPs, whereas Decode needs massive HBM memory bandwidth and capacity to hold large batch KV caches.

• Splitting the Cluster (Disaggregated Architecture).

• Modern hyperscale inference systems (such as Mooncake, DistServe, and NVIDIA Dynamo) split the GPU cluster into two dedicated pools:
• 1. Prefill Pool: Optimized for raw compute throughput (high Tensor Core utilization, smaller tensor parallelism) to crunch long prompts rapidly and hit aggressive Time-To-First-Token (TTFT) targets.
• 2. Decode Pool: Optimized for high memory bandwidth and large KV-cache capacity (wider tensor/expert parallelism, large batch sizes) to stream tokens smoothly at peak tokens-per-second.
• High-Speed KV Cache Transfer. Once a Prefill GPU finishes processing a prompt and builds its initial KV Cache, it streams that KV Cache directly across ultra-fast NVLink or InfiniBand RDMA interconnects into the VRAM of a Decode GPU, which immediately takes over token generation without any interference.