PagedAttention, speculative decoding, MoE and Blackwell

Modern AI deployment relies on memory-virtualized serving engines, sparse expert routing, and sub-byte hardware precision to serve massive models at high speed and low cost. These three technologies represent the forefront of production LLM infrastructure.

1. LLM Inference Serving Optimizations: PagedAttention & Speculative Decoding

During text generation, an LLM caches the Key (K) and Value (V) tensors of every previous token in VRAM so it does not have to recompute them at every step. Managing this KV Cache and overcoming the sequential bottleneck of token-by-token generation are the two biggest challenges in inference serving.

• PagedAttention (vLLM) – Virtual Memory for the KV Cache.

• The Fragmentation Problem. Historically, inference engines pre-allocated a contiguous chunk of VRAM for each request based on the model's maximum possible sequence length (e.g., 32,768 tokens). If a user only generated 200 tokens, over 99% of that reserved VRAM sat empty, wasting 60% to 80% of GPU memory through internal and external fragmentation.
• OS-Style Paging. Inspired by operating system virtual memory, PagedAttention divides each sequence’s KV cache into small, fixed-size blocks (e.g., 16 tokens per block) that can sit non-contiguously anywhere in physical VRAM. A lightweight block table maps logical token blocks to physical GPU memory addresses on the fly inside the attention kernel.
• Memory Sharing (Copy-on-Write). Because blocks are decoupled, multiple requests sharing the same system prompt or generating parallel completions (such as beam search) point to the exact same physical KV blocks in VRAM, slashing memory usage and enabling much larger batch sizes.

• Speculative Decoding – Breaking the Sequential Bottleneck.

• Why Autoregressive Generation Is Slow. Generating 100 tokens requires 100 sequential forward passes. Because each pass only processes one new token, the GPU spends most of its time reading model weights from VRAM while its Tensor Cores remain nearly idle.
• Draft-and-Verify Pipeline. Speculative decoding pairs the massive Target Model (e.g., 70B parameters) with a tiny, ultra-fast Draft Model (e.g., 1B–3B parameters, or lightweight prediction heads like Medusa / EAGLE).
• Parallel Verification. The tiny draft model quickly guesses the next K tokens (e.g., 5 tokens) sequentially. The massive target model then processes all 5 guessed tokens simultaneously in a single parallel forward pass, checking their probabilities. Because verifying 5 tokens in parallel takes almost the exact same time as generating 1 token, the system accepts all matching tokens at once—delivering 2× to 3× faster generation with mathematically identical output quality.

2. Mixture-of-Experts (MoE) & Expert Parallelism

Dense models activate 100% of their parameters for every single token, making massive models prohibitively expensive to run. Mixture-of-Experts (MoE) architectures decouple total model knowledge from per-token compute cost.

• How MoE Layers Work (Mixtral & DeepSeek).

• In an MoE Transformer block, the single dense Feed-Forward Network (FFN) is replaced by a pool of smaller independent FFNs called Experts (e.g., 8 experts in Mixtral 8x7B, or 256 fine-grained experts in DeepSeek-V3).
• A lightweight learned Router (Gating Network) inspects each incoming token and routes it to only the top-K most relevant experts (typically 2 to 8 experts). A model might hold 600B+ total parameters in memory, yet activate only 37B parameters per token, achieving massive model intelligence at a fraction of the compute cost.
• Shared Experts & Auxiliary-Loss-Free Balancing. Modern MoE designs like DeepSeek dedicate a few Shared Experts that process every token to capture common foundational knowledge, while routed experts specialize in niche domains. Dynamic bias adjustments prevent "routing collapse" (where the router overloads a few popular experts and ignores the rest).

• Expert Parallelism (EP) Across GPUs.

• Because an MoE model has hundreds of experts, they cannot fit on one GPU. Expert Parallelism distributes different experts across different GPUs (e.g., Experts 1–4 on GPU 0, Experts 5–8 on GPU 1).
• All-to-All Communication. When GPU 0 processes a batch of tokens, its router may assign some tokens to Experts living on GPU 3. Every MoE layer therefore executes an All-to-All dispatch, sending token hidden states across high-speed NVLink/InfiniBand interconnects to whichever GPU holds the assigned expert, computing the expert FFN, and performing a second All-to-All combine step to return the results.

3. NVIDIA Blackwell (B200) & Second-Gen Transformer Engine (FP4)

Introduced to scale trillion-parameter reasoning and MoE models, the NVIDIA Blackwell (B200) architecture redesigned both multi-die packaging and sub-byte numerical precision.

• Dual-Die Unified Architecture.

• A single B200 GPU packs 208 billion transistors across two reticle-limited dies fused together by a 10 TB/s NV-HBI (High-Bandwidth Interface). To CUDA kernels and software frameworks, both dies operate as one single unified GPU with zero cache-coherency penalties.
• HBM3e Memory Bandwidth. Equipped with 192 GB of HBM3e VRAM delivering 8 TB/s of memory bandwidth (more than double the H100's 3.35 TB/s), Blackwell directly targets the memory-bandwidth bottlenecks of KV caches and MoE weight loading.

• Second-Generation Transformer Engine & NVFP4.

• While Hopper introduced 8-bit floating-point (FP8), Blackwell’s second-generation Transformer Engine pushes Tensor Core compute down to 4-bit floating-point (FP4), doubling raw compute throughput again (up to 20 petaFLOPS of dense FP4 compute per GPU) and halving memory footprint compared to FP8.
• Solving FP4 Dynamic Range with Micro-Tensor Scaling (NVFP4). Standard 4-bit numbers only have 16 possible values, which normally destroys accuracy when extreme outlier numbers appear in neural activations. Blackwell solves this in hardware using two-level micro-scaling: instead of applying one scaling factor to a large matrix, Blackwell groups weights and activations into tiny blocks of 16 elements, assigning each tiny block its own high-precision FP8 scaling factor alongside a global FP32 tensor scale.
• Hardware-Accelerated Quantization. The Transformer Engine dynamically inspects tensor ranges layer-by-layer and converts values between FP16/BF16, FP8, and NVFP4 directly inside the Tensor Core pipeline without extra memory round-trips.