Choosing AI optimization strategies for your hardware
Illustrative satellite view of the Caucasus between the Black and Caspian seas, not GPU hardware.
Choosing the right optimization strategy starts with matching your hardware architecture to your primary performance bottleneck. Every platform enforces strict physical limits on compute throughput, memory bandwidth, and capacity.
1. Target Hardware Architectural Trade-Offs
Different hardware platforms solve the movement and multiplication of tensors in fundamentally different ways.
| Hardware Class | Memory Architecture | Compute Engines | Software & Kernel Ecosystem | Primary Strengths & Hard Limits |
|---|---|---|---|---|
| Data Centre NVIDIA GPUs (H100, B200, R100) | Discrete HBM3e / HBM4 (80 GB–288 GB, 3.35–19.2 TB/s) + fast on-chip SRAM. | Tensor Cores (FP16, BF16, FP8, NVFP4) + TMA async copy units. | CUDA, cuDNN, Triton, TensorRT-LLM, vLLM, NCCL. | Highest raw FLOPS and multi-GPU scaling; expensive and constrained by PCIe/NVLink boundaries. |
| Consumer NVIDIA GPUs (RTX 4090 / 5090) | Discrete GDDR6X / GDDR7 (24 GB–32 GB, ~1.0–1.8 TB/s). | Tensor Cores (FP16, FP8, FP4) with lower FP64/NVLink capabilities. | CUDA, PyTorch, Triton, llama.cpp, ExLlamaV2. | Unmatched local price-to-performance; severely bottlenecked by 24–32 GB VRAM ceiling and lack of NVLink. |
| Apple Silicon (M3/M4 Max & Ultra) | Unified Memory (LPDDR5) shared across CPU/GPU/NPU (up to 192–512 GB, ~400–800 GB/s). | GPU cores + Simdgroup Matrix units + Apple Neural Engine (ANE). | Metal (MSL), MLX, PyTorch MPS, llama.cpp. | Huge RAM capacity for loading massive models locally; moderate memory bandwidth limits prompt prefill speed. |
| Mobile SoCs (Snapdragon 8 Elite, Apple A18/A19) | Low-power LPDDR5X shared RAM (8 GB–16 GB, ~50–85 GB/s). | Mobile GPU (Adreno/Apple) + dedicated INT4/INT8 NPU (Hexagon/ANE). | Vulkan, Metal, ExecuTorch, LiteRT, MLC-LLM. | Ultra-low power (3–8 W); strict thermal throttling and tight 3–6 GB OS app memory limits. |
2. Solving Specific Bottlenecks & Engineering Goals
Once you identify your target hardware, every performance bottleneck falls into one of three engineering categories—each requiring a distinct technical playbook.
Goal A: Maximizing Training Speed (Throughput & Scaling)
Training is dominated by storing activations/gradients and keeping Tensor Cores fed across multiple steps.
- Eliminate Python & Memory Overhead: Enable torch.compile(mode="max-autotune") and BF16/FP8 mixed precision so Tensor Cores run at full speed while cutting activation memory in half.
- Overlap Communication with Compute: Across multiple GPUs, use PyTorch FSDP2 or ZeRO-3 with gradient accumulation so All-Gather and Reduce-Scatter transfers happen asynchronously behind matrix multiplications.
- Activation Checkpointing: If large batch sizes trigger Out-Of-Memory (OOM) errors, discard intermediate layer activations during the forward pass and recompute them during backpropagation—trading ~25% extra compute for massive batch-size headroom.
Goal B: Minimizing Inference Latency (TTFT & Tokens/Sec)
Inference splits into Prefill (compute-bound prompt ingestion) and Decode (memory-bandwidth-bound token generation).
- Cut Time-To-First-Token (TTFT): Use FlashAttention-2/3 and Prefix Caching (reusing KV cache blocks for shared system prompts) so long prompts saturate Tensor Cores without redundant math.
- Accelerate Token Generation (TPOT): Quantize weights to FP8 or INT4 (AWQ/Marlin) to shrink HBM reads by 2×–4×, enable CUDA Graphs to eliminate CPU kernel-launch gaps, and add Speculative Decoding (EAGLE/Medusa) to verify 3–5 tokens per forward pass.
- Eliminate Batch Interference: In multi-user production environments, use PagedAttention with continuous batching—or split Prefill and Decode onto separate GPU pools—so heavy prompts never stall active token streams.
Goal C: Shrinking Memory Footprint (Fitting Models on Constrained Devices)
When a model exceeds available VRAM or mobile RAM, aggressive compression and zero-copy execution become mandatory.
- Sub-Byte Weight Compression: Use 4-bit quantization (Q4_K_M or AWQ) to shrink a 70B model from 140 GB down to ~40 GB, or an 8B model from 16 GB down to ~4.9 GB.
- Compress the KV Cache: Long conversations often consume more RAM than the model weights themselves. Apply FP8/INT4 KV Cache quantization, Sliding Window Attention, or architectures with Multi-Head Latent Attention (MLA) to cut context memory by 50%–90%.
- Zero-Copy Streaming (mmap): On Unified Memory laptops and phones, memory-map quantized model files directly from NVMe/UFS storage so the OS pages weights in on demand without doubling RAM usage during startup.