Deciding Which Model Handles Each Request
Send classification to Haiku, reasoning to Opus. Routing requests to the right model saves money without sacrificing quality.
Blog
Deep dives into LLM inference optimization. Practical insights for developers and founders building with AI.
Managing Model Versions Without Downtime
Models change. Prompts change. How do you update without breaking clients? Immutable versions and controlled rollout.
Safe Rollouts for LLM Changes
Model changes are high-risk deployments. 1% traffic to new, compare outputs, then gradually expand. Here's the playbook.
What Production LLM Systems Need to Survive
The gap between 'works on my laptop' and 'survives production' is filled with timeouts, retries, fallbacks, and rate limits. Here's the checklist.
Attention That Fits in Memory
Standard attention needs O(n²) memory. Memory-efficient variants need O(n). Same output, 10x less peak memory.
Finding Memory Leaks in LLM Serving
Memory grows slowly over hours, then OOM. Here's how to find where the bytes are going before they crash your server.
The Performance Wins from Fusing Kernels
Every CUDA kernel launch has overhead. Fusing three operations into one can be 3x faster. Here's where fusion helps and how to get it.
What Flash Attention Actually Does
Flash Attention doesn't make attention faster. It makes attention fit in memory. The speedup is a side effect of better memory access.
Testing Quality After Quantization
Eval suites catch problems benchmarks miss. Here's how to build testing that prevents quantization regressions from reaching users.
When to Use FP8 for Inference
H100's FP8 gives near-FP16 quality at near-INT8 speed. It's becoming the new default. Here's when and how to use it.