#optimization

FlashAttention claims 2-4x speedup. CUDA graphs claim 10x. What actually helps in production, and what's just good marketing?

Every LLM request has two distinct phases with different performance characteristics. Understanding them is the key to targeted optimization.

Users don't perceive throughput. They perceive the silence before the first token appears. TTFT is the metric that determines whether your app feels fast.

Your LLM bill is one number. Your product has twenty features. Without cost attribution, you're optimizing in the dark.

A 128K context window doesn't mean you should use 128K tokens. Context is a budget with diminishing returns and escalating costs.

E2EL = TTFT + generation time sounds simple. But where does that time actually go? Understanding the equation reveals where to optimize.

A 2,000 token system prompt processed 10 million times a month. Without caching, you're paying to process the same tokens over and over.

Double your context window, quadruple your compute. The O(n²) attention cost catches teams off guard when they scale.