#attention

Attention visualization reveals which tokens influence outputs. Debug why the model ignored critical context or fixated on irrelevant tokens.

Attention scores decay with distance. By position 50K, tokens may have near-zero influence. Positional encodings have practical limits, regardless of window size.

Full attention is O(n²). Sliding window attention is O(n). The trade: lose long-range dependencies, gain linear scaling. Often worth it.

Self-attention lets a sequence talk to itself. Cross-attention lets one sequence attend to another. Understanding the difference enables better architectures.

Models advertise 128K context windows. But attention quality degrades with distance. The last 10% of context often contributes less than the first 10%.

Standard attention needs O(n²) memory. Memory-efficient variants need O(n). Same output, 10x less peak memory.

Flash Attention doesn't make attention faster. It makes attention fit in memory. The speedup is a side effect of better memory access.

A 128K context window doesn't mean you should use 128K tokens. Context is a budget with diminishing returns and escalating costs.

Double your context window, quadruple your compute. The O(n²) attention cost catches teams off guard when they scale.