How to Think About Context as a Budget
A 128K context window doesn't mean you should use 128K tokens. Context is a budget with diminishing returns and escalating costs.
23 posts tagged with "cost"
A 128K context window doesn't mean you should use 128K tokens. Context is a budget with diminishing returns and escalating costs.
A 2,000 token system prompt processed 10 million times a month. Without caching, you're paying to process the same tokens over and over.
Double your context window, quadruple your compute. The O(n²) attention cost catches teams off guard when they scale.