#tokens

100 requests sounds like 100 requests. But one 50k-token request consumes more resources than 99 short ones combined. Batch by tokens, not requests.

Your API has rate limits. Your database has connection limits. Your LLM endpoints should have token limits. Here's how to add them without breaking production.

Input tokens are cheap. Output tokens are expensive. The physics of transformer inference explains why, and what you can do about it.