Latency
Time to first token measures the wait. Generation speed measures how quickly the answer continues.
Practical guide
LLM inference, from first token to cost per task
Learn which metrics make an agent feel fast, how cache reuse changes economics, and when serverless or dedicated GPUs fit.
Capacity
Throughput only matters inside the latency and user speed your product requires.
Cache
Repeated agent context can move across GPU memory, CPU memory, and NVMe instead of being recomputed.
The buying model
Optimize for the completed task
A cheap token is not cheap when a weaker model retries, loops, or fails the task. Compare models using completion quality, generated tokens, elapsed time, and total inference cost for the same representative trace.
For interactive agents, set the minimum acceptable generation speed per user first. Measure how much aggregate throughput remains at that speed. This prevents a highly batched benchmark from hiding a poor user experience.
Prefill dominates an agent's bill because every turn re-sends the whole conversation. Prompt caching reuses the KV cache for the repeated prefix and bills it at a cached rate, but only when the turn reaches the same model and the same worker; see prompt caching and cache-aware routing.
Per-user speed depends on the serving stack as much as the GPU: speculative decoding multiplies decode speed 2-3x on the same hardware with identical output.
Serverless or dedicated inference?
Use serverless first
It fits uncertain or bursty demand. You pay for tokens and avoid reserving idle GPUs.
Reserve dedicated capacity
It fits steady production traffic that can use the reservation and needs isolated, predictable capacity.
LLM inference questions
What is LLM inference?
LLM inference is the work a trained language model performs to read a prompt and generate an answer. Production inference includes model execution, batching, cache management, networking, and request scheduling.
What makes LLM inference expensive?
Model size, prompt length, generated tokens, active users, hardware, and cache reuse drive cost. For agents, retries and task completion quality also matter.
How should an agent company measure inference?
Track time to first token, generation speed per active user, completed tasks, retries, prompt cache reuse, and total cost per completed task.