Latency
Time to first token measures the wait. Generation speed measures how quickly the answer continues.
Practical guide
LLM inference, from first token to cost per task
Learn which metrics make an agent feel fast, how cache reuse changes economics, and when serverless or dedicated GPUs fit.
Capacity
Throughput only matters inside the latency and user speed your product requires.
Cache
Repeated agent context can move across GPU memory, CPU memory, and NVMe instead of being recomputed.
Definition
What is LLM inference?
LLM inference is running a trained large language model to turn a prompt into output tokens. The weights stay fixed. Each request tokenizes the input, runs it through the model, and generates the answer one token at a time until the model emits a stop token or hits the length limit.
Training happens once per model version. Inference happens on every request from every user and agent, so it is the cost a product pays every month and the latency its users feel on every turn.
How LLM inference works: prefill and decode
Prefill
The model reads the entire prompt in one parallel pass and stores each token's attention keys and values in the KV cache. This phase is compute-bound. A longer prompt means a longer wait before the first output token.
Decode
The model generates one token per forward pass, reading the weights and the growing KV cache each step. This phase is bound by memory bandwidth, so serving engines batch many users' decode steps together with continuous batching.
Agents stress prefill more than chat does, because every turn resends the growing transcript. Prompt caching cuts that cost by reusing the cached prefix instead of recomputing it.
LLM inference metrics
| Metric | What it measures | Driven mostly by |
|---|---|---|
| Time to first token (TTFT) | Wait before the answer starts | Prompt length, prefill compute, queueing, cache hits |
| Tokens per second per user | How fast one answer streams | Memory bandwidth, batch size, speculative decoding |
| Aggregate throughput | Total tokens per second across all users | Batching, KV cache memory, number of GPUs |
| Cost per completed task | Spend to finish one agent task | Model price, tokens generated, retries, cache reuse |
LLM inference engines
An inference engine loads the weights, schedules requests, and manages KV cache memory. All four below serve an OpenAI-compatible API.
| Engine | Built for | License |
|---|---|---|
| vLLM | Multi-user GPU serving with PagedAttention and continuous batching | Apache 2.0 |
| SGLang | GPU serving with RadixAttention prefix reuse for agents and RL rollouts | Apache 2.0 |
| Ollama | Running GGUF models locally with a model library | MIT |
| llama.cpp | C/C++ GGUF runtime for laptops, Apple Silicon, CPUs, and edge | MIT |
Compare them head to head in vLLM vs Ollama and SGLang vs vLLM, or read LLM inference optimization for the techniques that sit on top of the engine.
The buying model
Optimize for the completed task
A cheap token is not cheap when a weaker model retries, loops, or fails the task. Compare models using completion quality, generated tokens, elapsed time, and total inference cost for the same representative trace.
For interactive agents, set the minimum acceptable generation speed per user first. Measure how much aggregate throughput remains at that speed. This prevents a highly batched benchmark from hiding a poor user experience.
Prefill dominates an agent's bill because every turn re-sends the whole conversation. Prompt caching reuses the KV cache for the repeated prefix and bills it at a cached rate, but only when the turn reaches the same model and the same worker; see prompt caching and cache-aware routing.
Per-user speed depends on the serving stack as much as the GPU: speculative decoding multiplies decode speed 2-3x on the same hardware with identical output.
Serverless or dedicated inference?
Use serverless first
It fits uncertain or bursty demand. You pay for tokens and avoid reserving idle GPUs.
Reserve dedicated capacity
It fits steady production traffic that can use the reservation and needs isolated, predictable capacity.
How to size a GPU clusterLLM inference questions
What is LLM inference?
LLM inference is the work a trained language model performs to read a prompt and generate an answer. Production inference includes model execution, batching, cache management, networking, and request scheduling.
What is the difference between prefill and decode?
Prefill processes the whole prompt in one parallel pass and writes its keys and values into the KV cache. Decode then generates one token per step, reading that cache each time. Prefill is limited by compute, decode by memory bandwidth, which is why long prompts raise time to first token and long answers lower tokens per second.
What is the difference between LLM training and inference?
Training updates a model's weights from data and runs once per model version on large GPU clusters. Inference runs the finished model, with weights frozen, every time a user or agent sends a prompt. For a deployed product, inference is the recurring cost.
Which software runs LLM inference?
Inference engines load the weights, schedule requests, and manage the KV cache. vLLM and SGLang are the common open-source choices for GPU serving with many concurrent users. Ollama and llama.cpp run GGUF models on laptops, Apple Silicon, and CPUs. All four expose an OpenAI-compatible API.
What makes LLM inference expensive?
Model size, prompt length, generated tokens, active users, hardware, and cache reuse drive cost. For agents, retries and task completion quality also matter.
How should an agent company measure inference?
Track time to first token, generation speed per active user, completed tasks, retries, prompt cache reuse, and total cost per completed task.
Private deployments
The fastest endpoints are private deployments
Morph's top speeds come from dedicated deployments, not shared public endpoints: speculators trained on your traffic, caching tuned to your workload, and volume discounts over public per-token rates. Over 500 billion tokens per day run this way.
Talk to us about a private deployment