Inference

How a model processes input and generates output through prefill, decode, sampling, and streaming.

Updated 28 Aug 2026

Inference is the execution of a trained model to produce output from new input. For a generative LLM, the practical path has two parts: process the input, then generate the output.

Figure 1. Prefill processes the input; decode then generates the response one token at a time.

Prefill

During prefill, the model processes the input tokens before it begins to answer. If a request contains 20K tokens across instructions, history, and documents, those tokens need to be processed before the first output token appears.

Larger input normally means more initial work. From the client side, time to first token can also include network delay, queueing, and request preparation, so the complete wait is not necessarily model computation.

In runtimes that use a KV cache, prefill also prepares state reused during generation. The practical effect is especially visible in local inference: more context takes more initial processing and more memory to retain.

Decode and autoregressive generation

After prefill, decode begins. The model generates one token, adds it to the sequence, then uses the updated sequence to generate the next. This is autoregressive generation.

The response is not calculated as one finished block of text. It is built step by step:

The model produces candidates for the next token.
The generation settings select one.
That token becomes part of the response context.
The process repeats until generation stops.

Generation may stop because the model emits an end token, reaches the output limit, or encounters a configured stop sequence. The maximum is a cap, not a target length.

Sampling

At every step, several next tokens are possible. Sampling controls how the runtime chooses among them.

Greedy selection takes the highest-scoring candidate. Sampling selects from a distribution, which allows runs to vary.

Temperature

Temperature changes how much weight lower-probability alternatives receive. A low value concentrates selection around the leading candidates. A higher value allows more variation.

It is not a control for intelligence or reasoning depth. It affects token selection during decode.

Top-p

Top-p limits candidates to the smallest set that contains a configured amount of cumulative probability, then samples within that set.

Temperature and top-p can be combined, but values should not be copied between models as if they were equivalent. Defaults and recommendations depend on the model and runtime. Exact reproducibility is also not guaranteed merely by setting temperature = 0; version, backend, and hardware can still matter.

Streaming

Without streaming, the application waits for generation to finish and receives the completed response. With streaming, it receives fragments as they are produced.

This lets a client display text or begin processing a tool call before the full response exists. It improves perceived responsiveness but does not make decode intrinsically faster.

Speed and latency

The two measurements I find most useful when reading local-inference results are:

Time to First Token (TTFT): delay before the first token arrives. It reflects prefill and, depending on the measurement, other request overhead.
Generation tokens/s: the rate at which output tokens arrive during decode.

A tokens/s number is only comparable when tools measure the same interval. Some separate prefill and decode; others report a wider average.

The Qwen3.8-27B and OpenCode experiment shows the distinction in practice. A short request with the model fully on GPU generated around 40–45 tokens/s, while a test with roughly 50K input tokens took much longer to process and generated around 21 tokens/s. The result reflected input size and runtime pressure as well as the model itself.

Local Inference covers memory, offload, and hardware effects.