dful.
Dful Dark
Theme
Browse notes
On this page

Inference

Prefill processes input and decode generates output token by token; sampling, streaming, and TTFT help interpret behavior and performance.
Updated 11 Oct 2026

Inference is the execution of a trained model on new input. For a generative LLM, two practical phases are worth separating: prefill, which processes the input, and decode, which generates output one token at a time.

Prefill and decode Prefill processes the existing input before the first output token. Decode then generates the response one token at a time. TTFT · wait for the first token Input tokens Prompt + context Prefill Process existing input First token The answer begins Decode Generate one token at a time 23n Output continues token by token Prefill and decode Prefill processes the existing input before the first output token. Decode then generates the response one token at a time. Input tokens Prompt + context Prefill Process existing input First token The answer begins Decode Generate one token at a time TTFT · wait for the first token 23n Output continues token by token
Figure 1. Prefill processes the input; decode then generates the response one token at a time.

Prefill

During prefill, the model processes input tokens before it begins to answer. If a request contains 20K tokens across instructions, history, and documents, that input has to be processed before the first output token appears.

Larger input usually means more initial work. From the client side, time to first token can also include network delay, queueing, and request preparation, so TTFT does not measure model computation alone.

In runtimes with a KV cache, prefill also prepares state reused during generation. This is especially visible in local inference: more context means more initial processing and a larger cache in memory.

Decode

After prefill, decode begins. The model produces candidates for the next token, the generation settings select one, and that token is added to the sequence before the process repeats. This step-by-step process is autoregressive generation.

Generation stops when the model emits an end token, reaches its output limit, or encounters a configured stop sequence. The output maximum is a cap, not a length the model is expected to fill.

Sampling

Sampling controls how the next token is selected from the model’s candidates. Greedy selection takes the highest-scoring candidate; sampling draws from a distribution, allowing runs to vary.

Temperature

Temperature changes how much weight lower-probability alternatives receive. Lower values concentrate selection around the leading candidates; higher values allow more variation.

It acts on token selection during decode. Raising temperature does not add intelligence or deeper reasoning.

Top-p

Top-p limits selection to candidates that together contain a configured amount of probability mass, then samples within that set.

Temperature and top-p can be combined, but their values are not directly portable across models. Defaults and recommendations depend on the model and runtime. temperature = 0 also does not guarantee exact reproducibility if the model version, backend, or hardware changes.

Streaming

With streaming, the application receives fragments as they are generated and can display them or begin processing a tool call before the complete response exists.

Streaming changes when the client receives output; by itself it does not make decode faster.

Speed and latency

The two measurements I use most when interpreting local-inference results are:

  • Time to First Token (TTFT): delay before the first token arrives, including prefill and, depending on the measurement, other request overhead.
  • Generation tokens/s: output rate during decode.

A tokens/s figure is comparable only when tools measure the same interval. Some separate prefill and decode, while others report a broader average.

Tests with Qwen3.8-27B and OpenCode showed the difference clearly. A short request with the model fully on GPU generated around 40–45 tokens/s, while a test with roughly 50K input tokens took much longer to process and generated around 21 tokens/s. Input size and runtime pressure were part of the observed difference.

References