dful.
Dful Dark
Theme
Browse notes
On this page

Local Inference

The settings that most affect a local model: memory, quantization, offload, context, KV cache, and how to measure whether the configuration works.
Updated 11 Oct 2026

Local inference runs the model on hardware you control directly. A runtime such as LM Studio or llama.cpp loads the weights and generates output on your machine, so memory, context, and CPU/GPU placement become part of the performance you get.

The practical problem is fitting the model and the state required for the session into available memory without making generation too slow.

Local inference memory composition A local host has separate VRAM and RAM pools. Weights, cache, buffers and system state compete for capacity; longer context increases cache pressure and reduces headroom. VRAM GPU-resident pool Weights / offloaded layersKV cacheRuntime buffers RAM CPU-side pool Non-offloaded weightsCPU cache / stateOS + other processes Placement can split weights, cache and buffers across both pools. Context length ↑ More active tokens KV cache ↑ More runtime state Headroom ↓ Less capacity for growth Local inference memory composition A local host has separate VRAM and RAM pools. Weights, cache, buffers and system state compete for capacity; longer context increases cache pressure and reduces headroom. VRAM GPU-resident pool Weights / offloaded layersKV cacheRuntime buffers RAM CPU-side pool Non-offloaded weightsCPU cache / stateOS + other processes Placement can split weights, cache and buffers across both pools. Context length ↑ More active tokens KV cache ↑ More runtime state Headroom ↓ Less capacity for growth
Figure 1. Weights, KV cache, buffers, and CPU/GPU placement share the available memory during local inference.

Weights, formats, and quantization

Model weights contain the learned parameters. Parameter count gives a first indication of size, but the precision used to represent those values changes memory requirements substantially.

A 27B model stored at 16-bit precision needs roughly 54 GB for weights alone. A 16 GB GPU cannot load that full representation. Quantization reduces value precision and allows the same model to be distributed in much smaller files.

Names such as Q4_K_M or Q3_K_XL identify particular schemes in the runtime ecosystem. Lower-bit quantization usually saves memory and can change quality or speed. The trade-off is better judged on real tasks than from file size alone.

GGUF is the format used by llama.cpp to store model tensors and metadata. It is not itself a quantization level; a GGUF file can contain weights encoded with different schemes.

VRAM, RAM, and offload

A weight file smaller than available VRAM does not guarantee the complete configuration will load. KV cache, runtime buffers, and the operating system also need memory.

When the model does not fit entirely in VRAM, llama.cpp can split layers between GPU and CPU through GPU offload. This makes larger models usable, but moving substantial compute to CPU can reduce generation speed sharply.

Tests with Qwen3.8-27B and OpenCode showed this clearly. The roughly 16.8 GB Q4 allowed 54/65 offload units and generated around 9 tokens/s. A roughly 13.4 GB Q3 left enough room for full GPU placement, and short requests moved to around 40–45 tokens/s.

This does not prove that Q3 is five times faster than Q4. Quantization and CPU/GPU placement changed together. In that setup, removing partial offload resolved the main bottleneck.

Context and KV cache

A larger context window also consumes more memory. During prefill and decode, the runtime keeps a KV cache with state computed for earlier tokens. As context grows, the cache leaves less headroom for weights and buffers.

LM Studio can adjust loaded context and, for compatible models, the precision or placement of K and V. Quantizing the cache can recover VRAM at the cost of another possible quality or performance trade-off.

In the Qwen experiment, moving K and V to Q4_0 allowed a large session to retain full GPU offload. Dedicated memory reached roughly 15.4 out of 16 GB. The setup remained usable, while making the physical cost of long context obvious.

Load the context the task needs and leave headroom for runtime state rather than using the advertised maximum by default.

Runtime settings

LM Studio provides an interface over llama.cpp for downloading, loading, and serving models. llama.cpp also provides tools such as llama-cli and llama-server.

The settings that have mattered most in my tests are:

  • GPU offload: how much work can remain on the GPU.
  • Context length and KV cache: how much context fits and how much VRAM it consumes.
  • Threads: relevant when significant work remains on CPU.
  • Flash Attention: a runtime optimization that can reduce memory or improve speed in compatible configurations.

Batch parameters can also affect prompt processing. I would tune them after measuring where the bottleneck is; changing several sliders at once makes an improvement harder to attribute.

Speculative decoding

Speculative decoding uses a cheaper path, often a smaller draft model, to propose several tokens that the main model verifies. It can accelerate decode when enough candidates are accepted and add overhead when the draft model is a poor match.

It is easier to evaluate after memory, context, and CPU/GPU placement are stable.

Local API

LM Studio and llama-server can expose OpenAI-compatible HTTP endpoints. A client can reuse an SDK or harness by pointing base_url at the local server.

LM Studio performs inference. OpenCode, Codex, or another client may provide the harness, tools, and loop. Serving a model through an API does not by itself give it filesystem or shell access.

The server can also listen on a LAN. In the Qwen experiment, LM Studio used the Windows host GPU while OpenCode and pentesting tools ran inside a Kali VM. Inference remained local even though the client ran on another machine.

Model APIs covers request shape and the limits of compatibility.

Benchmarking

A local benchmark is useful when it supports comparison and can be repeated. Record at least:

  • Exact model and quantization.
  • Runtime, hardware, and GPU offload.
  • Context length and KV-cache configuration.
  • Prompt and output size.
  • Generation tokens/s and, when relevant, prefill time or TTFT.
  • Peak VRAM and RAM use.

A short prompt does not necessarily represent an agent session with 50K active tokens. In the Qwen article, short Q3 generations reached around 40–45 tokens/s, while the long-context test generated around 21 tokens/s. Both measurements describe different conditions and remain useful when those conditions are kept with the result.

Also state whether timing includes model loading or starts from a warm server.

Practical tuning

A sequence that is easy to interpret is:

  1. Choose a model size appropriate for the task.
  2. Select a quantization with enough memory headroom.
  3. Keep as much work on GPU as possible and measure the effect of offload.
  4. Load a realistic context target for the workload.
  5. Inspect KV-cache and buffer usage.
  6. Measure prefill, generation speed, and memory with the real workload.
  7. Tune threads, batch, or speculative decoding when they address an observed bottleneck.

A useful configuration keeps enough quality and context at a reasonable speed for the available hardware.

References