dful.
Dful Dark
Theme
Knowledge tree
On this page

Local Inference

How to fit and tune a model on local hardware through quantization, memory, GPU offload, context, and runtime choices.
Updated 28 Aug 2026

Local inference runs the model on hardware you directly control. Instead of sending a request to an external provider, a runtime such as LM Studio or llama.cpp loads the weights and generates the response on the local machine.

The practical problem is fitting the model, context, and runtime state into available memory without making generation too slow.

Local inference memory composition A local host has separate VRAM and RAM pools. Weights, cache, buffers and system state compete for capacity; longer context increases cache pressure and reduces headroom. VRAM GPU-resident pool Weights / offloaded layersKV cacheRuntime buffers RAM CPU-side pool Non-offloaded weightsCPU cache / stateOS + other processes Placement can split weights, cache and buffers across both pools. Context length ↑ More active tokens KV cache ↑ More runtime state Headroom ↓ Less capacity for growth Local inference memory composition A local host has separate VRAM and RAM pools. Weights, cache, buffers and system state compete for capacity; longer context increases cache pressure and reduces headroom. VRAM GPU-resident pool Weights / offloaded layersKV cacheRuntime buffers RAM CPU-side pool Non-offloaded weightsCPU cache / stateOS + other processes Placement can split weights, cache and buffers across both pools. Context length ↑ More active tokens KV cache ↑ More runtime state Headroom ↓ Less capacity for growth
Figure 1. Weights, KV cache, buffers, and CPU/GPU placement share the available memory during local inference.

Weights, formats, and quantization

Model weights contain the learned parameters. Parameter count gives a first indication of size, but the precision used to represent those values changes memory requirements substantially.

A 27B model stored at 16-bit precision needs roughly 54 GB for weights alone. A 16 GB GPU cannot load that full representation. Quantization reduces value precision and allows the same model to be distributed in much smaller files.

Names such as Q4_K_M or Q3_K_XL identify particular schemes in the runtime ecosystem. Lower-bit quantization generally saves memory and may affect quality or speed. The useful test is the real task; the name alone does not establish whether the loss is acceptable.

GGUF is the format used by llama.cpp to store model tensors and metadata. It is not itself a quantization level. A GGUF file can hold weights encoded with different schemes.

VRAM, RAM, and offload

A weight file smaller than available VRAM does not guarantee the complete configuration will load. KV cache, runtime buffers, and the operating system also need memory.

When the model does not fit entirely in VRAM, llama.cpp can split layers between GPU and CPU through GPU offload. This makes larger models usable, but system RAM is not a free extension of VRAM. If substantial compute remains on CPU or repeatedly crosses the bus, generation may become much slower.

The Qwen3.8-27B and OpenCode experiment showed this clearly. The roughly 16.8 GB Q4 allowed 54/65 offload units and generated around 9 tokens/s. A roughly 13.4 GB Q3 left enough room for full GPU placement, and short requests moved to around 40–45 tokens/s.

This does not prove that Q3 is five times faster than Q4. Quantization and CPU/GPU placement changed together. The useful conclusion is that removing partial offload resolved the main bottleneck in that setup.

Context and KV cache

A larger context window also consumes more memory. During prefill and decode, the runtime keeps a KV cache with state computed for earlier tokens. As context grows, the cache grows and leaves less headroom for weights and buffers.

LM Studio can adjust loaded context and, for compatible models, the precision or placement of K and V. Quantizing the cache can recover VRAM, but it is another trade-off that may affect quality or performance.

In the Qwen experiment, moving K and V to Q4_0 allowed a large session to retain full GPU offload. Dedicated memory reached roughly 15.4 out of 16 GB. The setup remained usable but made the physical cost of long context obvious even though the model advertised a much larger maximum window.

Loading the maximum by default is rarely useful. The target is enough context for the task with headroom for runtime state.

Runtime settings

LM Studio provides an interface over llama.cpp for downloading, loading, and serving models. llama.cpp also provides tools such as llama-cli and llama-server.

The settings that have mattered most in my tests are:

  • GPU offload: how much work can stay on the GPU.
  • Context length and KV cache: how much context fits and how much VRAM it consumes.
  • Threads: relevant when significant work remains on the CPU.
  • Flash Attention: a runtime optimization that can reduce memory or improve speed in compatible configurations.

Batch parameters can also affect prompt processing, but they do not need attention until measurement points to that bottleneck. Changing many sliders at once makes the cause of an improvement hard to identify.

Speculative decoding

Speculative decoding uses a cheaper path, often a smaller draft model, to propose several tokens that the main model verifies. It can accelerate decode when enough candidates are accepted.

It can also add overhead when the draft is poorly matched. Test it after memory, context, and placement are stable rather than treating it as an automatic speedup.

Local API

LM Studio and llama-server can expose OpenAI-compatible HTTP endpoints. An application can reuse an SDK or agent client by pointing base_url at the local server.

This separates two pieces that are easy to conflate. LM Studio performs inference. OpenCode, Codex, or another client may provide the harness, tools, and loop. Serving a model through an API does not by itself give the model filesystem or shell access.

The server can listen on a LAN. In the Qwen experiment, LM Studio used the Windows host GPU while OpenCode and pentesting tools ran inside a Kali VM. Inference remained local even though the client ran on another machine.

Model APIs covers request shape and the limits of compatibility.

Benchmarking

A local benchmark is useful when it supports comparison and can be repeated. Record at least:

  • Exact model and quantization.
  • Runtime, hardware, and GPU offload.
  • Context length and KV-cache configuration.
  • Prompt and output size.
  • Generation tokens/s and, when relevant, prefill time or TTFT.
  • Peak VRAM and RAM use.

A short prompt does not represent an agent session with 50K active tokens. In the Qwen article, short Q3 generations reached around 40–45 tokens/s, while the long-context test generated around 21 tokens/s. Both measurements are valid for their conditions.

Also state whether timing includes model loading or starts from a warm server.

Practical tuning

The easiest sequence to interpret is:

  1. Choose a model size appropriate for the task.
  2. Select a quantization with enough memory headroom.
  3. Keep as much work on GPU as possible and measure the effect of offload.
  4. Load a realistic context target instead of the maximum by default.
  5. Inspect KV-cache and buffer usage.
  6. Measure prefill, generation speed, and memory with the real workload.
  7. Tune threads, batch, or speculative decoding only when they address an observed bottleneck.

The best configuration is not the one that maximizes one slider. It is the one that keeps enough quality and context at a usable speed on the available hardware.

References