Run LLMs on Your CPU with llama.cpp: GGUF Files, Quantization, Speed and Streaming

So far in Part V the models have lived in someone else's data centre, or on a GPU. But what if you have an ordinary laptop with no graphics card, a patchy internet connection, or data you would rather not send anywhere? You can still run a language model. This session shows how, with a real model on a real machine: a 1.5-billion-parameter chat model, generating text on a CPU.

The tool that made this practical is llama.cpp, a lightweight C and C++ engine for running quantized models on ordinary CPUs (and optionally GPUs or Apple Silicon). Georgi Gerganov started it in 2023, and it is the engine under many popular local-AI apps. It builds on the quantization ideas of Session 21 and the model we used for RAG in Session 22.

What you need

• The engine: llama.cpp itself (a command-line program and a server) or its Python bindings, llama-cpp-python.
• A model in GGUF format: a single file that holds the weights, the tokenizer and the settings. Download one from the Hugging Face Hub. Names look like qwen2.5-1.5b-instruct-q4_k_m.gguf.
• Enough RAM to hold the file, plus a little extra for the context.

pip install llama-cpp-python
# or the command-line tools:  brew install llama.cpp

What is a GGUF file, and what does Q4_K_M mean?

GGUF is a binary format built for fast loading and quantized weights. Quantization (Session 21) stores each weight in fewer bits. The suffix tells you the scheme. Q4_K_M means roughly 4-bit blocks with per-block scales ("K" is a family of smarter quantization methods) and "M" for medium, a balance of size and quality. Other common ones are Q8_0 (near lossless, twice the size), Q5_K_M and Q3_K_S (smaller, lossier).

Let's measure what that does to a real file:

import os
from llama_cpp import Llama

llm = Llama(model_path="data/qwen2.5-1.5b-instruct-q4_k_m.gguf", n_ctx=256, verbose=False)
path = "data/qwen2.5-1.5b-instruct-q4_k_m.gguf"
n_params = 1.54e9                                           # published size of the Qwen2.5-1.5B model
file_bytes = os.path.getsize(path)
print(f"file: {file_bytes / 1e9:.2f} GB for about {n_params / 1e9:.2f} billion weights")
print(f"-> {file_bytes * 8 / n_params:.1f} bits per weight on average")
print(f"FP16 would need {n_params * 2 / 1e9:.1f} GB, so this file is {n_params * 2 / file_bytes:.1f}x smaller")
print("\nthe 'Q4_K_M' in the name: 4-bit blocks with scales, and a few sensitive layers kept at higher precision,")
print("which, together with higher-precision embeddings, is why the average is well above 4 bits")

file: 1.12 GB for about 1.54 billion weights
-> 5.8 bits per weight on average
FP16 would need 3.1 GB, so this file is 2.8x smaller

the 'Q4_K_M' in the name: 4-bit blocks with scales, and a few sensitive layers kept at higher precision,
which, together with higher-precision embeddings, is why the average is well above 4 bits

The file is 1.12 GB for about 1.54 billion weights, which is 5.8 bits per weight on average, not 4. Two reasons: Q4_K_M keeps some sensitive layers at higher precision, and the large embedding and output tables are stored at higher precision too. Even so, the file is 2.8 times smaller than the 3.1 GB the same weights would take in FP16. When you read "4-bit", read "about 5 to 6 bits all in" for small models.

Your first chat

Loading the model and chatting uses the same message format as the cloud APIs:

import os, time
from llama_cpp import Llama

path = "data/qwen2.5-1.5b-instruct-q4_k_m.gguf"                 # a 4-bit (Q4_K_M) GGUF file
llm = Llama(model_path=path, n_ctx=2048, n_threads=8, verbose=False)

n_params = int(llm.metadata.get("general.size_label", "0").replace("B", "").replace(".", "") or 0)   # e.g. 1.5B
file_gb = os.path.getsize(path) / 1e9
print(f"model: {llm.metadata.get('general.name')}   file on disk: {file_gb:.2f} GB   context window: {llm.n_ctx()} tokens")

start = time.time()
out = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Explain quantization in one sentence."}], max_tokens=60, temperature=0)
seconds = time.time() - start
usage = out["usage"]
print("answer:", out["choices"][0]["message"]["content"])
print(f"{usage['prompt_tokens']} prompt tokens + {usage['completion_tokens']} generated tokens in {seconds:.1f} s "
      f"= {usage['completion_tokens'] / seconds:.0f} generated tokens per second (whole call, on this machine)")

model: qwen2.5-1.5b-instruct   file on disk: 1.12 GB   context window: 2048 tokens
answer: Quantization is the process of converting a continuous signal into a discrete set of values by rounding or truncating the signal to the nearest representable value.
37 prompt tokens + 30 generated tokens in 0.7 s = 40 generated tokens per second (whole call, on this machine)

The model loads from a 1.12 GB file, has a 2,048-token context window, and answers a question about quantization. The timing line says it took about 1.1 seconds for 30 generated tokens, around 27 tokens per second on this machine, which is roughly the speed of comfortable reading, with no GPU involved. (Your speed will differ: it depends on your CPU, memory and the model size.)

The settings you will use most are n_ctx (the context window, which costs memory), n_threads (how many CPU threads to use) and, if you have a GPU, n_gpu_layers (how many layers to offload to it).

from llama_cpp import Llama

llm = Llama(model_path="model.Q4_K_M.gguf", n_ctx=2048, n_threads=8)
out = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Explain quantization in one sentence."}],
    max_tokens=100)
print(out["choices"][0]["message"]["content"])

Streaming and the first-token delay

Waiting for a whole answer feels slow. Streaming delivers tokens as they are produced, so the reader starts reading at once. Two numbers describe a generation: the time to the first token (how long until anything appears, mostly spent reading the prompt) and the generation speed afterwards.

import time
from llama_cpp import Llama

llm = Llama(model_path="data/qwen2.5-1.5b-instruct-q4_k_m.gguf", n_ctx=1024, n_threads=8, verbose=False)
start = time.time(); first = None; pieces = []
for chunk in llm.create_chat_completion(messages=[{"role": "user", "content": "Give me three study tips, one short line each."}],
                                         max_tokens=70, temperature=0, stream=True):
    piece = chunk["choices"][0]["delta"].get("content", "")
    if piece:
        if first is None: first = time.time() - start
        pieces.append(piece)
total = time.time() - start
print("".join(pieces))
print(f"\ntime to the first token: {first:.2f} s   |   whole answer: {total:.1f} s   ({len(pieces)} tokens)")

1. Break down your study sessions into manageable chunks.
2. Use active learning techniques like summarizing or teaching the material to someone else.
3. Create a consistent study schedule and stick to it.

time to the first token: 0.16 s   |   whole answer: 0.9 s   (40 tokens)

The first token arrived in 0.17 seconds and the whole 40-token answer was done in 1.0 seconds. For a user, 0.17 seconds feels instant, and the rest streams in as they read. The same pattern, a fast start and a steady flow, is what makes cloud chatbots feel responsive.

Speed: it depends on memory and cores

Generating text is mostly limited by how fast the CPU can read the model's weights from memory, since every new token needs them all. So expect a few to tens of tokens per second for small models, and less for bigger ones. Two levers help: a smaller or lower-bit model, and the right number of threads. More threads is not always faster:

import time
from llama_cpp import Llama

print("threads   generated tokens/second")
for threads in (1, 2, 4, 8):
    llm = Llama(model_path="data/qwen2.5-1.5b-instruct-q4_k_m.gguf", n_ctx=512, n_threads=threads, verbose=False)
    start = time.time()
    out = llm.create_completion("Write a short note about studying for exams.", max_tokens=48, temperature=0)
    n = out["usage"]["completion_tokens"]
    print(f"{threads:6d}    {n / (time.time() - start):6.1f}")

threads   generated tokens/second
     1      26.4
     2      39.5
     4      53.3
     8      46.7

On this machine one thread gives 26 tokens a second and two give 36. Four threads is the best at 55. Then eight threads is slower (35). That is common on laptops with a mix of fast "performance" cores and slower "efficiency" cores: asking for more threads than there are fast cores lets slow ones drag the rest down. Always benchmark on your own machine and pick the best, which is often the number of performance cores rather than the total.

A local server for your apps

Besides the command-line chat (llama-cli), llama.cpp can run a local server that speaks the same API as OpenAI's, so any code from Session 21 works against it by changing the base URL:

llama-cli    -m model.Q4_K_M.gguf -p "Hello" -n 64          # chat from the terminal
llama-server -m model.Q4_K_M.gguf --port 8080               # an OpenAI-compatible server

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")
reply = client.chat.completions.create(
    model="local", messages=[{"role": "user", "content": "Name one benefit of running a model locally."}])
print(reply.choices[0].message.content)

(I did not run this server block for this article. The bundled Python server did not work with the library versions on the machine used to write it, so check the llama-server binary or Ollama, which offer the same API, and test it on your own system.)

Ollama, LM Studio and the other options

llama.cpp is the engine. Friendlier tools wrap it:

• Ollama: one command downloads and runs a model, and it exposes a local API.
• LM Studio: a desktop app with a chat window and a model browser.
• vLLM or TGI: high-throughput servers for production GPU deployments.
• Transformers in Python: the most flexible, and slowest on a CPU.

When local makes sense

Small models are also worse at some things. The 1.5B model here follows short instructions well but is easily out-reasoned and much more error-prone than a frontier model. In Session 24 you will see it call a tool correctly one moment and invent arguments the next.

Try it yourself

1. Download a different GGUF of the same model (for example Q8_0 and Q2_K). Compare the file size, the speed and the quality of the answers.
2. Run threads_speed.py with your own thread counts. Where is your best?
3. Change n_ctx from 2,048 to 512 in chat_cpu.py. What happens if the prompt is longer than the context?
4. Time the same prompt on a cloud API and on this local model. Which has the lower first-token delay? Which is faster overall?

What comes next

We can now read documents with RAG and run a model locally. But the model still only produces text. Session 24 shows how to let it call your functions, the building block of every AI agent.

Frequently asked questions