Large Language Models Explained: Parameters, Training, Quantization, LoRA and Calling an LLM API

In Session 20 we saw that a Transformer is a next-token predictor. Make it very large and train it on a huge amount of text, and you get a large language model (LLM). Everything you know as an AI assistant is one of these, with extra training on top.

This session answers the practical questions that come right after. What does "7B" mean, and why does a model need 14 GB of memory? How can a model be shrunk to run on a laptop? How do you adapt a model without retraining all of it? And what does a call to an LLM through an API actually look like? We run the memory maths, quantize a real model, build LoRA from scratch and look at a request on the wire.

What is an LLM?

• Parameters: the learned weights. "7B" means seven billion of them. More parameters usually means more capability, and always more memory and cost.
• An LLM is a large, usually decoder-only Transformer, pre-trained to predict the next token on a huge text corpus. That one skill lets it write, answer questions, translate and code.
• An SLM (small language model) is roughly 1 to 8 billion parameters. It runs on a laptop or phone, with lower cost and delay.
• A base model only continues text. An instruct (chat) model has been further trained to follow instructions and hold a conversation.
• A multimodal model accepts more than text (images, audio, documents) by converting each input type into embeddings the Transformer reads together with text. A text-only LLM does not draw pictures: image generators are a different family (diffusion models).

The model families

Models come in two access styles. Closed models are reachable only through the provider's API and the weights are never shared. Open-weight models let you download the weights and run or fine-tune them yourself, though the training data and code may not be public. (Truly open source, with weights, code and data all public, is rare. OLMo is an example.) Always read the licence: Apache 2.0 and MIT are permissive, others restrict commercial use.

Names, versions and sizes change every few months, so check each provider's current page before you build on a detail. At the time of writing (November 2026), published sizes include DeepSeek V4-Pro at about 1.6 trillion parameters in total with about 49 billion active per token (MIT licence), Qwen3.5 at 397 billion total with 17 billion active (Apache 2.0) and Gemma 4 with a 31-billion-parameter dense model (Apache 2.0). The closed providers do not publish parameter counts, so any number you see for GPT, Claude or Gemini is a guess.

That "total versus active" wording comes from mixture-of-experts (MoE) models. They hold many specialised feed-forward "experts" but send each token through only a few. The total decides how much memory you need to load the model. The active count decides how much computing each token costs.

How an LLM is built

Three stages turn raw text into an assistant:

1. Pre-training: learn language by predicting the next token over huge amounts of text. The result is the base model.
2. Supervised fine-tuning (SFT): train on instruction and answer pairs. The result is an instruct model.
3. Alignment (RLHF or similar): use human preferences so answers are helpful and safe (Session 13's bonus).

At run time the model only ever predicts the next token. Special tokens (role markers, "tool call", "end of turn") are just entries in the vocabulary that the surrounding program interprets. Reasoning models first generate "thinking" tokens before the final answer, and those tokens are billed as output, so APIs let you cap how much they think.

Every response ends with a stop reason that tells your code what to do next:

Memory: why a 7B model needs 14 GB

Every parameter is stored as a number, and the number of bytes per parameter is the precision. The memory just for the weights is simple arithmetic:

bytes_per = {"FP32": 4, "FP16 / BF16": 2, "INT8": 1, "INT4": 0.5}

print("memory needed just to hold the weights (GB):\n")
print(f"{'precision':12s} {'7B model':>10s} {'70B model':>11s}")
for name, b in bytes_per.items():
    print(f"{name:12s} {7e9 * b / 1e9:10.1f} {70e9 * b / 1e9:11.1f}")

# a mixture-of-experts model: you must LOAD every expert, but each token only USES a few
total, active = 1.6e12, 49e9                    # DeepSeek V4-Pro's published sizes
print(f"\nMoE example: {total / 1e12:.1f}T parameters to hold in memory, {active / 1e9:.0f}B used per token = {active / total:.1%} of the weights per step")

# the KV cache: extra memory that grows with the conversation (Session 20)
layers, kv_heads, head_dim, tokens = 32, 8, 128, 8000
kv = 2 * layers * kv_heads * head_dim * tokens * 2          # K and V, 2 bytes each
print(f"\nKV cache of a 32-layer model with 8 KV heads for {tokens:,} tokens: {kv / 1e9:.2f} GB (on top of the weights)")

memory needed just to hold the weights (GB):

precision      7B model   70B model
FP32               28.0       280.0
FP16 / BF16        14.0       140.0
INT8                7.0        70.0
INT4                3.5        35.0

MoE example: 1.6T parameters to hold in memory, 49B used per token = 3.1% of the weights per step

KV cache of a 32-layer model with 8 KV heads for 8,000 tokens: 1.05 GB (on top of the weights)

A 7B model needs 28 GB in FP32, 14 GB in FP16, 7 GB in INT8 and 3.5 GB in INT4. A 70B model needs 140 GB even in FP16, which is more than any single consumer GPU holds. The second part shows why MoE models are cheap to run but not to hold: DeepSeek V4-Pro must keep 1.6 trillion parameters in memory but uses only about 3.1 percent of them per token. The last line is the KV cache: roughly 1 GB for 8,000 tokens of conversation for a 32-layer model, on top of the weights.

Quantization: smaller numbers, smaller model

Quantization stores weights with fewer bits (say 4 instead of 16). The model shrinks and runs faster, and quality drops a little. The standard trick is to round weights to a small set of levels, using one scale per small group so that big and small weights are both represented well:

Let's quantize the real GPT-2 and measure what it costs. We round every large weight matrix to 8, 4 and 3 bits (in groups of 64 weights) and measure the model's loss on a short text. Lower loss is better:

import copy
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2").eval()
text = ("Quantization stores each weight with fewer bits. The model gets smaller and faster, and the answers change only a little. "
        "Photosynthesis turns sunlight, water and carbon dioxide into sugar and oxygen. The capital of France is Paris, and the river Seine runs through it.")
ids = tok(text, return_tensors="pt")["input_ids"]

def loss_of(m):
    with torch.no_grad():
        return m(ids, labels=ids).loss.item()

def fake_quantize(weight, bits, group=64):
    """Round weights to 2**bits levels, one scale per group of 64 values (the idea behind Q4 / Q8 formats)."""
    shape = weight.shape
    w = weight.reshape(-1, group)
    qmax = 2 ** (bits - 1) - 1
    scale = w.abs().amax(dim=1, keepdim=True) / qmax
    return ((w / scale).round().clamp(-qmax, qmax) * scale).reshape(shape)

print(f"{'precision':10s} {'weights stored in':>18s} {'text loss':>10s}")
print(f"{'FP32':10s} {124439808 * 4 / 1e6:15.0f} MB {loss_of(model):10.3f}")
for bits in (8, 4, 3):
    m = copy.deepcopy(model)
    for name, p in m.named_parameters():
        if p.dim() == 2 and p.shape[-1] % 64 == 0 and "wte" not in name and "wpe" not in name:
            p.data = fake_quantize(p.data, bits)                      # the big matrices; embeddings left alone
    mb = 124439808 * (bits / 8) / 1e6
    print(f"{'INT' + str(bits):10s} {mb:15.0f} MB {loss_of(m):10.3f}")

precision   weights stored in  text loss
FP32                   498 MB      3.945
INT8                   124 MB      3.934
INT4                    62 MB      4.364
INT3                    47 MB      5.576

Full precision gets a loss of 3.945. At 8 bits the model shrinks from 498 MB to about 124 MB and the loss is 3.934, effectively unchanged. At 4 bits (62 MB) the loss rises to 4.364, a visible but modest cost. At 3 bits (47 MB) it jumps to 5.576, where the model is clearly damaged. That curve is why 4-bit is the popular sweet spot. (The sizes in this table are an estimate of the stored weights. We left the embeddings alone in this experiment, so a real file is slightly larger.) Real quantized formats are smarter than plain rounding, but this is the core idea.

Fine-tuning, prompting and RAG

Three ways to change what a model does, in order of cost:

• Prompting: try this first. It is free.
• RAG: when the model needs fresh or private knowledge (Session 22).
• Fine-tuning: when you need to change behaviour, tone or format, not mainly to add facts.

Fine-tuning every weight of a big model needs huge GPU memory. Parameter-efficient fine-tuning (PEFT) freezes the model and trains a tiny number of extra parameters. The best-known method is LoRA (low-rank adaptation). The insight: the change a fine-tune makes to a big weight matrix $W$ is low-rank, so it can be written as the product of two thin matrices.

Let's build it ourselves and teach a frozen layer a new behaviour:

import torch
import torch.nn as nn

torch.manual_seed(0)
d, r = 64, 4

class LoRALinear(nn.Module):
    """A frozen linear layer plus a small trainable low-rank update:  W x + (alpha/r) * B A x"""
    def __init__(self, base, rank, alpha=8):
        super().__init__()
        self.base = base
        for p in self.base.parameters(): p.requires_grad = False        # the original weights never change
        self.A = nn.Parameter(torch.randn(rank, base.in_features) * 0.1)
        self.B = nn.Parameter(torch.zeros(base.out_features, rank))     # starts at zero: the model begins unchanged
        self.scale = alpha / rank
    def forward(self, x):
        return self.base(x) + self.scale * (x @ self.A.T) @ self.B.T

base = nn.Linear(d, d, bias=False)
# the "new behaviour" we want: the same layer plus a low-rank change
delta = torch.randn(d, r) @ torch.randn(r, d) * 0.15
target_W = base.weight.detach() + delta

lora = LoRALinear(base, r)
trainable = sum(p.numel() for p in lora.parameters() if p.requires_grad)
total = sum(p.numel() for p in lora.parameters())
print(f"trainable parameters: {trainable:,} of {total:,} ({trainable / total:.1%})")

opt = torch.optim.Adam([lora.A, lora.B], lr=0.02)
x = torch.randn(512, d)
y = x @ target_W.T
for step in range(1, 601):
    loss = nn.functional.mse_loss(lora(x), y)
    opt.zero_grad(); loss.backward(); opt.step()
    if step in (1, 100, 300, 600):
        print(f"step {step:3d}   loss {loss.item():.5f}")

# what the same trick saves on a big layer, as in a 7B model
D = 4096
print(f"\na {D} x {D} layer has {D * D:,} weights; LoRA with rank 8 trains {2 * 8 * D:,} ({2 * 8 * D / (D * D):.2%})")

trainable parameters: 512 of 4,608 (11.1%)
step   1   loss 5.29837
step 100   loss 0.00022
step 300   loss 0.00000
step 600   loss 0.00000

a 4096 x 4096 layer has 16,777,216 weights; LoRA with rank 8 trains 65,536 (0.39%)

On our small 64 by 64 layer, only 512 of 4,608 parameters train (11.1 percent), and the loss falls from 5.3 to essentially zero by step 300. The target change really was low-rank, so the tiny adapter captured it exactly. The big payoff is on real layers: a 4096 by 4096 layer has 16.8 million weights, and LoRA at rank 8 trains 65,536 of them, 0.39 percent. QLoRA goes further by holding the frozen base in 4-bit, so large models can be fine-tuned on a single consumer GPU. The adapter is a small file you can swap or merge.

from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model

base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B")
config = LoraConfig(r=8, lora_alpha=16, lora_dropout=0.05,
                    target_modules=["q_proj", "v_proj"], task_type="CAUSAL_LM")
model = get_peft_model(base, config)
model.print_trainable_parameters()        # typically well under 1% of all parameters

Model files: what you download

The Hugging Face Hub is the largest place to download from, and Ollama offers one-command models.

Calling an LLM through an API

A provider API is a hosted service. You send an HTTP request with your messages and receive the reply, with no hardware to manage. Providers include OpenAI, Anthropic, Google, Mistral, Groq and cloud platforms like AWS Bedrock, Azure and Vertex AI. Aggregators such as OpenRouter give one key for many models. And because many providers copy OpenAI's request format, you can often switch provider by changing only the base_url, the key and the model name.

A request has a few core ideas:

• Messages with roles: system (the rules), user (the question), assistant (the model's earlier replies).
• Parameters: temperature, top_p and max_tokens (Session 20).
• Streaming: receive tokens as they are generated instead of waiting for the full answer.
• Context window: input plus output tokens must fit.
• Pricing: charged per token, with input and output priced separately, and output usually costing more.
• Rate limits: caps on requests or tokens per minute.

Here is what actually crosses the wire. To keep this runnable without an account, the code below uses the real OpenAI SDK pointed at a stand-in server (a fake endpoint that records the request and replies in the standard format):

import json
import httpx
from openai import OpenAI

captured = {}

def fake_provider(request: httpx.Request) -> httpx.Response:
    """Stands in for a provider's server. It records what the SDK sent and replies in the OpenAI format."""
    captured["url"] = str(request.url)
    captured["body"] = json.loads(request.content)
    return httpx.Response(200, json={
        "id": "chatcmpl-demo", "object": "chat.completion", "model": "demo-model",
        "choices": [{"index": 0, "finish_reason": "stop",
                     "message": {"role": "assistant", "content": "Quantization stores weights with fewer bits."}}],
        "usage": {"prompt_tokens": 31, "completion_tokens": 8, "total_tokens": 39}})

client = OpenAI(api_key="not-a-real-key", base_url="https://api.example.com/v1",
                http_client=httpx.Client(transport=httpx.MockTransport(fake_provider)))

reply = client.chat.completions.create(
    model="demo-model", temperature=0.2, max_tokens=100,
    messages=[{"role": "system", "content": "Be concise."},
              {"role": "user", "content": "Explain quantization in one sentence."}])

print("POST", captured["url"])
print(json.dumps(captured["body"], indent=2))
print("\nreply     :", reply.choices[0].message.content)
print("stop reason:", reply.choices[0].finish_reason)
print("tokens used:", reply.usage.prompt_tokens, "in,", reply.usage.completion_tokens, "out")

POST https://api.example.com/v1/chat/completions
{
  "messages": [
    {
      "role": "system",
      "content": "Be concise."
    },
    {
      "role": "user",
      "content": "Explain quantization in one sentence."
    }
  ],
  "model": "demo-model",
  "max_tokens": 100,
  "temperature": 0.2
}

reply     : Quantization stores weights with fewer bits.
stop reason: stop
tokens used: 31 in, 8 out

The SDK turned our Python call into a POST to /v1/chat/completions with a JSON body holding the model, the messages and the parameters. The reply carried the text, a stop reason (stop) and token usage (31 in, 8 out). This is the entire interface. The calls below use the same shape against real providers:

import anthropic

client = anthropic.Anthropic()                      # reads ANTHROPIC_API_KEY from the environment

def ask(prompt, system="You are a helpful assistant.", max_tokens=512):
    msg = client.messages.create(model="claude-sonnet-5-5",      # check the provider's current model list
                                 max_tokens=max_tokens, system=system,
                                 messages=[{"role": "user", "content": prompt}])
    return msg.content[0].text

from openai import OpenAI

# any OpenAI-compatible provider or local server: change only base_url, key and model
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")     # for example Ollama
r = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "system", "content": "Be concise."},
              {"role": "user", "content": "What is the capital of France?"}],
    temperature=0.2, max_tokens=100)
print(r.choices[0].message.content)

Tokens, language and the bill

You pay per token, so how text splits into tokens matters. A tokenizer trained mostly on English handles English cheaply and other languages less so:

import tiktoken

enc = tiktoken.get_encoding("o200k_base")                  # the tokenizer family used by recent OpenAI models
samples = {
    "English sentence": "Machine learning lets computers learn patterns from data.",
    "Same idea in Urdu": "مشین لرننگ کمپیوٹرز کو ڈیٹا سے پیٹرن سیکھنے کے قابل بناتی ہے۔",
    "A line of Python": "for i in range(10): print(i ** 2)",
    "A long word": "internationalization",
}
print(f"{'text':20s} {'characters':>10s} {'tokens':>7s}")
for name, s in samples.items():
    print(f"{name:20s} {len(s):10d} {len(enc.encode(s)):7d}")

# a bill is tokens x price, with output tokens costing more. These prices are made-up placeholders.
price_in, price_out = 3.0, 15.0                            # dollars per million tokens (example only)
calls, tin, tout = 10_000, 1_200, 300
cost = calls * (tin * price_in + tout * price_out) / 1e6
print(f"\n{calls:,} calls of {tin} input + {tout} output tokens at ${price_in}/${price_out} per million: ${cost:,.2f}")

text                 characters  tokens
English sentence             57       9
Same idea in Urdu            61      25
A line of Python             33      13
A long word                  20       2

10,000 calls of 1200 input + 300 output tokens at $3.0/$15.0 per million: $81.00

A 57-character English sentence is 9 tokens. The same idea in Urdu (61 characters) is 25 tokens, nearly three times as many, with a modern tokenizer. That means more cost, more delay and less of the context window for the same meaning. The bill at the bottom is simple multiplication, and the prices there are made-up placeholders (look up your provider's real prices): 10,000 calls of 1,200 input and 300 output tokens cost $81 at the example rates. Output tokens dominate the cost even though there are fewer of them, which is why concise answers save money.

Free or paid, API or self-host?

Free API tiers exist (limits change often, and your data may be used for training, so check the terms). For self-hosting there is Ollama and LM Studio for easy use, llama.cpp for CPUs (the next-but-one session), vLLM for high-throughput GPU servers and Hugging Face Transformers in plain Python. Choose an API for best quality and quick starts at low volume. Choose self-hosting for privacy, high volume, offline use or custom fine-tuned models.

Attacks to know about

LLM applications have their own security problems:

• Prompt injection: an attacker hides instructions in user input or in content the model reads (web pages, documents, emails) so that the model follows them instead of yours. It is direct when typed by the user and indirect when it arrives through retrieved or tool data. It is the main risk in apps that read outside content.
• Prompt leakage: tricking the model into revealing its hidden system prompt. Never put secrets or API keys in a prompt, and assume the system prompt can be extracted.
• Jailbreaks: role-play, hypotheticals or encoded and multi-turn tricks that bypass safety training.
• Data poisoning, model extraction (copying a model through repeated queries) and denial of wallet (running up your bill).

Try it yourself

1. In memory_math.py, calculate the FP16 memory for a 13B and a 34B model. Which fit on a 24 GB GPU?
2. In quantize_gpt2.py, try group sizes of 32 and 128 (a smaller group means more scales). How does the 4-bit loss change?
3. In lora_from_scratch.py, change the rank of the LoRA layer from 4 to 1 while keeping the target at rank 4. Can it still reach zero loss?
4. In token_costs.py, count the tokens of a Roman-Urdu sentence and an Urdu-script sentence with the same meaning.

What comes next

An LLM only knows what it learned, which has a cutoff and does not include your private documents. Session 22 fixes that with retrieval-augmented generation: look up the facts first, then let the model answer from them.

Frequently asked questions