Quantization and context length explained
Understand the two memory controls with a worked example, without confusing smaller files with faster or better answers.
Published 16 September 2026 · HostingLinks · Editorial approach
Two controls that change different things
Quantization changes how model weights are represented. Context length changes how much text the model can consider within a request. A useful analogy is a reference book and a desk: compressing the book makes it smaller, while a larger desk gives you more space for the material you are currently working on. In an LLM the details are numerical, but the distinction helps explain why both settings affect memory.
What Q4, Q8 and FP16 mean here
Our catalogue offers Q4_K_M, Q8_0 and FP16 where supported. The Q4 and Q8 labels describe compressed weight formats; they are not promises that every parameter occupies exactly four or eight bits. Scaling data and mixed precision add overhead. Our estimates use 4.85 and 8.5 effective bits per weight respectively, while FP16 uses 16. Real downloaded files can differ.
Compression can change answer quality. A smaller file is useful when capacity is limited, but there is no universal rule that a particular format is best for every task. Try a few representative prompts, check factual correctness and compare the output that matters to you. A larger model at lower precision and a smaller model at higher precision are different choices, not automatically equivalent ones.
Context is a shared budget
Instructions, conversation history, supplied documents and generated text consume context. Tokens are pieces of text, not a fixed number of words. A long document can use much of the available window before an answer begins. Setting a large maximum also does not guarantee a model will use all of that context effectively.
Check both the model’s supported limit and your runtime configuration. Our calculator excludes models whose recorded context limit is below your selected value. It does not silently extend a model using experimental scaling. See Ollama’s context guidance for runtime controls and resource considerations.
Follow the numbers
For Llama 3.1 8B Q4_K_M, our weight allowance is 4.52 GiB. Moving from 4,096 to 32,768 tokens increases our FP16 cache estimate from 0.50 to 4.00 GiB. With runtime allowance and headroom, the GPU target moves from 6.7 to 10.6 GiB. The model did not gain parameters: it needed more working memory.
This example uses one conversation, full GPU loading and an uncompressed FP16 cache. Some runtimes support other cache formats or attention optimisations; our estimate does not model all of them. For Gemma and gpt-oss we deliberately allow full-context cache per layer without subtracting sliding-window savings. Read the methodology and limitations before treating an estimate as a purchase requirement.
A practical starting point
For an initial Ollama or LM Studio experiment, select balanced weights and a short context, then increase one setting at a time. If you change weight format, obtain the matching file. If you change context, apply it inside the runtime too. In vLLM, this calculator uses FP16 or supported native gpt-oss MXFP4 rather than assuming GGUF formats work identically everywhere.