Tag:llama.cpp

GGUF and the Art of Choosing a Quant: A Field Guide to Local Models

Quantization · Part 3

Open any popular model's page on Hugging Face and you'll find the GGUF listings: Q4KM, Q5KS, Q6K, Q80, IQ2XS — a wall of cryptic suffixes, each a different point on a size-quality curve, downloaded millions of times by people running models on gaming PCs an…