How to quantize a GGUF model with llama.cpp

llama.cpp quantization turns a higher-precision GGUF model into a smaller file that the same runtime can load for local inference. It is useful after converting or downloading a full-precision model that is too large for the available memory, disk budget, or deployment target.

The llama-quantize tool reads one GGUF input file and writes a separate output file, so the original model remains available for comparison or a different quantization pass. Source builds usually expose the binary under build/bin, while package-managed installs may place llama-quantize directly on the shell path.

Quantize from an F16, BF16, or F32 source when that file is available. Requantizing an already quantized model can reduce output quality, and large models need enough temporary disk and memory headroom for the input and output files before the load test starts.

1. Check the higher-precision input model before creating the quantized copy.

   $ ls -lh models/stories260K-f32.gguf
   -rw-r--r-- 1 user user 1.2M Jul  2 22:42 models/stories260K-f32.gguf

   Use an F16, BF16, or F32 GGUF file when possible. If the model still needs conversion from a Hugging Face directory, create the GGUF file first.
   Related: How to convert a Hugging Face model to GGUF for llama.cpp

2. Run llama-quantize with the input file, output file, and quantization type.

   $ ./build/bin/llama-quantize models/stories260K-f32.gguf models/stories260K-Q4_K_M.gguf Q4_K_M
   llama_print_build_info: build = 9859 (4fc4ec554)
   llama_quantize: quantizing 'models/stories260K-f32.gguf' to 'models/stories260K-Q4_K_M.gguf' as Q4_K_M
   ##### snipped #####
   llama_model_quantize_impl: model size  =     1.12 MiB (32.00 BPW)
   llama_model_quantize_impl: quant size  =     0.28 MiB (7.92 BPW)
   llama_quantize:    total time =     2.92 ms

   Q4_K_M is a common balanced output type for local inference. Use --allow-requantize only when no higher-precision source exists, because upstream llama.cpp warns that requantizing can reduce quality.

3. Compare the input and output file sizes.

   $ ls -lh models/stories260K-f32.gguf models/stories260K-Q4_K_M.gguf
   -rw-r--r-- 1 user user 297K Jul  2 22:42 models/stories260K-Q4_K_M.gguf
   -rw-r--r-- 1 user user 1.2M Jul  2 22:42 models/stories260K-f32.gguf

   The exact reduction depends on the source model and quantization type. Very small or unusual models can also show fallback tensor types in the quantization log.

4. Load the quantized file with llama-cli and run a short single-turn prompt.

   $ ./build/bin/llama-cli -m models/stories260K-Q4_K_M.gguf -p "Once upon a time" -n 16 -st --no-display-prompt
   Loading model...
   build      : b9859-4fc4ec554
   model      : models/stories260K-Q4_K_M.gguf
   modalities : text
   ##### snipped #####
   > Once upon a time
   e's mom told him that it's big ey
   
   [ Prompt: 2298.7 t/s | Generation: 85.7 t/s ]
   
   Exiting...

   The -st option runs one turn and exits after the predefined prompt. A successful load and generated text confirm that the quantized GGUF file is usable by llama.cpp.