How to quantize an Ollama model

Quantizing an Ollama model creates a smaller or faster local model from higher-precision weights. The tradeoff is reduced precision, so the result should be tested against the task it will actually run.

Ollama quantization is handled by ollama create with the --quantize flag when the Modelfile starts from FP16 or FP32 model weights. Common levels include q8_0 and K-means levels such as q4_K_M.

Quantization needs source weights and enough local resources to process them. Do not claim a quantized model is acceptable until it passes a real prompt or evaluation for the workload.

1. Create a Modelfile that points at the high-precision source model.

   FROM /srv/models/gemma-f16

2. Create a quantized model with the chosen level.

   $ ollama create --quantize q4_K_M example/gemma-q4 -f Modelfile
   transferring model data
   quantizing F16 model to Q4_K_M
   writing manifest
   success

3. Inspect the quantized model metadata.

   $ ollama show example/gemma-q4
     Model
       quantization        Q4_K_M
   ##### snipped #####

4. Run a short smoke test.

   $ ollama run example/gemma-q4 "Return only OK."
   OK

5. Keep the source weights until the quantized model passes workload checks.

   $ ollama list
   NAME              ID              SIZE
   example/gemma-q4  735e246cc1ab    4.8 GB