How to run post-training quantization for TFLite

Edge deployment often starts after a model has already been trained and exported, when the remaining constraint is the runtime artifact rather than the training loop. Post-training quantization converts that finished TensorFlow model into a smaller integer TFLite flatbuffer that LiteRT and TensorFlow Lite runtimes can load on mobile, embedded, or accelerator-backed targets.

Full-int8 quantization uses tf.lite.TFLiteConverter with tf.lite.Optimize.DEFAULT and a representative dataset to calibrate activation ranges before the flatbuffer is written. Restricting supported operations to tf.lite.OpsSet.TFLITE_BUILTINS_INT8 makes the conversion fail when an operation cannot stay integer-only, which is safer than discovering a float fallback inside a deployment runtime that expects int8 tensors.

Start from a SavedModel whose serving signature, input feature order, and preprocessing scale are already accepted. The support-score sample uses four numeric features so the conversion path is readable, but production calibration rows should come from normal training, validation, or logged inference data rather than invented values.

1. Open a terminal in the Python environment that can import TensorFlow and read the exported SavedModel directory.

   $ python3 -c "import tensorflow as tf; print(tf.__version__)"
   2.21.0

   The sample output was verified with TensorFlow 2.21.0 in the campaign CPU image. Use the same environment that exported or smoke-tested the SavedModel so converter behavior matches the model package set.
   Related: How to create a virtual environment for TensorFlow
   Related: How to install TensorFlow with pip

2. Inspect the default serving signature before choosing representative rows.

   $ saved_model_cli show --dir support_score_savedmodel --tag_set serve --signature_def serving_default
   The given SavedModel SignatureDef contains the following input(s):
     inputs['keras_tensor'] tensor_info:
         dtype: DT_FLOAT
         shape: (-1, 4)
         name: serving_default_keras_tensor:0
   The given SavedModel SignatureDef contains the following output(s):
     outputs['output_0'] tensor_info:
         dtype: DT_FLOAT
         shape: (-1, 1)
         name: StatefulPartitionedCall_1:0
   Method name is: tensorflow/serving/predict

   The representative rows and smoke-test sample must match this input shape, dtype, feature order, and preprocessing scale.

   Related: How to inspect a TensorFlow SavedModel with saved_model_cli

3. Save representative calibration rows beside the SavedModel directory.

   support_score_representative.csv

   feature_a,feature_b,feature_c,feature_d
   0.10,0.70,0.20,0.30
   0.18,0.62,0.25,0.35
   0.24,0.58,0.32,0.40
   0.36,0.46,0.42,0.45
   0.64,0.30,0.68,0.72
   0.72,0.24,0.76,0.82
   0.82,0.16,0.84,0.90
   0.92,0.08,0.90,0.96

   The short CSV keeps the demonstration compact. For a production model, use enough typical rows to cover the normal input range without adding outliers that the model should not accept.

   Do not reorder, rename, or rescale feature columns unless the exported model was trained with that exact input contract.

4. Save the quantization workflow as quantize_support_score.py.

   quantize_support_score.py

   import csv
   import os
   import pathlib
   import warnings
    
   os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3"
   warnings.filterwarnings("ignore", message=".*tf.lite.Interpreter is deprecated.*")
    
   import numpy as np
   import tensorflow as tf
    
   tf.get_logger().setLevel("ERROR")
    
   SAVEDMODEL_PATH = pathlib.Path("support_score_savedmodel")
   REPRESENTATIVE_PATH = pathlib.Path("support_score_representative.csv")
   FLOAT_TFLITE_PATH = pathlib.Path("support_score_float.tflite")
   INT8_TFLITE_PATH = pathlib.Path("support_score_int8.tflite")
   SAMPLE = np.array([[0.72, 0.24, 0.76, 0.82]], dtype=np.float32)
    
    
   def load_representative_rows():
       rows = []
    
       with REPRESENTATIVE_PATH.open("r", newline="", encoding="utf-8") as handle:
           reader = csv.DictReader(handle)
           for row in reader:
               rows.append(
                   [
                       float(row["feature_a"]),
                       float(row["feature_b"]),
                       float(row["feature_c"]),
                       float(row["feature_d"]),
                   ]
               )
    
       return np.asarray(rows, dtype=np.float32)
    
    
   representative_rows = load_representative_rows()
    
   float_converter = tf.lite.TFLiteConverter.from_saved_model(str(SAVEDMODEL_PATH))
   FLOAT_TFLITE_PATH.write_bytes(float_converter.convert())
    
    
   def representative_dataset():
       for row in representative_rows:
           yield [row.reshape(1, 4)]
    
    
   int8_converter = tf.lite.TFLiteConverter.from_saved_model(str(SAVEDMODEL_PATH))
   int8_converter.optimizations = [tf.lite.Optimize.DEFAULT]
   int8_converter.representative_dataset = representative_dataset
   int8_converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
   int8_converter.inference_input_type = tf.int8
   int8_converter.inference_output_type = tf.int8
   INT8_TFLITE_PATH.write_bytes(int8_converter.convert())
    
   interpreter = tf.lite.Interpreter(model_path=str(INT8_TFLITE_PATH))
   interpreter.allocate_tensors()
    
   input_details = interpreter.get_input_details()[0]
   output_details = interpreter.get_output_details()[0]
   input_scale, input_zero_point = input_details["quantization"]
   output_scale, output_zero_point = output_details["quantization"]
    
   quantized_sample = np.round(SAMPLE / input_scale + input_zero_point).astype(np.int8)
   interpreter.set_tensor(input_details["index"], quantized_sample)
   interpreter.invoke()
   raw_output = interpreter.get_tensor(output_details["index"])
   int8_score = (raw_output.astype(np.float32) - output_zero_point) * output_scale
    
   float_interpreter = tf.lite.Interpreter(model_path=str(FLOAT_TFLITE_PATH))
   float_interpreter.allocate_tensors()
   float_input = float_interpreter.get_input_details()[0]
   float_output = float_interpreter.get_output_details()[0]
   float_interpreter.set_tensor(float_input["index"], SAMPLE)
   float_interpreter.invoke()
   float_score = float_interpreter.get_tensor(float_output["index"])
    
   print(f"TensorFlow {tf.__version__}")
   print(f"saved_model_dir={SAVEDMODEL_PATH.name}")
   print(f"representative_rows={len(representative_rows)}")
   print(f"float_model_file={FLOAT_TFLITE_PATH.name}")
   print(f"int8_model_file={INT8_TFLITE_PATH.name}")
   print(f"float_model_bytes={FLOAT_TFLITE_PATH.stat().st_size}")
   print(f"int8_model_bytes={INT8_TFLITE_PATH.stat().st_size}")
   print(f"input_dtype={np.dtype(input_details['dtype']).name}")
   print(f"output_dtype={np.dtype(output_details['dtype']).name}")
   print(f"input_quantization=scale:{input_scale:.8f},zero_point:{input_zero_point}")
   print(f"output_quantization=scale:{output_scale:.8f},zero_point:{output_zero_point}")
   print(f"float_sample_output={float_score[0][0]:.6f}")
   print(f"int8_sample_output={int8_score[0][0]:.6f}")
   print(f"absolute_difference={abs(float_score[0][0] - int8_score[0][0]):.6f}")

   The script writes a float flatbuffer first so the same sample row can be compared against the int8 output after conversion.

5. Run the quantization script.

   $ python3 quantize_support_score.py
   ##### snipped #####
   TensorFlow 2.21.0
   saved_model_dir=support_score_savedmodel
   representative_rows=8
   float_model_file=support_score_float.tflite
   int8_model_file=support_score_int8.tflite
   float_model_bytes=5024
   int8_model_bytes=4552
   input_dtype=int8
   output_dtype=int8
   input_quantization=scale:0.00376471,zero_point:-128
   output_quantization=scale:0.00390625,zero_point:-128
   float_sample_output=0.563334
   int8_sample_output=0.562500
   absolute_difference=0.000834

   The int8_model_file line confirms the artifact was written, the input_dtype and output_dtype lines confirm the requested integer tensor contract, and the sample comparison confirms the converted model can run through the TFLite interpreter.

6. Check the int8 flatbuffer before handing it to the deployment build.

   $ ls -lh support_score_int8.tflite
   -rw-r--r-- 1 user user 4.5K Jun 29 04:45 support_score_int8.tflite

   A successful conversion does not guarantee acceptable model quality. Evaluate the int8 flatbuffer against the same acceptance data used for the float model before shipping it, refresh the representative dataset when accuracy drops, and move to quantization-aware training only when post-training quantization no longer meets the deployment target.