Large floating-point weight tensors can dominate a neural network's inference memory footprint. torchao can quantize a PyTorch model's linear weights to int8 on a Linux CPU, reducing their stored size while keeping floating-point inputs and outputs.

Int8 weight-only quantization stores integer weights alongside floating-point scale values. The per-row configuration converts supported Linear weights without requiring activation calibration, while biases and layers such as ReLU remain unchanged.

A smaller tensor footprint does not establish lower peak process memory or faster CPU inference. Comparing the original and quantized model in evaluation mode on identical inputs exposes numerical changes before a trained model is put into service.

Steps to quantize a PyTorch model with torchao:

  1. Create a new virtual environment for the project.
    $ python3 -m venv .venv

    These packages install on Python 3.11 through 3.14. An existing environment is unsuitable because the pinned install can replace its PyTorch and leave an incompatible torchvision. On Ubuntu or Debian, install python3-venv first if python3 -m venv reports that ensurepip is unavailable.
    Related: How to create a virtual environment for PyTorch
    Related: How to upgrade PyTorch

  2. Activate the virtual environment in your shell.
    $ source .venv/bin/activate
    (.venv) $
  3. Install the tested PyTorch and torchao CPU packages in the new environment.
    (.venv) $ python -m pip install \
      torch==2.15.0 torchao==0.18.0 numpy \
      --index-url https://download.pytorch.org/whl/cpu

    The --index-url value selects CPU wheels. Installing NumPy avoids an import warning from PyTorch.
    Related: torchao installation documentation

  4. Create a file named quantize_model.py in your text editor with the floating-point model and baseline calculation.
    quantize_model.py
    import copy
     
    import torch
    from torch import nn
    from torchao.quantization import (
        Int8Tensor,
        Int8WeightOnlyConfig,
        quantize_,
    )
    from torchao.utils import get_model_size_in_bytes
     
    torch.manual_seed(0)
    model = nn.Sequential(
        nn.Linear(128, 256),
        nn.ReLU(),
        nn.Linear(256, 64),
    ).cpu().eval()
    inputs = torch.randn(32, 128)
     
    with torch.inference_mode():
        baseline = model(inputs)
    before_bytes = get_model_size_in_bytes(model)
    print(f"Float32: {before_bytes:,} bytes")

    The small, untrained network accepts 32 rows of 128 floating-point features. For a trained model, the baseline comes from its loaded weights and representative validation inputs with the same preprocessing used in deployment.
    Related: How to save and load a PyTorch model

  5. Append the quantization section after the baseline storage print in quantize_model.py.
    quantized_model = copy.deepcopy(model)
    quantize_(quantized_model, Int8WeightOnlyConfig())
    for name, layer in quantized_model.named_modules():
        if isinstance(layer, nn.Linear):
            assert isinstance(layer.weight, Int8Tensor)
            print(f"Layer {name}: {type(layer.weight).__name__}")
     
    after_bytes = get_model_size_in_bytes(quantized_model)
    assert after_bytes < before_bytes
    print(f"Int8: {after_bytes:,} bytes")
    reduction = 100 * (1 - after_bytes / before_bytes)
    print(f"Size reduction: {reduction:.1f}%")

    quantize_ modifies its argument in place, so the copy preserves the floating-point model. The default configuration stores Int8Tensor weights with per-row scales. The size utility counts the underlying tensors, including quantization metadata and unchanged biases, rather than treating the quantized wrapper as a dense float tensor. This measures tensor storage, excluding Python objects, activations, temporary allocations, and checkpoint-file overhead. Keeping both models during comparison also increases process memory.
    Related: Int8WeightOnlyConfig reference

  6. Add the output comparison at the end of quantize_model.py.
    with torch.inference_mode():
        quantized_output = quantized_model(inputs)
    max_error = (baseline - quantized_output).abs().max().item()
    print(f"Output shape: {tuple(quantized_output.shape)}")
    print(f"Maximum error: {max_error:.6f}")
    torch.testing.assert_close(
        quantized_output, baseline, rtol=0.02, atol=0.01
    )
    print("Outputs within tolerance")

    The comparison allows an absolute difference of 0.01 plus 2% of each baseline value. It raises an AssertionError if the outputs exceed that tolerance. These values suit this small network, so a trained model needs rtol and atol set from its own validation limits.

  7. Save quantize_model.py in the current directory.
  8. Run quantize_model.py to confirm smaller int8 storage and inference outputs within tolerance.
    (.venv) $ python quantize_model.py
    ##### snipped #####
    Float32: 197,888 bytes
    Layer 0: Int8Tensor
    Layer 2: Int8Tensor
    Int8: 52,032 bytes
    Size reduction: 73.7%
    Output shape: (32, 64)
    Maximum error: 0.005172
    Outputs within tolerance

    Both linear layers now hold int8 weights, storage is smaller, and the quantized model still produces a 32 × 64 output on the original inputs. The first line is a harmless _pytree warning from importing torchao, omitted here. A failed storage assertion, missing Int8Tensor layer, or failed tolerance check means the conversion has not met these acceptance conditions.

Share this guide