How to enable deterministic TensorFlow operations

Repeatable TensorFlow runs matter when a model change, data-pipeline change, or hardware move needs a fair comparison. Operation determinism asks supported TensorFlow kernels to produce the same result each time they run with the same inputs, seed state, software stack, and hardware.

The process-level switch is TensorFlow's operation-determinism API. Pair it with the Keras random-seed helper near the top of the program so Python, NumPy, TensorFlow random state, layer initialization, and seeded dataset shuffling all start from the same state before training begins.

Deterministic operations are a debugging and comparison control, not a promise that every device or TensorFlow release gives identical numeric results. Unsupported deterministic kernels may raise an error, and tf.data pipelines that use stateful or random work can slow down because TensorFlow has to preserve a repeatable processing order.

1. Open a Python environment where TensorFlow imports successfully.

   $ python3 -c "import tensorflow as tf; print(tf.__version__)"
   2.21.0

   Use the same environment that runs the training job so the determinism check uses the same TensorFlow package.
   Related: How to create a virtual environment for TensorFlow
   Related: How to install TensorFlow with pip

2. Save a deterministic training smoke test as

   determinism-demo.py

   .

   determinism-demo.py

   import os
    
   os.environ["TF_CPP_MIN_LOG_LEVEL"] = "2"
    
   import numpy as np
   import tensorflow as tf
    
    
   SEED = 2026
    
   tf.keras.utils.set_random_seed(SEED)
   tf.config.experimental.enable_op_determinism()
    
    
   def train_once():
       tf.keras.backend.clear_session()
       tf.keras.utils.set_random_seed(SEED)
    
       features = np.array(
           [
               [0.0, 0.0],
               [0.0, 1.0],
               [1.0, 0.0],
               [1.0, 1.0],
               [2.0, 1.0],
               [1.0, 2.0],
               [2.0, 2.0],
               [3.0, 1.0],
           ],
           dtype=np.float32,
       )
       labels = np.array(
           [[0.0], [1.0], [1.0], [2.0], [3.0], [3.0], [4.0], [4.0]],
           dtype=np.float32,
       )
    
       dataset = (
           tf.data.Dataset.from_tensor_slices((features, labels))
           .shuffle(buffer_size=len(features), seed=SEED, reshuffle_each_iteration=False)
           .batch(4)
       )
    
       model = tf.keras.Sequential(
           [
               tf.keras.layers.Input(shape=(2,)),
               tf.keras.layers.Dense(4, activation="relu"),
               tf.keras.layers.Dense(1),
           ]
       )
       model.compile(optimizer=tf.keras.optimizers.SGD(learning_rate=0.05), loss="mse")
       model.fit(dataset, epochs=5, verbose=0, shuffle=False)
    
       weights = [weight.numpy().copy() for weight in model.weights]
       predictions = model(features, training=False).numpy()
       return weights, predictions
    
    
   first_weights, first_predictions = train_once()
   second_weights, second_predictions = train_once()
    
   weights_match = all(
       np.array_equal(left, right) for left, right in zip(first_weights, second_weights)
   )
   predictions_match = np.array_equal(first_predictions, second_predictions)
    
   print("TensorFlow:", tf.__version__)
   print("Deterministic operations enabled")
   print("Weights match:", weights_match)
   print("Predictions match:", predictions_match)
   print("Prediction sample:", np.round(first_predictions[:3, 0], 6).tolist())

   The script resets the seed before each run, enables deterministic operations before TensorFlow work starts, and uses a fixed tf.data shuffle seed.

3. Run the smoke test and confirm both comparisons return True.

   $ python3 determinism-demo.py
   TensorFlow: 2.21.0
   Deterministic operations enabled
   Weights match: True
   Predictions match: True
   Prediction sample: [0.6070209741592407, 1.6232589483261108, 1.7134809494018555]

   If a real model raises UnimplementedError after enabling determinism, the model uses an operation that TensorFlow cannot make deterministic in that environment. Replace the operation, run on another supported device path, or leave determinism disabled for that workload.

4. Add the seed and operation-determinism calls near the start of the real training program.

   import tensorflow as tf
    
   SEED = 2026
    
   tf.keras.utils.set_random_seed(SEED)
   tf.config.experimental.enable_op_determinism()
    
   # Create datasets, layers, models, and random tensors after this block.

   Call the determinism block before creating layers, loading models, building datasets, or running TensorFlow operations. Calls made after TensorFlow work starts cannot rewind earlier random choices or kernel execution.

5. Seed any dataset shuffle that should repeat between runs.

   train_dataset = train_dataset.shuffle(
       buffer_size=1000,
       seed=SEED,
       reshuffle_each_iteration=False,
   )

   tf.keras.utils.set_random_seed() covers TensorFlow random state, but explicit dataset seeds keep the input order clear when the pipeline contains random or shuffled stages.

6. Remove the temporary smoke-test script after the deterministic smoke test passes.

   $ rm determinism-demo.py