Training a neural network on several devices requires copies of the model to receive coordinated updates. A TensorFlow distribution strategy manages those copies, and setting it before creating a Keras model lets that model use the selected devices during training.

For one machine, tf.distribute.MirroredStrategy uses the GPUs visible to TensorFlow and synchronizes their updates after each batch. Without a visible GPU, it falls back to a single CPU replica, so the same program can run before GPU access is available.

The strategy is attached when the model is created, rather than switched on for an already-built model. An installed TensorFlow 2 environment is sufficient for a small training batch; using several GPUs also requires working GPU support in that environment.

Steps to set a TensorFlow distribution strategy with MirroredStrategy:

  1. Create a file named strategy-demo.py in your text editor.

    The Python environment needs TensorFlow installed.
    Related: How to install TensorFlow with pip

  2. Enter the device discovery and strategy initialization at the top of strategy-demo.py.
    import tensorflow as tf
     
    print("Visible GPUs:", tf.config.list_physical_devices("GPU"))
    strategy = tf.distribute.MirroredStrategy()
    print("Replicas:", strategy.num_replicas_in_sync)

    With two visible GPUs, MirroredStrategy(devices=[“/gpu:0”, “/gpu:1”]) restricts training to those devices. An empty GPU list means this process cannot see a GPU, even if one is installed.
    Related: How to enable GPU acceleration in TensorFlow

  3. Add model creation and compilation below the strategy initialization.
    with strategy.scope():
        model = tf.keras.Sequential([
            tf.keras.Input(shape=(4,)),
            tf.keras.layers.Dense(1, kernel_initializer="zeros"),
        ])
        model.compile(optimizer="sgd", loss="mse")

    The indentation keeps variable creation inside the strategy scope. Models, optimizers, metrics, and checkpoint restores that create variables belong in that scope; entering it later does not move existing variables. Zero weights make the initial loss predictable for this one-layer model.
    Related: How to compile a model in Keras

  4. Append a training batch below the with block at the outer indentation level.
    global_batch_size = 8 * strategy.num_replicas_in_sync
    features = tf.ones((global_batch_size, 4))
    labels = tf.ones((global_batch_size, 1))
    loss = model.train_on_batch(features, labels)
     
    print("Model uses strategy:", model.distribute_strategy is strategy)
    print("Training loss:", float(loss))
    print("Optimizer steps:", int(model.optimizer.iterations.numpy()))

    The global batch contains eight examples per replica. Keras handles distribution for its training methods, so this batch call can run outside the scope. A custom training loop needs strategy.run() to distribute computation.
    Related: How to run a custom training loop in TensorFlow

  5. Save strategy-demo.py.
  6. Run strategy-demo.py with the Python interpreter from your TensorFlow environment.
    $ python strategy-demo.py
    Visible GPUs: []
    Replicas: 1
    Model uses strategy: True
    Training loss: 1.0
    Optimizer steps: 1

    The output should show Model uses strategy: True for the scoped model and Optimizer steps: 1 after one training update. A False value means the model was created under a different strategy. The zero-initialized model produces a first-batch loss of 1.0 against labels of one. A CPU-only host reports the empty GPU list and one replica shown here; multiple visible GPUs produce a longer list and a larger replica count.

Share this guide