Training a neural network on several devices requires copies of the model to receive coordinated updates. A TensorFlow distribution strategy manages those copies, and setting it before creating a Keras model lets that model use the selected devices during training.
For one machine, tf.distribute.MirroredStrategy uses the GPUs visible to TensorFlow and synchronizes their updates after each batch. Without a visible GPU, it falls back to a single CPU replica, so the same program can run before GPU access is available.
The strategy is attached when the model is created, rather than switched on for an already-built model. An installed TensorFlow 2 environment is sufficient for a small training batch; using several GPUs also requires working GPU support in that environment.
Steps to set a TensorFlow distribution strategy with MirroredStrategy:
- Create a file named strategy-demo.py in your text editor.
The Python environment needs TensorFlow installed.
Related: How to install TensorFlow with pip - Enter the device discovery and strategy initialization at the top of strategy-demo.py.
import tensorflow as tf print("Visible GPUs:", tf.config.list_physical_devices("GPU")) strategy = tf.distribute.MirroredStrategy() print("Replicas:", strategy.num_replicas_in_sync)
With two visible GPUs, MirroredStrategy(devices=[“/gpu:0”, “/gpu:1”]) restricts training to those devices. An empty GPU list means this process cannot see a GPU, even if one is installed.
Related: How to enable GPU acceleration in TensorFlow - Add model creation and compilation below the strategy initialization.
with strategy.scope(): model = tf.keras.Sequential([ tf.keras.Input(shape=(4,)), tf.keras.layers.Dense(1, kernel_initializer="zeros"), ]) model.compile(optimizer="sgd", loss="mse")
The indentation keeps variable creation inside the strategy scope. Models, optimizers, metrics, and checkpoint restores that create variables belong in that scope; entering it later does not move existing variables. Zero weights make the initial loss predictable for this one-layer model.
Related: How to compile a model in Keras - Append a training batch below the with block at the outer indentation level.
global_batch_size = 8 * strategy.num_replicas_in_sync features = tf.ones((global_batch_size, 4)) labels = tf.ones((global_batch_size, 1)) loss = model.train_on_batch(features, labels) print("Model uses strategy:", model.distribute_strategy is strategy) print("Training loss:", float(loss)) print("Optimizer steps:", int(model.optimizer.iterations.numpy()))
The global batch contains eight examples per replica. Keras handles distribution for its training methods, so this batch call can run outside the scope. A custom training loop needs strategy.run() to distribute computation.
Related: How to run a custom training loop in TensorFlow - Save strategy-demo.py.
- Run strategy-demo.py with the Python interpreter from your TensorFlow environment.
$ python strategy-demo.py Visible GPUs: [] Replicas: 1 Model uses strategy: True Training loss: 1.0 Optimizer steps: 1
The output should show Model uses strategy: True for the scoped model and Optimizer steps: 1 after one training update. A False value means the model was created under a different strategy. The zero-initialized model produces a first-batch loss of 1.0 against labels of one. A CPU-only host reports the empty GPU list and one replica shown here; multiple visible GPUs produce a longer list and a larger replica count.