How to evaluate a Keras binary classifier with a confusion matrix

Aggregate accuracy can hide whether a binary classifier misses positive cases or raises false alarms on negative cases. A confusion matrix separates those errors into true-negative, false-positive, false-negative, and true-positive counts for the same holdout set used by Keras evaluation.

The saved model, /x-test.npy features, and /y-test.npy labels must use the same preprocessing and sample order. The script loads the model without its training configuration, then compiles it with binary cross-entropy and accuracy so the evaluation settings remain explicit; projects that used another binary loss or metric should use that matching configuration.

The binary evaluation assumes integer labels 0 and 1 plus one sigmoid score per sample, with 0.5 as the class threshold. Rows represent true labels and columns represent predicted labels; a fixed labels=[0, 1] order lets the custom matrix be checked against scikit-learn without changing cell positions. Multiclass softmax output needs argmax(axis=1) and a larger matrix instead.

1. Create /evaluate_confusion_matrix.py with the model, evaluation configuration, and holdout arrays.

   evaluate_confusion_matrix.py

   import keras
   import numpy as np
   from sklearn.metrics import confusion_matrix
    
   model = keras.saving.load_model("classifier.keras", compile=False)
   model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
   x_test = np.load("x-test.npy")
   y_test = np.load("y-test.npy").astype("int32").reshape(-1)

   The filenames and evaluation configuration must match the saved classifier and its holdout data.

2. Append the evaluation and threshold conversion below the holdout array loading code.

   metrics = model.evaluate(x_test, y_test, verbose=0, return_dict=True)
   probabilities = model.predict(x_test, verbose=0).reshape(-1)
   y_pred = (probabilities >= 0.5).astype("int32")

3. Append the matrix accumulation below the predicted-label conversion.

   matrix = np.zeros((2, 2), dtype="int32")
   np.add.at(matrix, (y_test, y_pred), 1)
   tn, fp, fn, tp = matrix.ravel()

4. Append the independent reference check and reporting block below the matrix unpacking.

   reference_matrix = confusion_matrix(y_test, y_pred, labels=[0, 1])
   matches_reference = np.array_equal(matrix, reference_matrix)
    
   print(f"loss: {metrics['loss']:.4f}")
   print(f"accuracy: {metrics['accuracy']:.4f}")
   print("confusion matrix rows=true columns=predicted labels=[0, 1]")
   print(matrix)
   print(f"tn={tn} fp={fp} fn={fn} tp={tp}")
   print(f"matches sklearn reference: {matches_reference}")
    
   if not matches_reference:
       raise RuntimeError("custom matrix differs from sklearn reference")

5. Compare the assembled script with this complete file.

   evaluate_confusion_matrix.py

   import keras
   import numpy as np
   from sklearn.metrics import confusion_matrix
    
   model = keras.saving.load_model("classifier.keras", compile=False)
   model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
   x_test = np.load("x-test.npy")
   y_test = np.load("y-test.npy").astype("int32").reshape(-1)
    
   metrics = model.evaluate(x_test, y_test, verbose=0, return_dict=True)
   probabilities = model.predict(x_test, verbose=0).reshape(-1)
   y_pred = (probabilities >= 0.5).astype("int32")
    
   matrix = np.zeros((2, 2), dtype="int32")
   np.add.at(matrix, (y_test, y_pred), 1)
   tn, fp, fn, tp = matrix.ravel()
   reference_matrix = confusion_matrix(y_test, y_pred, labels=[0, 1])
   matches_reference = np.array_equal(matrix, reference_matrix)
    
   print(f"loss: {metrics['loss']:.4f}")
   print(f"accuracy: {metrics['accuracy']:.4f}")
   print("confusion matrix rows=true columns=predicted labels=[0, 1]")
   print(matrix)
   print(f"tn={tn} fp={fp} fn={fn} tp={tp}")
   print(f"matches sklearn reference: {matches_reference}")
    
   if not matches_reference:
       raise RuntimeError("custom matrix differs from sklearn reference")

6. Run the completed evaluation script beside the saved model and holdout arrays.

   $ python evaluate_confusion_matrix.py
   loss: 0.4488
   accuracy: 0.6667
   confusion matrix rows=true columns=predicted labels=[0, 1]
   [[2 1]
    [1 2]]
   tn=2 fp=1 fn=1 tp=2
   matches sklearn reference: True

7. Read the matrix rows as true labels and columns as predicted labels.

   The displayed matrix contains two true negatives, one false positive, one false negative, and two true positives.

8. Confirm that the custom matrix matches the scikit-learn reference matrix for the fixed label order.

   A False result raises an error because the custom accumulation no longer agrees with the independent reference implementation.