How to evaluate a Keras binary classifier with a confusion matrix

Aggregate accuracy can hide whether a classifier is missing positive cases or over-flagging negative cases. A confusion matrix turns Keras predictions and holdout labels into class-level counts, so binary classifier review can separate false alarms from missed positives.

Use the same compiled model and preprocessed holdout arrays used for normal model.evaluate() checks. The evaluation call reports aggregate loss and metrics, while model.predict() produces the per-sample scores that become predicted class labels.

The binary setup assumes integer labels 0 and 1 plus a sigmoid output where scores at or above 0.5 count as class 1. Multiclass softmax models need argmax(axis=1) and a full class-by-class matrix instead of TN, FP, FN, and TP unpacking.

1. Save the evaluation script next to the saved classifier and holdout arrays.

   evaluate_confusion_matrix.py

   import numpy as np
   import keras
    
   model = keras.models.load_model("classifier.keras")
   x_test = np.load("x-test.npy")
   y_test = np.load("y-test.npy").astype("int32").reshape(-1)
    
   metrics = model.evaluate(x_test, y_test, verbose=0, return_dict=True)
    
   probabilities = model.predict(x_test, verbose=0).reshape(-1)
   y_pred = (probabilities >= 0.5).astype("int32")
    
   matrix = np.zeros((2, 2), dtype="int32")
   np.add.at(matrix, (y_test, y_pred), 1)
   tn, fp, fn, tp = matrix.ravel()
    
   print(f"loss: {metrics['loss']:.4f}")
   print(f"accuracy: {metrics['accuracy']:.4f}")
   print(f"predicted labels: {y_pred.tolist()}")
   print("confusion matrix rows=true columns=predicted labels=[0, 1]")
   print(matrix)
   print(f"tn={tn} fp={fp} fn={fn} tp={tp}")
   print(f"checked samples: {matrix.sum()} of {len(y_test)}")

   Replace the filenames with the saved classifier and holdout arrays from the project. Keep the same preprocessing and label encoding used for validation or test evaluation.

2. Run the script against the holdout set.

   $ python evaluate_confusion_matrix.py
   loss: 0.2738
   accuracy: 0.8333
   predicted labels: [0, 0, 1, 1, 1, 1]
   confusion matrix rows=true columns=predicted labels=[0, 1]
   [[2 1]
    [0 3]]
   tn=2 fp=1 fn=0 tp=3
   checked samples: 6 of 6

   model.evaluate() and model.predict() must use the same x_test and y_test sample order. A shuffled or filtered prediction array makes the matrix meaningless even when the shapes still match.

3. Read rows as true labels and columns as predicted labels.

   With labels 0 and 1, the top-left cell is TN, the top-right cell is FP, the bottom-left cell is FN, and the bottom-right cell is TP.

4. Confirm that the matrix total matches the holdout label count.

   If the last line does not match the number of labels in y-test.npy, check for shuffled features, filtered labels, or a generator that yields a different sample set.