How to tune a classifier decision threshold in scikit-learn

Binary classifiers often produce probabilities or scores before those scores become class labels. The default cut-off can be the wrong operating point when missed positive cases and false alarms have different costs.

scikit-learn provides TunedThresholdClassifierCV for post-training threshold selection. It keeps the model's score output and searches for the cut-off that optimizes a binary classification metric, so threshold tuning is a decision rule on top of the fitted classifier rather than a replacement for model selection or calibration.

Keep the threshold search separate from final evaluation. Internal cross-validation can choose the cut-off on the training data, but the accepted threshold should still be checked on held-out data because reusing the same rows for training, threshold selection, and reporting can overstate performance.

1. Create tune_threshold.py with a binary classifier and threshold tuner.

   tune_threshold.py

   import sklearn
   from sklearn.datasets import make_classification
   from sklearn.ensemble import RandomForestClassifier
   from sklearn.metrics import balanced_accuracy_score, classification_report, confusion_matrix
   from sklearn.model_selection import TunedThresholdClassifierCV, train_test_split
    
   X, y = make_classification(
       n_samples=1_200,
       n_features=12,
       n_informative=5,
       weights=[0.86, 0.14],
       class_sep=0.7,
       random_state=42,
   )
    
   X_train, X_test, y_train, y_test = train_test_split(
       X,
       y,
       stratify=y,
       test_size=0.25,
       random_state=42,
   )
    
   base_classifier = RandomForestClassifier(random_state=42)
   base_classifier.fit(X_train, y_train)
    
   default_predictions = base_classifier.predict(X_test)
   default_score = balanced_accuracy_score(y_test, default_predictions)
    
   tuned_classifier = TunedThresholdClassifierCV(
       RandomForestClassifier(random_state=42),
       scoring="balanced_accuracy",
       thresholds=50,
       cv=5,
       random_state=42,
   )
   tuned_classifier.fit(X_train, y_train)
    
   tuned_predictions = tuned_classifier.predict(X_test)
   tuned_score = balanced_accuracy_score(y_test, tuned_predictions)
    
   print(f"scikit-learn {sklearn.__version__}")
   print(f"default threshold balanced accuracy: {default_score:.3f}")
   print(f"tuned threshold: {tuned_classifier.best_threshold_:.3f}")
   print(f"cross-validated balanced accuracy: {tuned_classifier.best_score_:.3f}")
   print(f"test balanced accuracy after tuning: {tuned_score:.3f}")
   print()
   print("Default threshold report")
   print(classification_report(y_test, default_predictions, digits=3))
   print("Tuned threshold report")
   print(classification_report(y_test, tuned_predictions, digits=3))
   print("Tuned threshold confusion matrix")
   print(confusion_matrix(y_test, tuned_predictions))

   TunedThresholdClassifierCV is available in scikit-learn 1.5 and later. If the import fails, upgrade scikit-learn before using this threshold-tuning API.

2. Run the script and confirm that it prints a tuned threshold.

   $ python tune_threshold.py
   scikit-learn 1.9.0
   default threshold balanced accuracy: 0.736
   tuned threshold: 0.153
   cross-validated balanced accuracy: 0.811
   test balanced accuracy after tuning: 0.837
   ##### snipped #####

   The scoring value controls what the threshold search optimizes. Use a metric that matches the application tradeoff, such as recall when false negatives are more costly or precision when false positives are more costly.

3. Compare the default and tuned classification reports.

   Default threshold report
                 precision    recall  f1-score   support
   
              0      0.923     0.973     0.947       258
              1      0.750     0.500     0.600        42
   
       accuracy                          0.907       300
      macro avg      0.836     0.736     0.774       300
   weighted avg      0.899     0.907     0.899       300
   
   Tuned threshold report
                 precision    recall  f1-score   support
   
              0      0.969     0.841     0.900       258
              1      0.461     0.833     0.593        42
   
       accuracy                          0.840       300
      macro avg      0.715     0.837     0.747       300
   weighted avg      0.898     0.840     0.857       300

   The tuned threshold raises positive-class recall from 0.500 to 0.833 in this run. It also lowers positive-class precision from 0.750 to 0.461, so the threshold is only better when catching more positive cases is worth the extra false positives.

   Related: How to create a classification report with scikit-learn

4. Check the tuned confusion matrix before keeping the threshold.

   Tuned threshold confusion matrix
   [[217  41]
    [  7  35]]

   The matrix shows 35 true positives and 7 false negatives after tuning. Keep the threshold when that operating point matches the review, cost, or intervention policy for the model.

   Related: How to plot a confusion matrix with scikit-learn