How to tune a classifier decision threshold in scikit-learn

A binary classifier's score and the action taken from that score answer different questions. The default cut-off can miss too many positive cases when false negatives and false positives carry different costs, even when the fitted model ranks cases well.

The TunedThresholdClassifierCV meta-estimator searches candidate cut-offs with internal cross-validation and keeps the one that maximizes the selected scorer. Balanced accuracy gives the majority and minority classes equal weight here, but the production metric should reflect the real cost of each type of error.

Threshold selection stays inside the training split, and the held-out test split is evaluated only after the cut-off is chosen. This separation prevents the final report from selecting its own threshold and makes the added false positives visible beside the recovered positive cases.

1. Create tune_threshold.py with the imports and a reproducible imbalanced dataset.

   tune_threshold.py

   import sklearn
   from sklearn.datasets import make_classification
   from sklearn.ensemble import RandomForestClassifier
   from sklearn.metrics import balanced_accuracy_score, confusion_matrix
   from sklearn.model_selection import TunedThresholdClassifierCV, train_test_split
    
   X, y = make_classification(
       n_samples=1_200,
       n_features=12,
       n_informative=5,
       weights=[0.86, 0.14],
       class_sep=0.7,
       random_state=42,
   )

   TunedThresholdClassifierCV is available in scikit-learn 1.5 and later. An unavailable import indicates that the installed scikit-learn release predates this API.

2. Append a stratified training and test split to tune_threshold.py.

   X_train, X_test, y_train, y_test = train_test_split(
       X,
       y,
       stratify=y,
       test_size=0.25,
       random_state=42,
   )

   Stratification keeps the minority-class proportion similar in both partitions. The test rows remain outside model fitting and threshold selection.

3. Append the default classifier and its baseline predictions to tune_threshold.py.

   default_classifier = RandomForestClassifier(random_state=42)
   default_classifier.fit(X_train, y_train)
   default_predictions = default_classifier.predict(X_test)

4. Append the cross-validated threshold tuner and its predictions to tune_threshold.py.

   tuned_classifier = TunedThresholdClassifierCV(
       estimator=RandomForestClassifier(random_state=42),
       scoring="balanced_accuracy",
       thresholds=50,
       cv=5,
   )
   tuned_classifier.fit(X_train, y_train)
   tuned_predictions = tuned_classifier.predict(X_test)

   The scoring value defines the operating trade-off. A production scorer should reflect the real decision cost instead of treating balanced accuracy, precision, or recall as universally preferable.

5. Append held-out scores and confusion-matrix reporting to tune_threshold.py.

   print(f"scikit-learn {sklearn.__version__}")
   print(f"selected threshold: {tuned_classifier.best_threshold_:.3f}")
   print(
       "cross-validated balanced accuracy: "
       f"{tuned_classifier.best_score_:.3f}"
   )
   print(
       "default test balanced accuracy: "
       f"{balanced_accuracy_score(y_test, default_predictions):.3f}"
   )
   print(
       "tuned test balanced accuracy: "
       f"{balanced_accuracy_score(y_test, tuned_predictions):.3f}"
   )
   print("default confusion matrix:")
   print(confusion_matrix(y_test, default_predictions))
   print("tuned confusion matrix:")
   print(confusion_matrix(y_test, tuned_predictions))

6. Confirm the completed tune_threshold.py file matches the assembled program.

   tune_threshold.py

   import sklearn
   from sklearn.datasets import make_classification
   from sklearn.ensemble import RandomForestClassifier
   from sklearn.metrics import balanced_accuracy_score, confusion_matrix
   from sklearn.model_selection import TunedThresholdClassifierCV, train_test_split
    
   X, y = make_classification(
       n_samples=1_200,
       n_features=12,
       n_informative=5,
       weights=[0.86, 0.14],
       class_sep=0.7,
       random_state=42,
   )
    
   X_train, X_test, y_train, y_test = train_test_split(
       X,
       y,
       stratify=y,
       test_size=0.25,
       random_state=42,
   )
    
   default_classifier = RandomForestClassifier(random_state=42)
   default_classifier.fit(X_train, y_train)
   default_predictions = default_classifier.predict(X_test)
    
   tuned_classifier = TunedThresholdClassifierCV(
       estimator=RandomForestClassifier(random_state=42),
       scoring="balanced_accuracy",
       thresholds=50,
       cv=5,
   )
   tuned_classifier.fit(X_train, y_train)
   tuned_predictions = tuned_classifier.predict(X_test)
    
   print(f"scikit-learn {sklearn.__version__}")
   print(f"selected threshold: {tuned_classifier.best_threshold_:.3f}")
   print(
       "cross-validated balanced accuracy: "
       f"{tuned_classifier.best_score_:.3f}"
   )
   print(
       "default test balanced accuracy: "
       f"{balanced_accuracy_score(y_test, default_predictions):.3f}"
   )
   print(
       "tuned test balanced accuracy: "
       f"{balanced_accuracy_score(y_test, tuned_predictions):.3f}"
   )
   print("default confusion matrix:")
   print(confusion_matrix(y_test, default_predictions))
   print("tuned confusion matrix:")
   print(confusion_matrix(y_test, tuned_predictions))

7. Run tune_threshold.py to report the selected cut-off and held-out performance.

   $ python tune_threshold.py
   scikit-learn 1.9.0
   selected threshold: 0.153
   cross-validated balanced accuracy: 0.811
   default test balanced accuracy: 0.736
   tuned test balanced accuracy: 0.837
   default confusion matrix:
   [[251   7]
    [ 21  21]]
   tuned confusion matrix:
   [[217  41]
    [  7  35]]

   The tuned rule reduces false negatives from 21 to 7 in this run while increasing false positives from 7 to 41. The selected cut-off is suitable only when that exchange matches the cost of the two error types and remains acceptable on representative held-out data.

   Related: How to plot a confusion matrix with scikit-learn