How to tune a classifier decision threshold in scikit-learn

Binary classifiers often produce probabilities or scores before those scores become class labels. The default cut-off can be the wrong operating point when missed positive cases and false alarms have different costs.

scikit-learn provides TunedThresholdClassifierCV for post-training threshold selection. It keeps the model's score output and searches for the cut-off that optimizes a binary classification metric, so threshold tuning is a decision rule on top of the fitted classifier rather than a replacement for model selection or calibration.

Keep the threshold search separate from final evaluation. Internal cross-validation can choose the cut-off on the training data, but the accepted threshold should still be checked on held-out data because reusing the same rows for training, threshold selection, and reporting can overstate performance.

Steps to tune a scikit-learn classifier decision threshold:

Create tune_threshold.py with a binary classifier and threshold tuner.

tune_threshold.py

import sklearn
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import balanced_accuracy_score, classification_report, confusion_matrix
from sklearn.model_selection import TunedThresholdClassifierCV, train_test_split
 
X, y = make_classification(
    n_samples=1_200,
    n_features=12,
    n_informative=5,
    weights=[0.86, 0.14],
    class_sep=0.7,
    random_state=42,
)
 
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    stratify=y,
    test_size=0.25,
    random_state=42,
)
 
base_classifier = RandomForestClassifier(random_state=42)
base_classifier.fit(X_train, y_train)
 
default_predictions = base_classifier.predict(X_test)
default_score = balanced_accuracy_score(y_test, default_predictions)
 
tuned_classifier = TunedThresholdClassifierCV(
    RandomForestClassifier(random_state=42),
    scoring="balanced_accuracy",
    thresholds=50,
    cv=5,
    random_state=42,
)
tuned_classifier.fit(X_train, y_train)
 
tuned_predictions = tuned_classifier.predict(X_test)
tuned_score = balanced_accuracy_score(y_test, tuned_predictions)
 
print(f"scikit-learn {sklearn.__version__}")
print(f"default threshold balanced accuracy: {default_score:.3f}")
print(f"tuned threshold: {tuned_classifier.best_threshold_:.3f}")
print(f"cross-validated balanced accuracy: {tuned_classifier.best_score_:.3f}")
print(f"test balanced accuracy after tuning: {tuned_score:.3f}")
print()
print("Default threshold report")
print(classification_report(y_test, default_predictions, digits=3))
print("Tuned threshold report")
print(classification_report(y_test, tuned_predictions, digits=3))
print("Tuned threshold confusion matrix")
print(confusion_matrix(y_test, tuned_predictions))

TunedThresholdClassifierCV is available in scikit-learn 1.5 and later. If the import fails, upgrade scikit-learn before using this threshold-tuning API.

Run the script and confirm that it prints a tuned threshold.
```
$ python tune_threshold.py
scikit-learn 1.9.0
default threshold balanced accuracy: 0.736
tuned threshold: 0.153
cross-validated balanced accuracy: 0.811
test balanced accuracy after tuning: 0.837
##### snipped #####
```
The scoring value controls what the threshold search optimizes. Use a metric that matches the application tradeoff, such as recall when false negatives are more costly or precision when false positives are more costly.

Compare the default and tuned classification reports.

Default threshold report
              precision    recall  f1-score   support

           0      0.923     0.973     0.947       258
           1      0.750     0.500     0.600        42

    accuracy                          0.907       300
   macro avg      0.836     0.736     0.774       300
weighted avg      0.899     0.907     0.899       300

Tuned threshold report
              precision    recall  f1-score   support

           0      0.969     0.841     0.900       258
           1      0.461     0.833     0.593        42

    accuracy                          0.840       300
   macro avg      0.715     0.837     0.747       300
weighted avg      0.898     0.840     0.857       300

The tuned threshold raises positive-class recall from 0.500 to 0.833 in this run. It also lowers positive-class precision from 0.750 to 0.461, so the threshold is only better when catching more positive cases is worth the extra false positives.

Check the tuned confusion matrix before keeping the threshold.
```
Tuned threshold confusion matrix
[[217  41]
 [  7  35]]
```
The matrix shows 35 true positives and 7 false negatives after tuning. Keep the threshold when that operating point matches the review, cost, or intervention policy for the model.

Related: How to plot a confusion matrix with scikit-learn