How to calibrate a scikit-learn classifier

Classifier probabilities are easiest to trust when they line up with observed event rates. In scikit-learn, calibration fits a post-processing model around a classifier so predict_proba() values can be checked as probabilities rather than only as ranking scores.

CalibratedClassifierCV trains classifier and calibrator pairs with cross-validation. The base estimator still learns the decision pattern, while the calibrator maps its score output onto probabilities using a method such as sigmoid calibration.

Use data that was not used to fit the base classifier or the calibrator when judging the result. Brier loss and log loss check probability quality, while ROC AUC can stay similar because it measures ranking rather than probability scale.

1. Create calibrate_classifier.py with a base classifier, a calibrated classifier, and held-out probability checks.

   calibrate_classifier.py

   import numpy as np
   import sklearn
   from sklearn.calibration import CalibratedClassifierCV, calibration_curve
   from sklearn.datasets import make_classification
   from sklearn.ensemble import RandomForestClassifier
   from sklearn.metrics import brier_score_loss, log_loss, roc_auc_score
   from sklearn.model_selection import train_test_split
    
    
   X, y = make_classification(
       n_samples=4_000,
       n_features=20,
       n_informative=7,
       n_redundant=5,
       weights=[0.78, 0.22],
       class_sep=0.65,
       flip_y=0.08,
       random_state=42,
   )
    
   X_train, X_test, y_train, y_test = train_test_split(
       X,
       y,
       test_size=0.25,
       stratify=y,
       random_state=42,
   )
    
   base_classifier = RandomForestClassifier(
       n_estimators=200,
       min_samples_leaf=8,
       random_state=42,
   )
   base_classifier.fit(X_train, y_train)
   base_probability = base_classifier.predict_proba(X_test)[:, 1]
    
   calibrated_classifier = CalibratedClassifierCV(
       estimator=RandomForestClassifier(
           n_estimators=200,
           min_samples_leaf=8,
           random_state=42,
       ),
       method="sigmoid",
       cv=5,
   )
   calibrated_classifier.fit(X_train, y_train)
   calibrated_probability = calibrated_classifier.predict_proba(X_test)[:, 1]
    
    
   def print_scores(name, probability):
       print(f"{name} Brier loss: {brier_score_loss(y_test, probability):.4f}")
       print(f"{name} log loss: {log_loss(y_test, probability):.4f}")
       print(f"{name} ROC AUC: {roc_auc_score(y_test, probability):.4f}")
    
    
   print(f"scikit-learn {sklearn.__version__}")
   print(f"training rows: {X_train.shape[0]}")
   print(f"test rows: {X_test.shape[0]}")
   print(f"calibration folds: {len(calibrated_classifier.calibrated_classifiers_)}")
   print()
   print_scores("uncalibrated", base_probability)
   print_scores("calibrated", calibrated_probability)
   print()
   print("calibrated probability bins:")
   fraction_positive, mean_predicted = calibration_curve(
       y_test,
       calibrated_probability,
       n_bins=5,
       strategy="quantile",
   )
    
   for bin_number, (predicted, observed) in enumerate(
       zip(mean_predicted, fraction_positive),
       start=1,
   ):
       print(
           f"bin {bin_number}: "
           f"mean predicted={predicted:.3f}, "
           f"observed positive rate={observed:.3f}"
       )
    
   print()
   print("first five calibrated probabilities:")
   print(np.array2string(calibrated_probability[:5], precision=3, floatmode="fixed"))

   CalibratedClassifierCV uses stratified 5-fold cross-validation by default for binary and multiclass labels when cv is an integer or left unset. The explicit cv=5 keeps that split count visible in the output.

2. Run the calibration script.

   $ python3 calibrate_classifier.py
   scikit-learn 1.9.0
   training rows: 3000
   test rows: 1000
   calibration folds: 5
   
   uncalibrated Brier loss: 0.1076
   uncalibrated log loss: 0.3655
   uncalibrated ROC AUC: 0.8831
   calibrated Brier loss: 0.0949
   calibrated log loss: 0.3281
   calibrated ROC AUC: 0.8782
   
   calibrated probability bins:
   ##### snipped #####
   
   first five calibrated probabilities:
   [0.122 0.083 0.057 0.198 0.079]

3. Compare the probability losses before keeping the calibrated classifier.

   The calibrated run lowers Brier loss from 0.1076 to 0.0949 and log loss from 0.3655 to 0.3281 on held-out rows. ROC AUC moves from 0.8831 to 0.8782 because calibration can change probability scale without improving class ranking.

4. Check the calibrated probability bins.

   calibrated probability bins:
   bin 1: mean predicted=0.040, observed positive rate=0.050
   bin 2: mean predicted=0.072, observed positive rate=0.065
   bin 3: mean predicted=0.124, observed positive rate=0.090
   bin 4: mean predicted=0.250, observed positive rate=0.210
   bin 5: mean predicted=0.785, observed positive rate=0.815

   Closer predicted and observed rates indicate better calibration for that bin. Increase n_bins only when the held-out set is large enough to leave enough rows in each bin.

5. Confirm that the fitted calibrated classifier returns positive-class probabilities.

   first five calibrated probabilities:
   [0.122 0.083 0.057 0.198 0.079]

   Use calibrated_classifier.predict_proba(new_rows)[:, 1] for downstream probability thresholds after held-out Brier loss and log loss support the calibrated model.
   Related: How to tune a classifier decision threshold in scikit-learn