How to calculate permutation importance with scikit-learn

A fitted model can rely on inputs that its internal coefficients or split counts do not represent consistently across estimator types. Permutation importance measures that reliance by breaking one feature's relationship with the target and observing the score loss on data the model did not fit.

The calculation accepts any fitted estimator with a scoring method. A held-out validation set makes the ranking describe contribution to generalization, while scoring=“r2” keeps the score drop aligned with the regression model's validation metric.

Strongly correlated features can divide the apparent importance because the model may recover similar information from a feature that was not shuffled. Confirm that the estimator has predictive signal on the chosen validation set before interpreting small or negative importance values.

1. Create permutation_importance.py with the dataset imports and held-out split.

   permutation_importance.py

   from sklearn.datasets import load_diabetes
   from sklearn.inspection import permutation_importance
   from sklearn.linear_model import Ridge
   from sklearn.model_selection import train_test_split
    
    
   diabetes = load_diabetes()
   X = diabetes.data
   y = diabetes.target
   X_train, X_validation, y_train, y_validation = train_test_split(
       X,
       y,
       test_size=0.25,
       random_state=0,
   )

2. Append the fitted Ridge model and baseline validation score to permutation_importance.py.

   model = Ridge(alpha=1e-2)
   model.fit(X_train, y_train)
   baseline_r2 = model.score(X_validation, y_validation)

3. Append the held-out permutation calculation with 30 reproducible shuffles per feature.

   result = permutation_importance(
       model,
       X_validation,
       y_validation,
       scoring="r2",
       n_repeats=30,
       random_state=0,
   )

   importances_mean is the average decrease from the baseline score after shuffling a feature. importances_std records how much that decrease varies across repeats.

4. Append the stable positive-importance ranking and output formatting to permutation_importance.py.

   ranked_indices = result.importances_mean.argsort()[::-1]
   print(f"validation r2: {baseline_r2:.3f}")
   print("feature  mean drop  standard deviation")
    
   for index in ranked_indices:
       mean = result.importances_mean[index]
       standard_deviation = result.importances_std[index]
       if mean - 2 * standard_deviation > 0:
           print(
               f"{diabetes.feature_names[index]:<7} "
               f"{mean:>9.3f}  "
               f"{standard_deviation:.3f}"
           )

   The filter keeps features whose mean score decrease remains positive after subtracting twice the observed standard deviation.

5. Run permutation_importance.py to verify the validation score and ranked score decreases.

   $ python3 permutation_importance.py
   validation r2: 0.357
   feature  mean drop  standard deviation
   s5          0.204  0.050
   bmi         0.176  0.048
   bp          0.088  0.033
   sex         0.056  0.023

   The positive validation R2 establishes predictive signal for this held-out split. s5 ranks first because shuffling it causes the largest average decrease in that score.