How to calculate permutation importance with scikit-learn

Feature rankings need to reflect how a fitted model behaves on data it did not train on, not just how its training algorithm records internal splits or coefficients. In scikit-learn, permutation importance shuffles one feature column at a time and measures how much the model score drops, which ties the result to the estimator, dataset, and scoring metric being inspected.

The inspection function works with any fitted estimator that can be scored. A held-out regression run can fit a Ridge model, score validation rows, run permutation_importance, and sort features by the mean score decrease across repeated shuffles.

Calculate importances only after the model already shows predictive signal on held-out data. A low validation score can make important inputs appear unimportant, and strongly correlated features can share signal so permuting one column causes only a small drop.

1. Install scikit-learn in the active Python environment.

   $ python -m pip install scikit-learn

   Use the same environment that will fit or load the estimator being inspected.

2. Save the smoke-test script as perm_importance.py.

   perm_importance.py

   from sklearn.datasets import load_diabetes
   from sklearn.inspection import (
       permutation_importance,
   )
   from sklearn.linear_model import Ridge
   from sklearn.model_selection import (
       train_test_split,
   )
    
    
   diabetes = load_diabetes()
   split = train_test_split(
       diabetes.data,
       diabetes.target,
       test_size=0.25,
       random_state=0,
   )
   X_train, X_val, y_train, y_val = split
    
   model = Ridge(alpha=1e-2)
   model.fit(X_train, y_train)
    
   score = model.score(X_val, y_val)
   result = permutation_importance(
       model,
       X_val,
       y_val,
       n_repeats=30,
       random_state=0,
       scoring="r2",
   )
    
   print(f"validation r2: {score:.3f}")
   print("feature  mean_drop  std")
    
   ranked = result.importances_mean.argsort()[::-1]
   for index in ranked[:5]:
       name = diabetes.feature_names[index]
       mean = result.importances_mean[index]
       std = result.importances_std[index]
       print(
           f"{name:<7} {mean:>9.3f}  "
           f"{std:.3f}"
       )
    
   top_index = result.importances_mean.argmax()
   top_name = diabetes.feature_names[top_index]
   print(f"top feature: {top_name}")

   n_repeats=30 shuffles each feature 30 times. random_state=0 makes the ranking reproducible while the script is being checked.

3. Run the script from the same directory.

   $ python perm_importance.py
   validation r2: 0.357
   feature  mean_drop  std
   s5          0.204  0.050
   bmi         0.176  0.048
   bp          0.088  0.033
   sex         0.056  0.023
   s1          0.042  0.031
   top feature: s5

4. Check the validation score before reading the feature ranking.

   The held-out R2 score is positive in this run, so the model has predictive signal to inspect. A near-zero or negative score makes the importance table weak evidence, even when one feature sorts above another.

5. Read mean_drop as the average score decrease caused by shuffling a feature.

   s5 ranks first because shuffling that column reduces held-out R2 the most. std shows how much the score drop varied across the repeated shuffles.

6. Replace the sample estimator and validation arrays with the fitted model and held-out data from the project.

   result = permutation_importance(
       fitted_model,
       X_validation,
       y_validation,
       scoring="accuracy",
       n_repeats=20,
       random_state=42,
       n_jobs=-1,
   )

   Choose scoring to match the model decision being inspected, such as accuracy for a balanced classifier or r2 for a regression model.

7. Remove the smoke-test script if it was created only for validation.

   $ rm perm_importance.py