How to select features with SelectKBest in scikit-learn

Feature selection reduces a wide tabular matrix to the columns that have the strongest individual relationship with the target. In scikit-learn, SelectKBest is a univariate transformer for supervised screening before fitting an estimator or comparing a smaller feature set.

SelectKBest scores each feature independently with a scoring function such as f_classif for classification. The selector learns its scores during fit(), exposes the kept column mask through get_support(), and transforms later arrays to the same selected columns.

Fit the selector only on training rows when the reduced matrix feeds model evaluation. Putting SelectKBest before the classifier in a Pipeline keeps selection and prediction tied to the same fitted state, while held-out rows are only transformed and scored.

1. Create select_kbest_features.py with a SelectKBest selector before the classifier.

   select_kbest_features.py

   import sklearn
   from sklearn.datasets import load_breast_cancer
   from sklearn.feature_selection import SelectKBest, f_classif
   from sklearn.linear_model import LogisticRegression
   from sklearn.metrics import accuracy_score
   from sklearn.model_selection import train_test_split
   from sklearn.pipeline import Pipeline
    
    
   dataset = load_breast_cancer()
   X = dataset.data
   y = dataset.target
   feature_names = dataset.feature_names
    
   X_train, X_test, y_train, y_test = train_test_split(
       X,
       y,
       test_size=0.25,
       stratify=y,
       random_state=42,
   )
    
   selector = SelectKBest(score_func=f_classif, k=8)
   classifier = LogisticRegression(max_iter=5000)
   model = Pipeline(
       steps=[
           ("select", selector),
           ("classify", classifier),
       ]
   )
    
   model.fit(X_train, y_train)
    
   fitted_selector = model.named_steps["select"]
   selected_indices = fitted_selector.get_support(indices=True)
   selected_names = fitted_selector.get_feature_names_out(feature_names)
   scores = fitted_selector.scores_[selected_indices]
   ranked_features = sorted(
       zip(selected_indices, selected_names, scores),
       key=lambda item: item[2],
       reverse=True,
   )
    
   X_train_selected = fitted_selector.transform(X_train)
   X_test_selected = fitted_selector.transform(X_test)
   y_pred = model.predict(X_test)
    
   print(f"scikit-learn {sklearn.__version__}")
   print(f"original training shape: {X_train.shape}")
   print(f"selected training shape: {X_train_selected.shape}")
   print(f"selected test shape: {X_test_selected.shape}")
   print()
   print("selected features by score:")
   for index, name, score in ranked_features:
       print(f"{index:2d}  {name:<24} score={score:.2f}")
   print()
   print(f"held-out accuracy: {accuracy_score(y_test, y_pred):.3f}")

   The selector is inside the Pipeline so fit() learns feature scores from training rows before the classifier is trained. Replace f_classif with a regression score for continuous targets, and use chi2 only for non-negative feature values such as counts.

2. Run the smoke script from a Python environment with scikit-learn installed.

   $ python3 select_kbest_features.py
   scikit-learn 1.9.0
   original training shape: (426, 30)
   selected training shape: (426, 8)
   selected test shape: (143, 8)
   
   selected features by score:
   27  worst concave points     score=736.82
    7  mean concave points      score=675.65
   22  worst perimeter          score=661.44
   20  worst radius             score=634.14
    2  mean perimeter           score=530.66
   23  worst area               score=499.12
    0  mean radius              score=491.37
    3  mean area                score=441.36
   
   held-out accuracy: 0.951

3. Confirm that the selected matrices keep eight columns.

   The breast cancer dataset starts with 30 numeric features. The training and test shapes both end with 8 columns because k=8 was used and the fitted selector transformed both splits.

4. Check the selected feature names and scores.

   get_support(indices=True) returns the original column indexes that were kept. scores_ holds the univariate score for each original feature, and equal-score ties have no guaranteed order.

5. Confirm that the held-out rows reach the classifier after selection.

   The held-out accuracy line shows that the transformed test matrix can pass through the classifier. Use the project's own validation metric before deciding whether the reduced feature set is good enough for a real model.

6. Remove the scratch script when the smoke test is complete.

   $ rm select_kbest_features.py