How to impute missing values with scikit-learn

Blank cells are common in tabular data collected from forms, sensors, and joined systems, but most estimators need a complete numeric feature matrix. scikit-learn can learn replacement values from known rows and apply them consistently wherever the same features are transformed.

A single replacement strategy rarely suits a mixed table. SimpleImputer can fill numeric columns with a median while a separate branch fills categorical columns with their most frequent value before OneHotEncoder converts them to model-ready features.

Fit the preprocessing object on training rows only, then call transform() on validation, test, or future rows. That boundary prevents holdout values from influencing the learned statistics and keeps feature names and column order consistent across datasets.

1. Start impute_missing_values.py with the imports and distinct training and holdout tables.

   impute_missing_values.py

   import numpy as np
   import pandas as pd
   from sklearn.compose import ColumnTransformer
   from sklearn.impute import SimpleImputer
   from sklearn.pipeline import Pipeline
   from sklearn.preprocessing import OneHotEncoder
    
    
   numeric_features = ["age", "monthly_spend"]
   categorical_features = ["plan"]
    
   train = pd.DataFrame(
       {
           "age": [25, 32, np.nan, 45],
           "monthly_spend": [120, 210, 180, np.nan],
           "plan": ["basic", "premium", "basic", np.nan],
       }
   )
    
   holdout = pd.DataFrame(
       {
           "age": [np.nan, 42],
           "monthly_spend": [190, np.nan],
           "plan": [np.nan, "premium"],
       }
   )

2. Append numeric median imputation and categorical most-frequent imputation to impute_missing_values.py.

   numeric_imputer = SimpleImputer(strategy="median")
   categorical_pipeline = Pipeline(
       steps=[
           ("imputer", SimpleImputer(strategy="most_frequent")),
           ("encoder", OneHotEncoder(handle_unknown="ignore", sparse_output=False)),
       ]
   )

   Setting add_indicator=True adds missingness as another feature. An indicator is created only for a feature that contains missing values during fit().

3. Append a ColumnTransformer that sends each feature group through its matching branch.

   preprocessor = ColumnTransformer(
       transformers=[
           ("numeric", numeric_imputer, numeric_features),
           ("categorical", categorical_pipeline, categorical_features),
       ],
       verbose_feature_names_out=False,
   )

4. Append training-only fitting and holdout transformation to impute_missing_values.py.

   train_matrix = preprocessor.fit_transform(train)
   holdout_matrix = preprocessor.transform(holdout)
   feature_names = preprocessor.get_feature_names_out()
    
   numeric_statistics = preprocessor.named_transformers_["numeric"].statistics_
   categorical_statistics = (
       preprocessor.named_transformers_["categorical"]
       .named_steps["imputer"]
       .statistics_
   )
    
   train_ready = pd.DataFrame(train_matrix, columns=feature_names).round(1)
   holdout_ready = pd.DataFrame(holdout_matrix, columns=feature_names).round(1)

5. Append learned-statistic and missing-value checks to impute_missing_values.py.

   print(f"Numeric fill values: {numeric_statistics.tolist()}")
   print(f"Categorical fill values: {categorical_statistics.tolist()}")
   print()
   print("Transformed training rows:")
   print(train_ready.to_string(index=False))
   print(f"Training missing after transform: {int(np.isnan(train_matrix).sum())}")
   print()
   print("Transformed holdout rows:")
   print(holdout_ready.to_string(index=False))
   print(f"Holdout missing after transform: {int(np.isnan(holdout_matrix).sum())}")

6. Run the completed imputation script to require zero missing values in both transformed tables.

   $ python impute_missing_values.py
   Numeric fill values: [32.0, 180.0]
   Categorical fill values: ['basic']
   
   Transformed training rows:
    age  monthly_spend  plan_basic  plan_premium
   25.0          120.0         1.0           0.0
   32.0          210.0         0.0           1.0
   32.0          180.0         1.0           0.0
   45.0          180.0         1.0           0.0
   Training missing after transform: 0
   
   Transformed holdout rows:
    age  monthly_spend  plan_basic  plan_premium
   32.0          190.0         1.0           0.0
   42.0          180.0         0.0           1.0
   Holdout missing after transform: 0