How to create a ColumnTransformer in scikit-learn

Mixed tabular datasets rarely need one preprocessing rule for every column. ColumnTransformer in scikit-learn applies separate transformers to selected column groups, such as scaling numeric fields while one-hot encoding categorical fields, and joins the outputs into one feature matrix.

A column transformer is most useful when it sits inside a Pipeline with the estimator. The preprocessors learn medians, scaling values, categories, and imputation values during fit() on the training data, then the same fitted state is reused for later rows before prediction.

A pandas DataFrame keeps column names available for selector lists and output feature names. The categorical branch uses handle_unknown="ignore" so an unseen city in the holdout rows does not stop prediction, and sparse_output=False keeps the small output readable.

1. Create create_column_transformer.py with numeric and categorical preprocessing branches.

   create_column_transformer.py

   import numpy as np
   import pandas as pd
   import sklearn
   from sklearn.compose import ColumnTransformer
   from sklearn.impute import SimpleImputer
   from sklearn.linear_model import LogisticRegression
   from sklearn.pipeline import Pipeline
   from sklearn.preprocessing import OneHotEncoder, StandardScaler
    
    
   train = pd.DataFrame(
       {
           "age": [25, 31, 45, 22, 36, 52, 29, 41, 33, 48, 27, 39],
           "income": [
               52000,
               72000,
               68000,
               39000,
               85000,
               58000,
               np.nan,
               91000,
               62000,
               88000,
               45000,
               76000,
           ],
           "visits": [4, 7, 6, 2, 8, 3, 5, 9, 5, np.nan, 3, 6],
           "city": [
               "London",
               "Paris",
               "New York",
               "London",
               "Paris",
               "New York",
               "London",
               "Paris",
               "New York",
               "London",
               "Paris",
               "New York",
           ],
           "plan": [
               "basic",
               "premium",
               "standard",
               "basic",
               "premium",
               "standard",
               "standard",
               "premium",
               "standard",
               "premium",
               "basic",
               "premium",
           ],
           "converted": [0, 1, 1, 0, 1, 0, 0, 1, 0, 1, 0, 1],
       }
   )
    
   holdout = pd.DataFrame(
       {
           "age": [30, 50, 34],
           "income": [64000, 56000, 79000],
           "visits": [5, 4, 7],
           "city": ["Rome", "London", "Paris"],
           "plan": ["premium", "basic", "standard"],
       }
   )
    
   numeric_features = ["age", "income", "visits"]
   categorical_features = ["city", "plan"]
    
   numeric_transformer = Pipeline(
       steps=[
           ("imputer", SimpleImputer(strategy="median")),
           ("scaler", StandardScaler()),
       ]
   )
    
   categorical_transformer = Pipeline(
       steps=[
           ("imputer", SimpleImputer(strategy="most_frequent")),
           ("encoder", OneHotEncoder(handle_unknown="ignore", sparse_output=False)),
       ]
   )
    
   preprocessor = ColumnTransformer(
       transformers=[
           ("numeric", numeric_transformer, numeric_features),
           ("categorical", categorical_transformer, categorical_features),
       ],
       remainder="drop",
   )
    
   model = Pipeline(
       steps=[
           ("preprocess", preprocessor),
           ("classifier", LogisticRegression(max_iter=1000)),
       ]
   )
    
   X_train = train.drop(columns="converted")
   y_train = train["converted"]
   X_holdout = holdout
    
   model.fit(X_train, y_train)
    
   transformed_holdout = model.named_steps["preprocess"].transform(X_holdout)
   feature_names = model.named_steps["preprocess"].get_feature_names_out()
   predictions = model.predict(X_holdout)
    
   print(f"scikit-learn {sklearn.__version__}")
   print(
       "Transformed holdout shape: "
       f"{transformed_holdout.shape[0]} rows x {transformed_holdout.shape[1]} columns"
   )
   print()
   print("Feature names:")
   for name in feature_names:
       print(f"- {name}")
   print()
   print("Holdout categories:")
   print(X_holdout[["city", "plan"]].to_string(index=False))
   print()
   print(f"Predictions: {predictions.tolist()}")
   print(f"Pipeline accepted {len(X_holdout)} holdout rows")

   The transformers list uses ("name", transformer, columns) tuples. The names become feature-name prefixes and make nested parameters addressable later.

2. Run the transformer smoke test.

   $ python create_column_transformer.py
   scikit-learn 1.9.0
   Transformed holdout shape: 3 rows x 9 columns
   
   Feature names:
   - numeric__age
   - numeric__income
   - numeric__visits
   - categorical__city_London
   - categorical__city_New York
   - categorical__city_Paris
   - categorical__plan_basic
   - categorical__plan_premium
   - categorical__plan_standard
   
   Holdout categories:
     city     plan
     Rome  premium
   London    basic
    Paris standard
   
   Predictions: [0, 0, 1]
   Pipeline accepted 3 holdout rows

3. Check that the transformed feature names contain both preprocessing branches.

   The three numeric columns come from the imputed and scaled numeric branch. The six categorical columns come from the fitted one-hot encoder categories.

4. Verify that the held-out rows reach the classifier after transformation.

   Rome is not present in the fitted city categories, but handle_unknown="ignore" lets that row pass through the categorical encoder before prediction.

5. Remove the scratch file when the smoke test is complete.

   $ rm create_column_transformer.py