How to run cross validation with scikit-learn

A single train/test split can make an estimator look better or worse because of which rows happen to be held out. Cross validation reduces that dependence by training and scoring the estimator against several non-overlapping test folds.

The cross_val_score() helper fits a fresh estimator clone for every split and returns one test score per fold. Keeping StandardScaler and SVC in a Pipeline ensures that each scaler learns only from the training rows for its fold.

The iris dataset contains independent labeled rows, so a shuffled StratifiedKFold keeps all three classes represented while making the splits repeatable. Grouped observations and chronological records need splitters such as GroupKFold or TimeSeriesSplit instead; otherwise related or future rows can leak across the evaluation boundary.

Related: How to split data into train and test sets with scikit-learn
Related: How to run grid search with scikit-learn
Related: How to create a scikit-learn pipeline

Steps to run cross validation with scikit-learn:

  1. Create run_cross_validation.py with the imports and iris feature matrix.
    run_cross_validation.py
    import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
 
 
X, y = load_iris(return_X_y=True)
  2. Append the scaling and classifier pipeline after the data-loading line. 
    classifier = make_pipeline(
    StandardScaler(),
    SVC(kernel="linear", C=1.0),
)

    The pipeline fits StandardScaler separately on each fold's training rows before fitting the SVC classifier.

  3. Append the repeatable five-fold splitter after the pipeline definition. 
    cv = StratifiedKFold(
    n_splits=5,
    shuffle=True,
    random_state=42,
)

    An integer random_state preserves the same shuffled fold assignments across repeated runs.

  4. Append the accuracy-scoring call after the splitter definition. 
    scores = cross_val_score(
    classifier,
    X,
    y,
    cv=cv,
    scoring="accuracy",
    error_score="raise",
)

    error_score=“raise” stops the run when an estimator fails instead of placing a nan value in the score array.

  5. Append the fold summary after the scoring call. 
    print(
    "Fold accuracy:",
    np.array2string(scores, precision=3, floatmode="fixed"),
)
print(f"Mean accuracy: {scores.mean():.3f}")
print(f"Standard deviation: {scores.std():.3f}")
  6. Run the completed cross-validation script. 
    $ python run_cross_validation.py
Fold accuracy: [1.000 1.000 0.867 1.000 0.967]
Mean accuracy: 0.967
Standard deviation: 0.052