How to run grid search with scikit-learn

Hyperparameter search is most useful when the candidate space is small enough to examine completely. GridSearchCV fits every declared combination across cross-validation splits, making the computation predictable and the selected settings traceable.

A Pipeline keeps StandardScaler inside each training fold, so the scaler never learns from validation rows. Pipeline parameters use the step name followed by a double underscore, which gives grid keys such as svc__C and svc__gamma.

The breast cancer dataset provides a compact binary-classification run with a separate test split. Cross-validation selects the parameters from the training rows, while the held-out accuracy checks predictions from the refitted pipeline on rows that did not participate in the search.

1. Create run_grid_search.py with the dataset-loading section.

   run_grid_search.py

   import sklearn
   from sklearn.datasets import load_breast_cancer
   from sklearn.metrics import accuracy_score
   from sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split
   from sklearn.pipeline import make_pipeline
   from sklearn.preprocessing import StandardScaler
   from sklearn.svm import SVC
    
    
   X, y = load_breast_cancer(return_X_y=True)
    
   X_train, X_test, y_train, y_test = train_test_split(
       X,
       y,
       test_size=0.25,
       stratify=y,
       random_state=42,
   )

   The stratified split keeps the class proportions comparable while reserving 25 percent of the rows for the final prediction check.

2. Append the model-selection space to run_grid_search.py.

   model = make_pipeline(
       StandardScaler(),
       SVC(),
   )
    
   param_grid = {
       "svc__C": [0.1, 1, 10],
       "svc__gamma": ["scale", 0.01],
       "svc__kernel": ["rbf"],
   }

   Three C values multiplied by two gamma values and one kernel produce six candidates. Keeping scaling inside the pipeline fits it separately for every training fold.

3. Append the cross-validation search configuration to run_grid_search.py.

   cv = StratifiedKFold(
       n_splits=5,
       shuffle=True,
       random_state=42,
   )
    
   search = GridSearchCV(
       estimator=model,
       param_grid=param_grid,
       scoring="accuracy",
       cv=cv,
       refit=True,
   )

   refit=True rebuilds the best pipeline on the complete training split after candidate scoring finishes.

4. Append the held-out evaluation section to run_grid_search.py.

   search.fit(X_train, y_train)
   test_predictions = search.predict(X_test)
   held_out_accuracy = accuracy_score(y_test, test_predictions)
   candidate_count = len(search.cv_results_["params"])
    
   print(f"scikit-learn: {sklearn.__version__}")
   print(f"evaluated candidates: {candidate_count}")
   print(f"cross-validation fits: {candidate_count * search.n_splits_}")
   print(f"best parameters: {search.best_params_}")
   print(f"best mean CV accuracy: {search.best_score_:.3f}")
   print(f"held-out accuracy: {held_out_accuracy:.3f}")

5. Run run_grid_search.py in the Python environment that contains scikit-learn.

   $ python run_grid_search.py
   scikit-learn: 1.9.0
   evaluated candidates: 6
   cross-validation fits: 30
   best parameters: {'svc__C': 10, 'svc__gamma': 0.01, 'svc__kernel': 'rbf'}
   best mean CV accuracy: 0.972
   held-out accuracy: 0.979

   Six candidates across five folds produce thirty fits. The final accuracy comes from predictions made by the refitted search object on the held-out rows.