How to run grid search with scikit-learn

Grid search in scikit-learn compares a fixed set of hyperparameter combinations by fitting the estimator repeatedly through cross-validation. It suits small search spaces where every candidate should be scored instead of sampled, such as choosing a few SVC values before training a final classifier.

GridSearchCV wraps the estimator and behaves like the selected estimator after fitting. Parameter names follow the estimator path, so a pipeline step named svc accepts grid keys such as svc__C and svc__gamma.

A compact breast cancer classification run keeps the grid intentionally small and holds back a separate test split. Cross-validation chooses the best parameter combination on the training rows, and the held-out score checks that the refitted estimator still predicts unseen rows after the search.

1. Save the grid-search script as run_grid_search.py.

   run_grid_search.py

   import sklearn
   from sklearn.datasets import load_breast_cancer
   from sklearn.metrics import accuracy_score
   from sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split
   from sklearn.pipeline import make_pipeline
   from sklearn.preprocessing import StandardScaler
   from sklearn.svm import SVC
    
    
   X, y = load_breast_cancer(return_X_y=True)
    
   X_train, X_test, y_train, y_test = train_test_split(
       X,
       y,
       test_size=0.25,
       stratify=y,
       random_state=42,
   )
    
   model = make_pipeline(
       StandardScaler(),
       SVC(),
   )
    
   param_grid = {
       "svc__C": [0.1, 1, 10],
       "svc__gamma": ["scale", 0.01],
       "svc__kernel": ["rbf"],
   }
    
   cv = StratifiedKFold(
       n_splits=5,
       shuffle=True,
       random_state=42,
   )
    
   search = GridSearchCV(
       model,
       param_grid=param_grid,
       scoring="accuracy",
       cv=cv,
       refit=True,
   )
   search.fit(X_train, y_train)
    
   candidate_count = len(search.cv_results_["params"])
   test_predictions = search.predict(X_test)
   held_out_accuracy = accuracy_score(y_test, test_predictions)
    
   print(f"scikit-learn {sklearn.__version__}")
   print(f"scoring: {search.scoring}")
   print(f"cross-validation folds: {search.n_splits_}")
   print(f"evaluated candidates: {candidate_count}")
   print(f"total fits: {candidate_count * search.n_splits_}")
   print(f"best parameters: {search.best_params_}")
   print(f"best mean CV accuracy: {search.best_score_:.3f}")
   print(f"held-out accuracy: {held_out_accuracy:.3f}")
   print("top candidates:")
    
   ranked_candidates = sorted(
       zip(
           search.cv_results_["rank_test_score"],
           search.cv_results_["mean_test_score"],
           search.cv_results_["params"],
       ),
       key=lambda item: item[0],
   )
    
   for rank, mean_score, params in ranked_candidates[:3]:
       print(f"  rank {rank}: mean={mean_score:.3f}, params={params}")

   The parameter grid has three C values, two gamma values, and one kernel value. GridSearchCV evaluates their cross-product, so this run has six candidate settings before cross-validation expands them across folds.

2. Run the script from a Python environment that has scikit-learn installed.

   $ python run_grid_search.py
   scikit-learn 1.9.0
   scoring: accuracy
   cross-validation folds: 5
   evaluated candidates: 6
   total fits: 30
   best parameters: {'svc__C': 10, 'svc__gamma': 0.01, 'svc__kernel': 'rbf'}
   best mean CV accuracy: 0.972
   held-out accuracy: 0.979
   top candidates:
     rank 1: mean=0.972, params={'svc__C': 10, 'svc__gamma': 0.01, 'svc__kernel': 'rbf'}
     rank 2: mean=0.970, params={'svc__C': 10, 'svc__gamma': 'scale', 'svc__kernel': 'rbf'}
     rank 3: mean=0.967, params={'svc__C': 1, 'svc__gamma': 'scale', 'svc__kernel': 'rbf'}

3. Check the evaluated candidate and total fit counts.

   The output shows six candidates and thirty total fits because each candidate is evaluated across five cross-validation folds.

   Related: How to run cross validation with scikit-learn

4. Check the best parameter set before keeping the model.

   The output selects svc__C=10, svc__gamma=0.01, and svc__kernel=rbf for the configured accuracy scorer. The top-candidate rows show how close the next two settings were.

5. Check the held-out score before reusing the refitted estimator.

   refit=True retrains the best candidate on the full training split, so search.predict(X_test) uses the selected pipeline. The held-out accuracy line checks that the refitted estimator can score rows outside the grid-search folds.

6. Replace the sample grid with project-specific values after the smoke test works.

   param_grid = {
       "svc__C": [0.5, 1, 2],
       "svc__gamma": ["scale", 0.05],
       "svc__kernel": ["rbf"],
   }

   Keep the grid bounded. Each extra candidate is evaluated once per fold, so small-looking grid additions can multiply the fit count quickly.

7. Remove the smoke-test script if it is not part of the project.

   $ rm run_grid_search.py