How to run randomized search with scikit-learn

A randomized hyperparameter search trades exhaustive coverage for a fixed number of sampled candidates. That budget is especially useful for continuous SVC parameters such as C and gamma, where a small hand-written grid can overemphasize a few arbitrary values.

The estimator is a StandardScaler-to-SVC Pipeline, which keeps scaling inside each cross-validation training fold. Log-uniform distributions give each order of magnitude equal sampling weight, while nested parameter names target the classifier without separating it from preprocessing.

A stratified split reserves rows from every search decision. Five shuffled folds rank eight training-set candidates by mean accuracy; only after that ranking is inspected does a separate verifier score the refitted winner against the held-out rows.

1. Choose the training boundary and estimator pipeline in run_randomized_search.py.

   run_randomized_search.py

   from joblib import dump
   from scipy.stats import loguniform
   from sklearn.datasets import load_iris
   from sklearn.model_selection import RandomizedSearchCV, StratifiedKFold, train_test_split
   from sklearn.pipeline import Pipeline
   from sklearn.preprocessing import StandardScaler
   from sklearn.svm import SVC
    
    
   X, y = load_iris(return_X_y=True)
   X_train, X_test, y_train, y_test = train_test_split(
       X,
       y,
       test_size=0.25,
       stratify=y,
       random_state=42,
   )
    
   pipeline = Pipeline(
       [
           ("scaler", StandardScaler()),
           ("svc", SVC()),
       ]
   )

   The test split stays outside RandomizedSearchCV so it cannot influence candidate selection. Scaling remains inside the pipeline and is fitted separately for each training fold.

2. Define log-uniform C and gamma distributions beneath the pipeline.

   param_distributions = {
       "svc__C": loguniform(1e-2, 1e2),
       "svc__gamma": loguniform(1e-4, 1e0),
       "svc__kernel": ["rbf"],
   }

   The svc__ prefix addresses the named classifier step. Each log-uniform range samples across several orders of magnitude without enumerating a fixed grid.

3. Configure a five-fold stratified accuracy search with an eight-candidate budget.

   cv = StratifiedKFold(
       n_splits=5,
       shuffle=True,
       random_state=42,
   )
    
   search = RandomizedSearchCV(
       estimator=pipeline,
       param_distributions=param_distributions,
       n_iter=8,
       scoring="accuracy",
       cv=cv,
       random_state=42,
       refit=True,
   )

   n_iter=8 bounds the search at eight sampled settings. refit=True fits the selected pipeline again on all training rows after cross-validation.

4. Fit the bounded search on the training rows.

   search.fit(X_train, y_train)

5. Rank every sampled candidate through cv_results_.

   results = search.cv_results_
   ranked_indices = sorted(
       range(len(results["params"])),
       key=lambda index: results["rank_test_score"][index],
   )
    
   print("rank  mean CV accuracy  C         gamma")
   for index in ranked_indices:
       params = results["params"][index]
       print(
           f"{results['rank_test_score'][index]:>4}  "
           f"{results['mean_test_score'][index]:>16.3f}  "
           f"{params['svc__C']:>8.4f}  "
           f"{params['svc__gamma']:>8.4f}"
       )

   rank_test_score orders candidates by the configured accuracy scorer. Equal mean scores can share the same rank.

6. Persist the refitted winner and untouched rows after the ranking loop.

   dump(search.best_estimator_, "randomized_search_winner.joblib")
   dump((X_test, y_test), "randomized_search_holdout.joblib")

7. Run the bounded search to inspect its ordered candidate ranking.

   $ python3 run_randomized_search.py
   rank  mean CV accuracy  C         gamma
      1             0.965    0.3149    0.6351
      1             0.965    8.4718    0.0248
      1             0.965    2.5378    0.0680
      4             0.911   21.3683    0.0007
      5             0.588    0.0121    0.7579
      6             0.579    0.0171    0.2915
      7             0.570    0.0421    0.0004
      7             0.570    0.0534    0.0005

8. Build verify_randomized_winner.py to score the refitted winner against the reserved rows.

   verify_randomized_winner.py

   from joblib import load
   from sklearn.metrics import accuracy_score
   from sklearn.utils.validation import check_is_fitted
    
    
   winner = load("randomized_search_winner.joblib")
   X_test, y_test = load("randomized_search_holdout.joblib")
    
   check_is_fitted(winner)
   predictions = winner.predict(X_test)
   held_out_accuracy = accuracy_score(y_test, predictions)
   required_accuracy = 0.85
    
   if held_out_accuracy < required_accuracy:
       raise SystemExit(
           f"held-out accuracy {held_out_accuracy:.3f} "
           f"is below the required {required_accuracy:.3f}"
       )
    
   print(f"held-out accuracy: {held_out_accuracy:.3f}")
   print(f"required accuracy: {required_accuracy:.3f}")

   The verifier loads the already-refitted pipeline and predicts rows that were excluded from every candidate fit. The nonzero exit below the declared accuracy floor makes this check fail-capable.

9. Run the held-out verifier with the required accuracy floor.

   $ python3 verify_randomized_winner.py
   held-out accuracy: 0.921
   required accuracy: 0.850