Are You Still Using Grid Search for Hyperparameters Optimization?

A five-hyperparameter grid can quietly turn into 9,600 model fits and an afternoon of waiting. In this post I put grid search, random search, and evolutionary search with GASearchCV side by side — with runnable code — so you can decide when each one is worth your compute budget.

The Problem with Grid Search

Grid search is the first method everyone learns. The idea is simple: define a grid of values for each hyperparameter, and the algorithm tries them all. It's exhaustive, reproducible, and easy to understand.

But it has a fundamental problem: combinatorial explosion.

Imagine you're tuning a GradientBoostingClassifier with these hyperparameters:

param_grid = {
    "n_estimators": [100, 200, 300, 500],          # 4 values
    "max_depth": [2, 3, 4, 5, 6, 8],               # 6 values
    "learning_rate": [0.01, 0.05, 0.1, 0.2, 0.3], # 5 values
    "min_samples_split": [2, 5, 10, 20],            # 4 values
    "max_features": [0.5, 0.7, 0.8, 1.0],          # 4 values
}

Total combinations: 4 × 6 × 5 × 4 × 4 = 1,920 combinations. With 5-fold cross-validation, that's 9,600 model fits. If each one takes 2 seconds, the full search takes more than 5 hours.

Add one more hyperparameter with 3 values? Now you have 5,760 combinations. The time nearly triples.

This is the nature of grid search: it grows exponentially with the number of hyperparameters. It simply does not scale to complex models.

Random Search: An Improvement, But Still Blind

In 2012, Bergstra and Bengio showed in an influential paper that random search systematically outperforms grid search given the same evaluation budget. The intuition is elegant: in most problems, not all hyperparameters matter equally. Random search allows you to explore more unique values of each hyperparameter with the same number of evaluations.

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint, uniform

rs = RandomizedSearchCV(
    estimator=GradientBoostingClassifier(),
    param_distributions={
        "n_estimators": randint(100, 500),
        "max_depth": randint(2, 8),
        "learning_rate": uniform(0.01, 0.29),
        "min_samples_split": randint(2, 20),
        "max_features": uniform(0.5, 0.5),
    },
    n_iter=100,   # Only 100 evaluations, not 1,920
    cv=5,
    n_jobs=-1,
    random_state=42,
)
rs.fit(X_train, y_train)

Random search is clearly better than grid search in large spaces. But it has a critical limitation: it does not learn from previous results. Each evaluation is independent. The algorithm doesn't know that learning_rate=0.01 with n_estimators=100 performed poorly, and might explore that region again.

This leads to a natural question: can we run a search that learns as it searches?

Evolutionary Optimization with GASearchCV

sklearn-genetic-opt implements exactly that. Instead of a blind search, it uses genetic algorithms to guide exploration toward the most promising regions of the hyperparameter space. (If you're new to the library, the step-by-step walkthrough in Tune Your Scikit-learn Model Using Evolutionary Algorithms covers the full API.)

The key conceptual insight: if learning_rate=0.1 works well, it makes sense to explore values near 0.1, not go back and try 0.01. Genetic algorithms do this naturally through inheritance and variation.

from sklearn_genetic import GASearchCV, EvolutionConfig, RuntimeConfig
from sklearn_genetic.space import Integer, Continuous

search = GASearchCV(
    estimator=GradientBoostingClassifier(),
    param_grid={
        "n_estimators": Integer(100, 500),
        "max_depth": Integer(2, 8),
        "learning_rate": Continuous(0.01, 0.3),
        "min_samples_split": Integer(2, 20),
        "max_features": Continuous(0.5, 1.0),
    },
    cv=5,
    scoring="accuracy",
    evolution_config=EvolutionConfig(
        population_size=30,
        generations=25,
        crossover_probability=0.8,
        mutation_probability=0.1,
        elitism=True,
        keep_top_k=3,
    ),
    runtime_config=RuntimeConfig(n_jobs=-1, use_cache=True),
)

Notice that Continuous(0.01, 0.3) for learning_rate is fundamentally richer than any discrete list: it explores the continuous space of values, not just the points you manually defined.

Direct Comparison: Grid Search vs GASearchCV

Let's put all three methods side by side on the same dataset (make_classification with 1,000 samples and 20 features):

from sklearn.datasets import make_classification
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import (
    train_test_split, GridSearchCV, RandomizedSearchCV
)
from sklearn_genetic import GASearchCV, EvolutionConfig, RuntimeConfig, ConsecutiveStopping
from sklearn_genetic.space import Integer, Continuous
from scipy.stats import randint, uniform
import time

# Dataset
X, y = make_classification(n_samples=1000, n_features=20, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# ── 1. Grid Search ──────────────────────────────────────────
t0 = time.time()
gs = GridSearchCV(
    GradientBoostingClassifier(random_state=42),
    param_grid={
        "n_estimators": [100, 200, 300],
        "max_depth": [2, 3, 5],
        "learning_rate": [0.05, 0.1, 0.2],
    },
    cv=5, n_jobs=-1
)
gs.fit(X_train, y_train)
gs_time = time.time() - t0
print(f"Grid Search   | CV: {gs.best_score_:.4f} | Test: {gs.best_estimator_.score(X_test, y_test):.4f} | Time: {gs_time:.1f}s")

# ── 2. Random Search ────────────────────────────────────────
t0 = time.time()
rs = RandomizedSearchCV(
    GradientBoostingClassifier(random_state=42),
    param_distributions={
        "n_estimators": randint(100, 300),
        "max_depth": randint(2, 5),
        "learning_rate": uniform(0.05, 0.15),
    },
    n_iter=50, cv=5, n_jobs=-1, random_state=42
)
rs.fit(X_train, y_train)
rs_time = time.time() - t0
print(f"Random Search | CV: {rs.best_score_:.4f} | Test: {rs.best_estimator_.score(X_test, y_test):.4f} | Time: {rs_time:.1f}s")

# ── 3. GASearchCV ───────────────────────────────────────────
t0 = time.time()
ga = GASearchCV(
    GradientBoostingClassifier(random_state=42),
    param_grid={
        "n_estimators": Integer(100, 300),
        "max_depth": Integer(2, 5),
        "learning_rate": Continuous(0.05, 0.2),
    },
    cv=5, scoring="accuracy",
    evolution_config=EvolutionConfig(population_size=20, generations=20, elitism=True, keep_top_k=3),
    runtime_config=RuntimeConfig(n_jobs=-1, use_cache=True),
)
ga.fit(X_train, y_train, callbacks=[ConsecutiveStopping(generations=5, metric="fitness")])
ga_time = time.time() - t0
print(f"GASearchCV    | CV: {ga.best_score_:.4f} | Test: {ga.best_estimator_.score(X_test, y_test):.4f} | Time: {ga_time:.1f}s")

Typical results show that GASearchCV reaches an equal or higher score than grid search, with far fewer evaluations, thanks to early stopping and intelligent convergence.

Elitism and keep_top_k: Preserving the Best Genes

One risk with genetic algorithms is genetic regression: through unlucky mutations, weaker individuals might displace strong ones. The elitism=True parameter prevents this.

With keep_top_k=3, the 3 best individuals from each generation pass directly to the next without undergoing crossover or mutation. This guarantees the best solution found so far is never lost.

evolution_config = EvolutionConfig(
    population_size=30,
    generations=25,
    elitism=True,
    keep_top_k=3,        # The top 3 survive unchanged into the next generation
    mutation_probability=0.15,
    crossover_probability=0.85,
)

A keep_top_k value between 1 and 5 usually works well. If it's too high (say, 15 out of 30), the population loses diversity and the algorithm converges prematurely to a local optimum.

When Evolutionary Search Shines

Genetic algorithms are not the optimal solution in every scenario. Here's a practical guide:

Use GASearchCV when:

• The hyperparameter space has more than 3–4 dimensions.
• Some hyperparameters are continuous (learning rates, regularization strengths, dropout rates).
• Each evaluation is computationally expensive (large models, large datasets, many folds).
• You want to explore the space more intelligently than random search.
• You need early stopping to control your compute budget.

Consider grid search when:

• The space is small and discrete (2–3 hyperparameters, 3–5 values each).
• Exact reproducibility is critical and you need to document every evaluated combination.
• The cost per evaluation is low and exhaustiveness is affordable.

The rule of thumb: if the total number of combinations in your grid search exceeds 200–300, random search or GASearchCV are almost always better choices.

Conclusion and Next Steps

Grid search was the standard tool for years, and it remains valid for small spaces. But as models become more complex and hyperparameter spaces grow, its limitations become intolerable.

sklearn-genetic-opt offers a mature, well-integrated alternative for scikit-learn:

1. Install with pip install sklearn-genetic-opt.
2. Replace GridSearchCV with GASearchCV using the same estimator and scoring.
3. Define your param_grid with Integer, Continuous, and Categorical.
4. Configure EvolutionConfig with elitism=True and a couple of callbacks for early stopping.
5. Analyze cv_results_ to understand how the search evolved.

The next level is combining this with GAFeatureSelectionCV to simultaneously optimize which features enter the model — see Evolutionary Feature Selection for Machine Learning.

If you're still using exhaustive grid search for models with many hyperparameters, now is the right time to upgrade your workflow.

GASearchCV is part of sklearn-genetic-opt, an open-source scikit-learn-compatible library for evolutionary hyperparameter optimization.