Tune Your Scikit-learn Model Using Evolutionary Algorithms

If you already use GridSearchCV, you can switch to evolutionary hyperparameter tuning by changing a handful of lines. In this guide you'll set up GASearchCV from sklearn-genetic-opt end to end: typed search spaces, EvolutionConfig, adaptive probabilities, and early-stopping callbacks — all on a real scikit-learn model.

Why Genetic Algorithms for Hyperparameter Tuning?

Tuning hyperparameters is one of the most time-consuming — and most important — steps in machine learning. The gap between a mediocre model and a production-ready one often comes down to this fine-tuning process.

The classic approach, grid search, evaluates every possible combination of hyperparameters. It works well in small spaces, but it grows exponentially: 5 hyperparameters with 10 values each gives you 100,000 evaluations. That's simply not viable — I break down exactly how badly it scales in Are You Still Using Grid Search for Hyperparameters Optimization?

Random search improves on this by sampling combinations at random. It's more efficient, but it's still "blind" — it learns nothing from previous results.

Genetic algorithms change the game entirely. They work exactly like natural evolution:

1. An initial population of hyperparameter configurations (individuals) is created.
2. Each individual is evaluated — its "fitness" is the cross-validation score.
3. The best individuals are selected to reproduce.
4. Crossover combines two parent individuals to create offspring.
5. Mutation introduces random variations.
6. The process repeats for several generations.

The result: the algorithm converges toward regions of the hyperparameter space where performance is high — intelligently and efficiently.

Installation and Quick Start

Installing sklearn-genetic-opt is straightforward:

pip install sklearn-genetic-opt

The main interface is GASearchCV, which mirrors the API of scikit-learn's GridSearchCV and RandomizedSearchCV. If you already use those classes, the learning curve is minimal.

Defining the Search Space

Unlike grid search (which requires discrete value lists), sklearn-genetic-opt works with typed search spaces:

from sklearn_genetic.space import Integer, Continuous, Categorical

param_grid = {
    "n_estimators": Integer(50, 300),        # Integer between 50 and 300
    "max_depth": Integer(2, 15),             # Integer between 2 and 15
    "max_features": Continuous(0.3, 1.0),   # Continuous float between 0.3 and 1.0
    "criterion": Categorical(["gini", "entropy", "log_loss"]),  # Discrete choice
}

• Integer(low, high): Samples integers in the range [low, high].
• Continuous(low, high): Samples floats in the range [low, high].
• Categorical(choices): Samples from a discrete list of options.

The distinction matters: Continuous allows exploring values that no predefined list would cover — a genuine advantage over grid search.

Fitting the Model with GASearchCV

Here is a complete workflow using RandomForestClassifier on the Iris dataset:

from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn_genetic import GASearchCV, EvolutionConfig, PopulationConfig, RuntimeConfig
from sklearn_genetic.space import Integer, Continuous, Categorical

# Load data
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Base estimator
clf = RandomForestClassifier(random_state=42)

# Hyperparameter space
param_grid = {
    "n_estimators": Integer(50, 300),
    "max_depth": Integer(2, 15),
    "max_features": Continuous(0.3, 1.0),
    "criterion": Categorical(["gini", "entropy", "log_loss"]),
}

# Configure the evolutionary search
search = GASearchCV(
    estimator=clf,
    param_grid=param_grid,
    cv=5,
    scoring="accuracy",
    evolution_config=EvolutionConfig(
        population_size=30,
        generations=25,
        crossover_probability=0.8,
        mutation_probability=0.1,
        elitism=True,
        keep_top_k=3,
    ),
    population_config=PopulationConfig(initializer="smart"),
    runtime_config=RuntimeConfig(n_jobs=-1, use_cache=True),
)

# Fit
search.fit(X_train, y_train)

# Results
print("Best parameters:", search.best_params_)
print("Best CV score:", search.best_score_)

Interpreting the Results

After fitting, GASearchCV exposes the same attributes as GridSearchCV:

# The best hyperparameter set found
print(search.best_params_)
# {'n_estimators': 187, 'max_depth': 8, 'max_features': 0.74, 'criterion': 'gini'}

# The best cross-validation score
print(search.best_score_)
# 0.9667

# The best estimator, already fitted on all training data
best_model = search.best_estimator_
test_score = best_model.score(X_test, y_test)
print(f"Test accuracy: {test_score:.4f}")

# Full history of all evaluated individuals across generations
import pandas as pd
cv_results = pd.DataFrame(search.cv_results_)
print(cv_results.head())

The cv_results_ attribute is particularly valuable: it contains the full history of every individual evaluated across every generation, enabling you to visualize how the population evolves over time.

Fine-Grained Control with EvolutionConfig

EvolutionConfig is the heart of the genetic algorithm configuration. Its most important parameters:

from sklearn_genetic import EvolutionConfig

evolution_config = EvolutionConfig(
    population_size=30,         # Number of individuals per generation
    generations=25,             # Number of generations to run
    crossover_probability=0.8,  # Probability that two parents will crossover
    mutation_probability=0.1,   # Probability of mutation per gene
    elitism=True,               # Best individuals carry over to the next generation
    keep_top_k=3,               # How many top individuals to preserve with elitism
)

Practical tips:

• population_size: A value between 20 and 50 is sufficient for most problems. More individuals means more diversity but more computation time.
• generations: The algorithm converges faster with a larger population. Start with 20–30 generations and increase if you see continued improvement.
• elitism=True: Almost always worth enabling. It guarantees the best solution found is never lost between generations.
• crossover_probability and mutation_probability: The defaults (0.8 and 0.1) are a solid starting point. Higher mutation favors exploration; lower mutation favors exploitation.

Adaptive Probabilities

For long searches, you can use ExponentialAdapter to decrease mutation probability as evolution progresses — explore early, exploit late:

from sklearn_genetic import ExponentialAdapter

search = GASearchCV(
    estimator=clf,
    param_grid=param_grid,
    mutation_probability=ExponentialAdapter(
        initial_value=0.3,
        end_value=0.05,
        adaptive_rate=0.1
    ),
)

Early Stopping with Callbacks

On large datasets, each generation can take minutes. Callbacks let you stop the search early when certain conditions are met:

from sklearn_genetic import ThresholdStopping, ConsecutiveStopping

callbacks = [
    # Stop if fitness exceeds the threshold 0.97
    ThresholdStopping(threshold=0.97, metric="fitness"),

    # Stop if there's no improvement for 5 consecutive generations
    ConsecutiveStopping(generations=5, metric="fitness"),
]

search.fit(X_train, y_train, callbacks=callbacks)

• ThresholdStopping: Ideal when you have a clear objective (e.g., you need at least 95% accuracy).
• ConsecutiveStopping: Avoids wasting compute when the algorithm has already converged.

Comparison with Grid Search and Random Search

Here's the same task done with all three methods so you can compare directly:

from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from scipy.stats import randint, uniform
import time

# Grid Search
param_grid_gs = {
    "n_estimators": [50, 100, 200, 300],
    "max_depth": [2, 5, 8, 12, 15],
    "max_features": [0.3, 0.5, 0.7, 1.0],
    "criterion": ["gini", "entropy"],
}
# Total: 4 × 5 × 4 × 2 = 160 combinations × 5 folds = 800 evaluations

t0 = time.time()
gs = GridSearchCV(RandomForestClassifier(random_state=42), param_grid_gs, cv=5, n_jobs=-1)
gs.fit(X_train, y_train)
print(f"Grid Search — Best: {gs.best_score_:.4f} | Time: {time.time()-t0:.1f}s")

# Random Search
t0 = time.time()
rs = RandomizedSearchCV(
    RandomForestClassifier(random_state=42),
    {"n_estimators": randint(50, 300), "max_depth": randint(2, 15),
     "max_features": uniform(0.3, 0.7), "criterion": ["gini", "entropy"]},
    n_iter=50, cv=5, n_jobs=-1, random_state=42
)
rs.fit(X_train, y_train)
print(f"Random Search — Best: {rs.best_score_:.4f} | Time: {time.time()-t0:.1f}s")

# GASearchCV — at most 30 individuals × 25 generations = 750 evaluations
t0 = time.time()
search.fit(X_train, y_train, callbacks=[ConsecutiveStopping(generations=5, metric="fitness")])
print(f"GASearchCV — Best: {search.best_score_:.4f} | Time: {time.time()-t0:.1f}s")

Typical results show that GASearchCV matches or outperforms grid search with fewer evaluations, because it directs the search toward the most promising regions of the space. The advantage becomes more pronounced as the hyperparameter space grows larger.

Conclusion

sklearn-genetic-opt with GASearchCV provides a powerful and elegant alternative to classical hyperparameter tuning methods:

• Compatible with the scikit-learn API: if you already use GridSearchCV, migration is trivial.
• Efficient: Converges toward good solutions without exhaustively evaluating the space.
• Flexible: Continuous, integer, and categorical spaces in the same param_grid.
• Controllable: EvolutionConfig and callbacks give you full control over the process.

The natural next step is combining GASearchCV with GAFeatureSelectionCV to simultaneously optimize both model hyperparameters and dataset features — see Evolutionary Feature Selection for Machine Learning.

If you're working with scikit-learn models and still relying exclusively on grid search, now is the time to try evolution.

GASearchCV is part of sklearn-genetic-opt, an open-source scikit-learn-compatible library for evolutionary optimization.