Sklearn Genetic Opt

The Problem It Solves

In ML consulting work, hyperparameter optimization consistently appears as one of the most time-consuming phases — not because the problem is conceptually hard, but because the standard tools don't scale.

Grid search evaluates every combination exhaustively. With five hyperparameters and ten candidate values each, that's 100,000 cross-validation evaluations. In practice, it simply doesn't run to completion on real models.

Random search improves throughput but treats the search space as a black box: it samples blindly, learns nothing from prior results, and relies on luck rather than strategy.

The deeper issue is that both approaches were designed for small, well-understood search spaces. Production pipelines routinely have pipelines with preprocessing steps, custom estimators, and six or more interacting hyperparameters. Standard tools break down at exactly the point where good optimization matters most.

What I Built

sklearn-genetic-opt applies evolutionary algorithms to the hyperparameter search problem. The algorithm treats each configuration as an individual in a population, evaluates fitness via cross-validation, and evolves toward better solutions across generations — allocating more search budget to promising regions of the parameter space, not random corners of it.

The library exposes two estimators:

• GASearchCV — evolves hyperparameter configurations for any scikit-learn estimator, pipeline, or custom callable
• GAFeatureSelectionCV — evolves which feature subsets to include, optimizing for downstream model performance

Both are designed as drop-in replacements for GridSearchCV and RandomizedSearchCV. If you've used sklearn's search API, you already know how to use this library.

A Search That Learns

The core difference from random search is that sklearn-genetic-opt uses the DEAP evolutionary framework to run genuine genetic search — not random sampling with a fancier name.

from sklearn_genetic import GASearchCV, EvolutionConfig, RuntimeConfig
from sklearn_genetic.space import Integer, Continuous, Categorical
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("clf", GradientBoostingClassifier()),
])

param_grid = {
    "clf__n_estimators": Integer(50, 500),
    "clf__max_depth": Integer(2, 12),
    "clf__learning_rate": Continuous(0.001, 0.3),
    "clf__subsample": Continuous(0.5, 1.0),
    "clf__max_features": Categorical(["sqrt", "log2", None]),
}

search = GASearchCV(
    estimator=pipeline,
    cv=5,
    scoring="roc_auc",
    param_grid=param_grid,
    evolution_config=EvolutionConfig(
        population_size=25,
        generations=50,
    ),
    runtime_config=RuntimeConfig(n_jobs=-1),
)
search.fit(X_train, y_train)

The typed search space primitives — Integer, Continuous, Categorical — define the legal range for each parameter. Continuous lets the algorithm explore float values a grid would never sample; the genetic operators (crossover, mutation) respect these types during evolution.

Why the API Looks Like This

Every design decision in the library prioritized adoption over novelty.

Sklearn users already know the .fit() / .best_params_ / .best_estimator_ pattern. Requiring them to learn a new interface to get better search would eliminate most of the practical benefit. The API deliberately mirrors GridSearchCV so teams can swap it in with minimal friction.

The typed search space was also a deliberate departure from the grid-of-lists pattern. Specifying Continuous(0.001, 0.3) instead of [0.001, 0.01, 0.05, 0.1, 0.2, 0.3] removes the implicit assumption that the optimum falls on one of six handpicked values. Genetic operators can land anywhere in the continuous range.

For longer searches, the library includes adapter classes that schedule mutation and crossover probabilities dynamically across generations — high mutation early (broad exploration), low mutation late (local refinement around the best candidates found). This addresses the convergence problem that flat probability schedules cause on large search spaces.

Feature Selection as Optimization

The same evolutionary engine handles feature selection. Rather than searching hyperparameter space, GAFeatureSelectionCV searches the space of possible feature subsets — each individual is a binary mask over columns.

from sklearn_genetic import GAFeatureSelectionCV, EvolutionConfig
from sklearn.linear_model import LogisticRegression

selector = GAFeatureSelectionCV(
    estimator=LogisticRegression(max_iter=1000),
    cv=5,
    scoring="roc_auc",
    evolution_config=EvolutionConfig(
        population_size=30,
        generations=40,
    ),
)
selector.fit(X_train, y_train)

X_reduced = selector.transform(X_test)
print(f"Selected {selector.support_.sum()} of {X_train.shape[1]} features")

This matters in high-dimensional datasets where correlation between features makes manual selection unreliable, and where the downstream model's performance depends on which specific subset is passed in — not just on individual feature importances.

Where It Stands

sklearn-genetic-opt has been actively developed since 2020 and is currently at v0.13.1. It is published on PyPI, fully documented at sklearngeneticopt.rodrigo-arenas.com, and open to contributions.

The library is part of a broader set of tools I've built around production ML — the same problems that keep coming up in consulting engagements: search that scales, features that generalize, pipelines that can be tuned and shipped.