Sklearn Genetic Opt

The Problem It Solves

Hyperparameter search can consume a substantial training budget. The cost depends on the estimator, dataset, number of candidate configurations and cross-validation folds.

With five hyperparameters and ten candidate values each, grid search tests 100,000 configurations. Five-fold cross-validation would require 500,000 fits before the final refit. That can be impractical for some datasets and models.

Random search samples a chosen number of configurations and remains a useful baseline. Evolutionary search offers another approach: it uses previous evaluations to evolve a population of candidates. Compare these methods under the same evaluation budget; no method wins on every problem.

What I Built

sklearn-genetic-opt represents each configuration as an individual in a population. It evaluates candidates through cross-validation and uses selection, crossover and mutation to generate new configurations.

The library exposes two estimators:

• GASearchCV: searches hyperparameters for scikit-learn-compatible estimators and pipelines
• GAFeatureSelectionCV: evaluates feature subsets for an estimator

Both follow the scikit-learn .fit() pattern and provide the refitted estimator through .best_estimator_. GASearchCV exposes the chosen hyperparameters in .best_params_; GAFeatureSelectionCV exposes selected features in .support_. Configuration includes evolutionary parameters.

Evolutionary search with DEAP

DEAP supplies the evolutionary operators. The scikit-learn estimator and chosen scoring metric determine how each candidate is evaluated.

from sklearn_genetic import GASearchCV, EvolutionConfig, RuntimeConfig
from sklearn_genetic.space import Integer, Continuous, Categorical
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("clf", GradientBoostingClassifier()),
])

param_grid = {
    "clf__n_estimators": Integer(50, 500),
    "clf__max_depth": Integer(2, 12),
    "clf__learning_rate": Continuous(0.001, 0.3),
    "clf__subsample": Continuous(0.5, 1.0),
    "clf__max_features": Categorical(["sqrt", "log2", None]),
}

search = GASearchCV(
    estimator=pipeline,
    cv=5,
    scoring="roc_auc",
    param_grid=param_grid,
    evolution_config=EvolutionConfig(
        population_size=25,
        generations=50,
    ),
    runtime_config=RuntimeConfig(n_jobs=-1),
)
search.fit(X_train, y_train)

Integer, Continuous and Categorical define the permitted values for each parameter. For example, Continuous(0.001, 0.3) supports sampling within that interval without enumerating a list of values.

Configuration and results

After fitting, you can inspect the best configuration and refitted estimator through familiar scikit-learn attributes. Review cross-validation results to compare candidates and use separate data for the final evaluation.

Adapter classes let you vary crossover and mutation probabilities between generations. Choose a schedule for your budget and compare it with fixed probabilities; an adaptive schedule does not guarantee avoiding premature convergence.

Feature Selection as Optimization

The same evolutionary engine handles feature selection. Rather than searching hyperparameter space, GAFeatureSelectionCV searches the space of possible feature subsets: each individual is a binary mask over columns.

from sklearn_genetic import GAFeatureSelectionCV, EvolutionConfig
from sklearn.linear_model import LogisticRegression

selector = GAFeatureSelectionCV(
    estimator=LogisticRegression(max_iter=1000),
    cv=5,
    scoring="roc_auc",
    evolution_config=EvolutionConfig(
        population_size=30,
        generations=40,
    ),
)
selector.fit(X_train, y_train)

X_reduced = selector.transform(X_test)
print(f"Selected {selector.support_.sum()} of {X_train.shape[1]} features")

Evaluate feature selection within cross-validation so that selection does not use information from the test set. Compare the result with the same estimator trained on all features.

Where It Stands

sklearn-genetic-opt has been actively developed since 2020 and is currently at v0.13.1. It is published on PyPI, fully documented at sklearngeneticopt.rodrigo-arenas.com, and open to contributions.

The repository includes source code and tests. The documentation provides the API reference and examples for reproducing searches and feature selection.