Feature Selection with sklearn-genetic-opt

GAFeatureSelectionCV evaluates subsets of input columns through cross-validation. Its result depends on the estimator, scoring rule, data and search budget. This example uses sklearn-genetic-opt to select columns, compare two baselines and tune an SVM on the selected subset. Every reported value comes from the run.

Versions and execution

The four Python blocks form one file, selection_demo.py, in order. They were tested with Python 3.12, sklearn-genetic-opt 0.14.0, scikit-learn 1.9.1 and NumPy 2.5.3. After saving the blocks, run these PowerShell commands. The remaining dependencies are resolved by pip; use a complete environment lock for a release.

py -3.12 -m venv .venv
& .\.venv\Scripts\python.exe -m pip install sklearn-genetic-opt==0.14.0 scikit-learn==1.9.1 numpy==2.5.3
& .\.venv\Scripts\python.exe selection_demo.py

Split raw data; scale inside each fold

The built-in dataset contains 569 observations and 30 columns. Split the raw matrix first. Placing StandardScaler inside the estimator Pipeline lets each candidate's CV fold learn scaling from its own training rows. The final refit learns from all training rows; the held-out rows remain separate.

Fitting a scaler before the split makes held-out measurements influence training. Fitting it on the complete training matrix before CV also exposes each validation fold's measurements. The scikit-learn leakage guide explains both the split and pipeline pattern.

from functools import partial
import numpy as np
from sklearn.base import clone
from sklearn.datasets import load_breast_cancer
from sklearn.feature_selection import SelectKBest, mutual_info_classif
from sklearn.metrics import accuracy_score
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn_genetic import (
    EvolutionConfig, GAFeatureSelectionCV, GASearchCV,
    OptimizationConfig, RuntimeConfig,
)
from sklearn_genetic.space import Continuous

data = load_breast_cancer()
X, y = data.data, data.target
feature_names = data.feature_names
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)
cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=42)
estimator = Pipeline([
    ("scale", StandardScaler()),
    ("svc", SVC(kernel="rbf", C=1.0, gamma="scale")),
])
evolution = EvolutionConfig(population_size=8, generations=4)
runtime = RuntimeConfig(n_jobs=1, verbose=False, error_score="raise")
optimization = OptimizationConfig(diversity_control=False)
selector = GAFeatureSelectionCV(
    estimator=estimator, cv=cv, scoring="accuracy",
    evolution_config=evolution, runtime_config=runtime,
    optimization_config=optimization, random_state=42,
)
selector.fit(X_train, y_train)

Eight individuals, four generations and three folds make a small demonstration. Initial evaluations, offspring and cached repeats affect the actual number of fits; multiplying population by generations does not establish an exact total. This configuration disables diversity control to isolate the basic search and uses one job. Increasing the budget adds work and may change the result; it provides no guarantee of a global optimum.

Inspect the mask and its search score

support_ follows the original column order. transform() removes unselected columns and keeps their remaining values and order. The fitted estimator pipeline handles scaling. selector.predict(X_test) applies its mask internally; best_estimator_ expects an already selected matrix.

selected_features = feature_names[selector.support_]
print("Selected features:", selected_features.tolist())
print("Feature count:", int(selector.support_.sum()))
X_train_selected = selector.transform(X_train)
X_test_selected = selector.transform(X_test)
print("Training shapes:", X_train.shape, X_train_selected.shape)

# Match the chosen mask to its actual public CV results row.
selected_row = next(
    index for index, mask in enumerate(selector.cv_results_["features"])
    if np.array_equal(mask, selector.support_)
)
search_cv_score = float(selector.cv_results_["mean_test_score"][selected_row])
print(f"Selected-mask search CV: {search_cv_score:.4f}")

In version 0.14.0, best_score_ on the feature selector belongs to the optional final-selection validation (final_selection=True). This example uses the default search selection and reads the matching mask's row in cv_results_. The printed CV value is a search objective used to choose the mask; it is not an unbiased estimate of the chosen workflow's future accuracy. Feature counts and names vary with the data, search and versions.

Baselines and a second tuning stage

The all-feature baseline uses the same SVM pipeline. The filter baseline selects ten columns with mutual information inside each fold, then scales and fits. Ten is a demonstration choice; pick such settings using training data. Filter scores evaluate columns individually, while a wrapper scores subsets with the estimator. Their evaluation costs differ, and either can work better on a particular dataset.

The second search tunes svc__C and svc__gamma in the pipeline. The mask is fixed from the first search and only training rows enter tuning. The two stages below are a conditional workflow: its tuning folds share a mask previously chosen using the complete training set. These CV scores help that search choose parameters; they do not fairly estimate the full selection-plus-tuning process. For an unbiased comparison, repeat the entire selection and tuning workflow inside outer training folds, or evaluate a separate untouched holdout after choices are frozen.

all_features = clone(estimator)
filter_model = Pipeline([
    ("select", SelectKBest(partial(mutual_info_classif, random_state=42), k=10)),
    ("scale", StandardScaler()),
    ("svc", SVC(kernel="rbf", C=1.0, gamma="scale")),
])
for name, model in [("all_features", all_features), ("filter_10", filter_model)]:
    scores = cross_val_score(
        model, X_train, y_train, cv=cv, scoring="accuracy",
        n_jobs=1, error_score="raise",
    )
    print(f"{name} CV: {scores.mean():.4f} +/- {scores.std():.4f}")
    model.fit(X_train, y_train)

hp_search = GASearchCV(
    estimator=clone(estimator),
    param_grid={
        "svc__C": Continuous(0.1, 10.0, distribution="log-uniform"),
        "svc__gamma": Continuous(0.01, 1.0, distribution="log-uniform"),
    },
    cv=cv, scoring="accuracy", evolution_config=evolution,
    runtime_config=runtime, optimization_config=optimization, random_state=42,
)
hp_search.fit(X_train_selected, y_train)
print("Tuned parameters:", hp_search.best_params_)
print(f"Fixed-mask tuning CV: {hp_search.best_score_:.4f}")

The two stages need not be faster than other methods. Record actual fits, elapsed time and search seeds when comparing budgets. A smaller input matrix alone does not establish a better score or lower total cost.

Report the held-out results after training choices

All four workflows have now been fitted without using X_test or y_test. Run this final report after choosing the workflows from training evidence. Repeatedly changing settings based on these results would turn this holdout into development data. These are accuracies on one split of a small built-in dataset, without a claimed winner or deployment guarantee.

holdout_scores = {
    "all_features": float(accuracy_score(y_test, all_features.predict(X_test))),
    "filter_10": float(accuracy_score(y_test, filter_model.predict(X_test))),
    "genetic": float(accuracy_score(y_test, selector.predict(X_test))),
    "genetic_tuned": float(accuracy_score(
        y_test, hp_search.predict(X_test_selected)
    )),
}
for name, score in holdout_scores.items():
    print(f"{name}: {score:.4f}")

Continue with hyperparameter search or adaptive search probabilities. The published selector source describes the result attributes used here; scikit-learn pipelines explains step names and parameter prefixes.