Evolutionary Feature Selection for Machine Learning

Why Feature Selection Matters

In machine learning, more data is not always better. More features, however, often make performance worse. This is what is known as the curse of dimensionality: as the number of features grows, the input space becomes so sparse that models struggle to learn generalizable patterns.

The concrete problems caused by too many features are several:

• Overfitting: The model memorizes noise instead of learning real patterns.
• Training time: More features means more computation.
• Interpretability: A model with 200 variables is hard to explain.
• Multicollinearity: Correlated features confuse many algorithms.
• Noise: Irrelevant features act as noise and degrade predictions.

Feature selection solves all of these problems by choosing the subset of variables that maximizes model performance. In this article you'll use GAFeatureSelectionCV to evolve that subset automatically with a genetic algorithm, compare it against SelectKBest, and combine it with evolutionary hyperparameter tuning.

Traditional Methods and Their Limitations

There are three classical families of feature selection methods:

Filter methods (like SelectKBest): Evaluate each feature independently using a statistic (chi-squared, mutual information, Pearson correlation). They're fast, but they ignore interactions between variables. A feature that is irrelevant on its own can be very informative in combination with another.

Wrapper methods (like RFE): Train the model iteratively, removing features at each step. They capture interactions, but are computationally expensive — they require many full model fits.

Embedded methods (like LASSO, feature_importances_): Selection happens inside the model's own training process. They're efficient but tied to a specific model type.

None of these methods is a universal solution. Filter methods are naive, wrappers are slow, and embedded methods are algorithm-specific.

Evolutionary Feature Selection with GAFeatureSelectionCV

sklearn-genetic-opt offers a fourth approach: evolutionary feature selection via GAFeatureSelectionCV. The idea is brilliant in its simplicity:

Each feature in the dataset is represented as a bit in a binary chromosome. An individual with 20 features might look like: [1, 0, 1, 1, 0, 0, 1, ...], where 1 means "use this feature" and 0 means "ignore it."

The genetic algorithm evolves this population of binary masks, evaluating each feature subset via cross-validation. Over time, it converges toward the subset that maximizes model performance.

This approach has clear advantages over classical methods:

• Captures interactions between features (like wrapper methods).
• Is model-agnostic (like filter methods).
• Is efficient: No need to try all 2^n possible subsets.

Full Example with the Breast Cancer Dataset

We'll use the classic breast cancer diagnosis dataset from scikit-learn, which has 30 features and 569 samples:

from sklearn.datasets import load_breast_cancer
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn_genetic import GAFeatureSelectionCV, EvolutionConfig

# Load data
data = load_breast_cancer()
X, y = data.data, data.target
feature_names = data.feature_names

print(f"Original shape: {X.shape}")
# Original shape: (569, 30)

# Preprocessing
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Split
X_train, X_test, y_train, y_test = train_test_split(
    X_scaled, y, test_size=0.2, random_state=42, stratify=y
)

# Configure evolutionary feature selection
selector = GAFeatureSelectionCV(
    estimator=SVC(kernel="rbf", gamma="auto"),
    cv=5,
    scoring="accuracy",
    evolution_config=EvolutionConfig(
        population_size=20,
        generations=15,
        crossover_probability=0.8,
        mutation_probability=0.1,
        elitism=True,
        keep_top_k=2,
    ),
)

# Fit
selector.fit(X_train, y_train)

The process may take a few minutes depending on your hardware, since it's training 20 SVMs × 15 generations × 5 folds = up to 1,500 models (though the evaluation cache reduces this in practice).

Interpreting the Results

Once the selector is fitted, you have access to several key attributes:

# Boolean mask: True for selected features
print("Support mask:", selector.support_)
# [True, False, True, True, False, True, ...]

# Indices of the selected features
selected_indices = selector.support_.nonzero()[0]
print("Selected indices:", selected_indices)

# Names of the selected features
selected_features = feature_names[selector.support_]
print("Selected features:")
for feat in selected_features:
    print(f"  - {feat}")

# Number of selected features
n_selected = selector.support_.sum()
print(f"\nSelected features: {n_selected} out of {X.shape[1]}")

# Best cross-validation score
print(f"Best CV score: {selector.best_score_:.4f}")

# The best estimator trained with the selected features
best_model = selector.best_estimator_

A typical result might show that the algorithm selected 12–15 out of 30 features, maintaining or improving accuracy compared to the full-feature model.

Using transform() to Apply the Selection

The GAFeatureSelectionCV interface follows scikit-learn's transformer pattern. You can use transform() to apply the same feature selection to any dataset:

# Transform training data
X_train_selected = selector.transform(X_train)
print(f"Shape before: {X_train.shape}")    # (455, 30)
print(f"Shape after:  {X_train_selected.shape}")  # (455, 14)

# Transform test data (IMPORTANT: same transformation)
X_test_selected = selector.transform(X_test)

# Evaluate the best model on the test set
test_score = best_model.score(X_test_selected, y_test)
print(f"\nTest accuracy (with selection): {test_score:.4f}")

# Compare with the model without selection
baseline = SVC(kernel="rbf", gamma="auto")
baseline.fit(X_train, y_train)
baseline_score = baseline.score(X_test, y_test)
print(f"Test accuracy (no selection):   {baseline_score:.4f}")

A critical point: never apply fit_transform() on the test data. The selector learned which features to use from the training set. The test set must be transformed with exactly the same features for an honest evaluation.

You can also use selector.predict() directly, since it applies the transformation internally before predicting:

# Equivalent to selector.transform(X_test) + best_model.predict(...)
y_pred = selector.predict(X_test)

Comparing with SelectKBest

To see the value of evolutionary selection, let's compare with SelectKBest using mutual information:

from sklearn.feature_selection import SelectKBest, mutual_info_classif
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score

# Try different values of k
results = {}
for k in [5, 10, 14, 20, 30]:
    pipeline = Pipeline([
        ("selector", SelectKBest(mutual_info_classif, k=k)),
        ("clf", SVC(kernel="rbf", gamma="auto")),
    ])
    scores = cross_val_score(pipeline, X_train, y_train, cv=5, scoring="accuracy")
    results[k] = scores.mean()
    print(f"SelectKBest (k={k:2d}): {scores.mean():.4f} ± {scores.std():.4f}")

# GAFeatureSelectionCV already has its score
print(f"\nGAFeatureSelectionCV ({n_selected} features): {selector.best_score_:.4f}")

The key difference is that SelectKBest evaluates each feature in isolation (its statistical relationship with the target). GAFeatureSelectionCV evaluates complete subsets of features, capturing interaction effects. In datasets with correlated features or interaction effects, evolutionary selection typically wins.

Moreover, with SelectKBest you have to choose k upfront. With GAFeatureSelectionCV, the number of selected features emerges from the evolutionary process itself.

Tips for Better Results

Scale your data first. Many algorithms (SVM, KNN, logistic regression) are sensitive to feature scale. Apply StandardScaler or MinMaxScaler before fitting the selector.

Start with a small population. For datasets with many features, start with population_size=20 and generations=15 and increase if you see the algorithm hasn't converged.

Use a fast model as the estimator. For the feature selection phase, choose a fast estimator (like LogisticRegression or RandomForestClassifier with few trees). Once the best features are identified, you can train the final model — even a slower one like SVM or XGBoost — on those features.

from sklearn.linear_model import LogisticRegression

# Fast selection with LogisticRegression
selector_fast = GAFeatureSelectionCV(
    estimator=LogisticRegression(max_iter=1000),
    cv=5,
    scoring="accuracy",
    evolution_config=EvolutionConfig(population_size=20, generations=10),
)
selector_fast.fit(X_train, y_train)

# Final training with SVM on selected features
X_train_sel = selector_fast.transform(X_train)
X_test_sel = selector_fast.transform(X_test)
final_model = SVC(kernel="rbf", C=10)
final_model.fit(X_train_sel, y_train)
print(f"Final accuracy: {final_model.score(X_test_sel, y_test):.4f}")

Combine with GASearchCV. The most powerful pattern is to first run GAFeatureSelectionCV to identify the best feature subset, then run GASearchCV on that reduced dataset to tune hyperparameters. Each step becomes faster because the input dimensionality is lower.

from sklearn_genetic import GASearchCV
from sklearn_genetic.space import Continuous

# Step 1: Select features
selector.fit(X_train, y_train)
X_train_sel = selector.transform(X_train)
X_test_sel = selector.transform(X_test)

# Step 2: Tune hyperparameters on reduced dataset
hp_search = GASearchCV(
    estimator=SVC(kernel="rbf"),
    param_grid={
        "C": Continuous(0.1, 100),
        "gamma": Continuous(0.001, 1.0),
    },
    cv=5,
    scoring="accuracy",
    evolution_config=EvolutionConfig(population_size=20, generations=15),
)
hp_search.fit(X_train_sel, y_train)
print(f"Optimized test accuracy: {hp_search.best_estimator_.score(X_test_sel, y_test):.4f}")

Conclusion

GAFeatureSelectionCV takes feature selection to a higher level:

• Automatic: You don't need to decide how many features to use or which ones.
• Intelligent: It explores feature interactions, not just individual importances.
• Compatible: Follows the scikit-learn interface — fit, transform, predict.
• Composable: Can be used alongside GASearchCV to optimize both features and hyperparameters.

Feature selection is often the most overlooked step in machine learning projects, yet it can be the difference between a model that overfits and one that generalizes well. Natural evolution took millions of years to optimize genomes. You can optimize your feature set in minutes.

GAFeatureSelectionCV is part of sklearn-genetic-opt. To also tune hyperparameters on your reduced feature set, see Tune Your Scikit-learn Model Using Evolutionary Algorithms.