Adaptive Parameters Methods for Machine Learning

Genetic algorithms use two main operators to explore the hyperparameter space: mutation (random changes to individuals) and crossover (combining two solutions to generate offspring). Both operators have associated probabilities that control how frequently they are applied during evolution.

The natural question is: should these probabilities be fixed throughout the entire evolutionary process? The answer, in most cases, is no. In this post you'll see the four adapters sklearn-genetic-opt provides, the math behind ExponentialAdapter, and a practical rule for choosing adaptive_rate based on your generation budget.

Fixed vs. Adaptive Parameters

With fixed parameters, the mutation probability is the same in generation 1 as it is in generation 25. This presents an inherent problem: different phases of the optimization process have different needs.

• Early on: the hyperparameter space is unknown. We need broad exploration — high genetic diversity, frequent mutations to visit distant regions.
• Later: we've already identified promising regions. We need exploitation — fine-grained refinement around the best individuals, less random perturbation.

A high mutation probability sustained throughout the entire evolution prevents convergence. A low probability from the start restricts exploration and can get stuck in local optima. Adapters resolve this dilemma by dynamically adjusting parameters across generations.

The Exploration-Exploitation Trade-off

This trade-off is fundamental in all heuristic search methods. In the context of hyperparameter search with genetic algorithms:

• Exploration: trying diverse configurations, risking less promising individuals to discover unexpectedly good regions.
• Exploitation: concentrating computational resources on refining the best configurations found so far.

Adapters implement a smooth transition policy from exploration to exploitation. The intuition is simple: at the start of training, we want to operate more like a random search; at the end, we want to operate more like local gradient descent.

Available Adapters in sklearn-genetic-opt

The sklearn-genetic-opt library offers four types of adapters for mutation and crossover probabilities:

ConstantAdapter

The trivial case: the value is always the same. Equivalent to not using an adapter. Useful as a baseline or when you have evidence that a fixed value is optimal.

from sklearn_genetic import ConstantAdapter
# always returns 0.2, regardless of generation
adapter = ConstantAdapter(value=0.2)

ExponentialAdapter

Exponential decay from a high initial value to a low final value. Ideal for mutation probability: high at the start to explore, low at the end to refine.

from sklearn_genetic import ExponentialAdapter
adapter = ExponentialAdapter(initial_value=0.4, end_value=0.05, adaptive_rate=0.1)

InverseAdapter

Inverse growth from a low initial value to a high final value. Suitable for crossover probability: early on, individuals differ widely (mixing them isn't very useful); later, they become more similar and crossover can refine the solution.

from sklearn_genetic import InverseAdapter
adapter = InverseAdapter(initial_value=0.6, end_value=0.9, adaptive_rate=0.1)

PotentialAdapter

Growth via a potential function. Allows more abrupt transitions than the inverse adapter, useful when you want the change to occur mainly at the end of evolution.

from sklearn_genetic import PotentialAdapter
adapter = PotentialAdapter(initial_value=0.6, end_value=0.9, adaptive_rate=0.1)

Math Behind ExponentialAdapter

The exponential decay formula is:

value(t) = end_value + (initial_value - end_value) * exp(-adaptive_rate * t)

Where t is the current generation number. To understand the effect of adaptive_rate:

• With adaptive_rate=0.05: decay is slow, the transition happens gradually over many generations.
• With adaptive_rate=0.2: decay is fast, most of the transition happens in the first few generations.

import numpy as np
import matplotlib.pyplot as plt

def exponential_adapter(t, initial=0.4, end=0.05, rate=0.1):
    return end + (initial - end) * np.exp(-rate * t)

def inverse_adapter(t, initial=0.6, end=0.9, rate=0.1):
    return end - (end - initial) / (1 + rate * t)

generations = np.arange(0, 25)

plt.figure(figsize=(10, 4))
plt.subplot(1, 2, 1)
plt.plot(generations, [exponential_adapter(t) for t in generations])
plt.title("ExponentialAdapter (mutation)")
plt.xlabel("Generation"); plt.ylabel("Probability")

plt.subplot(1, 2, 2)
plt.plot(generations, [inverse_adapter(t) for t in generations])
plt.title("InverseAdapter (crossover)")
plt.xlabel("Generation"); plt.ylabel("Probability")
plt.tight_layout()
plt.show()

This visually illustrates how mutation probability decays from 0.4 to approximately 0.05 while crossover increases from 0.6 to 0.9 over 25 generations.

Complete Example: GASearchCV with Adapters

Here is how to integrate adapters into a real hyperparameter search with a scikit-learn classifier:

from sklearn_genetic import GASearchCV, ExponentialAdapter, InverseAdapter, EvolutionConfig
from sklearn_genetic.space import Integer, Continuous, Categorical
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, StratifiedKFold

# Sample data
X, y = make_classification(
    n_samples=1000, n_features=20, n_informative=10, random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Hyperparameter search space
param_grid = {
    "n_estimators": Integer(50, 300),
    "max_depth": Integer(3, 15),
    "min_samples_split": Integer(2, 20),
    "max_features": Continuous(0.1, 1.0),
    "criterion": Categorical(["gini", "entropy"])
}

clf = RandomForestClassifier(random_state=42)
cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=42)

# GASearchCV with adaptive parameters
search = GASearchCV(
    estimator=clf,
    param_grid=param_grid,
    scoring="accuracy",
    cv=cv,
    mutation_probability=ExponentialAdapter(
        initial_value=0.4,
        end_value=0.05,
        adaptive_rate=0.1
    ),
    crossover_probability=InverseAdapter(
        initial_value=0.6,
        end_value=0.9,
        adaptive_rate=0.1
    ),
    evolution_config=EvolutionConfig(
        population_size=30,
        generations=25
    ),
    verbose=True,
    n_jobs=-1
)

search.fit(X_train, y_train)

print(f"\nBest hyperparameters: {search.best_params_}")
print(f"Best CV score: {search.best_score_:.4f}")
print(f"Test score: {search.score(X_test, y_test):.4f}")

Using Only ExponentialAdapter for Mutation

If you prefer a more conservative approach and only want to adapt mutation:

search_mutation_only = GASearchCV(
    estimator=RandomForestClassifier(random_state=42),
    param_grid=param_grid,
    scoring="accuracy",
    cv=cv,
    mutation_probability=ExponentialAdapter(
        initial_value=0.4,
        end_value=0.05,
        adaptive_rate=0.1
    ),
    crossover_probability=0.7,  # fixed
    evolution_config=EvolutionConfig(population_size=30, generations=25),
    n_jobs=-1
)
search_mutation_only.fit(X_train, y_train)

This is useful when you have certainty about the optimal crossover rate but want adaptive exploration in mutation.

Comparing Fixed vs. Adaptive

To quantify the benefit of adapters, we can compare convergence:

# Search with fixed parameters
search_fixed = GASearchCV(
    estimator=RandomForestClassifier(random_state=42),
    param_grid=param_grid,
    scoring="accuracy",
    cv=cv,
    mutation_probability=0.1,   # fixed
    crossover_probability=0.8,  # fixed
    evolution_config=EvolutionConfig(population_size=30, generations=25),
    n_jobs=-1
)
search_fixed.fit(X_train, y_train)

# Compare fitness history
import pandas as pd

history_adaptive = pd.DataFrame(search.history)
history_fixed = pd.DataFrame(search_fixed.history)

print("Convergence — max fitness score per generation:")
print(f"{'Gen':>4} | {'Adaptive':>12} | {'Fixed':>12}")
print("-" * 35)
for gen in [0, 5, 10, 15, 20, 24]:
    a = history_adaptive.loc[gen, "fitness_max"]
    f = history_fixed.loc[gen, "fitness_max"]
    print(f"{gen:>4} | {a:>12.4f} | {f:>12.4f}")

In general, adaptive parameters tend to:

1. Converge faster: the initial exploration produces more diverse candidates.
2. Converge to better solutions: the final exploitation refines top candidates more effectively.
3. Be more robust: less sensitive to the initial choice of genetic algorithm hyperparameters.

When to Use Each Adapter

The practical choice depends on the problem. As a starting rule: use ExponentialAdapter for mutation and InverseAdapter for crossover, then tune adaptive_rate based on the number of generations available.

Choosing adaptive_rate

The adaptive_rate parameter controls how quickly the transition happens. A practical way to choose it is to think about at which generation you want to reach roughly half the transition:

• For ExponentialAdapter: half the decay from initial_value to end_value happens at t = ln(2) / adaptive_rate.
• With adaptive_rate=0.1 and 25 generations, the half-transition is at generation ~7, meaning most of the exploration happens in the first third.

For longer runs (50+ generations), reduce adaptive_rate proportionally so the exploration phase covers a meaningful fraction of the budget.

Conclusion

Adaptive parameters represent a meaningful improvement over fixed configuration in evolutionary hyperparameter search. sklearn-genetic-opt implements these adapters with a clean API that integrates without friction into existing scikit-learn workflows.

The underlying principle — high exploration early, high exploitation late — is universal in heuristic optimization and appears in analogous forms in simulated annealing (temperature schedule), particle swarm optimization (inertia weight), and reinforcement learning (epsilon-greedy decay). Understanding this trade-off is one of the most transferable skills in the field of metaheuristic optimization.

If you change just one thing in your next GASearchCV run, swap the fixed mutation_probability for an ExponentialAdapter — it's a one-line change that usually pays for itself in both convergence speed and final score.

ExponentialAdapter, InverseAdapter, and PotentialAdapter are part of sklearn-genetic-opt. If you're new to evolutionary hyperparameter search, start with Tune Your Scikit-learn Model Using Evolutionary Algorithms.