Your First Machine Learning Model with Scikit-learn
A step-by-step guide to building your first ML model: data exploration, preprocessing, cross-validation, hyperparameter tuning, and evaluation — all with scikit-learn.
Everyone starts somewhere. This article walks you through the complete workflow of building a machine learning model from scratch — no assumptions about prior ML experience, just Python and a willingness to experiment.
By the end you will have trained a classifier, validated it properly, tuned its hyperparameters, and understood what the metrics actually mean. You will also see why the classic approach has limits, which sets up the more advanced techniques in the other articles on this site.
The Dataset
We will use the classic Titanic dataset: predict whether a passenger survived based on features like age, sex, ticket class, and fare. It is small enough to run locally in seconds and large enough to expose real preprocessing challenges.
import pandas as pd
from sklearn.datasets import fetch_openml
# Load Titanic from OpenML
titanic = fetch_openml("titanic", version=1, as_frame=True)
df = titanic.frame
print(df.shape) # (1309, 14)
print(df.dtypes)
Step 1: Exploratory Data Analysis
Before writing a single line of model code, understand your data. Missing values, skewed distributions, and class imbalance all affect the model.
# Check missing values
print(df.isnull().sum().sort_values(ascending=False))
# body 1188
# cabin 1014
# boat 823
# age 263
# fare 1
# Target distribution
print(df['survived'].value_counts(normalize=True))
# 0 0.618 (did not survive)
# 1 0.382 (survived)
# Numerical summary
print(df[['age', 'fare', 'pclass']].describe())
Two immediate observations: cabin is mostly missing (drop it), and the dataset is imbalanced (62% did not survive — we need to pick the right metric). Also note that boat and body leak the target: they are only known after the disaster, so using them would make the model look great and be useless.
Step 2: Preprocessing
Raw data rarely goes straight into a model. We need to handle missing values, encode categoricals, and scale numerics.
import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OrdinalEncoder
from sklearn.impute import SimpleImputer
from sklearn.compose import ColumnTransformer
# Select and define feature types
NUMERIC_FEATURES = ['age', 'fare', 'sibsp', 'parch']
CATEGORICAL_FEATURES = ['sex', 'embarked', 'pclass']
# Build sub-pipelines for each type
numeric_pipeline = Pipeline([
('imputer', SimpleImputer(strategy='median')),
('scaler', StandardScaler()),
])
categorical_pipeline = Pipeline([
('imputer', SimpleImputer(strategy='most_frequent')),
('encoder', OrdinalEncoder(handle_unknown='use_encoded_value', unknown_value=-1)),
])
# Combine into a single transformer
preprocessor = ColumnTransformer([
('num', numeric_pipeline, NUMERIC_FEATURES),
('cat', categorical_pipeline, CATEGORICAL_FEATURES),
])
Using Pipeline and ColumnTransformer is not just clean code — it prevents data leakage by ensuring the imputer and scaler fit only on training data, not on the test set.
Step 3: Train/Test Split
from sklearn.model_selection import train_test_split
X = df[NUMERIC_FEATURES + CATEGORICAL_FEATURES]
y = df['survived'].astype(int)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)
print(f"Train: {X_train.shape[0]} samples")
print(f"Test: {X_test.shape[0]} samples")
stratify=y ensures both splits have the same class ratio — essential with imbalanced datasets.
Step 4: Your First Model
A Decision Tree is a good starting point: it is interpretable, makes no distributional assumptions, and exposes overfitting clearly.
from sklearn.tree import DecisionTreeClassifier
# Build the full pipeline: preprocessing + model
model = Pipeline([
('prep', preprocessor),
('clf', DecisionTreeClassifier(random_state=42)),
])
# Fit on training data only
model.fit(X_train, y_train)
# Quick check on training data
train_acc = model.score(X_train, y_train)
test_acc = model.score(X_test, y_test)
print(f"Train accuracy: {train_acc:.4f}")
print(f"Test accuracy: {test_acc:.4f}")
# Train accuracy: 0.9809 ← suspiciously high
# Test accuracy: 0.7710 ← much lower → overfitting
The large gap between train and test accuracy is a classic sign of overfitting. An unconstrained decision tree memorizes the training data.
Step 5: Cross-Validation
A single train/test split is fragile — you might have gotten lucky (or unlucky) with the split. Cross-validation gives a more reliable estimate of generalization.
from sklearn.model_selection import cross_val_score, StratifiedKFold
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
cv_scores = cross_val_score(
model, X_train, y_train,
cv=cv,
scoring='f1', # Better than accuracy for imbalanced data
)
print(f"F1 per fold: {cv_scores.round(3)}")
print(f"Mean F1: {cv_scores.mean():.4f} ± {cv_scores.std():.4f}")
# F1 per fold: [0.682 0.701 0.694 0.671 0.689]
# Mean F1: 0.6874 ± 0.0106
We use F1 (harmonic mean of precision and recall) instead of accuracy because accuracy on an imbalanced dataset is misleading — a model that always predicts "did not survive" would score 61.8%.
Step 6: Hyperparameter Tuning with GridSearchCV
Now we constrain the tree to fight overfitting. max_depth is the most important knob.
from sklearn.model_selection import GridSearchCV
param_grid = {
'clf__max_depth': [3, 5, 7, 10, None],
'clf__min_samples_leaf': [1, 5, 10, 20],
'clf__criterion': ['gini', 'entropy'],
}
grid_search = GridSearchCV(
estimator=model,
param_grid=param_grid,
cv=cv,
scoring='f1',
n_jobs=-1,
verbose=1,
)
grid_search.fit(X_train, y_train)
print(f"Best params: {grid_search.best_params_}")
print(f"Best CV F1: {grid_search.best_score_:.4f}")
# Best params: {'clf__criterion': 'entropy', 'clf__max_depth': 5, 'clf__min_samples_leaf': 10}
# Best CV F1: 0.7231
The grid tried 5 × 4 × 2 = 40 combinations, each evaluated with 5-fold CV — 200 model fits in total. With n_jobs=-1 this runs in parallel across all CPU cores.
Step 7: Evaluate on the Test Set
The test set is your final, honest estimate of performance. You look at it once, after all tuning decisions are made.
from sklearn.metrics import classification_report, confusion_matrix, ConfusionMatrixDisplay
import matplotlib.pyplot as plt
best_model = grid_search.best_estimator_
y_pred = best_model.predict(X_test)
print(classification_report(y_test, y_pred, target_names=['Did not survive', 'Survived']))
precision recall f1-score support
Did not survive 0.82 0.87 0.84 161
Survived 0.76 0.68 0.72 101
accuracy 0.80 262
macro avg 0.79 0.78 0.78 262
weighted avg 0.80 0.80 0.80 262
# Confusion matrix
cm = confusion_matrix(y_test, y_pred)
disp = ConfusionMatrixDisplay(cm, display_labels=['Not survived', 'Survived'])
disp.plot(cmap='Blues')
plt.tight_layout()
plt.savefig('confusion_matrix.png', dpi=150)
The model correctly identifies 68% of actual survivors (recall=0.68). For a medical use case you would want higher recall, even at the cost of precision.
Step 8: Feature Importance
Understanding which features drive predictions builds trust in the model and guides future feature engineering.
import pandas as pd
# Access the tree inside the pipeline
tree = best_model.named_steps['clf']
feature_names = NUMERIC_FEATURES + CATEGORICAL_FEATURES
importance_df = (
pd.DataFrame({'feature': feature_names, 'importance': tree.feature_importances_})
.sort_values('importance', ascending=False)
)
print(importance_df)
# feature importance
# 1 sex 0.3812
# 0 age 0.2134
# 5 pclass 0.1876
# 2 sibsp 0.0943
# ...
Sex, age, and passenger class dominate — consistent with historical accounts of the disaster ("women and children first").
Where to Go From Here
You now have a working, properly validated model. But GridSearchCV has a fundamental limitation: it exhausts a predefined grid, which does not scale well as the number of parameters grows. A grid of 5 parameters with 5 values each means 5⁵ = 3,125 fits.
A smarter approach is to use evolutionary algorithms to search the hyperparameter space — the same idea behind sklearn-genetic-opt. If you want to see how that compares in practice, the article Are You Still Using Grid Search for Hyperparameters Optimization? shows both methods side by side on the same dataset.
About the author
Rodrigo Arenas is a software architect and machine learning engineer in Medellín, Colombia. He builds products, platforms, and open-source software for AI, Data Engineering, and Machine Learning — creator of Ciaren, sklearn-genetic-opt, and PyWorkforce.
Recommended reading
Rodrigo Arenas — ML Engineer & Open Source Maintainer
Working on something similar? I can help you take it to production.