ML Model Monitoring in Production with Evidently
Learn how to detect data drift and model degradation in production using Evidently AI. Build automated reports and alerts before your model silently fails.
You trained a great model. It passed cross-validation with flying colors and sailed through your holdout set. You deployed it. And then, three months later, someone notices the predictions feel off.
This is not a hypothetical scenario — it is what happens to most production ML models. The world changes. Your model does not. This article shows how to detect that drift before it becomes a business problem, using Evidently: drift reports, model performance reports, and automated test suites you can wire into alerts.
Why Models Degrade
There are two main failure modes:
Data drift: the statistical distribution of the input features changes. Example: your fraud detection model was trained when most transactions were under 450. The model has never seen that region of the feature space.
Concept drift: the relationship between inputs and the target changes. Example: customer behavior after a major economic event shifts — features that used to predict churn no longer do.
Both manifest as silent degradation: the API keeps returning predictions, just worse ones. Monitoring catches this before stakeholders do.
Installing Evidently
pip install evidently scikit-learn pandas numpy
We will build a complete example: train a churn prediction model, simulate production data with drift, and generate Evidently reports.
Training the Reference Model
# train.py
import joblib
import numpy as np
import pandas as pd
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
np.random.seed(42)
n = 2000
# Simulate customer features
df_train = pd.DataFrame({
'tenure_months': np.random.exponential(24, n),
'monthly_charges': np.random.normal(65, 20, n).clip(20, 150),
'num_products': np.random.randint(1, 6, n),
'support_calls': np.random.poisson(1.5, n),
'last_login_days': np.random.exponential(15, n),
})
# Target: churn (higher support calls and fewer products → more likely to churn)
churn_prob = (
0.3
- 0.008 * df_train['tenure_months']
- 0.05 * df_train['num_products']
+ 0.04 * df_train['support_calls']
+ 0.003 * df_train['last_login_days']
).clip(0.05, 0.95)
df_train['churn'] = (np.random.random(n) < churn_prob).astype(int)
X = df_train.drop('churn', axis=1)
y = df_train['churn']
X_train, X_ref, y_train, y_ref = train_test_split(X, y, test_size=0.3, random_state=42)
model = Pipeline([
('scaler', StandardScaler()),
('clf', GradientBoostingClassifier(n_estimators=100, random_state=42)),
])
model.fit(X_train, y_train)
print(f"Train accuracy: {model.score(X_train, y_train):.4f}")
print(f"Val accuracy: {model.score(X_ref, y_ref):.4f}")
# Save model and reference dataset
joblib.dump(model, 'model.joblib')
X_ref.to_parquet('reference_data.parquet', index=False)
y_ref.to_frame().to_parquet('reference_labels.parquet', index=False)
The reference dataset is crucial: it is the statistical baseline that Evidently compares against when new data arrives.
Simulating Drifted Production Data
In a real scenario, you would collect live predictions. Here we simulate two months of production data where the distribution has drifted:
# simulate_production.py
import numpy as np
import pandas as pd
np.random.seed(99)
n = 500
# Month 1: similar to training (no drift yet)
df_month1 = pd.DataFrame({
'tenure_months': np.random.exponential(24, n),
'monthly_charges': np.random.normal(65, 20, n).clip(20, 150),
'num_products': np.random.randint(1, 6, n),
'support_calls': np.random.poisson(1.5, n),
'last_login_days': np.random.exponential(15, n),
})
# Month 2: drifted — new pricing pushed charges up, customers use fewer products
df_month2 = pd.DataFrame({
'tenure_months': np.random.exponential(18, n), # shorter tenures (new users)
'monthly_charges': np.random.normal(95, 25, n).clip(20, 150), # higher charges
'num_products': np.random.randint(1, 3, n), # fewer products (plan change)
'support_calls': np.random.poisson(3.0, n), # more support calls
'last_login_days': np.random.exponential(30, n), # less engagement
})
df_month1.to_parquet('production_month1.parquet', index=False)
df_month2.to_parquet('production_month2.parquet', index=False)
Data Drift Report
Evidently compares the reference dataset against your current production data and calculates statistical tests for each feature:
# report_drift.py
import pandas as pd
import joblib
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset, DataQualityPreset
reference = pd.read_parquet('reference_data.parquet')
month1 = pd.read_parquet('production_month1.parquet')
month2 = pd.read_parquet('production_month2.parquet')
# ── Month 1: should look clean ──────────────────────────────────────
report_m1 = Report(metrics=[DataDriftPreset(), DataQualityPreset()])
report_m1.run(reference_data=reference, current_data=month1)
report_m1.save_html('drift_month1.html')
# ── Month 2: should show drift ──────────────────────────────────────
report_m2 = Report(metrics=[DataDriftPreset(), DataQualityPreset()])
report_m2.run(reference_data=reference, current_data=month2)
report_m2.save_html('drift_month2.html')
Open drift_month2.html in your browser. You will see something like:
monthly_charges: DRIFT DETECTED (Wasserstein distance = 0.42)support_calls: DRIFT DETECTED (PSI = 0.31)tenure_months: borderline- Share of drifted features: 3/5 (60%)
Model Performance Report
If you have labels for the production data (common in batch scenarios with delayed feedback), you can also monitor model quality directly:
import numpy as np
from evidently import ColumnMapping
from evidently.metric_preset import ClassificationPreset
model = joblib.load('model.joblib')
# Add predictions and simulate labels for month 2
month2_eval = month2.copy()
month2_eval['prediction'] = model.predict(month2_eval)
month2_eval['target'] = (np.random.random(len(month2_eval)) < 0.45).astype(int) # degraded
reference_eval = reference.copy()
reference_eval['prediction'] = model.predict(reference_eval)
reference_eval['target'] = pd.read_parquet('reference_labels.parquet')['churn'].values
perf_report = Report(metrics=[ClassificationPreset()])
perf_report.run(
reference_data=reference_eval,
current_data=month2_eval,
column_mapping=ColumnMapping(target='target', prediction='prediction'),
)
perf_report.save_html('model_performance.html')
Programmatic Alerts
HTML reports are useful for exploration, but for automated monitoring you need programmatic access to the results:
from evidently.test_suite import TestSuite
from evidently.tests import (
TestShareOfDriftedColumns,
TestColumnDrift,
)
suite = TestSuite(tests=[
TestShareOfDriftedColumns(lt=0.3), # Fail if >30% of columns drift
TestColumnDrift(column_name='monthly_charges'),
TestColumnDrift(column_name='support_calls'),
])
suite.run(reference_data=reference, current_data=month2)
results = suite.as_dict()
for test in results['tests']:
status = test['status']
name = test['name']
print(f"[{status}] {name}")
# [FAIL] Share of Drifted Columns
# [FAIL] Column Drift: monthly_charges
# [FAIL] Column Drift: support_calls
You can wire results into any alerting system — Slack, PagerDuty, email — to get notified the moment drift exceeds your thresholds.
Scheduling Regular Drift Checks
In production, you typically run drift checks on a schedule. A simple cron approach:
# monitor.py — run daily via cron or Prefect
import datetime
import pandas as pd
from evidently.test_suite import TestSuite
from evidently.tests import TestShareOfDriftedColumns
def run_daily_check(production_parquet: str, reference_parquet: str) -> bool:
reference = pd.read_parquet(reference_parquet)
current = pd.read_parquet(production_parquet)
suite = TestSuite(tests=[TestShareOfDriftedColumns(lt=0.3)])
suite.run(reference_data=reference, current_data=current)
passed = suite.as_dict()['summary']['all_passed']
date = datetime.date.today().isoformat()
if not passed:
# trigger_alert(f"[{date}] Drift detected in production data!")
print(f"[{date}] ALERT: drift detected")
else:
print(f"[{date}] OK: no significant drift")
return passed
if __name__ == '__main__':
run_daily_check('production_today.parquet', 'reference_data.parquet')
If you already orchestrate retraining with Prefect, this check slots in as one more scheduled flow: Orchestrate ML Pipelines with Prefect.
What to Do When Drift Is Detected
Detection is only the first step. When an alert fires, the playbook is:
- Investigate: which features drifted? By how much?
- Diagnose root cause: upstream data pipeline change? Seasonality? Real-world event?
- Decide: retrain, adapt preprocessing, or escalate to a human decision?
- Retrain if needed: use the drifted data as the new reference after retraining
The key insight is that monitoring does not replace retraining — it tells you when to retrain, and with what data.
Conclusion
Silent model degradation is one of the most common and costly problems in production ML. Evidently gives you the infrastructure to detect it automatically, with minimal setup. The pattern is always the same: collect a representative reference dataset at training time, compare production data against it regularly, and act when statistical tests signal a significant shift.
To close the MLOps loop, track your retraining experiments from day one: Manage Your Machine Learning Lifecycle with MLflow in Python.
About the author
Rodrigo Arenas is a software architect and machine learning engineer in Medellín, Colombia. He builds products, platforms, and open-source software for AI, Data Engineering, and Machine Learning — creator of Ciaren, sklearn-genetic-opt, and PyWorkforce.
Recommended reading
Rodrigo Arenas — ML Engineer & Open Source Maintainer
Working on something similar? I can help you take it to production.