ML Monitoring with Evidently: Detect Data Drift in Python

Data drift means the distribution of an input, prediction or other logged value has changed relative to a reference period. It is a signal to investigate, not proof that model quality fell. Quality monitoring needs delayed ground-truth labels and a metric tied to the decision.

This example was tested with Python 3.13 and Evidently 0.7.23: python -m pip install "evidently==0.7.23" pandas numpy. The optional labeled-quality block also uses scikit-learn and pyarrow to read Parquet. Lock the complete dependency set in a real project and validate upgrades; Evidently’s older evidently.report and evidently.metric_preset imports belong to versions 0.6.7 and earlier.

Make a reference and a current batch

Use the same feature names, types and preprocessing stage in both datasets. The following data is deliberately synthetic: the changed monthly_charge distribution demonstrates a check, not a degraded production model.

import numpy as np
import pandas as pd

rng = np.random.default_rng(42)
reference = pd.DataFrame({
    "tenure_months": rng.exponential(24, 600),
    "monthly_charge": rng.normal(65, 12, 600),
    "support_calls": rng.poisson(1.5, 600),
})
current = pd.DataFrame({
    "tenure_months": rng.exponential(24, 300),
    "monthly_charge": rng.normal(90, 12, 300),  # intentional synthetic shift
    "support_calls": rng.poisson(1.5, 300),
})

A reference should represent a period in which the data pipeline and the model’s operating conditions were acceptable. Do not silently replace it after an alert: retain its identifier and version so comparisons remain interpretable.

Run and export a drift report

from evidently import DataDefinition, Dataset, Report
from evidently.presets import DataDriftPreset

definition = DataDefinition(numerical_columns=list(reference.columns))
current_dataset = Dataset.from_pandas(current, data_definition=definition)
reference_dataset = Dataset.from_pandas(reference, data_definition=definition)
report = Report([DataDriftPreset(columns=list(reference.columns))], include_tests=True)
snapshot = report.run(current_dataset, reference_dataset, name="daily-input-drift")

snapshot.save_html("input_drift.html")
with open("input_drift.json", "w", encoding="utf-8") as output:
    output.write(snapshot.json())

The data definition treats these three measurements as numeric, including the count support_calls. Automatic type inference can treat a low-cardinality integer column as categorical; in this synthetic sample, the resulting chi-square comparison encounters an empty reference category and divides by zero. An explicit type follows the meaning of the feature rather than suppressing that warning.

report.run(current_dataset, reference_dataset) returns a Snapshot; export from that object. include_tests=True enables automatic test conditions as well as drift metrics. Without it, the exported report has metrics but no tests in the pinned release. The preset's default dataset-level drift-share threshold is 0.5: a changed column can fail its test while the dataset-level test passes. Inspect the column results and calibrate thresholds with historical stable batches; that default is not a business alert policy.

Add quality only when labels arrive

Input drift can occur without a loss of model quality, and quality can fall without detectable input drift. When the true outcome arrives, log it with the prediction and evaluate it separately. The example below calculates a plain, reproducible metric; it does not manufacture labels to claim degradation.

from sklearn.metrics import roc_auc_score

# `label` and continuous `score` come from one completed cohort.
# Labels are binary and the batch contains at least one example of each class.
scored_batch = pd.read_parquet("labeled_production_batch.parquet")
auc = roc_auc_score(scored_batch["label"], scored_batch["score"])
print(f"batch ROC AUC: {auc:.3f}")

Compare that value with a predefined baseline and uncertainty appropriate to the volume and decision. A change can result from labels, sampling, missing joins, a changed score-producing model or a real relationship change; a decision threshold alone does not change ROC AUC computed from continuous scores. Check data quality and the serving path before retraining.

A practical response to an alert

1. Preserve the report, batch window, reference version and code version.
2. Inspect the changed columns and upstream transformations; seasonality and a schema change need different fixes.
3. Check prediction distribution and, once available, labeled performance.
4. Decide with the model owner whether to fix data, collect labels, adjust a documented threshold, retrain, or pause the decision.

This script creates evidence for review. It does not schedule jobs, deliver alerts, guarantee drift detection, or establish that a retrained model is safe for production.

Sources

• Evidently Report API
• Evidently DataDriftPreset API
• Evidently repository quickstart
• Evidently data definition