The Pipeline That Broke in Production

Preview mode. Browse milestones and edit starter code here. Sign in to run and submit.

0/8EDA

Python

intermediate~7h8 milestones

The problem

A churn model shipped three months ago and the intervention team says it has gone bad. The model has not changed and the relationship between behaviour and churn has not changed either: what changed is the data arriving at the door. Four separate things went wrong upstream, each of a different kind, and each needs a different check to catch. Your job is to build the validation and monitoring layer that names them, then harden the feature pipeline so the model survives contact with them.

Your task · Profile the Reference Period

Turn the clean training frame into a schema you can check future batches against.

Implement build_reference_profile(train_df) returning a dict with:

row_count: number of reference rows

numeric_columns: the five numeric feature names, sorted

categorical_columns: the two categorical feature names, sorted

ranges: dict mapping each numeric column to [min, max], each rounded to 4 places

category_values: dict mapping each categorical column to its sorted list of observed values

null_counts: dict mapping every feature column to its null count in the reference period

Exclude the label column "churned" throughout.

This dict is the contract. Everything in the next four milestones is a comparison against it, and the reason to build it explicitly is that "the data looks fine" is not something you can diff.

Constraints

▪The training frame is the clean reference period, the test frame is production
▪Only session_minutes has genuinely drifted; the other columns are stable
▪bandwidth_gb is logged in megabytes for a contiguous block of 1,200 rows
▪plan_tier gains a category that sorts second alphabetically
▪support_score is null for 901 rows because an upstream job failed
▪The label relationship is unchanged, so retraining is not the answer

Graded on

All four defects detected and correctly classified
PSI over 0.25 flagged on session_minutes and on nothing else
Naive pipeline scores at or below 0.50 F1; hardened scores at or above 0.54
Shard-merged mean and variance match the single-pass values

Evaluated server-side against a hidden test set.

Skills you'll demonstrate

Data ValidationSchema EnforcementPopulation Stability IndexDistribution DriftPipeline DesignDistributed StatisticsProduction Monitoring