AML Alert Triage: Precision vs Recall With Real Costs

Preview mode. Browse milestones and edit starter code here. Sign in to run and submit.

0/8EDA

Python

intermediate~6h8 milestones

The problem

A bank's transaction monitoring system raises tens of thousands of anti-money-laundering alerts a month, and about 92% of them are cleared as false positives after a human investigation that costs roughly $50 each. The compliance team cannot hire its way out. You are building the triage model that decides which alerts an analyst sees first and which are auto-closed. Send too few and confirmed laundering slips through, which is a regulatory fine. Send too many and the queue is the same queue you started with. Your job is to compute the metrics honestly, beat the legacy score, and defend an operating point in dollars.

Your task · Size the Queue

Establish how much of the legacy alert queue is noise, and which rule produces the most of it.

Implement explore_alerts(train_df) returning a dict with:

row_count: number of alerts in the training set
suspicious_count: number of alerts confirmed suspicious (is_suspicious == 1)
suspicious_rate: fraction confirmed, rounded to 6 places
legacy_false_positive_rate: 1 minus suspicious_rate, rounded to 6 places
rule_count: number of distinct rule_id values
noisiest_rule: the rule_id with the LOWEST confirmation rate
noisiest_rule_precision: that rule's confirmation rate, rounded to 6 places
first_time_alert_rate: confirmation rate where days_since_last_alert is NaN, rounded to 6 places
repeat_alert_rate: confirmation rate where days_since_last_alert is present, rounded to 6 places

The legacy false-positive rate is the number the compliance lead quotes in every meeting. The last two numbers are why you must not impute the missing history away.

Constraints

▪About 8% of alerts are confirmed suspicious, so accuracy is useless
▪days_since_last_alert is missing for accounts never alerted before, and first-time alerts are cleaner than repeat alerts
▪Analysts can review 25% of the legacy queue per day
▪A confirmed case is worth $1,500 in avoided regulatory exposure; a false alarm costs $50 of analyst time
▪Hidden test set is scored server-side, so you cannot tune against it

Graded on

Hand-computed precision, recall, F1 and average precision match the reference to 1e-4
PR-AUC >= 0.36 and ROC-AUC >= 0.80 on the hidden test set
Recall >= 0.62 while alerting on no more than 25% of alerts
Net benefit >= $1,250,000 with alerts capped at 40%
Batch scoring completes within 8 seconds

Evaluated server-side against a hidden test set.

Skills you'll demonstrate

Precision and RecallPrecision-Recall CurveClass ImbalanceThreshold SelectionCost-Sensitive LearningInformative MissingnessGradient Boosting