06 / Machine learning evaluation
Compare three fitted models and a simple baseline. Change the threshold to see which recorded outcomes each model identifies or misses.
Question
What errors accompany the validation-selected score threshold on held-out recorded outcomes?
Fixed comparison: L2 logistic regression at its 2015-validation-selected threshold of 0.23; 28,088 selected 2016 test loans. Controls below do not change this summary.
At threshold 0.23, the model flags 2,297 of 3,688 recorded Charged Off outcomes, or 62.28%, and also flags 9,159 Fully Paid loans. Only 20.05% of flagged records are Charged Off. These are retrospective labels, not credit decisions.
Before interpreting model ranking as a useful decision rule, an evaluation reviewer should inspect recall, precision and both error counts together. This example quantifies the trade-off; lending costs and an operational threshold are not established.
The extract includes only Fully Paid and Charged Off loans, with unknown follow-up and unresolved borrower independence. Scores are not validated future-default probabilities. No fairness, lending profitability or production-readiness claim is made.
Positive means the recorded Charged Off outcome. A lower threshold generally identifies more of these outcomes while also flagging more Fully Paid loans.
Retrospective evaluation: the extract includes only Fully Paid and Charged Off loans, with an unknown follow-up horizon. The 2016 test Charged Off rate is 13.13%, versus 24.90% in validation. Scores are not validated future-default probabilities or credit decisions.
Start here Move the threshold to compare flagged and missed outcomes. Changing model restores its validation threshold.
Move the slider in 0.01 steps or enter an exact threshold. The validation-selected threshold is shown after the data loads.
Loading held-out evaluation…
Precision = Charged Off among flagged records. Recall = flagged among actual Charged Off records. A false positive is a flagged Fully Paid record; a false negative is a missed Charged Off record. Precision is unavailable when nothing is flagged.
Every model uses the same held-out test cohort. These metrics do not change when the threshold moves.
ROC-AUC measures ranking discrimination. Average precision summarizes precision across recall levels; its no-skill reference is the test prevalence, 0.1313. Brier error is mean squared score error; lower is better. Small differences do not establish significant model superiority.
The new pipeline narrows leakage risks and exposes evaluation limits. Historical notebook, report and fairness claims are not reused.
Inspect aggregate results immediately. Full retraining requires authorized access to the exact historical source file identified in the package.
Python code and tests included. Full retraining needs the original loan CSV.
Python with the pinned packages. Synthetic tests run without borrower data.
Authorized exact CSV required for full retraining; borrower records are excluded.
Python pipeline, dependencies, model card, corrections, synthetic tests, results and independent metric checks.
Download reproduction ZIPZIP · 48 KBSplit definitions, feature exclusions, threshold rule and exact source requirements.
Download method notesMD · 6 KBpython -m pip install -r requirements.txt python -m unittest -v test_evaluation python evaluate.py --data YOUR_CSV --output results
Run inside the extracted package. Tests use synthetic fixtures; the last command retrains on the exact source.