Loan Default Prediction
Chose a logistic model (AUC 68.2%, 9.5% false negatives) and a 16% approval cutoff that maximizes expected profit.
Role Modeler and analystWhen Carnegie Mellon, ETE AnalyticsTeam Two people
A lender makes money on the loans it approves and loses it on the ones that default. This project asked how to decide who to approve, using borrower-level data on about 9,600 loans: FICO score, interest rate, whether the borrower met the lender’s credit policy, revolving utilization and similar fields. My teammate and I built the models and then turned the best one into a lending rule.
What to optimize
Before picking a model we had to decide what “good” meant. A loan that defaults and gets approved anyway costs far more than a good loan that gets declined, so we set the primary goal as keeping the false-negative rate low (defaults we fail to catch), with AUC as the second measure. Because lending is a mature industry where consistent returns matter more than growth, we put profit ahead of market share and return on investment when we had to choose.
Three models
We tried a decision tree, a logistic regression and a random forest, and used ROC curves and counterfactual tables to compare them.
- The tree looked strong at first. It used only interest rate, credit policy and FICO score, and the credit-policy split alone accounted for 84% of its predictive power. But its counterfactual false-negative rate was about 12%, against about 10% for the logistic model, so the higher AUC was misleading for our actual goal.
- The revised logistic regression kept only significant, uncorrelated variables, chosen with stepwise regression. It scored an AUC of 68.2% with a 9.5% false-negative rate.
- The random forest came out almost level on false negatives (9.4%) but lower on AUC (66.3%), and it is much harder to explain to an underwriter.
We chose the logistic model. Its AUC was a little higher than the forest’s, it missed almost the same share of defaults, and a lender can see why any application scored the way it did.
Setting the cutoff
A model only gives a probability, so someone has to pick the cutoff. I built an optimization model that tested approval cutoffs against expected profit and landed on 16%. At that cutoff, expected profit is maximized across the interest rates a lender would realistically charge, and it matches what the market does.
What I’d change
An AUC in the high 60s is modest, and I’d say so to anyone relying on it. With more time I’d test a gradient-boosted model, check performance across borrower groups before using it in credit decisions, and work out the cost of a false positive more carefully, since turning away a good borrower costs money too.
The presentation
