You build a gradient boosting model that performs significantly better than a simpler logistic regression on cross-validation, but when deployed, business stakeholders notice predictions are unstable over time and difficult to understand. How would you diagnose the cause, and what concrete steps would you take to improve both stability and interpretability without losing too much performance?

Question

VirtualInterview.AI · Accepted Answer

Problem: A gradient boosting model outperformed logistic regression by ~8 AUC points in CV, but production scores fluctuated week-to-week and stakeholders couldn’t interpret drivers. Solution: I’d first compare train/CV/production score distributions and feature drift using population stability index and Kolmogorov–Smirnov tests, plus SHAP drift over time. If instability is due to noisy or drifting features, I’d remove/transform them, add regularization (shrinkage, max_depth, min_samples_leaf), and retrain with time-based CV. For interpretability, I’d constrain the model (e.g., monotonic XGBoost, 50–100 trees), produce global SHAP summaries, and build a simpler surrogate model for explanations. Impact: On a similar project, this reduced weekly score volatility by ~40%, maintained ~95% of the AUC gain vs. logistic regression, and enabled clear top-5 driver explanations used in exec reviews.

Sample Answer

Keywords