Rare-event study warns that tuning can create illusory model gains

A peer-reviewed Scientific Reports benchmark evaluated 30 models and ten hyperparameter-optimisation strategies on rare failures in urban electric transport. The authors found mean cross-validation optimism of +0.17 ROC-AUC and reported that no tuned configuration beat a sensible default under matched evaluation; even the best model had no usable operational threshold at the observed failure rate.

Why it matters: Predictive-maintenance teams should use nested, leakage-resistant evaluation and business-cost thresholds; model-selection scores can look strong while remaining operationally useless.

Scientific Reports