Nested logistic regression models and ΔAUC applications : change-point analysis

Abstract

The area under the receiver operating characteristic curve (AUC) is one of the most popular measures for evaluating the performance of a predictive model. In nested models, the change in AUC (ΔAUC) can be a discriminatory measure of whether the newly added predictors provide significant improvement in terms of predictive accuracy. Recently, several authors have shown rigorously that ΔAUC can be degenerate and its asymptotic distribution is no longer normal when the reduced model is true, but it could be the distribution of a linear combination of some χ12 random variables [1,2]. Hence, the normality assumption and existing variance estimate cannot be applied directly for developing a statistical test under the nested models. In this paper, we first provide a brief review on the use of ΔAUC for comparing nested logistic models and the difficulty of retrieving the reference distribution behind. Then, we present a special case of the nested logistic regression models that the newly added predictor to the reduced model contains a change-point in its effects. A new test statistic based on ΔAUC is proposed in this setting. A simple resampling scheme is proposed to approximate the critical values for the test statistic. The inference of the change-point parameter is done via m-out-of-n bootstrap. Large-scale simulation is conducted to evaluate the finite-sample performance of the ΔAUC test for the change-point model. The proposed method is applied to two real-life datasets for illustration.