Agreement between heuristic shrinkage factor and optimal shrinkage factors in logistic regression for risk prediction: a simulation study across different sample sizes and settings

Abstract Introduction The heuristic shrinkage factor of Van Houwelingen and Le Cessie (𝑆 𝑉𝐻 ) is a commonly used closed-form solution to adjust for overfitting in unpenalised logistic regression models for risk prediction. It is also the basis of widely-adopted minimum sample size criteria for developing clinical prediction models. However, current evidence is lacking regarding the bias of 𝑆 𝑉𝐻 compared to the optimal shrinkage factor (𝑆 𝑜𝑝𝑡 ). Here, we examine this issue and also assess the bias of an alternative bootstrap-derived shrinkage factor (𝑆 𝑏𝑜𝑜𝑡 ). Methods We undertook two simulation studies. The first examined the bias of 𝑆 𝑉𝐻 and 𝑆 𝑏𝑜𝑜𝑡 as estimators of 𝑆 𝑜𝑝𝑡 across a range of different scenarios defined by 𝐶 𝑝𝑜𝑝 , the C-statistic of the model developed in a population sized dataset. The second examined the bias of 𝑆 𝑜𝑝𝑡 when using development sample sizes targeting a shrinkage of 0.9, based on a sample size calculation defined by 𝑆 𝑉𝐻 itself (𝑁 𝑜𝑟𝑖𝑔𝑖𝑛𝑎𝑙 ) or by an adapted simulation-based approach (𝑁 𝑠𝑖𝑚 ). Results For high C-statistics, 𝑆 𝑉𝐻 overestimated 𝑆 𝑜𝑝𝑡 , whereas for low C-statistics 𝑆 𝑉𝐻 underestimates 𝑆 𝑜𝑝𝑡 . For example, across scenarios when 0.8≤𝐶 𝑝𝑜𝑝 <0.85, the 95-percentile range in the bias was (0.005,0.387), compared to (−0.580,−0.007) across scenarios when 0.6≤𝐶 𝑝𝑜𝑝 <0.65. The magnitude of bias increased as 𝐶 𝑝𝑜𝑝 tended to either 0.5 or 1. As sample size increased and 𝑆 𝑜𝑝𝑡 →1, the magnitude of the bias in either direction reduced. 𝑆 𝑏𝑜𝑜𝑡 was less biased than 𝑆 𝑉𝐻 , with a median magnitude of bias across all scenarios of 0.007, compared to 0.032 for 𝑆 𝑉𝐻 . Developing models on datasets of size 𝑁 𝑠𝑖𝑚 gave 𝑚𝑒𝑎𝑛(𝑆 𝑜𝑝𝑡 ) closer to 0.9 (mean magnitude of bias across all scenarios 0.004) than 𝑁 𝑜𝑟𝑖𝑔𝑖𝑛𝑎𝑙 (mean magnitude of bias 0.041). Conclusions 𝑆 𝑉𝐻 is often a poor estimator of the optimal global shrinkage factor. If global shrinkage is needed, we recommend using the bootstrap shrinkage estimate. The bootstrap estimate shows minimal bias in most scenarios, though in small samples the variability is large so provides no guarantees to address overfitting in a single dataset. A sample size calculation based on simulation is often preferable over formula dependent on targeting 𝑆 𝑉𝐻 .