Ming-Hsun Lee, Wei-Yang Lin, Yen-Jung Chiu
Frozen foundation-model features are promoted as an annotation-sparing route to medical-image classification, yet whether their performance reflects diagnostic signal or site-specific shortcuts is rarely tested. On a single-site hospital cohort (187 cancer and 194 benign scans, 303 patients), frozen Models-Genesis whole-scan features classified cancer versus benign at an out-of-fold AUC of 0.994 under patient-grouped cross-validation and transferred near chance to two external cohorts. One is LIDC-IDRI, where malignancy is a median radiologist rating, not a confirmed diagnosis. The other carries confirmed diagnoses matching the development endpoint. We tested acquisition as a competing explanation. The two classes had been reconstructed under different protocols (0.7 mm versus 1.0 mm slice spacing), and a classifier reading six acquisition header fields, and no image data, reached 0.996, within 0.001 of the frozen-feature result ( p = 0.73 ). The near-perfect in-site AUC is therefore most plausibly explained by acquisition-related confounding, which extends beyond reconstruction protocol to scanner model and age, and is not evidence of malignancy discrimination. A second, independently pretrained CT foundation model reproduced the pattern, so it is a property of the data, not one checkpoint. Malignancy-related signal appeared at finer spatial granularity. Narrowing the field of view to one prespecified nodule per scan raised AUC from 0.584 to 0.808, and a nodule-level classifier transferred between external cohorts at 0.682 (95% CI 0.559-0.792) against confirmed benign controls under one of the two encoders. Whole-scan embeddings exposed a cohort-assembly shortcut strong enough to mimic a solved task; the signal that transferred was carried by lesion-level representations.