On the illusion of scRNA-seq batch effect correction

Batch correction methods in single-cell RNA sequencing are essential for removing technical variation that can otherwise lead to misleading downstream analyses. The reliability of these methods is typically evaluated using unsupervised metrics. Here, we apply a Machine Learning (ML) technique called probing, formalized as the Batch Probing Score (BPS), to empirically demonstrate across six datasets that the most popular batch correctors fail to fully remove batch signal. In most cases, the batch of origin remains clearly identifiable after correction, even though standard evaluation metrics cannot detect it. We show that existing unsupervised metrics lack the sensitivity and specificity required to capture residual batch signal, whereas ML-based approaches can still detect it. This residual signal can similarly be picked up by downstream analysis tools, potentially leading to biased results. Because BPS is supervised, it directly quantifies batch signal strength by measuring how accurately the batch of origin can be predicted for each sample. It therefore provides an upper bound on the residual ML-actionable batch signal that could otherwise remain unnoticed. Our findings suggest that probing-based metrics should become a standard for assessing batch correction methods in single-cell RNA-seq and other areas of genomics.