Matthew D. Montierth, Hao Yan, Liyang Xie, Kinga Nemeth, Xiaoxi Pan, Ruonan Li, Caner Ercan, Peng Yang, Ansam Sinjab, Tieling Zhou, Fuduan Peng, Manisha Singh, Linghua Wang, Scott Kopetz, Humam Kadara, Yinyin Yuan, George A. Calin, Wenyi Wang
Abstract Estimating tumor-specific transcript proportions from mixed bulk samples has potential to inform novel biology. However, estimation accuracy using existing methods in sparse-count data such as microRNA-seq and spatial transcriptomics has yet to be established. We generate a mixed small RNA benchmark dataset to demonstrate analytical challenges. To resolve them, we develop DeMixNB, a semi-reference-based deconvolution model assuming a sum of negative binomial distributions. Applications to miRNA-seq from 885 patients with breast cancer and 4,709 spatial spots from lung cancer generates clinical and mechanistic insights into tumor cell plasticity. This supports the important utility of DeMixNB to investigate cancer RNomes.