Ariana Peck, Joshua Hutchings, Jonathan Schwartz, Yue Yu, Utz H Ermel, Saugat Kandel, Dari Kimanius, Zhuowen Zhao, Shawn Zheng, Brendan Artley, David List, Sergio A Silva, Walter Reade, Jeremy Asuncion, Kira Evans, Jessica Gadling, Kandarp Khandwala, Suzette McCanny, Dannielle G McCarthy, Jun Xi Ni, Janeece Pourroy, Manasa Venkatakrishnan, Zun Shi Wang, David A Agard, Clinton S Potter, Bridget Carragher, Kyle I S Harrington, Mohammadreza Paraan
The difficulty of particle picking in cryo-electron tomography remains a barrier to routine in situ structure determination. Machine learning is well suited to overcome this bottleneck with efficient algorithms that generalize across molecular species. To spur new algorithm development, we held a 3-month Kaggle challenge that tasked contestants with annotating five molecular species across hundreds of experimental tomograms. Here we analyze the results of this competition, which successfully engaged >1,000 participants and delivered particle pickers that outperformed existing state of the art. Systematic comparisons of the contestants' submissions revealed the tolerance of subtomogram averaging to moderate but not severe over-picking and underscored the need for more robust measures of annotation quality. The winning models also highlighted the importance of data augmentation to overcome limited training data. All competition tomograms along with the ground truth and winning teams' annotations have been released on the CryoET Data Portal as a resource to benchmark current and future particle picking algorithms.