Novel document-level measures and their performance for active learning uncertainty sampling: a use case for automatic CDSS ontology curation

Human-in-the-loop active learning (AL) can identify key phrases to facilitate biomedical ontology development, maintenance, and other curation tasks. However, determining which documents to annotate by humans is not straightforward. We explored new strategies to make the document selection process transparent, reproducible, and effective. Our AL pipeline modified a BiLSTM-CRF model using PubMed abstracts. We tested four novel document-level uncertainty aggregation strategies: KPSum, KPAvg, DocSum, and DocAvg, that operate over standard token-level uncertainty scores: Minimum Token Probability (MTP), Token Entropy (TE), and Margin. All strategies show significant improvement in early active learning cycles ({theta} to {theta}2) for recall and F1. The systematic evaluations show that KPSum (actual order) shows consistent improvement in both recall and F1. The weighted F1 ({beta} = 5, 10) provided complementary results to raw recall and F1 ({beta} = 1). Our work advances uncertainty sampling by introducing document-level uncertainty aggregation and shows promise in automating ontology curation.