Hamed Movahed, Reza Donald Mirza, Romina Brignardello-Petersen, Alfonso Iorio, Niveditha Devasenapathy, Hossein Movahed, Qinwei Fu, Mobina Bagherianlemraski, Sara Ibrahim, Erfan Taherifard, Mahya Dorri, Maria-Jose Oliveros, Md Kamrul Hasan, Shahab Sayfi, Rachel J Couban, Gordon Guyatt
Use of thresholds in systematic reviews using the GRADE framework remains suboptimal, particularly in non-Cochrane reviews. For systematic reviews that do use thresholds, reviewers choose the null and the MID equally and very seldom use multiple thresholds, suggesting the perceived utility of the available options.
OBJECTIVE: Current guidance regarding applying the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) framework mandates establishing thresholds to assess certainty of evidence. These thresholds include the null, the minimally important difference (MID), and moderate and large effect thresholds. GRADE experts offer a rationale for use of each particular threshold. However, GRADE methodologists continue to debate the optimal choice of threshold, particularly whether to use the null threshold. We aimed to describe how systematic review authors used thresholds and in doing so to provide insight into the perceived utility of the available thresholds and to inform ongoing debate regarding the optimal choice of threshold.
STUDY DESIGN AND SETTING: We conducted a systematic survey sampling the 200 most recently published Cochrane and 200 non-Cochrane systematic reviews that used GRADE, which we identified from the Cochrane Database of Systematic Reviews (2018 to July 2024) and MEDLINE (2018 to December 2024). We documented whether review authors used thresholds and the thresholds they chose.
RESULTS: Among the sampled reviews, 118 of 200 (59%) Cochrane and 63 of 200 (31.5%) non-Cochrane reviews used thresholds. Among the reviews with an identifiable threshold type, 55 of 112 (49.1%) Cochrane and 26 of 55 (47.3%) non-Cochrane reviews used the null, while 54 of 112 (48.2%) Cochrane and 25 of 55 (45.5%) non-Cochrane reviews used the MID. Few reviews used the MID, moderate, and large effect thresholds together to form ranges of effects (2 of 112 [1.8%] Cochrane; 3 of 55 [5.5%] non-Cochrane).
CONCLUSION: Use of thresholds in systematic reviews using the GRADE framework remains suboptimal, particularly in non-Cochrane reviews. For systematic reviews that do use thresholds, reviewers choose the null and the MID equally and very seldom use multiple thresholds, suggesting the perceived utility of the available options.