Methodologic quality and risk-of-bias in systematic reviews of healthcare interventions: a review of methods

Authors

DOI:

https://doi.org/10.17267/2675-021Xevidence.2022.e4067

Keywords:

Systematic Review, Healthcare, Methodologic Quality, Risk of bias, Critical appraisal

Abstract

OBJECTIVE: To compare the characteristics of systematic reviews of healthcare interventions that assessed or did not methodologic quality/risk-of-bias of included studies. Additionally, to analyze methodologic features of those assessing the methodologic quality/risk-of-bias. METHODS: PubMed database was searched. From 25,571 systematic reviews retrieved, a random sample of 1,025 was screened. Frequencies were used to describe outcomes. Unadjusted and adjusted logistic regressions were performed to test the associations with methodologic quality/risk-of-bias results assessment. In a second analysis, systematic reviews that assessed methodologic quality/risk of bias were dichotomized according to the design of included studies (randomized clinical trials-only versus non-randomized studies of intervention or a combination of both). RESULTS: 303 systematic reviews were fully analyzed. Methodologic quality/risk of bias was assessed by 278 (92%). Methodologic quality/risk-of-bias assessment was associated with a higher number of databases searched (>4, P= 0.008), the presence of meta-analysis (P= 0.005), and the design of included studies (randomized clinical trials-only, P= 0.042). The chance of using a suitable tool and a tool designed for risk-of-bias assessment rather than methodologic quality was higher for randomized clinical trials-only systematic reviews (P< 0.05). The most used tool was Cochrane’s RoB Tool without a clear studies’ overall risk classification system. CONCLUSION: methodologic quality/risk-of-bias assessment was associated with included studies’ design (randomized clinical trials-only), a meta-analysis of data, and the number of databases searched (>4). The most used tool was Cochrane’s RoB Tool, with no clearly defined rating system. Methodologic quality/risk-of-bias assessment methods description, results, and impacts on meta-analysis, the certainty of evidence, and systematic reviews’ results are still to be consistently addressed.

 

Downloads

Download data is not yet available.

References

Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn't. BMJ. 1996;312(7023):71-2. https://doi.org/10.1136/bmj.312.7023.71

Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097. https://doi.org/10.1371/journal.pmed.1000097

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008. https://doi.org/10.1136/bmj.j4008

Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924-6. https://doi.org/10.1136/bmj.39489.470347.ad

Lasserson TJ, Thomas J, Higgins JPT. Chapter 1: Starting a review [Internet]. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.1 (updated September 2020). London: Cochrane; 2020. Available from: www.training.cochrane.org/handbook

Boutron I, Page MJ, Higgins JPT, Altman DG, Lundh A, Hróbjartsson A. Chapter 7: Considering bias and conflicts of interest among the included studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.1 (updated September 2020). London: Cochrane; 2020. Available from: www.training.cochrane.org/handbook

Page MJ, Higgins JP, Clayton G, Sterne JA, Hrobjartsson A, Savovic J. Empirical Evidence of Study Design Biases in Randomized Trials: Systematic Review of Meta-Epidemiological Studies. PloS one. 2016;11(7):e0159267. https://doi.org/10.1371/journal.pone.0159267

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. https://doi.org/10.1136/bmj.n71

Pussegoda K, Turner L, Garritty C, Mayhew A, Skidmore B, Stevens A, et al. Systematic review adherence to methodological or reporting quality. Syst Rev. 2017;6(1):131. https://doi.org/10.1186/s13643-017-0527-2

Saltaji H, Ospina MB, Armijo-Olivo S, Agarwal S, Cummings GG, Amin M, et al. Evaluation of risk-of-bias assessment of trials in systematic reviews of oral health interventions, 1991-2014: A methodology study. J Am Dent Assoc. 2016;147(9):720-8.e1. https://doi.org/10.1016/j.adaj.2016.03.017

Munn Z, Stern C, Aromataris E, Lockwood C, Jordan Z. What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences. BMC Med Res Methodol. 2018;18(1):5. https://doi.org/10.1186/s12874-017-0468-4

Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011] [Internet]. London: The Cochrane Collaboration; 2011. Available from: http://handbook-5-1.cochrane.org/

Martinic MK, Pieper D, Glatt A, Puljak L. Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks. BMC Med Res Methodol. 2019;19(1):203. https://doi.org/10.1186/s12874-019-0855-0

Stefani CM, Rezende LVML, Honnef LR, Oliveira LB, Lopes NC, Oliveira JMD, et al. Methodologic quality and risk of bias in Systematic Reviews of Healthcare Interventions: a methodologic review. 2021. https://doi.org/10.17605/OSF.IO/CH5T9

Green S, Higgins JPT, Alderson P, et al. Cochrane handbook: cochrane reviews: Ch 8: assessing risk of bias in included studies. Cochr Handbook Syst Rev Intervent. 2011:3–10. https://training.cochrane.org/handbook/archive/v5.1/

Wells G, Shea B, O'Connell D, Peterson J, Welch V, Losos M, et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomized studies in meta-analysis, 2000 [Internet]. Available from: http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp

Guyatt GH, Oxman AD, Montori V, Vist G, Kunz R, Brozek J, et al. GRADE guidelines: 5. Rating the quality of evidence--publication bias. J Clin Epidemiol. 2011;64(12):1277-82. https://doi.org/10.1016/j.jclinepi.2011.01.011

Santos MBF, Agostini BA, Bassani R, Pereira GKR, Sarkis-Onofre R. Protocol registration improves reporting quality of systematic reviews in dentistry. BMC Med Res Methodol. 2020;20(1):57. https://doi.org/10.1186/s12874-020-00939-7

Higgins JPT, Sterne JAC, Savovi? J, Page MJ, Hróbjartsson A, Boutron I, et al. A revised tool for assessing risk of bias in randomized trials In: Chandler J, McKenzie J, Boutron I, Welch V, editors. [Internet] Cochrane Methods. Cochrane Database of Systematic Reviews. 2016. https://doi.org/10.1002/14651858.CD201601

Sterne JAC, Savovi? J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019;366:4898. https://doi.org/10.1136/bmj.l4898

Higgins JPT, Savovi? J, Page MJ, Elbers RG, Sterne JAC. Chapter 8: Assessing risk of bias in a randomized trial [Internet]. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). London: Cochrane; 2022. Available from: www.training.cochrane.org/handbook

Reeves BC, Deeks JJ, Higgins JPT, Shea B, Tugwell P, Wells GA. Chapter 24: Including non-randomized studies on intervention effects. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.1 (updated September 2020). London: Cochrane; 2020. Available from: www.training.cochrane.org/handbook

Sterne JAC, Hernán MA, Reeves BC, Savovi? J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomized studies of interventions. BMJ 2016;355:i4919. https://doi.org/10.1136/bmj.i4919

Sterne JAC, Hernán MA, McAleenan A, Reeves BC, Higgins JPT. Chapter 25: Assessing risk of bias in a non-randomized study. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.1 (updated September 2020). London: Cochrane; 2020. Available from: www.training.cochrane.org/handbook

Sterne J, Higgins J, Reeves B, Savovi? J, Turner L. Cochrane Scientific Committee. Recommendation statement/report. Review of the development of the risk of bias tool for nonrandomised studies for interventions – ROBINS-I [Internet]. Available from: https://methods.cochrane.org/sites/default/files/public/uploads/scientific_committee_statement_report_robins_i_fin.pdf

Coleman BD, Khan KM, Maffulli N, Cook JL, Wark JD. Studies of surgical outcome after patellar tendinopathy: clinical significance of methodological deficiencies and guidelines for future studies. Victorian Institute of Sport Tendon Study Group. Scand J Med Sci Sports 2000;10(1):2–11. https://doi.org/10.1034/j.1600-0838.2000.010001002.x

Downs SH, Black N. The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health. 1998;52(6):377-84. https://doi.org/10.1136/jech.52.6.377

Tufanaru C, Munn Z, Aromataris E, Campbell J, Hopp L. Chapter 3: Systematic reviews of effectiveness. In: Aromataris E, Munn Z, editors. JBI Manual for Evidence Synthesis. JBI; 2020. https://doi.org/10.46658/JBIMES-20-04

Aromataris E, Munn Z, editors. JBI Manual for Evidence Synthesis. JBI; 2020. https://doi.org/10.46658/JBIMES-20-01

Downloads

Published

2022-05-18

How to Cite

Miron Stefani, C., Vicente Melo de Lucas Rezende , L., Honnef, L. R., Butini Oliveira , L., da Cruz Lopes , N., Massignan, C., De Luca Canto, G., & Meller Dias de Oliveira , J. . (2022). Methodologic quality and risk-of-bias in systematic reviews of healthcare interventions: a review of methods . Journal of Evidence-Based Healthcare, 4, e4067. https://doi.org/10.17267/2675-021Xevidence.2022.e4067

Issue

Section

Research Articles