Subject Guides: Evidence Synthesis Reviews: 6. Critical appraisal

Critical Appraisal / Risk of Bias Assessment

Critical Appraisal, also referred to as the Risk of Bias Assessment, is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context. This usually requires a team of at least two using a validated Risk of Bias tool.

Validated Critical Appraisal / Risk of Bias Assessment Tools

Cochrane Risk of Bias Tool (RoB 2)
The Risk of Bias 2 (RoB 2) tool is an update to the original risk of bias tool that launched in 2008. It provides a framework for assessing the risk of bias in a single result from any type of randomized trial.
JBI Critical Appraisal Tools
The Joanna Briggs Institute has various checklists to assist in assessing the trustworthiness, relevance, and results from a wide variety of study designs.
Cochrane ROBINS-I tool
The ROBINS-I tool (“Risk Of Bias In Non-randomised Studies - of Interventions”) is concerned with evaluating risk of bias in estimates of the effectiveness or safety (benefit or harm) of an intervention from studies that did not use randomisation to allocate interventions.

AI for Systematic Reviews: Critical Appraisal

Generative AI in Critical Appraisal

There is some literature to suggest that large language model generative AI tools perform comparably to humans in conducting critical appraisal. The tools are observed to be considerably faster than humans at critical appraisal and were found be consistent in their appraisal of articles. However, these tools were also found to hallucinate data and make mistakes at points.

Furthermore, as AI tools change rapidly, it is impossible to conduct critical appraisal using an generative AI tool in a way that would meet the standards for replicability required of a systematic review.

Lastly, uploading full-text articles into a generative AI tool would constitute copyright infringement.

Given the above, NSLHD Libraries would instead recommend the use of a specialised systematic review tools which utilise AI features, such as PICO Portal and RobotReviewer.

Further reading:

Below are some articles on the use of Generative AI tools in critical appraisal

Roberts, H. R. R., Stephen, R. A., Hayley, A. H., Thomas, D. D., & Iain, S. W. (2023). Comparative study of ChatGPT and human evaluators on the assessment of medical literature according to recognised reporting standards. BMJ Health & Care Informatics, 30(1), e100830. https://doi.org/10.1136/bmjhci-2023-100830
Lai, H., Ge, L., Sun, M., Pan, B., Huang, J., Hou, L., Yang, Q., Liu, J., Liu, J., Ye, Z., Xia, D., Zhao, W., Wang, X., Liu, M., Talukdar, J. R., Tian, J., Yang, K., & Estill, J. (2024). Assessing the Risk of Bias in Randomized Clinical Trials With Large Language Models. JAMA Netw Open, 7(5), e2412687. https://doi.org/10.1001/jamanetworkopen.2024.12687

PICO Portal

Pico Portal is an AI-powered literature review platform. Its features include:

deduplication
priority screening (sorting)
highlighted titles, abstracts, and full text
AI-assisted quality assessment
data extraction

Built on Cochrane's Risk of Bias 2 Tool AI-assisted quality assessment detects and highlights the relevant section of a paper to streamline for the reviewer to find quickly. While this does not replace human judgement in critical appraisal, it automates and streamlines the process.

NSLHD does not provide access to PICO Portal.

PICO Portal

RobotReviewer

RobotReviewer is a machine learning system aimed at supporting evidence synthesis.

Using Cochraine's Risk of Bias 1 Tool, the platform allows users to upload Randomized Controlled Trial articles, and its machine learning system automatically determines information concerning the trial conduct, including:

the PICO
study design
risk of bias

RobotReviewer is not a substitute for human critical appraisal, instead it assists researchers by semi-automating the critical appraisal process.

NSLHD does not provide access to RobotReviewer.

RobotReviewer

Undertaking Systematic, Scoping and Other Evidence Synthesis Reviews, Part II

Navigate the maze of evidence synthesis including systematic reviews, scoping reviews, and more.

Part II covers the later stages of the evidence synthesis review including critical appraisal, data extraction, data synthesis and more. The session will also cover useful automation and AI tools. Offered by NSLHD Libraries in conjunction with RNS ICU Research’s A/Prof Rosalind Elliott.

The content of the Overview class and Part I is assumed knowledge for this class.

Watch a recording of this class or see upcoming training in the NSLHD Libraries intranet calendar.