<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href="static/style.xsl"?><OAI-PMH xmlns="http://www.openarchives.org/OAI/2.0/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/ http://www.openarchives.org/OAI/2.0/OAI-PMH.xsd"><responseDate>2026-11-21T01:43:25Z</responseDate><request verb="GetRecord" identifier="oai:macsphere.mcmaster.ca:11375/33876" metadataPrefix="dim">https://prod-ms-be.lib.mcmaster.ca/server/oai/request</request><GetRecord><record><header><identifier>oai:macsphere.mcmaster.ca:11375/33876</identifier><datestamp>2026-10-30T14:08:57Z</datestamp><setSpec>com_11375_271</setSpec><setSpec>col_11375_272</setSpec></header><metadata><dim:dim xmlns:dim="http://www.dspace.org/xmlns/dspace/dim" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:doc="http://www.lyncode.com/xoai" xsi:schemaLocation="http://www.dspace.org/xmlns/dspace/dim http://www.dspace.org/schema/dim.xsd">
   <dim:field mdschema="dc" element="contributor" qualifier="advisor">Nieuwlaat, Robby</dim:field>
   <dim:field mdschema="dc" element="contributor" qualifier="author">Green-Campbell, Dana</dim:field>
   <dim:field mdschema="dc" element="contributor" qualifier="department">Health Research Methodology</dim:field>
   <dim:field mdschema="dc" element="date" qualifier="accessioned">2026-10-19T18:51:03Z</dim:field>
   <dim:field mdschema="dc" element="date" qualifier="issued">2026</dim:field>
   <dim:field mdschema="dc" element="identifier" qualifier="uri">https://hdl.handle.net/11375/33876</dim:field>
   <dim:field mdschema="dc" element="identifier" qualifier="uri">https://doi.org/10.71548/25645</dim:field>
   <dim:field mdschema="dc" element="description" qualifier="abstract">Background: Health outcome identification and prioritization are important steps for developing health guidelines that reflect the perspective of the target end-users. Research is needed on the methodology for prioritizing health guidelines. With the increasing integration of artificial intelligence (AI) tools into the evidence synthesis process, AI tools have the potential to aid in the other steps of the guideline development process beyond evidence synthesis. While guideline development groups have started to use AI for the identification of priority health outcomes, there are currently no studies examining the use and validation of AI tools during the outcome prioritization stage of the guideline development process.
Methods: We conducted an AI validation study to investigate the accuracy and reproducibility of outcome identification and prioritization steps completed by custom GPTs created on ChatGPT, Google Gemini, and Microsoft Copilot models named “Outcome Prioritization for Importance (OPI)” to identify and prioritize relative importance of health outcomes compared to human guideline panels and untrained GPTs. Prompts were run at multiple timepoints. Jaccard Index Scores, Cohen’s Kappa, and Fleiss’ Kappa were used to measure similarity and agreement between the GPTs and humans, with Jaccard Index Scores and Kappa values of 0.5 and 0.61 as the cut-off indicators of concordance, respectively. Jaccard Index Scores and Fleiss’ Kappa were also used to assess reproducibility. Percent agreement was used to determine agreement between outcome importance ratings for prioritization. Time spent running prompts through GPTs was summarized narratively.
Results: A total of 3501 outcomes were prioritized among the GPTs and human guideline panels. Gemini-OPI and ChatGPT-OPI had the highest mean Jaccard scores during internal and external validation, but none of the GPTs had a mean Jaccard score above 0.5. Cohen’s kappa values indicated there was no agreement between the custom GPTs versus humans and the untrained GPTs versus humans. ChatGPT-OPI had the highest similarity per recommendation question across all days, and a majority of the outcome lists generated per recommendation question by the GPTs had moderate-to-high similarity across all days. None of the GPTs had substantial or high agreement across all 3 days for outcome prioritization. Two hundred and ninety-two out of 421 (69.3%) of the outcome importance ratings had 100% agreement per recommendation question for all GPTs. Time spent running prompts through GPTs varied depending on optimization of workflow. 
Conclusion: This AI validation study used validation methods to assess outputs generated by customized GPT models built in ChatGPT, Gemini, and Copilot to human guideline panel members and untrained GPTs for the purpose of identifying and prioritizing health outcomes that are most important for decision-making. The results of this study can be used to refine the existing custom GPTs, design additional studies to assess model performance, and apply techniques to other validation studies for LLMs in the guideline development process.</dim:field>
   <dim:field mdschema="dc" element="description" qualifier="degreetype">Thesis</dim:field>
   <dim:field mdschema="dc" element="description" qualifier="degree">Master of Science (MSc)</dim:field>
   <dim:field mdschema="dc" element="language" qualifier="iso">en</dim:field>
   <dim:field mdschema="dc" element="title">Automating Outcome Prioritization in Guideline Development: An artificial intelligence validation study</dim:field>
   <dim:field mdschema="dc" element="type" lang="en">Thesis</dim:field>
   <dim:field mdschema="others" element="access-status">open.access</dim:field>
</dim:dim></metadata></record></GetRecord></OAI-PMH>