
A study by University of Illinois Cancer Center members and others evaluated the performance of generative pre-trained transformer (GPT) models in generating accurate next-generation sequencing reports and treatment recommendations for non-small cell lung cancer (NSCLC).
Published in JCO Clinical Cancer Informatics, an American Society of Clinical Oncology journal, the study also introduced a novel metric that can be used to evaluate the quality of large language models (LLMs). That metric, the Generative artificial intelligence Performance Score (G-PS), considers AI accuracy, relevancy and hallucinations (false or misleading information.)
The study, “Comparative Analysis of Generative Pre-Trained Transformer Models in Oncogene-Driven Non–Small Cell Lung Cancer: Introducing the Generative Artificial Intelligence Performance Score,” was led by fourth-year University of Illinois College of Medicine Chicago medical student Zacharie Hamilton under the mentorship of Cancer Center member and senior author Ryan Nguyen, DO.
“Our study underscores the potential of generative AI, like GPT-4, to become a part of precision oncology care, especially for a NSCLC with driver mutations. By introducing the G-PS, we offer a novel approach for evaluating LLMs, emphasizing the importance of detecting hallucinations in performance metric,” study authors wrote.
Other Cancer Center authors are Deputy Director VK Gadi, MD, PhD; Natalie Reizine, MD; Frank Weinberg, MD, PhD; and Shikha Jain, MD, FACP, all of whom are UI Health oncologists, as well as Zhengjia Chen, PhD, Program Director of the Cancer Center’s Biostatistics Shared Resource Core.
The Cancer Center is part of UI Health, the academic health enterprise of the University of Illinois Chicago (UIC). Other UIC authors are Noor Naffakh, PharmD, MS, co-director of the Precision Oncology Tumor Board, and Aseem Aseem, MD. Author Larry G. Kessler, ScD, is at the University of Washington in Seattle, and Christopher Bun, PhD, is at Kirkland & Ellis in Chicago.