Scientists at the University of Oxford have evaluated the capability of Artificial Intelligence (AI) tools to automatically process information from patients’ Electronic Health Records (EHRs). This is a crucial step for conducting large-scale, confidential medical research.
In the modern digital age, millions of EHR records represent a massive resource for improving research, education, and health care quality. However, the growing interest in using EHRs to train AI models raises questions about how robust existing methods are in fully protecting patient privacy.
The Oxford team, led by Dr. Rachel Kuo, conducted a study to examine whether software could meet acceptable data protection standards.
The study authors note that patient privacy is essential for building public trust in health care research. They explained that manual deletion of personally identifiable information (such as names or addresses) is time-consuming and expensive, and machine learning models can help alleviate this burden.
The study, published in the journal iScience, aimed to assess the ability of Large Language Models (LLMs) and purpose-built software tools to identify and separate patient identifiers (names, dates, medical record numbers) from real records without altering the clinical content.
The Oxford team first manually edited 3,650 medical records to create a reliable reference dataset. They then compared this to two specific de-identification tools (Microsoft Azure and AnonCAT) and five general-purpose LLMs (including GPT-4, GPT-3.5, Llama-3, Phi-3, and Gemma).
As a result of the study, Microsoft Azure’s de-identification service showed the highest overall performance, which almost perfectly matched the editing performed by humans. GPT-4 was also strong, proving that modern language models can accurately extract identifiers with minimal prior training.
However, the study also revealed risks. Dr. Soltan explained that while some Large Language Models perform impressively, others may generate false or misleading text—or “hallucinations”—which poses a risk in a clinical context, and careful validation is necessary before use.
The scientists conclude that automating de-identification can significantly reduce the time and cost of preparing clinical data for research while preserving patient confidentiality.
