Secure Yet Fragile: Privacy-Preserving Healthcare AI Remains Vulnerable to Cyberattacks

Artificial intelligence is creating new opportunities for healthcare organizations to collaborate without sharing sensitive patient data. But protecting the privacy of the data used to train an AI system does not necessarily mean the AI itself is secure.

New research from the Institute for Smarter Cities, Spaces, and Health is uncovering vulnerabilities that could affect emerging AI systems designed to analyze medical images and clinical information — and identifying where stronger safeguards are needed before these technologies are widely deployed in healthcare.

Published in Scientific Reports, the study, “Secure yet fragile: adversarial vulnerabilities of federated vision–language models in medical AI,” was led in part by researchers in FAU’s SPEED Lab (Secure AI, Multimodal LLM and Edge Intelligence Lab), including Ahmed Imteaj, Ph.D., faculty fellow of the Institute and assistant professor in the FAU College of Engineering and Computer Science, and graduate students Awal Ahmed Fime and Md Zarif Hossain.

The research focuses on vision-language models, or VLMs, a form of artificial intelligence capable of connecting visual information with language. In healthcare, these models can analyze medical images and translate visual findings into clinically meaningful concepts and descriptions.

The researchers examined these models in combination with federated learning, an approach that allows organizations such as hospitals to collaboratively train an AI model while keeping sensitive patient data within their own systems. Instead of sharing the underlying data, each participant contributes what its local model has learned to improve a shared model.

While that approach offers important privacy benefits, the researchers found it can leave the resulting AI vulnerable in another way: an attack targeting one participant can potentially compromise the performance of the shared model, affecting other participants that rely on it.

“Federated AI offers an important advantage for healthcare because institutions can collaborate without directly sharing sensitive patient data, but our study shows that privacy alone does not make these systems secure or reliable,” Imteaj said. “A vulnerability introduced at one participating site can potentially spread through the collaborative training process and affect the shared model. As federated AI moves closer to clinical use, healthcare providers and AI developers need to build protections against these kinds of attacks into these systems from the beginning. Ultimately, trustworthy medical AI requires us to protect both the patient data and the integrity of the model making decisions from that data.”

To test vulnerabilities, the researchers simulated various cyberattacks against AI models trained on medical imaging data. They found that federated medical AI models can remain highly vulnerable to subtle adversarial manipulations, with small changes to medical images causing substantial degradation in model performance

In a healthcare setting, those vulnerabilities could have significant consequences. The models studied are designed to recognize and interpret medical imagery, meaning an attack could interfere with their ability to correctly distinguish medically relevant features, including pathological tissue or organ abnormalities.

The team also tested existing methods for defending the models against these attacks. While the defenses improved their resilience, vulnerabilities remained — underscoring the need to consider security alongside privacy as these technologies continue to advance.

For Imteaj and his students in the SPEED Lab, the study is part of a broader effort to develop artificial intelligence that can operate securely and reliably outside the laboratory. Their work combines expertise in AI security, multimodal AI, federated learning and edge intelligence to better understand how increasingly sophisticated AI systems behave in complex, real-world environments.

As AI becomes increasingly capable of supporting medical imaging and clinical decision-making, ensuring that these systems are both secure and accurate will be critical to their responsible adoption.

The study was co-authored by Awal Ahmed Fime, Tasfia Zaman Samiha, Md Zarif Hossain, Saika Zaman, Ashfak Md Shibli, Abdur R Shahid, Zhen Ni and Ahmed Imteaj.

Read the full study, “Secure yet fragile: adversarial vulnerabilities of federated vision–language models in medical AI,” in Scientific Reports.