Research Summary: Memorisation bias in medical AI

Research from arXiv highlights a critical issue termed 'memorisation bias' in medical AI models. This bias occurs when models unintentionally memorize individual patient records from training datasets, leading to significant changes in predictions for a patient's future, unseen data if their historical data was included in training. This phenomenon impacts diverse data modalities and model architectures, and its consequences for clinical deployment are still being understood.

Why it matters

This finding reveals a fundamental vulnerability in medical AI applications that can undermine diagnostic and prognostic accuracy, posing significant risks to patient safety and trust. Addressing memorisation bias is critical for ensuring the reliability and ethical deployment of AI in healthcare, impacting regulatory frameworks and development strategies for AI systems.

Key insights

• Medical AI models are susceptible to 'memorisation bias', where they unintentionally retain individual patient data from training sets.
• Predictions for a patient's future, unseen data can be significantly altered if their anonymised historical data was part of the model's training.
• This bias is prevalent across various data modalities and model architectures.
• The phenomenon can persist over prolonged time spans, impacting long-term clinical assessments.
• The full clinical implications of memorisation bias, particularly concerning patients being assessed by models trained on their own historical data, are not yet fully understood.
• Previous concerns about memorisation primarily focused on privacy attacks, but this research identifies a new dimension related to clinical deployment accuracy.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2609.17223