Wiki · Translation · concept
EHR foundation model
A model pretrained on millions of longitudinal patient records, then fine-tuned to predict what happens next in the clinic.
An EHR foundation model borrows the language-model playbook for medicine. Instead of words, it pretrains a transformer on the sequence of a patient’s life — the diagnoses, labs, medications, and visits recorded in electronic health records — then fine-tunes that general model for specific predictions. It is a foundation model whose corpus is the longitudinal medical record rather than the web.
How it works
Each patient becomes a sequence of coded events ordered in time, and the model learns to predict held-out or future events from the surrounding context — the same self-supervised trick that lets language models learn from raw text, with no labels required. Med-BERT pretrained on the structured codes of millions of patients and produced contextual embeddings that improved downstream disease prediction (Rasmy et al., 2021). Newer systems model the timeline generatively: Foresight is a pretrained transformer that forecasts a patient’s likely future events from their history, effectively simulating the clinical trajectory forward (Kraljevic et al., 2024). Once pretrained, one model fine-tunes to many tasks — risk scores, phenotyping, next-event prediction — instead of training a separate model per question.
Why it matters (for bio × AI)
These models are powerful precisely because they ingest everything — the whole record, not a hand-picked feature set — which lets them capture patterns no task-specific model would. That same appetite is the privacy problem. Large models memorize their training data: Carlini et al. extracted verbatim training examples — real names, phone numbers, addresses — from a production language model, and showed bigger models memorize more (Carlini et al., 2021). Pointed at clinical notes, that means a model can emit a real patient’s record. This is why an EHR foundation model is only shippable on top of the rest of the privacy stack — de-identified training data, private training, and access controls — rather than in place of it.
Related
See foundation model for the general pretrain-then-adapt pattern this specializes, transformer for the architecture it runs on, and differential privacy for the guarantee that bounds what such a model can leak about any one patient.