Wiki · AI & Machine Learning · concept
Foundation model
A large model pre-trained on broad data once, then reused for many downstream tasks.
A foundation model is trained once on a broad pile of data, then adapted to many tasks. Rather than building a fresh model per problem, you pre-train one large model on a general objective (predict the next token, fill in the masked amino acid), and it learns representations general enough to fine-tune — or just prompt — for specific jobs.
How it works
The engine is self-supervised learning: the data provides its own labels (hide part of the input, predict it), so training can scale to enormous unlabeled corpora — all of GitHub, all of UniProt. That scale is the point: with enough data and parameters, capabilities emerge that weren’t explicitly trained for. The model ends up with a rich latent space you can reuse.
Why it matters (for bio × AI)
The foundation-model recipe is what carried AI into biology. Protein language models (ESM), DNA models (Evo), and single-cell models (scGPT, Geneformer) are all the same bet: pre-train on life’s “text,” then read out structure, function, or effect. The open question the blog keeps returning to — do these models understand biology, or interpolate it? — is a foundation-model question.
Related
Latent space and embeddings are what a foundation model produces and reuses.