Wiki · Bio × AI Methods · concept
Protein language model
A language model trained on protein sequences, treating amino acids like tokens to learn the 'grammar' of proteins.
A protein language model is what you get when you point a large language model at proteins instead of text. Each amino acid becomes a token, each protein a “sentence,” and the model learns to read them the way an LLM reads English. Nobody tells it what a binding site or a fold is. By training on enormous numbers of natural sequences, it picks up the statistical grammar of proteins — which residues tend to co-occur, which substitutions are tolerated, which patterns evolution keeps reusing.
How it works
The dominant recipe is masked-token pretraining, borrowed straight from language models. Hide some amino acids in a sequence, ask the model to fill them back in, and repeat across millions of proteins. To guess a masked residue well, the model has to internalize context — the residues around it, and the constraints they impose. Out of this self-supervised task, with no structural or functional labels, fall two useful things: a per-residue embedding for every position, and a per-protein vector summarizing the whole chain. Signals of structure and function emerge inside these representations for free.
Why it matters (for bio × AI)
Those embeddings are the payload. Feed them to a small downstream model and you can predict function, estimate how a mutation will help or harm (variant-effect prediction), or steer protein design. Meta’s transformer-based ESM and ESM-2 are the exemplars here — foundation models for protein sequence. ESM-2 also powers ESMFold, a structure head that folds a protein straight from its sequence, skipping the multiple-sequence alignment that older predictors leaned on. One learned representation, many jobs.
Related
See transformer for the architecture underneath, embeddings for the vectors it produces, foundation model for the pretrain-once, reuse-everywhere pattern it follows, and large language model for the same idea applied to human text.