Wiki · AI & Machine Learning · concept
Transformer
A neural network that reads a whole sequence at once and lets every element decide which others matter.
A transformer is a neural network built around one trick: letting every part of a sequence look at every other part. Earlier models read text or sequence data one step at a time, passing a memory forward like a game of telephone. The transformer threw that out. It takes the whole sequence in at once and, for each element, asks: which of the others should I be paying attention to right now? That question — answered in parallel, everywhere — is the entire idea.
How it works
Break the input into pieces called tokens — words, sub-words, or amino acids. Each token starts as a vector. Then comes attention: every token compares itself to every other token and pulls in information from the ones it deems relevant, ignoring the rest. A token for “it” can reach back and grab the noun it refers to; an amino acid can attend to a distant partner that folds beside it. Because there’s no step-by-step recurrence, all these comparisons happen at once — which is what lets transformers train on enormous data using GPUs. Stack this operation in layers, and the model builds richer and richer context: early layers catch local patterns, later ones assemble long-range meaning.
Why it matters (for bio × AI)
The transformer is the architecture that carried modern AI into biology. Protein language models like ESM and ProtT5 are transformers reading sequences the way a language model reads sentences. AlphaFold’s Evoformer is a transformer variant that reasons over evolutionary alignments to predict structure. Single-cell foundation models use the same backbone to place cells in a shared map. When people say a field “had its ChatGPT moment,” they usually mean someone pointed a transformer at their data and it worked — because attention treats a genome, a protein, or a cell’s gene expression as just another sequence to make sense of.
Related
See attention for the mechanism at its core, large language model for what transformers became at scale, and foundation model for the reusable models they make possible.