Wiki · AI & Machine Learning · concept
Attention
Each token deciding which other tokens to look at, weighted by relevance.
Attention lets every element in a sequence decide which other elements matter to it. Instead of reading a sentence — or a protein — strictly left to right, a model using attention lets each token glance at every other token at once and pull in the ones that are relevant, ignoring the rest. It is the mechanism that turns a flat list of words or amino acids into a web of relationships the model can reason over.
How it works
Picture each token asking a question and every other token holding up an answer. The
question is called a query, each answer a key, and the actual content it carries
a value. A token compares its query against all the keys; where they match well, that
token gets a high weight, and the model builds the token’s new representation as a
weighted blend of everyone’s values. Strong matches dominate; irrelevant tokens fade
out. The one formula worth seeing is softmax(QKᵀ/√d)·V: QKᵀ scores every
query-key pair, softmax turns those scores into weights that sum to one, and
multiplying by V mixes the values accordingly. The √d just keeps the scores from
blowing up as vectors get longer. That is the whole idea — a learned, content-based
lookup done for every token in parallel.
Why it matters (for bio × AI)
Biology is full of long-range dependencies. Two residues far apart in a protein’s sequence can sit side by side in its folded 3D structure and jointly control function. Recurrent networks, which pass information step by step, tend to forget across such distances; attention connects any two positions directly, no matter how far apart. That is why transformer-based protein language models can spot residues that co-evolve or physically interact — the same trick that reshaped language modeling.
Related
See transformer for the architecture built around attention, and embeddings for the vectors that queries, keys, and values operate on.