Attention and Transformers

A transformer lets every token look at every other token and decide how much to pay attention to each. In a vision-language model, a word like "red" can attend to the image patches that are red. The attention weights come from comparing a query from one token with keys from all the others.

Word (query) Temperature

Scaled dot-product attention

Each token makes a query $\mathbf{q}$, and every token makes a key $\mathbf{k}_j$ and a value $\mathbf{v}_j$. The attention weights are a softmax of the query-key dot products, and the output is the weighted average of the values:

$$\alpha_j = \frac{\exp(\mathbf{q}^\top \mathbf{k}_j / \sqrt{d})}{\sum_m \exp(\mathbf{q}^\top \mathbf{k}_m / \sqrt{d})} \qquad \text{out} = \sum_j \alpha_j\,\mathbf{v}_j$$

What to notice

Here each patch's key is a small hand-made feature vector: how red, how blue, how far left, how round, how boxy, how floor-like. The word's query points along one of those directions, so the matching patches light up. Low temperature makes attention sharp, nearly picking one patch. High temperature spreads it out. A real model learns these vectors, with hundreds of dimensions and many attention heads running in parallel, but each head works exactly like this.