How to use the Transformer Attention Visualizer
- 1. Type a sentenceUp to 16 words; each word becomes one token.
- 2. Pick settingsChoose random weights or tied query/key weights, a seed, and whether to apply a causal mask.
- 3. Compute attentionRead the heatmap and click any word to see its scores and softmax weights.
What self-attention computes
In a transformer, each token's vector is multiplied by three weight matrices to make a query (what it's looking for), a key (what it offers) and a value (what it passes on). The attention weights are softmax(Q·Kᵀ ÷ √d_k): each query is compared with every key, scaled and turned into weights that add up to 1. Each token's output is the weighted sum of the value vectors. A causal mask, used by GPT-style models, stops tokens from looking at later tokens.
Here each word gets a fixed random vector plus a sine/cosine position signal, with d_model = 16 and d_k = 8, in a single head.
Worked example
Take "the cat sat on the mat because the cat was tired" (11 words). With the causal mask on, the first word can only attend to itself, so its row is 1.00 followed by masked cells. The second word, cat, splits its attention between the and itself (0.24 and 0.76). Switch weights to Query = key weights and turn the mask off, and every word attends most strongly to itself: mat → mat with weight 0.67, and the two cats each give 0.36 to cat. That shows that attention scores measure similarity between queries and keys.
What this does and doesn't show
- The maths is exact and is checked against NumPy in our tests.
- The weights are random, not trained, so the patterns don't reflect meaning. Trained models learn weights that make heads track grammar, references and topics.
- Real models use many heads and layers, much larger vectors, and subword tokens instead of whole words.
Frequently asked questions
Why divide by √d_k?
Dot products grow with vector length; dividing by √d_k keeps scores in a range where softmax doesn't saturate, which keeps training stable.
Can I see a real model's attention?
That needs the model's trained weights, which this browser tool doesn't include. Libraries such as BertViz show attention from real models in Python.
Is the transformer attention visualizer free?
Yes. You can open the transformer attention visualizer and fill in everything without signing up. When you compute attention for the first time, we ask once for your name and WhatsApp number; after that it, and every other Uncodemy tool, opens straight away.