Transformer Architecture Explained in a Minute

Why Not RNNs?

Before Transformers, Recurrent Neural Networks (RNNs) dominated sequence modeling. They had two critical flaws:

• Context collapse — earlier tokens are forgotten in long sequences (vanishing gradients).
• No parallelism — tokens processed one-by-one, making training painfully slow.

Transformers solved both. Every token attends to every other token simultaneously.

	RNN / LSTM	Transformer
Token processing	Sequential, one at a time	All tokens in parallel
Long-range context	Degrades with distance (vanishing gradients)	Direct attention between any two tokens
Training speed	Slow — cannot parallelize across sequence	Fast — full-sequence parallelism on GPUs
Path between distant tokens	O(n) steps	O(1) — one attention hop
Order awareness	Built into recurrence	Added via positional encoding

The 6 Core Components

The Attention Mechanism

Attention uses three learned projections of the input — Query, Key, and Value:

What Do Q, K, and V Actually Mean?

The formula is compact, so it's worth slowing down on what each piece is doing. Start with an ordinary Python dictionary: you hand it a key, it returns exactly one value, and the match must be exact. Attention is the soft, differentiable version of that lookup. Instead of retrieving one value, every query retrieves a little bit of every value, weighted by how well its key matches.

Concretely: each token's embedding is multiplied by three learned weight matrices — W_Q, W_K, W_V — producing three different views of the same token. The query is the token asking a question ("I'm the word it — what do I refer to?"). The key is how each token advertises what it can answer ("I'm the animal, a concrete noun"). The value is the actual information a token contributes once selected. Separating key from value matters: a token can be easy to find for one reason and useful for a completely different reason once found.

The dot product QK^T measures alignment between every query and every key — one similarity score per token pair, forming an n×n matrix. Dividing by √d_k keeps those scores in a sane range: as the vector dimension grows, raw dot products grow with it, and large scores push softmax into a near one-hot regime where gradients vanish. The softmax then converts each row of scores into weights that sum to 1, and the final multiplication by V blends the value vectors using those weights. The output for the word it is no longer a generic "it" vector — it's a context-mixed vector leaning heavily toward whatever it refers to. Because every step is matrix multiplication and softmax, the whole lookup is differentiable — the model learns what to ask and what to advertise, end to end.

Tokenization

Before any embedding, text is tokenized via Byte Pair Encoding (BPE). Words are split into subword units — "unbelievable" might become ["un", "believ", "able"]. This keeps vocabulary size manageable while gracefully handling rare words and domain-specific terminology.

Cross-Attention

In encoder-decoder models (T5, BART), the decoder's queries attend to the encoder's keys and values — not its own. This lets the decoder "read" the full encoded input at every generation step, enabling translation and summarization.

BERT vs GPT vs T5: Which Architecture Family Does What?

The comparison image above summarizes the three families; here's the reasoning behind each split, because "which half of the transformer you keep" is really a question about which direction attention is allowed to look.

Encoder-only (BERT, RoBERTa, and most embedding models). Every token attends to every other token — left and right context — because the model's job is understanding, not generation. BERT is trained with masked language modeling: hide 15% of the tokens, ask the model to reconstruct them from both sides. The result is a model that produces rich contextual representations but has no natural way to generate text — there's no mechanism for producing the next token. Use this family for classification, named-entity recognition, semantic search, and sentence embeddings.

Decoder-only (GPT, Claude, LLaMA, Mistral). A causal mask forces each token to attend only to tokens before it. Training is next-token prediction on raw text — no labels needed, just scale. That restriction sounds like a weakness, but it makes the model a native text generator, and it turns out that predicting the next token at sufficient scale teaches translation, summarization, reasoning, and everything else as side effects. This is why virtually every modern LLM is decoder-only: one simple objective, unlimited training data, and generation built in.

Encoder-decoder (T5, BART, and most translation models). The encoder reads the full input bidirectionally; the decoder generates output while cross-attending to the encoded input. This is the natural fit for sequence-to-sequence tasks where input and output are distinct texts — translation, summarization, question answering over a document. T5 reframed every NLP task as text-to-text under this architecture. In practice, large decoder-only models have absorbed much of this territory by simply placing the input in the prompt — but for constrained, input-grounded transformations, encoder-decoder models remain strong and often cheaper.

Computational Complexity

Self-attention is O(n²) — every token pair is scored. Double the sequence length → 4× the compute. This bottleneck spawned efficient variants:

• Flash Attention — IO-aware exact attention, no approximation
• Sparse Attention — attends to a subset of tokens per head
• Grouped-Query Attention (GQA) — shares K/V heads across Q heads, used in LLaMA 2+

What Has Changed Since 2017?

The 2017 blueprint is still recognizable in every modern LLM, but four refinements matter enough to know by name:

• Rotary Position Embeddings (RoPE) — the original paper added a sinusoidal position signal to each embedding. RoPE instead rotates the query and key vectors by an angle proportional to position, so attention scores depend on the relative distance between tokens rather than absolute slots. This generalizes better to sequence lengths never seen in training and is now standard in LLaMA, Mistral, Qwen, and most open models.
• Multi-Query and Grouped-Query Attention (MQA / GQA) — at inference time, the K and V vectors of every previous token must be cached, and that KV cache dominates GPU memory for long contexts. MQA shares one K/V head across all query heads; GQA is the middle ground, sharing K/V across small groups. Quality stays close to full multi-head attention while the cache shrinks several-fold — this is a big part of how long-context models are servable at all.
• FlashAttention — not a new formula, but a new implementation. It computes exact attention in tiles that fit GPU on-chip SRAM, never materializing the full n×n score matrix in slow memory. Same math, several times faster and with memory that scales linearly in sequence length — the reason "O(n²)" stopped being a practical wall for moderate contexts.
• Mixture of Experts (MoE) — replaces each dense feed-forward layer with many parallel "expert" FFNs and a router that sends each token to just one or two of them. A model can hold hundreds of billions of parameters while activating only a fraction per token — more capacity without proportional compute. Mixtral, DeepSeek, and (reportedly) several frontier models use this design.

Notice what hasn't changed: embeddings, residual connections, layer norm, and scaled dot-product attention are all still there. A 2017 reader could skim a 2026 architecture diagram and recognize almost everything.

What Are Common Misconceptions About Transformers?

A few beliefs that survive far too many blog posts:

• "Attention tells you what the model is thinking." Attention weights show where information flows, not why a prediction was made. Research on attention-as-explanation is decidedly mixed — different weight patterns can produce identical predictions. Treat attention maps as a debugging aid, not an explanation.
• "Transformers made CNNs obsolete." Vision Transformers are excellent at scale, but convolutional networks remain highly competitive on smaller datasets, edge devices, and tasks where their built-in translation bias is an advantage rather than a limitation. The two are converging, not replacing each other.
• "The model processes words." It processes subword tokens. "Unbelievable" may be three tokens; a rare name may be five. This is why LLMs historically struggled to count letters in a word — the letters were never individually visible.
• "Transformers understand order because of word position in the input." Self-attention is permutation-invariant — shuffle the tokens and, without positional encoding, the outputs shuffle identically. Order awareness is entirely injected by the positional signal; it is not free.
• "Bigger context windows mean the model reads everything equally well." Long-context models routinely show "lost in the middle" behavior — information at the start and end of the context is retrieved more reliably than information buried in the center. A million-token window is not a million-token memory.

Frequently Asked Questions