Transformer Architecture Principles

The first two posts covered Python internals and neural network fundamentals. This one is about the architecture behind modern LLMs: the Transformer. It follows the path of the data, from raw text to a prediction, then asks why Transformers beat RNNs and CNNs, and then looks closely at attention and at how a model knows the order of its tokens.

The third post covers four topics:

• Transformer model basics, from raw text to a prediction
• Why use Transformers?
• The core essence of attention
• Positional encoding methods

#Transformer model basics

#Input Processing

#How is raw text tokenized into integer IDs?

To understand how raw text is broken down, we first need Byte Pair Encoding (BPE), a subword tokenization method.

• BPE builds a vocabulary of subword units in a greedy way. It starts from a vocabulary of individual characters and repeatedly merges the most frequent adjacent pairs of characters (or existing subword units) into new, larger units, until the vocabulary reaches the desired size.
• The procedure:
  1. Create a base vocabulary from all the unique characters.
  2. Find the most frequent adjacent pair of symbols.
  3. Replace every occurrence of that pair with a new, combined symbol, and add the symbol to the vocabulary.
  4. Repeat until the target vocabulary size is reached.

Raw text tokenization turns unstructured language into numbers a neural network can consume. It has two main stages: segmenting the raw text into tokens, then mapping those tokens to unique numerical IDs using a predefined vocabulary. There are several subword methods:

• Each subword in the sequence is looked up in a pre-built vocabulary, to get its corresponding integer ID.
• Modern libraries, including the Rust-based tokenizers from Hugging Face, give state-of-the-art, efficient and production-ready implementations of these algorithms, with consistent text preprocessing.

from transformers import AutoTokenizer

# GPT-2 uses a BPE tokenizer.
# Hugging Face's tokenizers library (written in Rust) does the work.
tok = AutoTokenizer.from_pretrained("gpt2")

ids = tok("The cat sat")["input_ids"]  # text -> a list of integer IDs
print(ids)
print(tok.convert_ids_to_tokens(ids))  # the subword pieces behind each ID
print(tok.decode(ids))                 # IDs -> text again (detokenization)

#How are special tokens (e.g., [bos], [eos], [pad]) handled for different tasks?

• Special tokens are fundamental to Transformers. They act as sentinels or delimiters that give structural and semantic cues for processing a sequence.
• They are distinguished tokens, each assigned a unique integer ID in the model's vocabulary. They tell the model about the structure, boundaries and intent of the input sequences, which is critical for the self-attention mechanism and the task-specific layers after it.
• Special tokens are typically prefixed, suffixed or inserted into the token ID sequence, according to the model's pre-training configuration and the specific task.

[pad] is crucial for batching, but the model shouldn't attend to meaningless padding. So an attention mask is generated: 1 for real tokens and 0 for padding tokens. During self-attention, the scores at padding positions are effectively zeroed out, or set to a very large negative value before the softmax.

#Embedding layer

#How are token IDs mapped to dense vectors (word embeddings)?

• This is commonly known as word embeddings or token embeddings. It transforms discrete, symbolic representations into continuous, semantically rich vector spaces.
• Each unique token has a unique integer ID, which acts as a direct index into a large embedding matrix of shape V × D, where V is the vocabulary size and D is the chosen dimension of the dense embedding vectors. Each row is the dense vector for one token.
• From the embedding matrix E, the vector e for token t is retrieved by selecting the t-th row of E.
• The matrix is often initialised randomly and adjusted step by step to minimise the loss function. Pre-trained embeddings can initialise it, or the embeddings can be learned as an integral part of larger, context-aware language models that capture richer, dynamic, contextualised meanings for tokens.

import torch
from torch import nn

emb = nn.Embedding(num_embeddings=50000, embedding_dim=768)  # the V x D matrix

ids = torch.tensor([[15, 2201, 7]])  # a batch with one sequence of three token IDs
vecs = emb(ids)                      # picks rows 15, 2201 and 7: shape (1, 3, 768)

#How are position encodings added (absolute, relative, rotary)?

• Self-attention is permutation-invariant: on its own, it can't tell the order of the tokens, so position has to be added somewhere. The last section of this post explains why, and compares the methods.
• Absolute encodings are added to the embeddings right here, before the first block: x_i = embedding(token_i) + p_i, where p_i is a fixed (sinusoidal) or learned vector for position i.
• Relative encodings (RoPE, ALiBi, T5's bias) are applied inside every attention layer instead, to the queries and keys or to the attention scores. Most modern LLMs use RoPE.

#Transformer Stack

#For each block, how does data flow through multi-head attention → residual connection → layer normalization → feed-forward network → residual connection → layer normalization?

1. The block's input X first goes through multi-head attention (MHA). MHA lets the model attend jointly to information from different representation subspaces at different positions. It computes the attention outputs independently, concatenates them, and projects them linearly. The output Z1 = MHA(X) has shape (S, D): S is the sequence length and D the model's hidden dimension.
2. Residual connection. Z1 is added directly to its input: Y1 = X + Z1. This skip connection lets information flow through the network more easily, addresses the vanishing gradient problem and makes it possible to train deeper models. It lets the sub-layer learn a residual function F(X), so the output is X + F(X).
3. Layer normalization. It stabilises training by normalising the activations, which reduces internal covariate shift, smooths optimisation and gives the next layers inputs with a consistent distribution.
4. The normalised output passes through the feed-forward network (FFN): a position-wise, fully connected network with two linear transformations and a non-linear activation in between. It lets the model add non-linearity and do further per-token transformations.
5. The FFN's output is added to its input (the second residual connection), which keeps gradients flowing through the FFN sub-layer and helps learning.
6. The result of the second residual connection, Y2, goes through another layer normalization. This gives the next block a well-conditioned input and keeps the benefits of normalisation throughout the deep network. The result is the output for the next block, or for the final layer.

#What does a modern LLM block change, and where are its parameters?

Llama-style decoders keep the same skeleton, with a few upgrades:

A block of model width d holds about 12 d² parameters: 4 d² in the attention projections (Q, K, V and the output) and about 8 d² in the feed-forward network. That gives a handy rule of thumb:

parameters ≈ 12 × L × d²  +  embeddings

Llama-2-7B:  L = 32, d = 4,096   →   12 × 32 × 4,096² = 6.44 B
             + input embedding and output head, 2 × 32,000 × 4,096 = 0.26 B   →   6.70 B   (actual 6.74 B)

#How is masking applied differently in encoder-only, decoder-only, and encoder–decoder models?

• Masking selectively stops certain tokens from being attended to by others, by changing the attention scores before the softmax.
• The values for masked tokens are set to a very large negative number before the softmax in the attention mechanism, so their attention weights become effectively zero.

#Task-Specific Head

#Text generation (decoder): How does the output projection map hidden states back to vocabulary logits?

• The output projection layer is a linear mapping that turns the model's internal contextual representation into a distribution over the whole vocabulary, which is what makes text generation possible.
• It takes the high-dimensional hidden state vector from the final decoder layer and maps it linearly to a vector whose size matches the model's vocabulary. The result is the logits.
• A sampling strategy such as greedy decoding, beam search or top-k sampling uses the output of the projection layer to select the next token. That token is appended to the input sequence for the next decoding step, forming a loop until a termination condition is met.

#Classification (encoder): How is the [CLS] token or pooled vector used for label prediction?

• The [CLS] token acts as a condensed semantic summary of the whole input sequence, which is then projected into the label space for classification.
• A special [CLS] token is prepended to the input sequence. Its final state is taken as the aggregate representation of the entire input, designed to capture the contextual information that summarises the sentence or document for the classification task.
• That aggregate hidden state is fed to a task-specific classification head: a simple feed-forward neural network followed by a softmax activation. It turns the high-dimensional hidden state into a lower-dimensional vector whose entries are the unnormalised scores for each target class.
• The [CLS] approach is the most common (for example in BERT), but some architectures or fine-tuning strategies use a pooled output instead. A pooling operation (mean or max pooling) over all the token hidden states of the sequence gives a single aggregate vector, which then goes through a classification head in the same way.

#Embedding extraction: How is the final hidden state used directly without projection?

• The final hidden state means the output vectors of the very last Transformer layer (or block) for each input token. Each token's hidden state holds its own meaning, strongly influenced by its context in the input sequence, and captures semantic, syntactic and even pragmatic information learned during pre-training.
• When extracting embeddings directly, we deliberately skip the task-specific projection layers, because the hidden states themselves are the high-level features wanted for many downstream tasks.
  • The hidden states are already a compressed, contextualised representation of the input, so they suit general-purpose embeddings without further task-specific transformation.
  • Using them directly avoids the biases or constraints of a particular downstream task's output layer, and keeps a more versatile, generalisable representation.

#Output Post-Processing

#How are logits converted to predictions (softmax for classification, sampling/greedy/beam search for generation)?

• Logits are the pre-activation outputs of a neural network layer, often the final layer before a non-linear activation turns them into probabilities or discrete choices.
  • They can be any real number, positive or negative, and their size indicates the model's confidence in a class or token before normalisation.
• For classification, logits are typically turned into a probability distribution over the classes with the softmax function. Softmax takes a vector of arbitrary real-valued logits and squashes it into a vector of real values between 0 and 1 that sum to 1, a categorical probability distribution.
• For generation, logits are often turned into probabilities with softmax too, but the next step uses a sampling strategy and not a simple argmax:

import torch

logits = torch.tensor([2.0, 1.0, 0.1, -1.0])  # raw scores for cat, dog, car, the
probs = torch.softmax(logits, dim=0)          # about [0.638, 0.235, 0.095, 0.032]; they sum to 1

greedy = torch.argmax(probs)                  # always the top token: index 0, "cat"
sampled = torch.multinomial(probs, 1)         # one random draw, weighted by the probabilities

Temperature, top-k and top-p reshape the distribution before a token is sampled:

p_i = e^(z_i / τ) ÷ Σ_j e^(z_j / τ)          τ < 1 sharpens, τ > 1 flattens, τ → 0 approaches greedy

top-k:  keep only the k most likely tokens, renormalise, then sample
top-p:  keep the smallest set of tokens whose probabilities add up to p (the "nucleus"), renormalise, then sample

def sample(logits, temperature=1.0, top_p=0.9):
    probs = torch.softmax(logits / temperature, dim=-1)
    sorted_p, idx = probs.sort(descending=True)
    keep = sorted_p.cumsum(-1) - sorted_p < top_p      # the tokens needed to reach top_p
    sorted_p[~keep] = 0
    choice = torch.multinomial(sorted_p / sorted_p.sum(), 1)
    return idx[choice]

#How is detokenization applied for text outputs?

1. Map token IDs to strings. The sequence of numerical token IDs is converted back to the matching string pieces, using the tokenizer's internal vocabulary mapping.
2. Apply tokenizer-specific rules. The tokenizer's own detokenization logic handles subword prefixes, leading spaces and how punctuation attaches.
3. Handle special tokens. Special tokens are processed according to the tokenizer's configuration (removed or replaced, for example).
4. Concatenate and clean. The processed token strings are joined together. A final cleanup pass may fix leftover problems such as redundant spaces or misplaced punctuation that the earlier rules couldn't resolve.

#Special Cases

#How do cross-attention and separate encoder–decoder stacks handle seq2seq tasks like translation?

1. Input encoding. The source sentence (for example "The cat sat on the mat.") is tokenized and embedded, then fed into the encoder stack. The encoder processes the sequence and produces a rich set of contextual embeddings that capture the meaning of the English sentence. These encodings are stored and passed to the decoder.
2. Iterative decoding with cross-attention. The decoder generates the target sentence (French) token by token:
   • Initial state. The decoder starts with a special start-of-sequence token (such as [SOS]) as its initial input.
   • First prediction. In the first step, the decoder runs masked self-attention over [SOS] (which only sees [SOS]), then performs cross-attention by querying the encoder's full output. This lets it find which parts of the English sentence matter most for predicting the first French word. A feed-forward network and a final linear layer with softmax then predict a probability distribution over the French vocabulary, and the highest-probability word (for example "Le") is chosen.
   • Subsequent predictions. The newly predicted word ("Le") is fed back into the decoder's input (along with [SOS]). In the next step, masked self-attention can attend to both [SOS] and "Le" to understand the target context so far. At the same time, cross-attention is performed against the encoder's output again, to find the relevant source information for the next prediction (for example "chat").
   • Termination. The loop continues until the decoder predicts a special end-of-sequence token (such as [EOS]), which marks the end of the translation.

#How does the architecture adapt to non-text inputs (vision, audio) using modality-specific embedding layers?

• To handle non-text inputs like images or audio, the core strategy is to transform these continuous, often multi-dimensional data into a sequence of continuous embeddings that mimic the token embeddings used for text.
• The key is a modality-specific embedding layer, which preprocesses and projects the raw data into a latent vector space where the standard Transformer architecture can operate.
• Once the data is in that space, the self-attention mechanism, which is permutation-invariant to the original data structure (within a segment), can learn the relationships.

• The concept extends to multimodal models, where different modalities (text, image, audio) are each processed by their own modality-specific embedding layers, then concatenated, fused or integrated at different stages of a single Transformer, often with shared or specialised cross-attention.
• The core advantage: by turning diverse modalities into one unified sequence-embedding format, Transformers can use their self-attention to capture complex, long-range dependencies within and across modalities, without redesigning the core attention and feed-forward layers for every new data type.
  • The Transformer block itself becomes largely modality-agnostic. It relies on the embedding layer to supply a suitable, semantically rich input representation that keeps the essential information of the original data.

#Why use Transformers?

Background: Transformers have almost completely taken over NLP and LLM applications. Why not CNNs or RNNs?

• Transformer models handle long-range dependencies and allow parallel computation at scale.
• RNNs, including LSTMs and GRUs, process sequences token by token. That brings challenges that limit scalability and efficiency for complex, long sequences:
  • They can't be parallelised, which significantly slows training on long sequences and large datasets and hurts scalability.
  • Gradients tend to shrink or explode over many time steps, which makes it hard for RNNs to capture and use long-range dependencies.
• CNNs use fixed-size convolutional filters, so they only capture local context. To capture long-range dependencies, the layers must be stacked, which increases model depth, computational cost and the indirectness of information flow, and makes it hard to build direct, arbitrary-distance relationships.
  • Unlike RNNs, CNNs don't process tokens in sequence, and on their own they don't encode token order across a long sequence, so language tasks need explicit positional information or architectural changes.

#How do Transformers solve the long-range dependency problem of RNNs?

• Self-attention mechanism. Instead of processing tokens one after another as RNNs do, self-attention computes the output for each token by weighing its relevance to all other tokens in the input sequence at the same time, whatever their distance.
• Positional encodings. Self-attention is permutation-invariant: it processes all tokens in parallel, with no sequential bias. So Transformers add positional encodings to inject order information.

#What are the fundamental differences between Attention and convolution/recurrent mechanisms?

They differ in how they process sequential data, capture dependencies and scale computation.

• Attention directly computes the relevance between all pairs of elements in an input sequence, so it models global, direct dependencies whatever the distance in the sequence. It builds direct weighted connections, and the output is a dynamically computed weighted sum of all input elements, between any two positions i and j. This makes it highly parallelisable, because the query-key-value interaction can be computed for all positions at the same time.
• Convolution (CNNs) processes data with local receptive fields that slide over the input. It excels at capturing local patterns and hierarchical features. Each output element depends only on a fixed-size window of input elements, defined by the kernel. Global dependencies are learned only implicitly, by stacking many convolutional layers so the effective receptive field grows. It is highly parallelisable, since each operation is independent across positions.
• Recurrence (RNNs) processes a sequence one element at a time while keeping a hidden state that carries information from earlier steps. It is inherently sequential, which suits temporal dynamics but is problematic for long-range dependencies.

#Why can the Transformer architecture scale to hundreds of billions of parameters and still be trainable?

It comes from a combination of parallelisability, robust architectural components, and the evolution of hardware and distributed training methods.

• Parallelism in self-attention. Self-attention processes all input tokens at the same time, which is fundamental to its scalability. It computes, for each output position, a weighted sum of all input values, with weights set by the similarity between queries and keys. This can be expressed entirely with matrix multiplications, which parallelise very well on modern accelerators. The feed-forward networks add to this, since the output of the attention sub-layer is processed independently per position.

• Components for deep, stable training: the residual connections and layer normalization from the block above. Residuals give gradients a direct path through every layer, and normalization keeps the activations in a stable range, so very deep stacks still train.

• Hardware and distributed infrastructure. GPUs and TPUs keep getting faster at exactly the matrix multiplications a Transformer is made of. A model with hundreds of billions of parameters is spread over thousands of them, using the strategies from post 01: data, tensor and pipeline parallelism, and sharding.

• Optimisation. Adaptive optimizers with dynamic learning-rate schedules are crucial for navigating the complex loss landscapes of enormous models and ensuring stable convergence.

• Data-driven inductive bias. A Transformer's ability to learn patterns from massive datasets lets it generalise effectively.

#The Core Essence of Attention

#Is Attention just matrix multiplication? Where does the memory blow-up occur?

• Matrix multiplication is the bedrock, but attention is more accurately a structured sequence of matrix operations that together implement one specific functional transformation.
• At its core, self-attention projects the input embeddings into query, key and value matrices, then applies a scaled dot-product operation and a weighted summation, implemented with matrix multiplication or related linear algebra.
• The main source of the memory blow-up is the explicit materialisation and storage of the attention score matrix produced by the query-key-transpose product. That product is itself a critical matrix multiplication, computing the raw attention score (compatibility) between every query vector and every key vector.

#How do Query / Key / Value determine the output?

1. Compute similarity scores (Q against K). For each query vector, compute its dot product with every key vector in the sequence. This measures the relevance, or semantic similarity, between the query element and every other element (including itself).
2. Scale the scores. The raw scores are divided by the square root of the key dimension, √d_k. This is crucial to stop the dot products growing too large, which can push the softmax into regions with extremely small gradients and make learning hard.
3. Apply softmax to get attention weights. The scaled scores go through a softmax along the key dimension, which turns them into a probability distribution: the attention weights.
4. Compute the weighted sum of values (A against V). The attention weights A are multiplied with the value matrix V. This takes a weighted sum of all the value vectors, where the weights are exactly the attention weights from the previous step, and gives a new, context-aware representation.

All four steps in one line, with the shapes for one head:

Attention(Q, K, V) = softmax( Q·Kᵀ / √d_k  +  M ) · V

Q, K, V: (N, d_k)      Q·Kᵀ: (N, N)      M: the mask, 0 or −∞      output: (N, d_k)

import math

import torch


def attention(q, k, v):
    d_k = q.size(-1)
    scores = q @ k.transpose(-2, -1)         # (N, N): every query against every key
    scores = scores / math.sqrt(d_k)         # scale, to keep softmax out of its flat regions
    weights = torch.softmax(scores, dim=-1)  # each row becomes weights that sum to 1
    return weights @ v                       # a weighted sum of the value vectors

#Why is softmax needed? Are there alternatives?

• Probability distribution. Softmax makes the outputs non-negative and sum to 1, a basic requirement for reading them as class probabilities or attention weights. It allows a direct comparison and a clear picture of the model's confidence in different outcomes.
• Differentiability. It is fully differentiable, so gradient-based optimisation (such as backpropagation) can train the model. This is crucial for learning from error signals.
• Emphasising dominant values. The exponential function amplifies larger inputs and suppresses smaller ones, so the largest logit gets a much higher probability. This makes the "most likely" option stand out and supports clear decisions.

Alternatives to softmax:

Techniques like layer normalization are often applied to the logits or to intermediate activations inside an attention mechanism, to stabilise training and improve performance before softmax is applied. They do a different job: feature scaling and regularisation, not probability conversion.

#Why split into multiple heads? Does it actually improve performance or is it just for efficiency?

• Multi-head attention lets the model attend jointly to information from different representation subspaces at different positions. It is like having several specialised lenses on the same data at once, each extracting a different kind of relevant information and capturing distinct semantic or syntactic relationships.
• Without it, a single attention head would have to capture every possible relationship in the input sequence, which can be a bottleneck for complex tasks.
• Each head runs an independent attention calculation after projecting the Q, K and V vectors into a lower-dimensional space with its own linear transformations.
• With several heads, the model can learn different types of dependencies at the same time. For example, one head might attend to syntactic dependencies while another focuses on semantic relationships.
• The outputs of the individual heads are concatenated into the final multi-head output, combining the different perspectives each head learned.
• The efficiency side. It isn't the main semantic driver, but multi-head attention does help parallel computation. Each head can compute its output independently and at the same time, which is a big advantage for efficient use of modern GPUs and TPUs.

head_i        = Attention( X·W_Q,i ,  X·W_K,i ,  X·W_V,i )          each W: (d, d / h)
MultiHead(X)  = Concat(head_1, …, head_h) · W_O                     W_O: (d, d)

PauseData

#Positional Encoding Methods

#Why does the vanilla Transformer have no sense of sequence order?

• The vanilla Transformer inherently has no sense of order, because of its foundation, the self-attention mechanism, which treats all the input tokens as an unordered set.
• Core principle: permutation invariance. Self-attention computes relationships between tokens without any built-in knowledge of their original positions. If you shuffle the input sequence, self-attention produces the same set of outputs, just reordered. This is called permutation invariance.

• Why it matters. Understanding sequence is critical for language (subject-verb agreement, temporal relationships) and many other sequence tasks. Without it, a model can't tell "I saw the man with the telescope" from "I saw the man on the telescope".
• The solution: positional encodings. They are added to the input embeddings. They are unique, fixed patterns that convey the absolute or relative position of each token, giving the model the sequence context it needs.

#What is the essential difference between Sinusoidal and Learnable position encodings?

Both give each position its own vector, which is added to the token embedding before the first block. They differ in where that vector comes from.

Sinusoidal encodings use fixed, predefined sine and cosine functions of different frequencies to generate a unique positional vector for each position. Each dimension of the encoding is a sinusoid with a different frequency and offset. This makes it easy for the model to learn to attend to relative positions.

PE(pos, 2i)    = sin( pos ÷ 10000^(2i / d) )
PE(pos, 2i+1)  = cos( pos ÷ 10000^(2i / d) )          low dimensions oscillate fast, high dimensions slowly

Learnable encodings treat positions as parameters learned during training, like word embeddings. Typically an embedding or lookup layer gives each possible position a unique, randomly initialised vector, which is then updated by backpropagation along with the rest of the model's parameters to optimise the task objective.

#What problem does Rotary Position Embedding (RoPE) solve?

RoPE addresses the challenge of encoding relative positional information directly and efficiently into the self-attention mechanism. This eases the limits of earlier absolute positional encodings, particularly extrapolation to longer sequences and the explicit representation of how far apart tokens are.

The problem: limits of earlier positional encodings. Vanilla Transformers process sequences as unordered sets of tokens. Auxiliary encodings (sinusoidal or learnable embeddings) provide absolute position signals, but they bring several conceptual and practical difficulties:

• No built-in relative-position bias. Standard dot-product attention computes interaction scores from the absolute positions of tokens, after the positional embeddings are added. It has no inherent bias for the relative distance between a query token at position m and a key token at position n. This makes it harder to generalise to different sequence lengths or to model local dependencies robustly, where relative distance is crucial.
• Poor extrapolation. Models that rely only on absolute positional embeddings often degrade a lot on sequences much longer than the ones seen in training. The model hasn't learned, or can't infer, positional information for larger, unseen absolute indices, so its grasp of long-range dependencies breaks down.
• Inefficient relative encoding. Other methods, such as Relative Positional Embeddings, try to bring in relative position, but they often need complex changes to the attention scoring function or extra trainable parameters, which raises computational overhead and model complexity.

RoPE's solution: rotational positional encoding. RoPE injects relative positional information by applying rotations to the query and key vectors, with the rotation depending on each token's absolute position. The key insight is that the dot product of the rotated vectors becomes a function of their relative distance, so relative positional knowledge sits directly in the attention mechanism.

• Core mechanism. RoPE treats pairs of dimensions in the query and key vectors as complex numbers and applies a phase rotation determined by the token's absolute position. This makes the inner product, which forms the attention score, implicitly depend on the relative offset.

for each pair of dimensions (2i, 2i+1), with θ_i = 10000^(−2i / d), a token at position m is rotated by m·θ_i:

x'_2i    = x_2i · cos(m·θ_i) − x_2i+1 · sin(m·θ_i)
x'_2i+1  = x_2i · sin(m·θ_i) + x_2i+1 · cos(m·θ_i)

score(m, n) = RoPE(q, m) · RoPE(k, n)      depends only on q, k and the distance n − m

Advantages and the problems it solves:

#What other positional encoding variants exist? (e.g., ALiBi, T5's relative position)

#Summary

Text becomes subword tokens, integer IDs and then embedding vectors, with special tokens and masks for structure and padding. Each Transformer block is attention followed by a feed-forward network, each wrapped in a residual connection and a normalization, and masks decide what each token is allowed to see. Modern decoders keep that skeleton with pre-norm RMSNorm, SwiGLU, RoPE and grouped-query attention, and hold about 12 d² parameters per block. A task head turns the final hidden states into logits and a sampled token, a class, or an embedding. Transformers beat RNNs and CNNs because attention connects every pair of tokens directly and in parallel, at an O(N²) cost. Attention is softmax(Q·Kᵀ / √d_k) · V, split into several heads, and since it has no sense of order, position is added: absolutely (sinusoidal or learned) or relatively (RoPE, ALiBi).

#Concepts at a glance

#Test yourself

Next in the series: 04 · Transformer Inference Optimization Strategies.