In 2017, the paper "Attention Is All You Need" introduced the Transformer, an architecture that has since revolutionized the field of NLP. It completely discards the recurrence found in RNNs and LSTMs and relies entirely on a mechanism called self-attention.
1. Moving Beyond Recurrence
RNNs process text sequentially, word by word. This has two major drawbacks:
- It's slow: The computation for the 10th word cannot begin until the 9th word is processed. This sequential nature prevents parallelization.
- Long-range dependencies are still hard: Even with LSTMs, capturing context between distant words in a very long text is challenging.
The Transformer was designed to solve both of these problems.
2. The Core Idea: Self-Attention
The heart of the Transformer is self-attention. It's a mechanism that allows the model to weigh the importance of all other words in the same sequence when encoding a particular word. It helps the model build a richer, more context-aware representation of each word.
The Intuition: Consider the sentence: "The animal didn't cross the street because it was too tired."
When the model processes the word "it," it needs to understand what "it" refers to. Does it refer to the "street" or the "animal"? Self-attention allows the model to calculate an "attention score" between "it" and every other word in the sentence. It will learn to assign a very high score to "animal," effectively linking the two words together and understanding the context. It does this for every word in the sentence simultaneously.
3. Queries, Keys, and Values
How does self-attention calculate these scores? For each input word vector, it creates three new vectors:
- Query (Q): Represents the current word's "question" or what it's looking for. (e.g., "I am the pronoun 'it', who do I refer to?")
- Key (K): Represents a word's "label" or what it offers. (e.g., "I am the noun 'animal', I can be referred to.")
- Value (V): Represents the actual content or meaning of the word.
The process is like a database retrieval system:
- The Query vector of the current word is compared with the Key vector of every other word in the sentence. This comparison produces the attention scores (how much a word should pay attention to every other word).
- These scores are then used to create a weighted sum of all the Value vectors.
- The result is a new vector for the current word that is a blend of all other words, weighted by their relevance.
4. The Transformer Architecture
The full Transformer model uses this self-attention mechanism in its Encoder-Decoder structure. It also adds a few key ingredients:
- Multi-Head Attention: Instead of performing self-attention once, it does it multiple times in parallel (in different "heads"). Each head can learn different types of relationships, making the model more powerful.
- Positional Encodings: Since the model has no recurrence, it has no inherent sense of word order. Special vectors called positional encodings are added to the word embeddings to give the model information about the position of each word in the sequence.
- Feed-Forward Networks: Each attention layer is followed by a simple feed-forward network to add further processing.
Because the calculations for each word don't depend on the previous word's output, the entire sequence can be processed in parallel, making Transformers incredibly fast and scalable to train on massive datasets.