Getting meaning from text: self-attention step-by-step video

• Token relationships
  The words in a sentence sometimes relate to each other, like river and bank, and sometimes they don't. To determine how related two tokens are, attention simply calculates the scalar product between their embeddings.
  We can imagine that the embeddings of bank and river are more similar, since they should both encode the aspect of nature, so that their scalar product should be higher than if the tokens were completely unrelated.  
• Keys, queries, and values
  Unfortunately, calculating the scalar product directly on the embeddings would simply tend to give higher values when two tokens are very similar tokens and give smaller values otherwise. But what grammatical analysis taught us is that important relationships can happen between words that can be completely different: a subject and a verb, a preposition and a complement, etc.
  To have more flexibility, the embeddings go through different linear projections so that one embedding creates a key, a query, and a value vector. The projections allow us to select which components of the embeddings to focus on and to orient them, so that the scalar products between the keys and the queries represent the relationships that matter.
• Activations
  The scalar products between a query and the keys, which give the level of relationship between the query’s token and every other token, are typically scaled down for numerical stability, then passed through a softmax activation function.
  The softmax makes large relationships exponentially more significant. Since this operation is non-linear, it also means that self-attention can be re-applied several times to achieve more and more complex transformations, making the process deep learning.
• Linear combinations
  New contextualizedembeddings are created by combining the values corresponding to every input token, in proportions given by the results of the softmax function: if the query of the river token has a strong relationship with the key of the bank token, the value of bank is added in large part to the contextualized embedding for river.

Multi-head attention and BERT

A single sequence of input embeddings can be projected using many different sets of key, query, and value projections in what is called multi-head attention. Each projection set can focus on calculating different types of relationships between the tokens and create specific contextualized embeddings. 

The contextualized embeddings coming from different attention heads are simply concatenated together. 

Deep learning models for Natural Language Processing typically apply many layers of multi-head attention, and mix in extra operations to get robust results. 

For instance, the BERT Encoder uses the WordPiece embeddings of tokens but always begins by adding them to positional embeddings. This step gives information about the order of the tokens in the input sentence, which self-attention would not consider otherwise.

Additional linear projections, normalization, and feed-forward layers give the whole model more flexibility and stability.

The result is a model that can remove ambiguity from natural text and reduce it to precise values that you can use to automatically search, classify, or even annotate text content.