The Multilingual BERT block implements the BERT—Bidirectional Encoder Representations from Transformers—network in its base size, as published in BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
BERT pushes the state of the art in Natural Language Processing by combining two powerful technologies:
What’s more, the original authors have released pre-trained weights, so that you can use it with minimal training work.
Why use a multilingual model?
More than a simple convenience, multilingual models often perform better than monolingual models.
One reason is that the training data available is generally more limited in any single language. In addition, many languages share common patterns that the model can pick up more easily when it is trained with a variety of languages.
The input of the Multilingual BERT encoder must come from a BERT Tokenizer block.
The tokenizer must use Multilingual cased as Vocabulary, so that the tokenized numerical values are compatible with the Multilingual BERT encoder block.
The Multilingual BERT encoder returns the so-called CLS output. This output is a vector that can be passed to other blocks to perform regression or classification.
The BERT Encoder block implements the base version of the BERT network. It is composed of 12 successive transformer layers, each having 12 attention heads.
The total number of parameters is 110 million.
Every token in the input of the block is first embedded into a learned 768-long embedding vector.
Each embedding vector is then transformed progressively every time it traverses one of the BERT Encoder layers:
Through linear projections, every embedding vector creates a triplet of 64-long vectors, called the key, query, and value vectors
The key, query, and value vectors from all the embeddings pass through a self-attention head, which outputs one 64-long vector for each input triplet.
Every output vector from the self-attention head is a function of the whole input sequence, which is what makes BERT context-aware.
A single embedding vector uses different linear projections to create 12 unique triplets of key, query, and value vectors, which all go through their own self-attention head.
This allows each self-attention head to focus on different aspects of how the tokens interact with each other.
The output from all the self-attention heads are first concatenated together, then they go through another linear projection and a feed-forward layer, which helps to utilize deep non-linearity. Residual connections from previous states are also used to increase robustness.
The result is a sequence of transformed embedding vectors, which are sent through the same layer structure 11 more times.
After the 12th encoding layer, the embedding vectors have been transformed to contain more accurate information about each token. This block returns only the first one (corresponding to the
[CLS] token), which is often sufficient for classification tasks.
The BERT tokenizer converts plain text into a sequence of numerical values, which AI models love to handle.*
How does text tokenization work?
The tokenizer splits the input text into small pieces, called tokens.
There can be more tokens than words if parts of a word (like prefixes and suffixes) are more common than the word itself.
The Sequence length is enforced by truncating or padding the sequence of tokens.
The sequence of integers is ready to be processed by one of the language processing blocks.
Sequence length: The total number of tokens kept in the sequence. It’s necessary to fix the sequence length, since models require fixed size inputs.
If the text input is longer than the Sequence length, the end of the text will be ignored. If the text input is smaller, the sequence will be padded with
Choose a length that matches your typical text size to utilize all the data while avoiding unnecessary calculations on the padding tokens.
Trainable: whether we want the training algorithm to change the value of the weights during training. In some cases, one will want to keep parts of the network static, e.g., when using the encoder part of an autoencoder as preprocessing for another model.
Jacob Devlin et al.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2019.