How does text tokenization work?
The tokenizer splits the input text into small pieces, called tokens.
There can be more tokens than words if parts of a word (like prefixes and suffixes) are more common than the word itself.
The Sequence length is enforced by truncating or padding the sequence of tokens.
Special tokens required by the XLM-R Encoder block are added, and every token is then replaced with an integer value.
The sequence of integers is ready to be processed by the XLM-R Encoder block.
Sequence length: The total number of tokens kept in the sequence. It’s necessary to fix the sequence length, since models require fixed size inputs.
If the text input is longer than the Sequence length, the end of the text will be ignored.
If the text input is smaller, the sequence will be padded with
Choose a length that matches your typical text size to utilize all the data while avoiding unnecessary calculations on the padding tokens.