Extending BERT-like Language Models for Multiple Languages and Longer Documents

Though these models are very performant in the representations that they learn, this is also a limiting factor inherent in their model architecture. Because computing attention is so computationally expensive, most models limit this to around 512 tokens (sub-parts of words). Because the models train with a short attention span, they don't learn long-term dependencies between long sentences. More recent research has focused on methods for improving this in existing Transformers, such as the Reformer, Transformer-XL, or Longformer models. These models, as with most deep learning research, have focused exclusively on the English language, making them limited for real-world use. To use these models on, e.g., Swedish data, we would need vast amounts of long-context data, which might not be available. This process would then need to be repeated for every new language we want to use the model for.

It is not feasible to re-purpose even a fraction of the research models into the over 7000 languages spoken in the world. However, what some recent models have done (XLM-R and mBERT) is to train existing models on over 100 of the most common languages in the world. These models have shown to be on par with existing monolingual models or even out-performing them in some instances. Additionally, they have proven to adapt surprisingly well to new languages, previously not seen by the model.

Training a language model from scratch takes a long time, a large-context model more so and a multilingual model exuberantly more. We speculated that the ideal way to create a model that could leverage longer context in multiple languages would be if we could start from a multilingual model and extend its context with additional training. Luckily, there does exist such models: one of which is called the Longformer.

The Longformer model, made by researchers from Allen AI, is a Transformer agnostic model that can re-use part of the attention mapping a language model, such as BERT has already learned, and re-purpose that to efficiently learn contexts over longer passages. After extending the attention span of a model it is trained on a dataset with long sentences to learn those new mappings quickly. Using this method, we extended the attention span of a multilingual XLM-R model to 8 times its original size and pre-trained it on the English long-context dataset WikiText-103. To verify if the model had learned the mapping, we fine-tuned it on several question-answering tasks in multiple languages.

The reasoning behind this is that question-answering generally requires the model to have a deep understanding of languages, and is also the task with the most data available in multiple languages.

I first encountered the problem when creating a website that would allow users to upload their own documents and search the documents by asking specific questions you had about them. The goal was to create an application that would make it easier for people, like my grandpa, to quickly find out in terms of service agreement, or similar documents, what he is and isn't allowed to do on a website - Taking document search into the 21st century.

Along with some friends, we created a simple app that showed promise but was not as performant as we would have expected. From this initial experimentation, we noted, among other things, the imitating attention span of current pre-trained Transformer models, and the (at the time) a limited number of languages a model could handle. Since then, there has been a huge initiative on democratizing these models to more languages and researchers are actively researching the optimal architecture for performance and long-context. In Sweden, researching initiatives, such as VINNOVA and Wallenberg AI (WASP) has bolstered the effort and progress of making AI models available for everyone in Swedish. The outcome of the thesis is a direct result of the VINNOVA financing of Swedish language models.