Data science /

Peltarion awarded research funding for building state-of-the-art Swedish language model

October 18 2019/5 min read

We're very excited to announce that Peltarion has been awarded funding from the Swedish research institute, Vinnova, for building a Swedish language model based on state-of-the-art NLP techniques. This project is to take place over the course of the next three years, together with Research Institutes of Sweden (RISE), AI Innovation of Sweden, Swedish Agency for Economic and Regional Growth (Tillväxtverket) and Swedish Public Employment Service (Arbetsförmedlingen). The purpose of the project is to enhance service delivery and increase the efficiency of operations throughout the public sector. 

We truly believe that building next-generation AI models for analyzing Swedish text will contribute to pushing forward the AI agenda and adoption rate in Sweden significantly. 

Why now?

The large-scale language modeling breakthroughs that have been done within NLP in recent years have created many new technical possibilities, allowing for understanding and working with text in ways never possible before. These new models, based on self-attention instead of recurrent and convolutional layers and containing hundreds of millions of parameters, are able to understand the true contextual intent, i.e the semantics of text. This creates huge opportunities for public sector authorities to streamline their businesses and even find new ways of creating benefits — for customers and for society.

To exemplify what is possible with these techniques, we at Peltarion fine-tuned a model based on Google BERT on a dataset called QQP. The prototype seen below makes it possible to submit an arbitrary question (e.g., “Will artificial intelligence end the world”) and then find semantically similar questions from

Prototype of fine-tuned Peltarion model based on Google BERT and QQP dataset

The first and most similar question is very close to the posed question, both syntactically with many words in common, and also semantically in asking a question with very similar intent. However, what is more interesting is the second to most similar question, “Is technology destroying humanity?”. This question has none of the words in common with the posed question, but the semantic of the question is still very similar. This is what we want, to find similar questions that are similar in intent, independent of what words that were used and how the posed question was formulated.

Why the need for a Swedish language model?

Right now, most of the existing language models are developed primarily for the English language. This is partly due to the fact that there’s a lot of English text data that can be used to train the models. Also, it’s because the work done to date has been done by the leading tech giants, i.e., the Google’s and Facebook’s of the world. 

The dominance of English-language models puts Swedish-language models at a disadvantage since the downstream NLP applications based on them are dependent on the quality of the training data. Building a Swedish language model from scratch is difficult, time-consuming and expensive. However, if we collaborate and share our work, we can open up opportunities for our Swedish authorities and other companies to build AI solutions making use of text in ways never seen before.

To date, there are a few Swedish ELMo models and a multilingual BERT model. Arbetsförmedlingen has made amazing work with an initial version of a native Swedish BERT model based on Swedish Wikipedia articles. However, lots of work remains to ensure a bias-free and high-quality language model. The progress seen in NLP since the Google BERT model launched in October 2018 has been extreme, with new models coming out more or less every month. In addition, to be able to make use of the Swedish language model, we also need to find datasets for downstream tasks such as sentiment analysis, textual similarity, question answering, and many more.

Areas of impact for a Swedish language model

The range of areas of impact for a Swedish BERT model is huge. For example, the Swedish Public Employment Service can use NLP solutions to better understand the labor market and enhance the future competitiveness of Swedish companies. The gap between existing and in-demand skills will be narrowed, and Arbetsförmedlingen could use the information gleaned from the model to promote continuous further education in an innovative new way.

The Swedish Agency for Economic and Regional Growth (Tillväxtverket), could use a Swedish BERT to engage a more advanced conversational system to assist entrepreneurs throughout the company start-up process — a typically confusing experience laden with tons of conflicting information. Users need more hands-on assistance through this process, and a native BERT model could help provide a more enhanced, adaptable conversational system based on natural language and a deeper understanding of users. 

Thanks to the Vinnova funding and our partnerships with Research Institutes of Sweden, AI Innovation of Sweden and Swedish Public Employment Service, we look forward to what these advancements will bring and what it will contribute to Swedish society! 

  • Pauline Norden

    Pauline Norden

    Communications Manager

    Pauline Norden is the Communications Manager at Peltarion, with experience from working with social media, content and influencer marketing within the tech sector. She holds a degree from Stockholm School of Economics, and before joining Peltarion, she worked with heading up the online community at the private finance company Tink.

02/ More on Data science