Making soundtracks easy to tag & find, through AI

Four years ago, digital marketing expert Gary Vaynerchuk boldly stated[1] that “The single most important strategy in content marketing today is video.” Since then, video has only become a bigger and bigger force in creative productions, from small content producers to giant productions. This trend, which seems unstoppable, has profoundly impacted the music industry, because overlaid on nearly all these videos are songs. And choosing the right soundtrack for a video can be an enormous undertaking for creators.

To address this need, Epidemic Sound has built a platform where people can search for music and license it for videos. Key to that mission is creating a match between what people are looking for and what they find as consistent and accurate as possible.

To help Epidemic Sound in this quest, we put our heads together to find a way of classifying music. We were hoping to streamline the process, make the quality better, and get away from personal opinions. We found that we could transform sound waves from songs to spectrograms, and treat them as images. This way, we could easily create models to classify different moods; Smooth, Hopeful, Romantic, Eccentric…. The model we created performed an 91.8% accuracy and thanks to skills from the music industry together with experience from our AI sphere, we could save an immense amount of time, and get to more consistent results.

Want to try this for yourself? Build a music tagger model on the Peltarion Platform with this tutorial. Or, watch this episode where our Andreas guests Seth Juarez in the Microsoft’s AI Show and talks us through this case.

In the old record shops, classifying music was a manual task. Record-store workers assigned vinyl to different trays depending on a subjective classification. Much later, this technique evolved when companies like Spotify and Pandora started classifying digital versions of songs — still completely manually. Manual classification is not just a ridiculously time-consuming task. It’s also one filled with personal bias, with the consequence of inconsistency.

To help improve the quality in classification of their music library, and to decrease the time it took to classify the entire library, Epidemic Sound worked with data scientists at Peltarion and created deep learning models on the Peltarion Platform.

Spectrograms, fortunately, can be used as normal images, which makes them an accessible data type for deep learning. As a base for building models on the platform, the FCN-6 model[1], which consists of a number of convolutional layers followed by a final set of densely connected layers, was chosen. The model was created with the help of Convolutional Neural Networks (CNN) and then evaluated and tweaked in several steps through different versions to find the best result. All these iterations were done with training data, which by default in the Peltarion Platform is 80% of the total available data, leaving the final 20% for validation of the model.

The biggest initial concern for Epidemic was to find a higher level of accuracy when searching for songs. The better the accuracy, the easier it would be to find the right song to fulfill a particular need, and the bigger the chance for more licenses and better business. The models created in the Peltarion Platform achieved an accuracy rate of 91.8%. But well, this number doesn’t mean much if you don’t know how accurately a human might be able to calculate song categories. While we don’t have data on that, we do know that in the medical field, even highly skilled radiologist error rates can be as high as 30%. 

Besides the accuracy and consistency made possible for searching for songs that match moods, the process for using deep learning to categorize music genres will be a game-changer for Epidemic Sound. Höglund’s rough estimate is that it will be ten times faster for the company to conduct the same task from now on, giving them a chance to sell many more licenses.

References

1. The rise of video marketing on social and how it affects your business — Vaynerchuck (2015)
2. FCN-6 from “Automatic tagging using deep convolutional neural networks” — Choi, Fazekas and Sandler (2016)