Machine and deep learning: Experimentation stage

Recently, I got the opportunity to speak at the initiative seminar on AI organized by the Chalmers AI Research Centre (CHAIR). The key message in my presentation (see video here) was that working with artificial intelligence (AI), more specifically machine and deep learning (ML/DL), constitutes a major software engineering challenge that is severely underestimated by companies that start to experiment with machine and deep learning.

Although I have discussed some of the challenges in a previous blog post, we have continued to conduct research in this area and have collected additional data concerning the specific challenges. In addition, we have developed a model that captures how companies typically evolve in their adoption of AI/ML/DL. In the figure below, the steps that companies typically evolve through are illustrated. In this blog series, I intend to discuss the challenges associated with each step, with supporting research from an article that was recently accepted for publication in the proceedings of the XP 2019 conference.

How the use of AI/ML/DL evolves in industry

As the figure illustrates, the first step that most companies engage in is experimentation and prototyping. In this case, the work on machine and deep learning models is conducted purely in-house and without any connection to the products and services that the company offers to its customers. The work with basically any ML/DL approach follows the process shown in the figure below. In the simplest view, here are four stages: assemble datasets (or data pipes), create models, train and evaluate and deployment. There are two iterative processes represented by the inner loop and the outer loop. The inner loop is concerned with the typical activity of creating a model, training it, evaluating it and then tweaking the model with the intent of increasing accuracy and reducing error rates. The outer loop illustrates the periodic or continuous retraining of models based on the most recent data and the subsequent continuous deployment of models into operation.

The basic ML/DL development process

Our research shows that companies experience various challenges in each of the steps of the process and that these challenges depend on the evolution stage where the company finds itself. For companies in the experimentation and prototyping stage, I’ll describe the key challenge in each process step.

For the “assemble datasets” step (which in later stages becomes the data pipelines step), the very activity of assembling the right datasets for training and validation purposes often proves to be a significant challenge. Although all companies tend to drown in data, this data often has unclear semantics and the way it has been collected is often ambiguous, resulting in datasets that are not necessarily representative of the operational data that would be used during operations. As a data point: a company that I recently visited claimed that more than 90% of all effort in the data analytics team went to assembling datasets and setting up reliable data pipelines. Although easily underestimated, this is a major challenge.

The “create models” step is concerned with creating ML/DL models that perform well for the data that the problem domain is characterized by. As a well-performing model is highly dependent on the characteristics of the input data, any issues during the previous step automatically affect the quality of the model. In addition, especially in this early stage, companies often encounter a lack of talent with experience, exacerbating the situation.

The “train and evaluate” step typically struggles with establishing the problem specification and desired outcome as well as having datasets that capture a solid ground truth that can be used as a reference for training and evaluating models. As a consequence, it can prove to be difficult to determine which model is superior as well as whether any of the models are of sufficient accuracy.

Due to the nature of this stage, there is no deployment mechanism yet. The challenges with setting up a deployment mechanism are discussed in future articles focused on the higher stages in the evolution model.

In conclusion, the first stage in adopting ML/DL in your products, systems and solutions is concerned with experimentation and prototyping. During this stage, the predominant challenge is the establishment of datasets of sufficient quality as a basis for model creation, training and evaluation. These datasets need to be representative of the data that will, during operations, come through the data pipelines. Our research shows that companies struggle with data quality in this stage and, as a result, the subsequent steps in the development process are negatively impacted. So, begin your journey with ML/DL today, but be sure to focus your energy where it counts: high-quality datasets.

This article was originally published on janbosch.com

References

  1. 01/ Lucy Ellen Lwakatare, Aiswarya Raj, Jan Bosch, Helena Holmström Olsson and Ivica Crnkovic, "A taxonomy of software engineering challenges for machine learning systems: An empirical investigation"  — XP 2019 (forthcoming), 2019
Jan Bosch
Member of the Board

About

Jan Bosch is a professor in Software Engineering at Chalmers University of Technology in Gothenburg and has been a member of the board at Peltarion since September 2017. Apart from this, he is the director of the Software Center, runs the consulting firm Boschonian AB and is the author of several books as well as the editor for Journal of Systems and Software and Science of Computer Programming.

More from the author

Get started for free