Out of the deep learning (DL) projects our data scientists have collaborated upon with customers, common challenges have emerged. These challenges can be mapped across a grid to take the form of a quadrant. The horizontal axis denotes the level of business impact from high to low, and the vertical axis denotes the difficulty in solving from high to low.
Although these challenges may not emerge in every AI project, by mapping DL project challenges in this way, organizations can plan to navigate these issues, mitigate the challenges and risks and increase the the likelihood of success.
This article is partly based on the academic paper “Software Engineering Challenges of Deep Learning” by Anders Arpteg, Björn Brinne, Luka Crnkovic-Friis, Jan Bosch.
High business impact, hard to solve
In the top right quadrant of our illustration lie the majority of our challenges – seven out of twelve. These are the challenges that can be described as both having a big impact on business value and being difficult to solve. They are top areas of concern for businesses embarking upon DL projects today, because of their nature and because they require specialist thinking linked to AI/deep learning, which may not be prevalent in most organizations.
1. Effort estimation – The challenge lies in knowing how much effort and how many resources will be required to complete an AI project. For example, it is nearly impossible to gauge how many iterations a model will need to achieve the desired goal.
2. Resource limitations – All AI projects are dependent on talent. Demand for skilled data scientists and data engineers is high, and competition (and budget) for top talent fierce. Data science gaps are a common and significant restraint.
3. Privacy and safety – These are important considerations for most companies especially if dealing with personal and-or regulated data. The way knowledge is distributed across a large neural network involves poorly understood mechanisms, which makes it difficult for designers to control where and how information is stored.
4. Cultural differences – A lot of different teams have to come together to collaborate on a production-ready AI system. This can create friction in terms of expectations of outcomes, which impacts the effective operations of these projects.
5. Limited transparency – Deep learning systems inherently make it difficult to determine how results were reached. In heavily regulated industries or where the investment case is large, transparency on how and why a model operates may be needed..
6. Unintentional feedback loops – In models deployed in a big data context (as AI systems often are), there is a risk of creating unintended feedback loops where, over time, your real-world system adapts to your model rather than the other way around.
7. Glue code and supporting systems – Keeping the code that interacts with external services (like cloud systems) up to date and aligned with changes introduced by external providers can introduce unexpected challenges in production, which require development or support resources.
Modeling and prototyping quicker can mean project milestones can be hit earlier
The mitigating action may vary by challenge or due to the actual AI project scope. For example, the right software choice could make the whole project cycle simpler and more intuitive, transparent and auditable. Modeling and prototyping quicker can mean project milestones can be hit earlier and quicker. Getting models running, trained and evaluated quickly and in faster experimentation cycles will get the team to a point where, although it may not be possible to get rid of all uncertainty, milestones can be hit earlier and quicker.
Learning from each project will expedite the next one - for example, getting data in a format that work for modelling. Generating and gaining access to data in the right formats is often a time-consuming upfront task in any AI project.
Lower business impact, hard to solve
In the upper left quadrant of our grid are challenges that are extremely difficult to overcome and don’t have the highest business impact on the project: dependency management, troubleshooting and testing.
These are challenges best consulted upon with an external resource in order to determine if they will be an area of concern, so as to put measures in place with the help of an outside expert. The level of experience required to solve these kinds of challenges is frequently beyond the knowledge set held within more traditional engineering teams.
High business impact, easier to solve
Experiment management and monitoring and logging are the two challenges that lie in the lower right side of the quadrant, where high impact meets low difficulty.
Version control is crucial in the development of DL models, where many experiments are typically performed to get to the optimal model. But versioning is challenging in AI projects because every model has its own parameters and metrics
Monitoring and logging become an issue as the behavior of the external world changes and drifts occur in incoming data, causing the behavior of the live AI system to change.
Platforms or processes that support clear audit trails and ongoing monitoring of the project and live models will support mitigation in this area.
Challenges to DL projects are not always obvious at the outset, and can be more difficult to resolve if a business has not thought through how to navigate the project as it evolves and manage or mitigate challenges. Knowing what challenges are inherent to DL projects, and how to categorize them in terms of their impact and difficulty level, can help you plan ahead and remove or mitigate the challenge(s) to give the project the best chance of success.
For a detailed breakdown on each of these 12 challenges and the use cases they arose from, download our free whitepaper “Deep Learning Challenges: Lessons From The Field.”