High business impact, hard to solve
In the top right quadrant of our illustration lie the majority of our challenges — seven out of twelve. These are the challenges that can be described as both having a big impact on business value and being difficult to solve. They are top areas of concern for businesses embarking upon DL projects today, because of their nature and because they require specialist thinking linked to AI/deep learning, which may not be prevalent in most organizations.
- Effort estimation. The challenge lies in knowing how much effort and how many resources will be required to complete an AI project. For example, it is nearly impossible to gauge how many iterations a model will need to achieve the desired goal.
- Resource limitations. All AI projects are dependent on talent. Demand for skilled data scientists and data engineers is high, and competition (and budget) for top talent fierce. Data science gaps are a common and significant restraint.
- Privacy and safety. These are important considerations for most companies especially if dealing with personal and-or regulated data. The way knowledge is distributed across a large neural network involves poorly understood mechanisms, which makes it difficult for designers to control where and how information is stored.
- Cultural differences. A lot of different teams have to come together to collaborate on a production-ready AI system. This can create friction in terms of expectations of outcomes, which impacts the effective operations of these projects.
- Limited transparency. Deep learning systems inherently make it difficult to determine how results were reached. In heavily regulated industries or where the investment case is large, transparency on how and why a model operates may be needed.
- Unintentional feedback loops. In models deployed in a big data context (as AI systems often are), there is a risk of creating unintended feedback loops where, over time, your real-world system adapts to your model rather than the other way around.
- Glue code and supporting systems. Keeping the code that interacts with external services (like cloud systems) up to date and aligned with changes introduced by external providers can introduce unexpected challenges in production, which require development or support resources.
The mitigating action may vary by challenge or due to the actual AI project scope. For example, the right software choice could make the whole project cycle simpler and more intuitive, transparent and auditable. Modeling and prototyping quicker can mean project milestones can be hit earlier and quicker. Getting models running, trained and evaluated quickly and in faster experimentation cycles will get the team to a point where, although it may not be possible to get rid of all uncertainty, milestones can be hit earlier and quicker.
Learning from each project will expedite the next one - for example, getting data in a format that work for modeling. Generating and gaining access to data in the right formats is often a time-consuming upfront task in any AI project.