Challenges of implementing AI in healthcare

Privacy

Privacy, while important in every industry, is typically enforced especially vigorously when it comes to medical data. Since patient data in European countries is typically not allowed to leave Europe, many hospitals and research institutions are wary of cloud platforms and prefer to use their own servers.

For startup companies, it’s hard to get access to patient data to develop products or business cases. Usually, this is easier for medical researchers, who can make use of standard application procedures meant to facilitate research based on patient clinical data.

Regulation

AI algorithms meant to be used in healthcare (in Europe) must apply for CE marking. More specifically, they need to be classified according to the Medical Device Directive, as explained very well in this blog post by Hugh Harvey. Stand-alone algorithms (algorithms that are not integrated into a physical medical device) are typically classified as Class II medical devices.

The General Data Protection Regulation (GDPR) directives introduced in May 2018 will also lead to a number of new regulations that needs to be complied with and that are, in some cases, not clear-cut. For example, some degree of transparency in automated decision-making (see below) will be required, but it‘s hard to tell from the directives what level of transparency will be enough, so we’ll probably need to await the first court cases to learn where the border lies. Other issues are likely to result from the requirement for informed consent. For example, will it still be possible to perform research on dementia under the new regulations, considering some of the participating individuals may not be able to give informed consent?

Transparency

Despite potential difficulties in establishing parameters, transparency of decision support is, of course, paramount to medical AI. A doctor needs to be able to understand and explain why a certain procedure was recommended by an algorithm. This necessitates the development of more intuitive and transparent prediction-explanation tools. There is often a trade-off between predictive accuracy and model transparency, especially with the latest generation of AI techniques that make use of neural networks, which makes this issue even more pressing. An interesting viewpoint on transparency and algorithmic decision-making is given in a paper named Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR, which was co-written by a lawyer, a computer scientist and an ethicist.

Sociocultural

Doctors make decisions based on learned knowledge, previous experience and intuition, and problem-solving skills. Getting doctors to consider suggestions from an automated system can be difficult. It’s likely that some elements of AI literacy need to be introduced into medical curricula so that AI is not perceived as a threat to doctors, but as an aid and amplifier of medical knowledge. In fact, if AI is introduced in a way that empowers human workers rather than displacing them, it could free up their time to perform more meaningful tasks or grant more resources to employ more workers.

Engineering/technical debt

The latest techniques in AI making use of deep neural networks have reached amazing performance in the last five to seven years. However, the tooling and infrastructure needed to support these techniques are still immature, and few people have the necessary technical competence to deal with the whole range of data and software engineering issues. Especially in medicine, AI solutions will often face problems related to limited data and variable data quality. Predictive models will need to be re-trained when new data comes in, keeping a close eye on changes in data-generation practices and other real-world issues that may cause the data distributions to drift over time. If several data sources are used to train models, additional types of “data dependencies,” which are seldom documented or explicitly handled, are introduced.

In medical applications, transfer learning — using a pre-trained model and adapting it to one’s specific use case — is often applied, but then a “model dependency” is introduced where the underlying model may need to be retrained or change its configuration over time. The large amount of “glue code” typically needed to hold together an AI solution, together with potential model and data dependencies, makes it very difficult to perform integration tests on the whole system and make sure that the solution is working properly at any given time.

An operational AI platform such as the one we are building at Peltarion, handling the entire modeling process including software dependencies, data and experiment versioning as well as deployment, has the potential to solve many of these engineering and technical debt issues.