Audio analysis for industrial maintenance

Apply deep learning to real business problems

This tutorial will show you how to process audio data and use an AI model that can suggest maintenance needs.

Person - Target audience: Intermediate Users
Clock - Estimated time: Setup - 15 minutes. Training - 30 minutes

You will learn to
Peltarion logo - Transform audio files to mel spectrograms.
Peltarion logo - Solve a binary image classification problem.
Peltarion logo - Analyze the performance of your model.
Peltarion logo - Test the deployment of your model.


The problem

factory machinery

A key part of smart manufacturing and a modern factory approach involves real-time monitoring of machinery operating conditions. This is a perfect use case for deep learning. The goal of this tutorial is to build a deep learning model that can identify whether a machine is operating under normal or abnormal conditions.

Once trained, a model like this could be used to detect failures and automatically create work schedules for maintenance workers.


The data

The audio recordings are produced by Hitachi, and the original dataset can be found here. The dataset contains sounds from four types of industrial machines: valves, pumps, fans, and slide rails.

To analyze audio files on the Peltarion Platform, the recordings are converted to mel spectrograms. A spectrogram displays the strength of a signal at different frequencies and how this varies over time.

Image classification models can then be used to gain insight into the complicated patterns in the audio data to predict if a machine is operating as it should.

You can import spectrograms that we made on the 6dB Valve data straight from our Data library.

Colab notebook

If you are interested in seeing how this is done or do this yourself, you can have a look at our Machinery spectrogram Colab notebook. By doing the data processing yourself, you can also choose what type of machine you would like to build your model on.

Spectrogram examples

To gain a little bit of insight into what a spectrogram is and how it works, listen to the examples in the table below and compare them to the corresponding spectrograms.

The recordings are of solenoid valves opening and closing repeatedly. The exact nature of the abnormality is unknown. Low frequencies are shown at the top, and the time progresses from left to right. The lighter section at the top represents the low background noise from the factory, and the sharp vertical lines are the distinct clicks from the valves.

Normal Abnormal

normal spectrogram
abnormal spectrogram

Create a project

Let’s begin!

Navigate to the Peltarion Platform and click New project.

Add the spectrograms to your project

You will then be taken to the Dataset view.

  • If you are using our ready-made spectrograms, click the Import free datasets option and look for the Industrial machinery operating conditions dataset.

  • Otherwise, click Upload file to upload model_data.zip file made using the Colab notebook.

Set binary encoding

This problem is an example of binary image classification. This means that the model assigns each photo one of two possible labels in the Abnormal feature, true or false.

  • Binary encoding with the positive class as true should be the default setting, but you can check by clicking the wrench icon next to the Abnormal feature to change the settings.

You are now ready to begin building the model. Click Use in new experiment.

Use in new experiment button

Build the model

You will be greeted by the Experiment Wizard, which will take you through the following steps:

  • Dataset tab.
    Double-check that you are using the latest dataset version and that you have the appropriate subsets selected as training and validation subsets.

  • Inputs/target tab.
    Make sure that your input feature is the Image and that your output feature is the label Abnormal.

  • Problem type tab.
    Select Single-label image classification. This is when a deep learning model predicts one class for each example.

  • Click Create.

Create button

Run the experiment

It is now time to run the model. The default settings work well for this experiment, but feel free to play around with the optimization settings as much as you would like to see how they impact performance.

You have now finished setting up the experiment! Once you click Run, the model will take a while to run, so have a break and check back later to see how well it is doing.

Run button

Evaluating the experiment

In the Evaluation view, you will find several ways of looking at how your model has performed.

Binrary accuracy and recall

For this particular situation, we are interested in binary accuracy and recall:

  • Binary accuracy measures the proportion of correct predictions.

  • Recall measures the proportion of actual positives (abnormal operation) that were marked as anomalous.

The reason for looking at both of these metrics is that while it is important to have a high accuracy overall, in this case, false negatives are more harmful than false positives.

A false positive means undergoing maintenance on a machine that is actually not malfunctioning. A false negative, however, means incorrectly allowing a malfunctioning machine to continue operating which will likely lead to bad production or more severe failure of the machine down the line.

Confusion matrix and ROC curve

To inspect the predictions of your model, select the subset and checkpoint (epoch) you wish inspect and click Inspect.

The confusion matrix is a good way of understanding exactly how your model has performed. It shows you how many predictions fall into each possible outcome: (true positive, true negative, false positive, and false negative).

The threshold slider allows you to change the sensitivity for making a positive anomaly prediction.

Since we consider that false negatives are more harmful than false positives, decrease the Threshold until few false negative predictions remain.

The ROC curve shows at a glance the proportion of anomalous examples that positively identified, as a function of the false positive rate.


Deploy your model

The last step in this process is to deploy your model and see how it might perform if used in the real world.

In the Evaluation view click Create deployment.

Create deployment button
  1. In the popup, select your last Experiment and your best epoch as your Checkpoint.

  2. Click the Enable button to deploy the experiment.

Test how it works using our web app

To test your model, use the web app we made specifically for this case.

  1. If you accessed the data from the data library, download and unzip these demo images.
    If you used the Colab notebook to create the dataset, use the demo_images.zip file created by the notebook.

    The demo images are the spectrograms of 5 normal and 5 anomalous audio recordings. These 10 examples are not in the dataset used for training, meaning that that they have never been seen by the model before and represent new, unknown examples that we can use to test the model.

  2. Copy the URL and Token from the right-hand side of the Deployment view and paste them into the corresponding fields in the app.
    Deployment API

  3. Click the image icon to upload one of the test images. Once you have tried it, you can upload another image by clicking the image again.

You can, of course, use the Open web app button on the right-hand side of the Deployment view to test the deployment.


Tutorial recap and next steps

Congratulations, you have completed the Audio analysis for industrial maintenance tutorial! In this experiment, you have solved a binary image classification problem that can be implemented in important real-life applications.

A model like this can be used to improve how maintenance is done, reducing costs and machine downtime by creating tailor-made maintenance schedules.

Next steps

To take this a step further, you would need to create models that link machinery condition data (like the model built in this tutorial) to other data including input parameters and other sensors such as temperatures, weather conditions, etc., depending on the specific application.

This would in turn allow predictive models that can say if a machine will break ahead of time.

Was this page helpful?
YesNo