Story

In any industry, the worker should need to report accidents that should happen in the workspace. Here we are developing a device where employees can speak into a microphone to report an accident. The device will be loaded with a machine-learning model that can recognize a set of accident-reporting keywords. This technology can speed up the reporting process and make it easier for employees who may not be able to write.

Hardware

The development board used in this project is Nordic Thingy:53.It allows you to create prototypes and proof of concept without building customized hardware.Its twin Arm Cortex-M33 processors provide enough processing power and memory to execute embedded machine learning (ML) models directly on the development platform.
Through a mobile device, customers can link their Nordic Thingy:53 to their Edge Impulse studio account using the nRF Edge Impulse app. It enables them to download trained ML models to the Thingy:53 for deployment and inferencing and wirelessly communicate sensor data to the mobile device through Bluetooth LE. It also supports serial connection for connecting with the Edge Impulse studio account. For setting up the device in any way ,please read the doc here.

Data Collection

It is one of the main step in the machine learning process. Here we used nRF Edge Impulse app for the Data collection. The main keyword used here for reporting the accident is Accident itself. This is our data collection settings. Here we recorded 10s data at 16Khz frequency.
This is our keyword collected with some noise.
Then we omiited the noise by splitting sample. So we got our exact 1s keyword. To make the machine learning model robust we collected the data from different ages from different gender. In addition to the keyword we'll also need audio that is not our keyword. Like background noise, the TV playing ('noise' class), and humans saying other words ('unknown' class). A machine learning model requires a certain level of "uniformity" in the data it is exposed to, as otherwise it will not be able to learn effectively. The more diverse your data is, the better your machine learning algorithm will perform. So for the Unknown and Noise we used this Edge Impulse dataset. For this data we used direct upload method by browsing from the computer. For more information regarding this dataset please read here.
We made around 24 minutes of data which is split it between the testing and training.
19m 35s data for the testing.

4m 47s data for the training.

Data Explorer

With the data explorer, you can explore your dataset visually, detect outliers, and label unlabeled data. This gives us a one-look overview of our complete dataset. To know more about this tool, please have a look here.

This is our dataset visualisation that we generated using the Data explorer. In this project we only care about the accident keyword because unknown and Noise are actually treated as the same.As you can see some of our accident keywords are in the Noise cluster and one or two are in the unknwon. Upon examining the Noise cluster we saw that, some of them are noises itself which we got it from improper keywprd splitting. The remaining keywords which are unclear, that's why they included in the Noise. Here is one example which is a accident keyword that is in Noise cluster. So we deleted the extreme outlying accident keywords in each cluster and rebalanced the datset, now it looks clean.

Impulse Design

This is our Impulse. Impulse is actually the machine learning pipeline termed by the Edge Impulse.

For processing block we used MFCC which is actually great for the keyword spotting and it works very well with our data.
This is the MFCC feature generation block.
On the Right side, we can see the Melcepsteral co-efficients of the above data and on the left side the hyper parameters for generating the Mel Frequency Cepstral Coefficients. We are going with the default parameters because they work very well with the keyword spotting.
These are the features generated for our data. They are not mingled as much, so our machine learning model will perform very well.

With all data processed it's time to start training a neural network.

Neural Network

These are our neural network settings and architecture, thats work very well for our data.


By enabling the data augmentation, the Edge Impulse will make our dataset that can work better in real life and also prevents the over-fitting. Here we used 1-D convolutional architecture which works very well for the kwyord spotting. Lets look on the last training perfomance to validate our model.
For such a small dataset we got 94.4% accuracy which is pretty good,so we can proceeed with this model.

Classifying New Data

It's time to test the model on new, unseen data before deploying it in the real world.
In the model testing, we got around 87% accuracy. That's really awesome.

Deployment

To deploy our model in the device, just switch over to the deployment tab and click on the build option.The app will actually start building our project and uploading the firmware to our Thingy 53.

Real world inferencing

After sucessfully deploying the model, we can start our inferencing on real world by switching onto the inferencing tab. This video shows the real time inferencing in the Thingy53.