Nekhil R / Accident reporting by Keyword spotting Public

Nekhil R / Accident reporting by Keyword spotting

This is your Edge Impulse project. From here you acquire new training data, design impulses and train models.

Keyword spotting

About this project


In any industry, the worker should need to report accidents that should happen in the workspace. Here we are developing a device where employees can speak into a microphone to report an accident. The device will be loaded with a machine-learning model that can recognize a set of accident-reporting keywords. This technology can speed up the reporting process and make it easier for employees who may not be able to write.


Nordic thingy 53.jpg The development board used in this project is Nordic Thingy:53.It allows you to create prototypes and proof of concept without building customized hardware.Its twin Arm Cortex-M33 processors provide enough processing power and memory to execute embedded machine learning (ML) models directly on the development platform.
Through a mobile device, customers can link their Nordic Thingy:53 to their Edge Impulse studio account using the nRF Edge Impulse app. It enables them to download trained ML models to the Thingy:53 for deployment and inferencing and wirelessly communicate sensor data to the mobile device through Bluetooth LE. It also supports serial connection for connecting with the Edge Impulse studio account. For setting up the device in any way ,please read the doc here.

Data Collection

It is one of the main step in the machine learning process. Here we used nRF Edge Impulse app for the Data collection. The main keyword used here for reporting the accident is Accident itself. This is our data collection settings. Here we recorded 10s data at 16Khz frequency. WhatsApp Image 2022-12-27 at 12.45.42 PM.jpeg
This is our keyword collected with some noise.
Collected Data.jpg Then we omiited the noise by splitting sample. So we got our exact 1s keyword. To make the machine learning model robust we collected the data from different ages from different gender. splitting data.jpg In addition to the keyword we'll also need audio that is not our keyword. Like background noise, the TV playing ('noise' class), and humans saying other words ('unknown' class). A machine learning model requires a certain level of "uniformity" in the data it is exposed to, as otherwise it will not be able to learn effectively. The more diverse your data is, the better your machine learning algorithm will perform. So for the Unknown and Noise we used this Edge Impulse dataset. For this data we used direct upload method by browsing from the computer. For more information regarding this dataset please read here.
We made around 24 minutes of data which is split it between the testing and training.
19m 35s data for the testing.
Train data.jpg
4m 47s data for the training.
test dataset.jpg

Data Explorer

With the data explorer, you can explore your dataset visually, detect outliers, and label unlabeled data. This gives us a one-look overview of our complete dataset. To know more about this tool, please have a look here.

This is our dataset visualisation that we generated using the Data explorer. data explorer output.jpg In this project we only care about the accident keyword because unknown and Noise are actually treated as the same.As you can see some of our accident keywords are in the Noise cluster and one or two are in the unknwon. Upon examining the Noise cluster we saw that, some of them are noises itself which we got it from improper keywprd splitting. The remaining keywords which are unclear, that's why they included in the Noise. Here is one example which is a accident keyword that is in Noise cluster. outlier.jpg So we deleted the extreme outlying accident keywords in each cluster and rebalanced the datset, now it looks clean.
dataexplorer final.jpg

Impulse Design

This is our Impulse. Impulse is actually the machine learning pipeline termed by the Edge Impulse.
For processing block we used MFCC which is actually great for the keyword spotting and it works very well with our data.
This is the MFCC feature generation block.
MFCC block.jpg On the Right side, we can see the Melcepsteral co-efficients of the above data and on the left side the hyper parameters for generating the Mel Frequency Cepstral Coefficients. We are going with the default parameters because they work very well with the keyword spotting.
These are the features generated for our data. They are not mingled as much, so our machine learning model will perform very well.
Feature generation tab.jpg
With all data processed it's time to start training a neural network.

Neural Network

These are our neural network settings and architecture, thats work very well for our data.
NN settings.jpg
NN architecture.jpg
By enabling the data augmentation, the Edge Impulse will make our dataset that can work better in real life and also prevents the over-fitting. Here we used 1-D convolutional architecture which works very well for the kwyord spotting. Lets look on the last training perfomance to validate our model.
Model output.jpg For such a small dataset we got 94.4% accuracy which is pretty good,so we can proceeed with this model.

Classifying New Data

It's time to test the model on new, unseen data before deploying it in the real world. testing.jpg
In the model testing, we got around 87% accuracy. That's really awesome.


To deploy our model in the device, just switch over to the deployment tab and click on the build option.The app will actually start building our project and uploading the firmware to our Thingy 53. WhatsApp Image 2022-12-30 at 8.57.43 PM.jpeg

Real world inferencing

After sucessfully deploying the model, we can start our inferencing on real world by switching onto the inferencing tab. This video shows the real time inferencing in the Thingy53.

Download block output

Title Type Size
MFCC training data NPY file 1153 windows
MFCC training labels NPY file 1153 windows
MFCC testing data NPY file 304 windows
MFCC testing labels NPY file 304 windows
Classifier model TensorFlow Lite (float32) 9 KB
Classifier model TensorFlow Lite (int8 quantized) 6 KB
Classifier model TensorFlow SavedModel 13 KB
Classifier model Keras h5 model 8 KB

Clone project

You are viewing a public Edge Impulse project. Clone this project to add data or make changes.


Data collected
24m 19s

Project info

Project ID 170378
Project version 3
License Apache 2.0