Story

The direct annual cost of vandalism runs in billions of dollars annually in the United States only. Breaking glass and defacing properties are some of the serious forms of vandalism. Conventional security techniques such as direct lighting and intruder alarms are ineffective in many locations and cases. So by means of this project, we are able to detect the sound of glass breaking and can alert the user instantly about the event. In this project, we mainly focus on vandal activity such as glass breaking, however, this project can be applied to any vandal activity that produces sounds.

How Does It Work

The device will work as follows, suppose a vandal tried to break the glass then the tinyML algorithm running in the device can recognise the event using a microphone unit. Then the device will give mail notifications to the registered users regarding the vandalism detection.

Hardware

Arduino Nano 33 BLE Sense

For this project we are using, Arduino nano 33 ble sense. It's a 3.3V AI-enabled board in the smallest available form factor. It comes with a series of embedded sensors including MP34DT05 (Digital Microphone).

ESP -01

The ESP-01 is used for adding the wifi capability for the device because the Arduino nano 33 ble sense has only Bluetooth. Actually, the wifi is used for sending mail alerts. The serial communication is established between the Arduino and esp 01 for giving the mail alert.

Software

Data acquistion

One of the most important parts of any machine learning model is its dataset. The edge impulse offers us two options to create our dataset either direct uploading the file or recording via the device itself. Here we proceeded with recording the device itself because the first reason is that it's just a prototype so the data will be limited and the second reason is that it produces more accuracy. For connecting the BLE sense to the edge impulse please read this tutorial.
In this scenario, we have only two classes Glass break and Noise. Glass breaking sounds that we have used are from the vivid online resources and the major noise datasets are from this Microsoft Scalable Noisy Speech Dataset (MS-SNSD). We also included the natural noise in the room apart from MS-SNSD.
The sound recording was done for 20 seconds at 16KHz sampling rate. Something to keep in mind is that you must keep the sampling rate the same between your training dataset and your deployment device. When you are planning to use the 44.1Khz sound, you should need to downsample it to the 16KHz(when you are going to deploy in Arduino).
We collected around 10 minutes of data and split it between training and test set. In the training data we split the sample to 2s otherwise, the inferencing will fail because the BLE sense has limited memory to handle the data.

Impulse Design

This is our Impulse. Impulse is actually the machine learning pipeline termed by the Edge Impulse.
Here we used MFE as the processing block ,because it is very suitable for non-human voices. We have used default parameters of the MFE block.

Neural Network

These are our Neural Network settings which is most suitable for our data.If your are tinkering with your own dataset you should need to change this parameters if required. We enabled the Data augmentation which helps us to randomnly transform data during training. So by means of this we are able to run more training cycles without overfitting and also it leads to the improved accuracy. This is our Neural Network architecture. We have used the default 1D convolutional. Then we trained the model and we got 97% accuracy,which is very awesome. By looking the confusion matrix it is clear that there is no sign of underfitting and overfitting.

Model Testing

Before deploying the model it's a good practice to run the inference on the test dataset. In the model testing, we got around 92% accuracy.
Let's look into some of the misclassifications,
In this case, the noise very well resembled the Glass break that's why it is misclassified. In this case too, the model performed very well in classifying the data, the data contains both the Glass_Break and the noise but the majority of the data was noise, that's why its misclassified. In these two cases(below shown) also the noise was major.


The model is performing very well and can be deployed it in the real world.

Deployment

For deploying the impulse to BLE Sense,we created a arduino library with this optimizations.
Then we add that library to the Arduino IDE. Then we modified the example sketch that is provided. You can find entire code and assets including circuit in this Github repo.

IFTT

For triggering the mail we used IFTT service. To setup the mail triggering upon any poitive detections please have a look at this tutorial.
This is the applet created by me.

Case

All the components were fitted inside this case.

Real world Testing

Let's test the model after deploying it on the BLE sense board. You can check it out in the below video. We have actually played the Glass breaking sound via a large speaker.