Story

As elevators become more and more indispensable in people's lives, safety has become more of a concern. Overloading is an important cause of elevator accidents. The existing elevators use weighing sensors to find the load in elevators, they may sometimes fail. And their maintenance is too expensive. To avoid such accidents, we are going to design a device which can count passengers in real-time at high speed and can give an alert if the no of passengers is above the threshold.
This device can be attached anywhere in the elevators. In comparison with existing technology, its implementation cost is too low, and maintenance is easy.

How does it work?

In this prototype, we only consider two floors, the ground floor and 1st floor. After entering all the passengers in the elevators, anyone needs to press the close button. Then we count the number of passengers, if it is above the threshold the device will make an alarm. Otherwise, the elevator will move on. The threshold passenger limit can be set by the user. Check out the Demo Video

In addition to the overload alert, we also provide elevator statistics. This means that the device can upload the count in elevators with the specific time stamp to the excel software. The count will be always updated after pressing the close button.
One of the interesting aspects of this data is that it can be easily visualised utilizing any graphs or charts. In this way, it can be useful for any person who analyses elevator usage. This is the count coming from the device and it is logged in CSV format(opened in Microsoft excel).
Below are the various graphs generated for the same data.

Clustered Column

Line chart

Pie chart

Consider the case of elevators in a shopping mall, so by using these statistics, the mall owner can easily add up elevators, if the usage of the elevator is too high and can also remove the elevator if usage is too low.

Nicla Vision

In this project we are using Nicla Vision a tiny AI board from arduino. It features a 2MP color camera and has the intelligence to process and extract useful information from anything it sees.

Data collection and Labelling

For the purpose of the data collection, we mounted the board on a tripod and connected it to a laptop using a lengthy USB cable. The below image shows the data acquistion setup.
The whole setup was on one end of the room and we actullay stood on the opposite end. So our micro controller unit can easily pickup the passengers. You can follow this tutorial to connect Nicla vision to the Edge Impulse.
We captured 73 images and split it between testing and training. The images will contain only one person or two person at a time. Then we labelled each images one by one and here we have only one class named "people". The orientation of the nicla vision provides inverted images and that's not a problem at all.

Impulse Design

This is the machine learning pipeline for this project
We choose the image width and height as 96x96 and the resize mode to "Fit the shortest axis". After saving the impulse we moved onto the image tab and chosen "Grayscale" as the colour depth and saved the parameters and generated features for our images. The image below shows the generated features.

Model Training

This is our Neural network training settings and architecture for generating the model.
We only changed the training cycle from 60 to 70. Further increasing of training cycle or learning rate can overfit the data, so we sticked on to this.
Add image of training settings. As the Neural network architecture we used FOMO (MobileNet V2 0.35). The results are actually surprising. We got around 96% accuracy for the model(used quantized int version).

Testing Model

It's time for testing the model. First we tested the test data which we seperated earlier and We got around 84 % accuracy. That seeems to be fine.
Now let's move onto the Live classification. So we are testing 3 sample images that captured from Nicla vision and let's see hoe our model perfomrms.

Test 1

Test 2

Test 3

In all our testing samples, the model performed very well.

Deployment

Now we have our ML model, we need to deploy it to our Nicla vision. We just created a arduino library by pressing the building button, so a zip file will be downloaded. Then we added that library to the Arduino. Then we modified the example sketch to get the project done. You can find code and assets in this github repo.

Additional Hardware and Casing

In addition to the Nicla Vision we used a buzzer and a LED to make alarm.
But the output current of Nicla(4.7 mA) is not enough to properly power up the LED and buzzer. So we used a 2N222A transistor to drive theee devices.
So we used an external power supply of 5V inaddition to the USB power supply for powering the Nicla vision itself. A push button is also used to check whether the door is closed or not. Finally We made a nice tiny case for this device.
Then we inserted each components into it...
Our device is ready to implement.

TeraTerm

We used this software to stream data from the Nicla vision. The streaming data can be logged anywhere in any format.Here we proceeded with CSV(comma seperated value) format, so the file can be easily opened with Micorsoft excel.

Make sure to tick the timestamp before logging the data. The below image shows a sample data streamed from our device which is actually opened n excel.
We can easily generate graphical reports from this data by selecting that text. The below figure represents the line chart for the above data. There are wide variety of options available and they are shown below.

Conclusion

This device can be easily integrated with any elevators,so the elevator will only starts off when the passengers are in the permissible range. For reducing the cost we can use ESP32 Eye like microcontroller unit instead of the Nicla Vision.