Code for MobiCom'22 paper 'TinyML-CAM: 80 FPS Image Recognition in 1 Kb RAM'
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
Bharath Sudharsan, PhD 83be1ce907
Update README.md
7 months ago
ESP32-image-object-classification-live-demo.mp4 Added demo video 11 months ago
LICENSE Initial commit 11 months ago
README.md Update README.md 7 months ago
[h]-HogClassifier.h Updated code 10 months ago
[h]-HogPipeline.h Updated code 10 months ago
[ino]-CameraWebServer.ino Update [ino]-CameraWebServer.ino 10 months ago
[ino]-arduino-ESP32-code.ino added code 10 months ago
[ipynb]-TinyML-CAM-full-code-with-markdown.ipynb added code 10 months ago

README.md

TinyML-CAM - Image Recognition System that Runs at 80 FPS in 1 Kb RAM

Demo - HOG and Random Forest based Image Recognition on ESP32

ESP32 classifying Raspberry Pi Pico, Portenta H7, Wio Terminal from image frames

https://user-images.githubusercontent.com/16524846/179447640-d7f5efa9-3a44-431c-922d-348ee526c782.mp4

Results

Following can be observed from the video:

  • Time. For image frames, the digital signal processing (DSP) based features extraction time is ≈ 12 ms, while classification time is ≈ < 20 𝜇𝑠 (1/1000th of DSP).

  • FPS. It is 1000/12 ms = 83.3 FPS, which is the time taken by the TinyML-CAM system for HOG features extraction (using DSP) plus classification. Since the ESP32 has a 30 FPS frame rate, just to capture frames, it takes 1000/30 = 33 ms. Since the DSP plus classification time is only ≈ 12 ms, the image recognition happens in real-time between two consecutive frames, thus not altering the ESP32 camera's FPS.

  • Accuracy. As expected during Pairplot analysis, Portenta and Pi (features overlapped) are mislabelled quite often, which can be rectified by improving dataset quality.

  • Memory. Consumes only 1 kB of RAM - difference between the RAM calculated by Arduino IDE before and after adding the TinyML-CAM image recognition system.

Paper

https://dl.acm.org/doi/pdf/10.1145/3495243.3558264

Requirements

  • To capture images from the ESP32 with ease, install Eloquent library via Arduino IDE library manager.
  • To collect images on a PC and train an ML classifier, install EverywhereML Python package.
  • To test the TinyML-CAM pipeline, users only require an ESP32 of any variant:

Code

Future Work

To lower the DSP time (currently 12 ms) by implementing mathematical approximation methods, which will boost the frame rate - i.e., if reduced to 6 ms, then 1000/6 ms = 166.6 FPS.

Similar to the TinyML benchmark, we plan to test the pipeline on a range of datasets, ML algorithms, and IoT boards.