{ "cells": [ { "cell_type": "markdown", "id": "ffa8fbc6", "metadata": {}, "source": [ "# Esp32-Cam Image Object Recognition in 30 minutes\n", "\n", "\n", "![Esp32-cam image recognition](assets/esp32-cam-image-recognition-cover.png)\n", "\n", "\n", "Have you ever wanted to perform **object recognition** on your cheap Esp32-cam in a matter of minutes?\n", "\n", "Do you want it to be easy and fast?\n", "\n", "This project is for you!\n", "\n", "Learn how to quickly implement your own **object recognition system on the Esp32-cam** by: \n", "\n", " 1. collect images from Esp32-cam to create a dataset\n", " 2. train a Machine Learning classifier on your PC to recognize objects in the images\n", " 3. deploy that classifier to your Esp32-cam for real-time object recognition" ] }, { "cell_type": "markdown", "id": "8b98bc2b", "metadata": {}, "source": [ "## Image Recognition that is *Fast*\n", "\n", "Image and object recognition is not something entirely new on the Esp32-cam and other microcontrollers, thanks to [TensorFlow for Microcontrollers](https://www.tensorflow.org/lite/microcontrollers) and no-code platforms like [Edge Impulse](https://edgeimpulse.com).\n", "\n", "They come with pre-trained Neural networks of varying size and complexity that you can leverage to implement your own image recognition system.\n", "\n", "*But...*\n", "\n", "**Neural Networks for image recognition are heavyweight: they can take anywhere from 50 Kb to 500 Kb of RAM.**\n", "\n", "Since your cheap Esp32-cam usually comes with limited RAM, you will often be forced to opt for a low complexity, low accuracy network.\n", "\n", "Even more, with weight it comes **time complexity**: classifying an image on the Esp32-cam usually takes about 500 ms (*source: [Edge Impulse blog](https://www.edgeimpulse.com/blog/add-sight-to-your-esp32)*).\n", "\n", "Can we do better?\n", "\n", "Can we do *faster*?\n", "\n", "Yes, we can!\n", "\n", "**Image and object recognition on Esp32-cam can be implemented in 30 minutes, with minimal code configuration, thanks to the Eloquent Arduino ecosystem of libraries: once deployed, it takes 1 kb of RAM and runs at 60 FPS.**\n", "\n", "Follow the next steps to get up and running!" ] }, { "cell_type": "markdown", "id": "d21cdacb", "metadata": {}, "source": [ "## Hardware Requirements\n", "\n", "To follow this project the only requirement is an Esp32 camera.\n", "\n", "You can find many models on the market:\n", "\n", " - [from Ai Thinker](http://www.ai-thinker.com/pro_view-24.html) (the most widely used)\n", " - [from Espressif](https://www.espressif.com/en/products/devkits/esp-eye/overview)\n", " - [from M5Stack](https://shop.m5stack.com/products/esp32-camera)\n", "\n", "I can't recommend enough the cameras from M5Stack because they come with 4 Mb external PSRAM, but any from the above list should work.\n", "\n", "" ] }, { "cell_type": "markdown", "id": "e985c552", "metadata": {}, "source": [ "## Software requirements\n", "\n", "To capture the images from the Esp32-cam with ease, I suggest you to install the **[Eloquent Arduino library version 2.1.2](https://github.com/eloquentarduino/EloquentArduino)**. It is available on the Arduino IDE Library Manager.\n", "\n", "![Eloquent Arduino library](assets/eloquent-arduino-2.x.y.png)\n", "\n", "To collect the images on your PC and train the Machine Learning model, you have to install the **[everywhereml Python package](https://github.com/eloquentarduino/everywhereml)**.\n", "\n", "Create a new Python project and run\n", "\n", "```bash\n", "pip install everywhereml==0.0.5\n", "```" ] }, { "cell_type": "markdown", "id": "32a97c62", "metadata": {}, "source": [ "## Step 1 of 5: Load the CameraWebServer sketch\n", "\n", "First step to create a Machine Learning model is to collect data.\n", "\n", "Since the Esp32-cam quality is pretty low, I recommend you to:\n", "\n", " 1. fix the camera in position with tape and don't let it move\n", " 2. use artificial illumination if possible (image quality degrades in low light conditions)\n", " \n", "\n", "
\n", " \n", "To keep acquisition speed fast, we will capture at QQVGA resolution (160 x 120). If your project requires you to capture at higher resolutions, change the sketch accordingly.\n", "\n", "\n", " | hog0 | \n", "hog1 | \n", "hog2 | \n", "hog3 | \n", "hog4 | \n", "hog5 | \n", "hog6 | \n", "hog7 | \n", "hog8 | \n", "hog9 | \n", "... | \n", "hog126 | \n", "hog127 | \n", "hog128 | \n", "hog129 | \n", "hog130 | \n", "hog131 | \n", "hog132 | \n", "hog133 | \n", "hog134 | \n", "target | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
count | \n", "1978.000000 | \n", "1978.000000 | \n", "1978.000000 | \n", "1978.000000 | \n", "1978.000000 | \n", "1978.000000 | \n", "1978.000000 | \n", "1978.000000 | \n", "1978.000000 | \n", "1978.000000 | \n", "... | \n", "1978.0 | \n", "1978.0 | \n", "1978.0 | \n", "1978.0 | \n", "1978.0 | \n", "1978.0 | \n", "1978.000000 | \n", "1978.000000 | \n", "1978.000000 | \n", "1978.000000 | \n", "
mean | \n", "0.000961 | \n", "0.004506 | \n", "0.052086 | \n", "0.041644 | \n", "0.031181 | \n", "0.020685 | \n", "0.016865 | \n", "0.109291 | \n", "0.118905 | \n", "0.004492 | \n", "... | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.000036 | \n", "0.647984 | \n", "0.138959 | \n", "1.498483 | \n", "
std | \n", "0.003348 | \n", "0.007184 | \n", "0.031365 | \n", "0.025717 | \n", "0.028305 | \n", "0.037702 | \n", "0.028823 | \n", "0.058904 | \n", "0.037382 | \n", "0.018272 | \n", "... | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.000612 | \n", "0.187835 | \n", "0.052473 | \n", "1.116958 | \n", "
min | \n", "0.000000 | \n", "0.000000 | \n", "0.000000 | \n", "0.000000 | \n", "0.000000 | \n", "0.000000 | \n", "0.000000 | \n", "0.013274 | \n", "0.023559 | \n", "0.000000 | \n", "... | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.000000 | \n", "0.280464 | \n", "0.023285 | \n", "0.000000 | \n", "
25% | \n", "0.000000 | \n", "0.000000 | \n", "0.027400 | \n", "0.022143 | \n", "0.014863 | \n", "0.006805 | \n", "0.003621 | \n", "0.077048 | \n", "0.092610 | \n", "0.000000 | \n", "... | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.000000 | \n", "0.471291 | \n", "0.108158 | \n", "1.000000 | \n", "
50% | \n", "0.000000 | \n", "0.000000 | \n", "0.052826 | \n", "0.038823 | \n", "0.025297 | \n", "0.013129 | \n", "0.008883 | \n", "0.100355 | \n", "0.119885 | \n", "0.000000 | \n", "... | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.000000 | \n", "0.722219 | \n", "0.140522 | \n", "1.000000 | \n", "
75% | \n", "0.000000 | \n", "0.007689 | \n", "0.076274 | \n", "0.058344 | \n", "0.037194 | \n", "0.021973 | \n", "0.016421 | \n", "0.122752 | \n", "0.143970 | \n", "0.000000 | \n", "... | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.000000 | \n", "0.774513 | \n", "0.167540 | \n", "2.000000 | \n", "
max | \n", "0.034322 | \n", "0.038407 | \n", "0.141044 | \n", "0.157112 | \n", "0.277901 | \n", "0.395260 | \n", "0.199861 | \n", "0.426787 | \n", "0.271960 | \n", "0.155243 | \n", "... | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.015333 | \n", "1.000000 | \n", "0.436249 | \n", "3.000000 | \n", "
8 rows × 136 columns
\n", "