AI Gesture Smart Wand Tutorial

We've had plenty of experience with machine learning projects on computers. But can we create a project that allows machine learning models to run on hardware, independent of a computer?

Recently, there's an open-source project that uses the STM32 microcontroller to run NNoM. It deploys a convolutional neural network motion classification model on the STM32. By detecting the attitude data from the six-axis gyroscope MPU6050, it returns the correct classification.

The original address of the open-source project is: GitHub - lyg09270/CyberryPotter_ElectromagicWand_Basic_Project: Cyberry Potter electromagic wand. Basic project that can light up LEDs with different (spell) motions, and can also modify the LEDs to other functions as needed.

This tutorial and the open-source project's tutorial will be more detailed, filling in the pitfalls we've encountered, so everyone can successfully replicate it! Also, the open-source project returns results as serial data, which is not very intuitive. Therefore, we've added two Microbits to visualize the returned data, making the project more vivid and interesting.

Block Diagram of the Project

What is NNoM？

NNoM is a lightweight neural network framework designed specifically for microcontrollers. It supports common deep learning functions, such as Convolutional Neural Networks (CNNs) and fully connected layers, and has the following features:

• Efficient and Lightweight: Suitable for embedded devices with limited resources.
• Easy to Use: Provides intuitive APIs and a variety of example codes.
• Open Source: Supports free modification and extension.

In this project, NNoM will be used to run a gesture classification model on the STM32 microcontroller, enabling offline gesture recognition.

Model Files

Wands (four styles, shared by netizens):

https://makerworld.com/zh/models/519053?from=search#profileId-440711

Matching wand casings (designed by KittenBot):

https://makerworld.com/zh/models/1034922#profileId-1018455

Project Breakdown

The core of this project lies in training the model. However, setting up the environment can be quite troublesome and often poses a challenge for beginners. Therefore, this tutorial will only teach you how to download the pre-trained program (which includes 12 standard gestures, generally sufficient for most use cases). Once you have successfully completed this project and want to delve deeper, such as adding or modifying gestures, you can explore the open-source project repository on your own.

The project can be broken down into the following steps:

• Wiring the STM32 with the MPU6050
• Connecting the STlink programmer to the STM32 and downloading the program
• Connecting the USB serial downloader to the STM32 to test the serial return data
• Writing and testing the 2.4G wireless communication program between Microbits
• Modifying the Microbit 2.4G wireless communication program to add serial data reception and judgment
• Connecting the STM32's serial port to the Microbit's serial port and testing
• Assembling the printed casing to form the wand

Step 1 - Wiring

Connecting the STM32 with the MPU6050

Please strictly follow the wiring diagram below for connection.

You can use the 3D-printed parts I designed for fixing and soldering.

Step 2 - Connecting the Programmer

Connect the STLink programmer and download the compiled hex file.

Wiring for the programmer:

ST-LINK Utility Software Download:

https://www.st.com.cn/zh/development-tools/stsw-link004.html#get-software

Usage:

Connect the main controller

Select the programming file

Download the provided file "CyberryPotter.hex" and click "Start"

Prompt that the download is successful

After the download is complete, you can unplug the STLink programmer.

Step 3 - Serial Port Testing

Connect the serial debugger to test whether the program and hardware are running correctly.

Wiring between CH340 and STM32:

Physical Connection:

Connect using a serial tool:

The baud rate is 115200. After opening the serial port, press the reset button on the STM32 board.

The window will receive

"dmp init complete

Hello!"

[!NOTE]

If the serial port does not receive any information, please check:

• Whether the program has been successfully downloaded
• Whether the serial wiring is correct

When you press the button without making any gestures, the window will receive "Unrecognized, " which indicates that your board is working properly. At this point, when you perform the corresponding gestures, it will return the corresponding results.

[!NOTE]

If the serial port does not receive any information, please check:

• Whether the wiring between the gyroscope and the STM32 is correct

For example, perform a →↓ motion. Press the button (just tap it, do not hold it down), and complete the motion within 1.5 seconds.

The serial port will return the result, which also indicates that your MPU6050 and the trained model are working properly.

At this point, the core functionality of the AI Gesture Smart Wand has been successfully implemented.

When the corresponding gesture is performed, the serial port will return the corresponding string. You can use another microcontroller to interface with the serial command, thus enabling various applications.

The gesture results are shown in the figure below:

Step 4 - Microbit Wireless Communication

Transmitter

Assume Microbit_A acts as the transmitter, connecting to the STM32's serial port to receive the serial results returned by the gestures.

Here, an IObit is used for interfacing.

Microbit —— STM32

P12————A10

P13————A9

5V————5V

GND————GND

The lithium battery supplies 5V to the IObit.

Program address:

https://makecode.microbit.org/_5Pi3AiUAKJv2

Receiver

Only one Microbit is needed, which is plugged into the Robotbit with an external cube LED strip module (P1).

Program address:

https://makecode.microbit.org/_WvyiVaMzuPze

Physical Testing

When gestures 1-6 are triggered, the LED strip turns off.

When gestures 7-12 are triggered, the LED strip lights up.

[!NOTE]

Tip: Press the button and release it, then complete the gesture within 1.5 seconds. Make the gesture more forceful.

Assemble the AI Wand, paying attention to the installation positions of the screws. Use double-sided tape to attach the lithium battery pack to the side of the casing.

Assembly Complete

Testing

The core basic interaction has been successfully tested.

The rest is up to your creativity.