Blindness-aiding environment sensing device

The present paper will examine the work of the creator X, namely the Blindness Aid Environment Sensing Device. The device employs the Grove Vision AI V2-Camera module to identify objects within the surrounding environment, while the XIAO ESP32S3 and Raspberry Pi are utilised for speech output. The device combines object detection and text-to-speech technology in order to provide the visually impaired with environmental information.

Background

Individuals with visual impairments frequently encounter difficulties in both seeing and moving around, which can significantly impede their ability to navigate their surroundings. This project will demonstrate how artificial intelligence and computer vision technology can assist in addressing this challenge. The objective of this project is to assist blind individuals in becoming less reliant on their immediate surroundings and external assistance.

The project unites object detection and text-to-speech technology to provide visually impaired individuals with a comprehensive understanding of their surroundings. The converted speech can be listened to through headphones, thus facilitating a more comprehensive comprehension of the surrounding environment.

Developing Object Detection Models with SenseCraft AI

I utilise the SenseCraft AI platform for the training of object detection models. It is the platform of choice for the development and deployment of models.

To commence work on a project, it is simply necessary to log in to one's SenseCraft AI account (or alternatively, to create a free account). The platform offers a straightforward approach to utilising existing models or creating new ones.

It is possible to upload existing data or record new data using a connected device in SenseCraft AI. A dataset comprising common items such as chairs, tables and pedestrians was assembled for the project. The greater the number of items included in the dataset, the more effective the model will be. Furthermore, consideration should be given to the size of the dataset. The greater the number of images of a specific object, the more precise the resulting model.

The initial dataset comprised 312 images of three distinct objects. In the future, additional images of additional objects will be incorporated into the dataset with the objective of further enhancing the model's accuracy.

Upload item identification models to Grove Vision AI

• Open the SenseCraft platform: Home - SenseCraft AI
• Open the SenseCraft homepage and find the Model Training option, click on it to enter

In the model training interface, first choose the type of classification and recognition you want to use, then select the Grove -Vision Al Module V2 at '2' and click on the connection. Then add categories according to what you need and name them.

Next, choose the Grove -Vision Al Module V2 device, then click Start Training, Advanced Settings On Demand. You can also set the default here. Once training is complete, you can select the Grove -Vision Al Module V2 for model deployment. While we're waiting for the deployment to finish, we can keep an eye on the camera on the right-hand side of the page to see how the preview of the object changes in real time. This step is complete, which means the model has been successfully deployed.

Object recognition via serial output using XIAO ESP32S3

Just connect the XIAO ESP32S3 to the Grove-Vision Al Module V2 with the uploaded model and you're good to go.

You'll find the full program below that outputs the serial number of the recognized object. Just compile and upload this program for the XIAO ESP32 series.

#include <Seeed_Arduino_SSCMA.h>
SSCMA AI;
void setup()
{
AI.begin();
Serial.begin(600);
}
void loop()
{
if (!AI.invoke())
{
Serial.println("invoke success");
Serial.printf("perf: prepocess=%d, inference=%d, postprocess=%d\n",
AI.perf().prepocess, AI.perf().inference,
AI.perf().postprocess);
for (int i = 0; i < AI.boxes().size(); i++)
{
Serial.printf(
"box %d: x=%d, y=%d, w=%d, h=%d, score=%d, target=%d\n", i,
AI.boxes()[i].x, AI.boxes()[i].y, AI.boxes()[i].w,
AI.boxes()[i].h, AI.boxes()[i].score, AI.boxes()[i].target);
}
for (int i = 0; i < AI.classes().size(); i++)
{
Serial.printf("class %d: target=%d, score=%d\n", i,
AI.classes()[i].target, AI.classes()[i].score);
delay(2000);
}
for (int i = 0; i < AI.points().size(); i++)0
{
Serial.printf("point %d: x=%d, y=%d, z=%d, score=%d, target=%d\n",
i, AI.points()[i].x, AI.points()[i].y,
AI.points()[i].z, AI.points()[i].score,
AI.points()[i].target);
}
}

Output results:

• l When I point the camera at someone, the serial port sends target1 so the system can recognise them.

• l When I point the camera at the table, the serial port outputs target2 so that the table can be recognised.

• l When I point the camera at the chair, the serial port outputs target0 to recognise it.

Configuring the Raspberry Pi

The XIAO ESP32S3 Sense with Grove -Vision Al Module V2 is a nifty little device that detects objects in your surroundings and tells you what they are and where they are. It uses a Raspberry Pi to receive the data and then convert it into speech.

We're using a Raspberry Pi 4 B, which is working well. Once I'd installed the operating system on the Raspberry Pi, I configured the audio control system and set the volume to 100%.

sudo raspi-config

Subsequently, the open-source software package Festival was installed on the Raspberry Pi. Developed by the UK's Speech Technology Research Centre, Festival provides a framework for the construction of speech synthesis systems. It offers comprehensive text-to-speech functionality through a range of application programming interfaces (APIs), including those accessible at the shell level, via the command interpreter, as a C++ library, and from the Java and Emacs editor interfaces.

To install Festival, enter the following command:

sudo apt-get install -y libasound2-plugins festival

Following the installation of Festival, a wired headset was connected and tested with the following audio.

echo "Hello" | festival --tts

Subsequently, the Python serial module was installed on the Raspberry Pi.

It is recommended that the XIAO ESP32S3 Sense be connected to the Raspberry Pi via a USB-C cable.

The process of writing code for the Raspberry Pi requires the identification of the serial port number of the XIAO Sense board as a preliminary step.

Once the XIAO Sense board has been connected and plugged into the Raspberry Pi, the following command can be executed in the terminal.

dmesg | grep tty

Results：

The serial port number has now been determined. The next step is to create the code. The following code was written for the Raspberry Pi to facilitate the conversion of incoming text to speech.

#!/usr/bin/env python
import time
import serial
import os
#Set the parameters of the serial port connection for communicating with the device
ser = serial.Serial(
        port='/dev/ttyACM1', # Specifies port to which the device is connected.
baudrate = 115200, # Setting the baud rate for serial communication
parity=serial.PARITY_NONE, # No parity bit is used
stopbits=serial.STOPBITS_ONE, # Using a stop bit
bytesize=serial.EIGHTBITS, # Each byte has 8 data bits
timeout=1 # Set the timeout for reading the serial port to 1 second
)
# Enter an infinite loop that keeps reading the serial port data
while True:
        receive_msg=ser.readline() # Read a line of data from the serial port
        print(receive_msg) # Print received data
# If the received data contains the word "1", perform the following operations:
        if b'1' in receive_msg.lower():
            os.system('echo "people in front" | festival --tts')
# If the received data contains the word ‘0’, do the following:
        if b'0' in receive_msg.lower():
            os.system('echo "chair in front" | festival --tts')
# If the received data contains the word ‘2’, perform the following operations:
        if b'2' in receive_msg.lower():
            os.system('echo " table in front" | festival --tts')

Test results:

Script to set up automatic boot up

.bashrc configuration

Modify the.bashrc file.

sudo nano /home/pi/.bashrc

Add the startup command text to the end of the /home/pi/.bashrc file.

echo Running at boot
sudo python /home/pi/sample.py

The echo command above is used to show that the scripts in the.bashrc file have started running.

The Bash RC configurator is run automatically when the Raspberry Pi starts up. The commands in the.bashrc file open a new terminal window when run.

But when I am configuring the above, I get the following error:

Solution:

For me, I had to add create the file /etc/asound.conf

and in this file and add this:

pcm.!default {
type asym
playback.pcm {
type plug
slave.pcm "hw:2,0"
}
}

This will run automatically when the Raspberry Pi is powered on!

Housing assembly

Based on the Seeed Raspberry Pi base case, the XIAO ESP32S3 with Grove V2 is mounted on the outside of the case by punching holes.

Final presentation: