Object detection with TensorFlow Lite

This project is designed to help you to use OpenCV to do objection detection on Qualcomm® Robotics RB5 development kit with a USB camera.

Objective

The main objective of this project is to start your artificial intelligence (AI) application development for robotics using the Qualcomm Robotics RB5 development kit. It will walk you through the following steps:

• Compile the TensorFlow lite libraries
• Test USB camera
• Run an objection detection application on the CPU of the Qualcomm Robotics RB5 development kit.

Materials Required / Parts List / Tools

• An Ubuntu 18.04 PC
• Qualcomm Robotics RB5 Development kit
• A USB camera
• A display monitor

Source Code / Source Examples / Application Executable

• TensorFlow
• Object detection application
• OpenCV for Wayland

Build / Assembly Instructions

Hardware set up.

• Connect the Qualcomm Robotics RB5 development kit to the monitor through HDMI cable.
• Plugin a keyboard and a mouse to the development board.
• Connect the USB camera module to the development board.

2. Wi-Fi setup on Qualcomm Robotics RB5 Development kit

Use the below command to setup the Wi-Fi connection.

root@qrb5165-rb5:/# vi /data/misc/wifi/wpa_supplicant.conf

Please modify the details listed below to set Wi-Fi host name and password, ten save it and reboot the device.

network={

ssid="YOUR_SSID_NAME"

key_mgmt=WPA-PSK

pairwise=TKIP CCMP

group=TKIP CCMP

psk="PASSWD"

}

After the reboot is finished, run the command “ifconfig” to check if the Wi-Fi is connected.

3. Run Weston desktop on Qualcomm Robotics RB5 Development kit

If the HDMI cable connects properly, after the device boot up, you will see the device boot up logs and then see a shell is ready for command input. Run the following command to launch Weston desktop.

root@qrb5165-rb5:/# Weston --connector=29

If it fails to run, you can try the following commands with adb on an Ubuntu PC. Please make sure the Qualcomm Robotics RB5 development kit is connected to the Ubuntu PC with a USB cable. Please ignore this step if the above command runs successfully.

adb shell

mkdir -p /usr/bin/weston_socket

chmod 700 /usr/bin/weston_socket

export XDG_RUNTIME_DIR=/usr/bin/weston_socket

export LD_LIBRARY_PATH=/usr/lib:/usr/lib/aarch64-linux-gnu/

weston --tty=1 --connector=29 --backend=drm-backend.so

4. Build the TF lite library on the Ubuntu host machine.

Here are the commands. For more information see the link: https://www.tensorflow.org/lite/guide/build_arm64

git clone https://github.com/tensorflow/tensorflow.git

cd tensorflow

git branch r2.1 remotes/origin/r2.1

git checkout r2.1

./tensorflow/lite/tools/make/build_generic_aarch64_lib.sh

Please make sure the libtensorflow-lite.a is generated after the above steps are done.

5. Build OpenCV for Wayland

The Qualcomm Robotics RB5 development board supports Wayland/Weston as the display server, this OpenCV is a special version for Wayland compiled on the development board.

For More info about OpenCV refer:

https://docs.opencv.org/master/d7/d9f/tutorial_linux_install.html.

Here are the steps to compile OpenCV on the target Qualcomm Robotics RB5 Development kit

root@qrb5165-rb5:/# sudo apt-get install build-essential curl unzip

root@qrb5165-rb5:/# sudo apt-get install cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev

root@qrb5165-rb5:/# sudo apt-get install libjpeg-dev libpng-dev

root@qrb5165-rb5:/# sudo apt-get install python-numpy libxkbcommon-dev libwayland-client0 libwayland-dev

root@qrb5165-rb5:/# mkdir /home/src

root@qrb5165-rb5:/# cd /home/src

root@qrb5165-rb5:/# git clone https://github.com/pfpacket/opencv-wayland.git

root@qrb5165-rb5:/# cd opencv-wayland/

root@qrb5165-rb5:/# mkdir build

root@qrb5165-rb5:/# cd build

root@qrb5165-rb5:/# cmake -D CMAKE_BUILD_TYPE=Debug -D CMAKE_INSTALL_PREFIX=/usr/local -DWITH_IPP=OFF -DWITH_WAYLAND=ON -DWITH_GTK=OFF ..

root@qrb5165-rb5:/# make -j7

6. Build the object detection application on target

Qualcomm Robotics RB5 Development kit.

Before you build the application, install TFlite dependencies first.

Here are commands to install TFlite dependencies.

root@qrb5165-rb5:/# cd /home/src

root@qrb5165-rb5:/# git clone https://github.com/tensorflow/tensorflow.git

root@qrb5165-rb5:/# cd tensorflow

root@qrb5165-rb5:/# git branch r2.1 remotes/origin/r2.1

root@qrb5165-rb5:/# git checkout r2.1

root@qrb5165-rb5:/#./tensorflow/lite/tools/make/download_dependencies.sh

The following steps are to build the application.

root@qrb5165-rb5:/# cd /home/src

root@qrb5165-rb5:/# git clone https://github.com/mattn/webcam-detect-tflite.git

root@qrb5165-rb5:/# cd webcam-detect-tflite

root@qrb5165-rb5:/# mkdir libs

Place the libtensorflow-lite.a file generated on the step #4 to the libs folder using the command adb push libtensorflow-lite.a

Copy the following content and save it as CMakeList.txt to /home/src/webcam-detect-tflite

cmake_minimum_required(VERSION 3.0)

project(webcam-detector)

set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++17 -O2 -g")

set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -ldl -lrt")

message(STATUS "optional:-std=c++17")

#set(OpenCV_DIR "/home/src/opencv-wayland/build/")

include_directories(/usr/include/freetype2/)

include_directories(/home/src/tensorflow/)

include_directories(/home/src/tensorflow/tensorflow/lite/tools/make/downloads/flatbuffers/include/)

find_package(OpenCV REQUIRED)

message(STATUS "OpenCV library status:")

message(STATUS " version: ${OpenCV_VERSION}")

message(STATUS " libraries: ${OpenCV_LIBS}")

message(STATUS " include path: ${OpenCV_INCLUDE_DIRS}")

include_directories( ${OpenCV_INCLUDE_DIRS} )

ADD_LIBRARY(TFlite_LIB STATIC IMPORTED)

SET_TARGET_PROPERTIES(TFlite_LIB PROPERTIES

IMPORTED_LOCATION /home/rb5/webcam-detect-tflite/libs/libtensorflow-lite.a)

set(SOURCE_FILES

main.cxx)

add_executable(${PROJECT_NAME} ${SOURCE_FILES})

target_link_libraries(${PROJECT_NAME} ${OpenCV_LIBS} )

target_link_libraries(${PROJECT_NAME}

freetype

pthread

dl

)

target_link_libraries(${PROJECT_NAME} TFlite_LIB)

After saving the above content to CMakeList.txt, execute the following commands:

root@qrb5165-rb5:/# mkdir build

root@qrb5165-rb5:/# cd build

root@qrb5165-rb5:/# cmake ..

root@qrb5165-rb5:/# make

If the above steps run successfully; the application webcam-detector will be generated under the folder build.

If you want to know more details about Tensorflow lite, please visit:

https://www.tensorflow.org/lite

Usage Instructions

Before running the application, please check the USB camera node.

By default, the device node in kernel should be /dev/video2. You can reconnect the USB camera module to check if this node appears and disappears accordingly.

Run the following commands on the Qualcomm Robotics RB5 Development kit to run the application.

root@qrb5165-rb5:/# cd home/src/webcam-detect-tflite/build

root@qrb5165-rb5:/# ./webcam-detector

Then you will see the classification result as shown below.

Qualcomm Robotics RB5 and Qualcomm QRB6165 are products of Qualcomm Technologies, Inc. and/or its subsidiaries