Make a Robot Follow a Ball with Color Detection

From exploring dangerous environments to picking up litter, you can do so many things with a wheeled robot and computer vision. This tutorial is a great starting place for a wide variety of robotics projects.

In this tutorial, you’ll learn how to make a rover follow a colored object using the Viam vision service. I used a SCUTTLE rover with a Raspberry Pi 4, but you can use any rover with a webcam.

Set up your robot

Turn on power to the robot.

We'll be using Viam software to configure and control the robot. Go to the Viam app and create a new robot.

Configure your robot as a wheeled base with a webcam camera. Regardless of the type of base you are using, Setting up a SCUTTLE with Viam is a good place to start if you haven’t already configured your base.

Configure color detection

Navigate to the robot page on the Viam app. Click on the robot you're using for this tutorial. Select the Config tab, and click on Services.

Scroll to the Create Service section. To create a color detector vision service:

• Select vision as the Type.
• Enter my_color_detector as the Name.
• Select Color Detector as the Model.
• Click Create Service.

In your vision service’s panel, set the following Attributes:

• Set the color to #a13b4c or rgb(161, 59, 76)
• Set hue tolerance to 0.06
• Set the segment size to 100px

Click Save config and head to the Components tab.

You have configured a heuristic-based detector that draws boxes around objects according to their color.

I used the color #a13b4c or rgb(161, 59, 76) (a reddish color) to match the red ball I had. If you want to detect other colors, change the color parameter detect_color. Object colors can vary dramatically based on the light source. I recommend that you verify the desired color detection value under actual lighting conditions. To determine the color value from the actual cam component image, you can use a pixel color tool, like Color Picker for Chrome.

Test your color detector

You can test your detector in the Viam app:

• On the Config tab, configure a transform camera with the following attributes:

{
  "pipeline": [
      {
      "type": "detections",
      "attributes": {
          "confidence_threshold": 0.5,
          "detector_name": "my_color_detector"
      }
      }
  ],
  "source": "<camera-name>"
}

• For <camera-name>, insert the name of your configured physical camera.
• Click Save config.
• Navigate to the Control tab, click on your transform camera and toggle it on. The transform camera will now show detections with bounding boxes around the detected colors.

Tip: If the color is not reliably detected, try increasing the hue_tolerance_pct or adjusting the lighting of the area to make the color being detected more visible. Note that the detector does not detect black, perfect greys (greys where the red, green, and blue color component values are equal), or white.

Set up your code environment

We are going to use Virtualenv to set up a virtual environment for this project, in order to isolate the dependencies of this project from other projects. Run the following commands in your command line to install `virtualenv`, set up an environment `venv` and activate it:

python3 -m pip install --user virtualenv
python3 -m venv env
source env/bin/activate

Then, install the Viam Python SDK:

pip3 install viam-sdk

Connect

Next, go to the Code sample tab on your robot page and select Python, then click Copy.

Note about location secrets:By default, the sample code does not include your robot location secret. I strongly recommend that you add your location secret as an environment variable and import this variable into your development environment as needed.To show your robot’s location secret in the sample code, toggle Include secret on the Code sample tab. You can also see your location secret on the locations page.

This code snippet imports all the necessary packages and sets up a connection with the Viam app.

Next, create a file named "main.py" and paste the boilerplate code from the Code sample tab of the Viam app into your file. Then, save your file.

Run the code to verify that the Viam SDK is properly installed and that the viam-server instance on your robot is live. If you haven’t yet installed viam-server, follow the installation guide to install viam-server on your robot before proceeding with this tutorial.

You can run your code by typing the following into your terminal from the same directory as your main.py file:

python3 main.py

The program prints a list of robot resources.

On top of the packages that the code sample snippet imports, add the random and the vlc package to the imports. The top of your code should now look like this:

import asyncio
import random
import vlc

from viam.robot.client import RobotClient
from viam.rpc.dial import Credentials, DialOptions
from viam.components.board import Board
from viam.components.camera import Camera
from viam.components.servo import Servo
from viam.services.vision import VisionClient

async def connect():
    creds = Credentials(
        type="robot-location-secret",
        payload="LOCATION SECRET FROM THE VIAM APP")
    opts = RobotClient.Options(
        refresh_interval=0,
        dial_options=DialOptions(credentials=creds)
    )
    return await RobotClient.at_address("ADDRESS FROM THE VIAM APP", opts)

async def main():
    # Other code

if __name__ == "__main__":
  print("Starting up... ")
  asyncio.run(main())
  print("Done.")

You will update the main() function later.

Detect the location of a colored object

With the configured color detector, you can programmatically retrieve a list of detections. Each detection comes with information about where in the camera’s picture it is detected.

The following leftOrRight function checks where in the picture the largest detection is and returns a responding integer: 0 for left, 1 for center, and 2 for right.

Add the leftOrRight function below your connect function:

# Get largest detection box and see if it's center is in the left, center, or right third
def leftOrRight(detections, midpoint):
   largest_area = 0
   largest = Detection()
   if not detections:
      print("nothing detected :(")
      return -1
   for d in detections:
      a = (d.x_max - d.x_min) * (d.y_max-d.y_min)
      if a > largest_area:
         a = largest_area
         largest = d
   centerX = largest.x_min + largest.x_max/2
   if centerX < midpoint-midpoint/6:
      return 0 # on the left
   if centerX > midpoint+midpoint/6:
      return 2 # on the right
   else:
      return 1  # basically centered

Add the main function

The main function:

• defines variables for how the robot should move,
• connects to the robot,
• initializes the base, the camera, and the detector, and
• repeatedly calls the leftOrRight function and turns the rover’s base in the respective direction.

Replace the main function with the following code:

async def main():
   spinNum = 10         # when turning, spin the motor this much
   straightNum = 300    # when going straight, spin motor this much
   numCycles = 200      # run the loop X times
   vel = 500            # go this fast when moving motor
​
   # Connect to robot client and set up components
   robot = await connect()
   base = Base.from_robot(robot, "base")
   camera = Camera.from_robot(robot, "<camera-name>")

   # Grab the vision service for the detector
   my_detector = VisionClient.from_robot(robot, "my_color_detector")

   # Main loop. Detect the ball, determine if it's on the left or right, and head that way.
   # Repeat this for numCycles
   for i in range(numCycles):
      detections = await my_detector.get_detections_from_camera(camera)
​
      answer = leftOrRight(detections, frame.size[0]/2)
      if answer == 0:
         print("left")
         await base.spin(spinNum, vel)  # CCW is positive
         await base.move_straight(straightNum, vel)
      if answer == 1:
         print("center")
         await base.move_straight(straightNum, vel)
      if answer == 2:
         print("right")
         await base.spin(-spinNum, vel)
      # If nothing is detected, nothing moves

   await robot.close()​

For <camera-name>, insert the name of your configured physical camera.

Run the code

Now, run the code again, from the same directory as your main.py file:

python3 main.py

Your rover should detect and navigate towards any red objects that come into view of its camera. Use something like a red sports ball or book cover as a target to follow to test your rover.

Congratulations! You now have a robot that can follow colored objects!

If you’re ready for more, try making your rover detect other colors. You could also write some code with a Viam SDK to make your rover move in a square.