Follow a Colored Object with a Rover (like SCUTTLE)

In this tutorial, you’ll learn how to use the vision service to make a rover follow a colored object. We’re using a SCUTTLE rover for this tutorial but you can use any rover, including the Viam rover.

You can see the full code at the end of the tutorial.

Requirements

You don’t need to buy or own any hardware to complete this tutorial.

You will need the following hardware to complete this tutorial:

• A wheeled rover, configured with a base component. If you have your own Viam rover follow the Viam Rover configuration instructions. If you own another mobile robot follow the general configuration instructions.

  This tutorial uses a SCUTTLE rover as an example but you can complete this tutorial using a Yahboom 4WD Smart Robot or any other wheeled robot that can be configured as a base component.

• An attached and configured webcam camera.

Set up the hardware

Connect the camera to the rover’s board. Turn on the power to the rover.

Configure color detection

This tutorial uses the color #a13b4c or rgb(161,59,76) (a reddish color).

To create a color detector vision service:

"services": [
  {
    "name": "my_color_detector",
    "api": "rdk:service:vision",
    "model": "color_detector",
    "attributes": {
      "segment_size_px": 100,
      "detect_color": "#a13b4c",
      "hue_tolerance_pct": 0.06
    }
  },
  ... // Other services
]

Click Save.

You have configured a heuristic-based detector that draws boxes around objects according to their color.

Test your color detector

You can test your detector by clicking on the Test area of the vision service’s configuration panel or from the CONTROL tab:

The camera stream will show detections with bounding boxes around the detected colors.

Program your rover

Set up your code environment

We are going to use Virtualenv to set up a virtual environment for this project, in order to isolate the dependencies of this project from other projects. Run the following commands in your command-line to install virtualenv, set up an environment venv and activate it:

python3 -m pip install --user virtualenv
python3 -m venv env
source env/bin/activate

Then, install the Viam Python SDK:

pip3 install viam-sdk

Connect

Next, go to the Code sample page of the CONNECT tab on your machine page and select Python.

To show your machine’s API key in the sample code, toggle Include API key.

This code snippet imports all the necessary packages and sets up a connection with the Viam app.

Next, create a file named main.py and paste the sample code from the Code sample page of the Viam app into your file. Then, save your file.

Run the code to verify that the Viam SDK is properly installed and that the viam-server instance on your robot is live. If you haven’t yet installed viam-server, follow the installation guide to install viam-server on your robot before proceeding with this tutorial.

You can run your code by typing the following into your terminal from the same directory as your main.py file:

python3 main.py

The program prints a list of robot resources.

On top of the packages that the code sample snippet imports, import pil_to_viam_image and viam_to_pil_image from viam.media.utils.pil. The top of your code should now look like this:

import asyncio

from viam.robot.client import RobotClient
from viam.rpc.dial import Credentials, DialOptions
from viam.services.vision import VisionClient
from viam.components.camera import Camera
from viam.components.base import Base
from viam.media.utils.pil import pil_to_viam_image, viam_to_pil_image


async def connect():
    opts = RobotClient.Options.with_api_key(
        # Replace "<API-KEY>" (including brackets) with your machine's API key
        api_key='<API-KEY>',
        # Replace "<API-KEY-ID>" (including brackets) with your machine's
        # API key ID
        api_key_id='<API-KEY-ID>'
    )
    return await RobotClient.at_address("ADDRESS FROM THE VIAM APP", opts)


async def main():
    # Other code
    print("Starting main")

if __name__ == "__main__":
    print("Starting up... ")
    asyncio.run(main())
    print("Done.")

You will update the main() function later.

Detect the location of a colored object

With the configured color detector, you can programmatically retrieve a list of detections. Each detection comes with information about where in the camera’s picture it is detected.

The following leftOrRight function checks where in the picture the largest detection is and returns a responding integer: 0 for left, 1 for center, and 2 for right.

Add the leftOrRight function below your connect function:

# Get largest detection box and see if it's center is in the left, center, or
# right third
def leftOrRight(detections, midpoint):
    largest_area = 0
    largest = {"x_max": 0, "x_min": 0, "y_max": 0, "y_min": 0}
    if not detections:
        print("nothing detected :(")
        return -1
    for d in detections:
        a = (d.x_max - d.x_min) * (d.y_max-d.y_min)
        if a > largest_area:
            a = largest_area
            largest = d
    centerX = largest.x_min + largest.x_max/2
    if centerX < midpoint-midpoint/6:
        return 0  # on the left
    if centerX > midpoint+midpoint/6:
        return 2  # on the right
    else:
        return 1  # basically centered

Add the main function

The main function:

• defines variables for how the robot should move,
• connects to the robot,
• initializes the base, the camera, and the detector, and
• repeatedly calls the leftOrRight function and turns the rover’s base in the respective direction.

Replace the main function with the following code:

async def main():
    spinNum = 10         # when turning, spin the motor this much
    straightNum = 300    # when going straight, spin motor this much
    numCycles = 200      # run the loop X times
    vel = 500            # go this fast when moving motor

    # Connect to robot client and set up components
    machine = await connect()
    base = Base.from_robot(machine, "my_base")
    camera_name = "<camera-name>"
    camera = Camera.from_robot(machine, camera_name)
    frame = await camera.get_image(mime_type="image/jpeg")

    # Convert to PIL Image
    pil_frame = viam_to_pil_image(frame)

    # Grab the vision service for the detector
    my_detector = VisionClient.from_robot(machine, "my_color_detector")

    # Main loop. Detect the ball, determine if it's on the left or right, and
    # head that way. Repeat this for numCycles
    for i in range(numCycles):
        detections = await my_detector.get_detections_from_camera(camera_name)

        answer = leftOrRight(detections, pil_frame.size[0]/2)
        if answer == 0:
            print("left")
            await base.spin(spinNum, vel)     # CCW is positive
            await base.move_straight(straightNum, vel)
        if answer == 1:
            print("center")
            await base.move_straight(straightNum, vel)
        if answer == 2:
            print("right")
            await base.spin(-spinNum, vel)
        # If nothing is detected, nothing moves

    await robot.close()

For <camera-name>, insert the name of your configured physical camera.

Run the code

Now, run the code again, from the same directory as your main.py file:

python3 main.py

Your rover should detect and navigate towards any red objects that come into view of its camera. Use something like a red sports ball or book cover as a target to follow to test your rover:

Next steps

Congratulations! If you’re ready for more, try making your rover detect other colors. You could also write some code with a Viam SDK to make your rover move in a square.

You can also ask questions in the Community Discord and we will be happy to help.

Full code

import asyncio

from viam.robot.client import RobotClient
from viam.rpc.dial import Credentials, DialOptions
from viam.services.vision import VisionClient
from viam.components.camera import Camera
from viam.components.base import Base
from viam.media.utils.pil import pil_to_viam_image, viam_to_pil_image


async def connect():
    opts = RobotClient.Options.with_api_key(
        # Replace "<API-KEY>" (including brackets) with your machine's API key
        api_key='<API-KEY>',
        # Replace "<API-KEY-ID>" (including brackets) with your machine's
        # API key ID
        api_key_id='<API-KEY-ID>'
    )
    return await RobotClient.at_address("ADDRESS FROM THE VIAM APP", opts)


# Get largest detection box and see if it's center is in the left, center, or
# right third
def leftOrRight(detections, midpoint):
    largest_area = 0
    largest = {"x_max": 0, "x_min": 0, "y_max": 0, "y_min": 0}
    if not detections:
        print("nothing detected :(")
        return -1
    for d in detections:
        a = (d.x_max - d.x_min) * (d.y_max-d.y_min)
        if a > largest_area:
            a = largest_area
            largest = d
    centerX = largest.x_min + largest.x_max/2
    if centerX < midpoint-midpoint/6:
        return 0  # on the left
    if centerX > midpoint+midpoint/6:
        return 2  # on the right
    else:
        return 1  # basically centered


async def main():
    spinNum = 10         # when turning, spin the motor this much
    straightNum = 300    # when going straight, spin motor this much
    numCycles = 200      # run the loop X times
    vel = 500            # go this fast when moving motor

    # Connect to robot client and set up components
    machine = await connect()
    base = Base.from_robot(machine, "my_base")
    camera_name = "<camera-name>"
    camera = Camera.from_robot(machine, camera_name)
    frame = await camera.get_image(mime_type="image/jpeg")

    # Convert to PIL Image
    pil_frame = viam_to_pil_image(frame)

    # Grab the vision service for the detector
    my_detector = VisionClient.from_robot(machine, "my_color_detector")

    # Main loop. Detect the ball, determine if it's on the left or right, and
    # head that way. Repeat this for numCycles
    for i in range(numCycles):
        detections = await my_detector.get_detections_from_camera(camera_name)

        answer = leftOrRight(detections, pil_frame.size[0]/2)
        if answer == 0:
            print("left")
            await base.spin(spinNum, vel)     # CCW is positive
            await base.move_straight(straightNum, vel)
        if answer == 1:
            print("center")
            await base.move_straight(straightNum, vel)
        if answer == 2:
            print("right")
            await base.spin(-spinNum, vel)
        # If nothing is detected, nothing moves

    await robot.close()

if __name__ == "__main__":
    print("Starting up... ")
    asyncio.run(main())
    print("Done.")