Detection (or 2D object detection)

Changed in RDK v0.2.36 and API v0.1.118

2D Object Detection is the process of taking a 2D image from a camera and identifying and drawing a box around the distinct “objects” of interest in the scene. Any camera that can return 2D images can use 2D object detection.

The service provides different types of detectors, both based on heuristics and machine learning, so that you can create, register, and use detectors for any object you may need to identify.

The returned detections consist of the bounding box around the identified object, as well as its label and confidence score:

• x_min, y_min, x_max, y_max (int): specify the bounding box around the object.
• class_name (string): specifies the label of the found object.
• confidence (float): specifies the confidence of the assigned label. Between 0.0 and 1.0, inclusive.

You can use the following types of detectors:

• color_detector: A heuristic detector that draws boxes around objects according to their hue (does not detect black, gray, and white).
• mlmodel: A machine learning detector that draws bounding boxes according to a ML model available on the robot’s hard drive.

Configure a color_detector

A heuristic detector that draws boxes around objects according to their hue. Color detectors do not detect black, perfect grays (grays where the red, green, and blue color component values are equal), or white. It only detects hues found on the color wheel.

"services": [
  {
    "name": "<service_name>",
    "type": "vision",
    "model": "color_detector",
    "attributes": {
      "segment_size_px": <integer>,
      "detect_color": "#ABCDEF",
      "hue_tolerance_pct": <number>,
      "saturation_cutoff_pct": <number>,
      "value_cutoff_pct": <number>
    }
  },
  ... // Other services
]

"services": [
  {
    "name": "blue_square",
    "type": "vision",
    "model": "color_detector",
    "attributes": {
      "segment_size_px": 100,
      "detect_color": "#1C4599",
      "hue_tolerance_pct": 0.07,
      "value_cutoff_pct": 0.15
    }
  },
  {
    "name": "green_triangle",
    "type": "vision",
    "model": "color_detector",
    "attributes": {
      "segment_size_px": 200,
      "detect_color": "#62963F",
      "hue_tolerance_pct": 0.05,
      "value_cutoff_pct": 0.20
    }
  }
]

The following parameters are available for a "color_detector".

Click Save config and head to the Components tab. Proceed to Add a camera component and a “transform” model.

Configure a mlmodel detector

A machine learning detector that draws bounding boxes according to the specified tensorflow-lite model file available on the robot’s hard drive. To create a mlmodel classifier, you need an ML Model Service with a suitable model.

Navigate to the robot page on the Viam app. Click on the robot you wish to add the Vision Service to. Select the Config tab, and click on Services.

Scroll to the Create Service section.

{
  "mlmodel_name": "<detector_name>"
}

"services": [
  {
    "name": "<service_name>",
    "type": "vision",
    "model": "mlmodel",
    "attributes": {
      "mlmodel_name": "<detector_name>"
    }
  }
]

"services": [
  {
    "name": "person_detector",
    "type": "vision",
    "model": "mlmodel",
    "attributes": {
      "mlmodel_name": "person_detector"
    }
  }
]

Click Save config and head to the Components tab.

Add a camera component and a “transform” model

You cannot interact directly with the Vision Service. To be able to interact with the Vision Service you must:

1. Configure a physical camera component.
2. Configure a transform camera to view output from the detector overlaid on images from the physical camera.

After adding the component and its attributes, click Save config.

Wait for the robot to reload, and then go to the Control tab to test the stream of detections.

Code

The following code gets the robot’s vision service and then runs a color detector vision model on an image from the robot’s camera "camera_1":

from viam.services.vision import VisionClient, VisModelConfig, VisModelType

robot = await connect()
# grab camera from the robot
cam1 = Camera.from_robot(robot, "cam1")
# grab Viam's vision service for the detector
my_detector = VisionClient.from_robot(robot, "my_detector")

img = await cam1.get_image()
detections = await my_detector.get_detections(img)

await robot.close()

import (
"go.viam.com/rdk/config"
"go.viam.com/rdk/services/vision"
"go.viam.com/rdk/components/camera"
)

// grab the camera from the robot
cameraName := "cam1" // make sure to use the same component name that you have in your robot configuration
myCam, err := camera.FromRobot(robot, cameraName)
if err != nil {
  logger.Fatalf("cannot get camera: %v", err)
}

visService, err := vision.from_robot(robot=robot, name='my_detector')
if err != nil {
    logger.Fatalf("Cannot get Vision Service: %v", err)
}

// gets the stream from a camera
camStream, err := myCam.Stream(context.Background())

// gets an image from the camera stream
img, release, err := camStream.Next(context.Background())
defer release()

// Apply the color classifier to the image from your camera (configured as "cam1")
detections, err := visService.GetDetections(context.Background(), img)
if err != nil {
    logger.Fatalf("Could not get detections: %v", err)
}
if len(detections) > 0 {
    logger.Info(detections[0])
}

Next Steps

Detect Color

Use the Vision Service in the Viam app to detect a color with the Viam Rover.

Colored Object Follower

Instructions for detecting and following a colored object with a SCUTTLE Robot on Viam software.

RGB Line Follower

Build a line-following robot that relies on a webcam and color detection.