Classification (or 2D image classification)

Changed in RDK v0.2.36 and API v0.1.118

2D Image Classification is the process of taking a 2D image from a camera and deciding which class label, out of many, best describes the given image. Any camera that can return 2D images can use 2D image classification.

The class labels used for classification vary and depend on the machine learning model and how it was trained.

The returned classifications consist of the image’s class label and confidence score.

• class_name (string): specifies the label of the found object.
• confidence (float): specifies the confidence of the assigned label. Between 0.0 and 1.0, inclusive.

The types of classifiers supported are:

• mlmodel: a machine learning classifier that returns a class label and confidence score according to the specified tensorflow-lite model file available on the robot’s hard drive.

Configure a mlmodel classifier

To create a mlmodel classifier, you need an ML Model Service with a suitable model.

Navigate to the robot page on the Viam app. Click on the robot you wish to add the Vision Service to. Select the Config tab, and click on Services.

Scroll to the Create Service section.

{
  "mlmodel_name": "<classifier_name>"
}

"services": [
  {
    "name": "<service_name>",
    "type": "vision",
    "model": "mlmodel",
    "attributes": {
      "mlmodel_name": "<classifier_name>"
    }
  },
  ... // Other services
]

"services": [
  {
    "name": "fruit_classifier",
    "type": "vision",
    "model": "mlmodel",
    "attributes": {
      "mlmodel_name": "fruit_classifier"
    }
  }
]

Click Save config and head to the Components tab.

Add a camera component and a “transform” model

You cannot interact directly with the Vision Service. To be able to interact with the Vision Service you must:

1. Configure a physical camera component.

2. Configure a transform camera with the following attributes to view output from the classifier overlaid on images from the physical camera:

   {
   "pipeline": [
       {
       "type": "classifications",
       "attributes": {
           "confidence_threshold": 0.5,
           "classifier_name": "my_classifier"
       }
       }
   ],
   "source": "<camera-name>"
   }
   

   After adding the component and its attributes, click Save config. Wait for the robot to reload, and then go to the Control tab to test the stream of detections.

   

Code

The following code gets the robot’s Vision Service and then runs a classifier vision model on an image from the robot’s camera "cam1":

from viam.services.vision import VisionClient, VisModelConfig, VisModelType

robot = await connect()
# grab camera from the robot
cam1 = Camera.from_robot(robot, "cam1")
# grab Viam's vision service for the classifier
my_classifier = VisionClient.from_robot(robot, "my_classifier")

img = await cam1.get_image()
classifications = await my_classifier.get_classifications(img)

await robot.close()

import (
"go.viam.com/rdk/config"
"go.viam.com/rdk/services/vision"
"go.viam.com/rdk/components/camera"
)

// grab the camera from the robot
cameraName := "cam1" // make sure to use the same component name that you have in your robot configuration
myCam, err := camera.FromRobot(robot, cameraName)
if err != nil {
  logger.Fatalf("cannot get camera: %v", err)
}

visService, err := vision.from_robot(robot=robot, name='my_classifier')
if err != nil {
    logger.Fatalf("Cannot get Vision Service: %v", err)
}

// gets the stream from a camera
camStream, err := myCam.Stream(context.Background())

// gets an image from the camera stream
img, release, err := camStream.Next(context.Background())
defer release()

// Apply the color classifier to the image from your camera (configured as "cam1")
classifications, err := visService.GetClassifications(context.Background(), img)
if err != nil {
    logger.Fatalf("Could not get classifications: %v", err)
}
if len(classifications) > 0 {
    logger.Info(classifications[0])
}