Classification (or 2D image classification)
Changed in RDK v0.2.36 and API v0.1.118
2D Image Classification is the process of taking a 2D image from a camera and deciding which class label, out of many, best describes the given image. Any camera that can return 2D images can use 2D image classification.
The class labels used for classification vary and depend on the machine learning model and how it was trained.
The returned classifications consist of the image’s class label and confidence score.
class_name
(string): specifies the label of the found object.confidence
(float): specifies the confidence of the assigned label. Between0.0
and1.0
, inclusive.
The types of classifiers supported are:
- mlmodel: a machine learning classifier that returns a class label and confidence score according to the specified
tensorflow-lite
model file available on the robot’s hard drive.
Configure a mlmodel
classifier
To create a mlmodel
classifier, you need an ML Model Service with a suitable model.
Navigate to the robot page on the Viam app. Click on the robot you wish to add the Vision Service to. Select the Config tab, and click on Services.
Scroll to the Create Service section.
- Select
vision
as the Type. - Enter a name as the Name.
- Select ML Model as the Model.
- Click Create Service.
In your Vision Service’s panel, fill in the Attributes field.
{
"mlmodel_name": "<classifier_name>"
}
Add the Vision Service object to the services array in your raw JSON configuration:
"services": [
{
"name": "<service_name>",
"type": "vision",
"model": "mlmodel",
"attributes": {
"mlmodel_name": "<classifier_name>"
}
},
... // Other services
]
"services": [
{
"name": "fruit_classifier",
"type": "vision",
"model": "mlmodel",
"attributes": {
"mlmodel_name": "fruit_classifier"
}
}
]
Click Save config and head to the Components tab.
Add a camera component and a “transform” model
You cannot interact directly with the Vision Service. To be able to interact with the Vision Service you must:
Configure a physical camera component.
Configure a transform camera with the following attributes to view output from the classifier overlaid on images from the physical camera:
{ "pipeline": [ { "type": "classifications", "attributes": { "confidence_threshold": 0.5, "classifier_name": "my_classifier" } } ], "source": "<camera-name>" }
After adding the component and its attributes, click Save config. Wait for the robot to reload, and then go to the Control tab to test the stream of detections.
Code
The following code gets the robot’s Vision Service and then runs a classifier vision model on an image from the robot’s camera "cam1"
:
from viam.services.vision import VisionClient, VisModelConfig, VisModelType
robot = await connect()
# grab camera from the robot
cam1 = Camera.from_robot(robot, "cam1")
# grab Viam's vision service for the classifier
my_classifier = VisionClient.from_robot(robot, "my_classifier")
img = await cam1.get_image()
classifications = await my_classifier.get_classifications(img)
await robot.close()
To learn more about how to use classification, see the Python SDK docs.
import (
"go.viam.com/rdk/config"
"go.viam.com/rdk/services/vision"
"go.viam.com/rdk/components/camera"
)
// grab the camera from the robot
cameraName := "cam1" // make sure to use the same component name that you have in your robot configuration
myCam, err := camera.FromRobot(robot, cameraName)
if err != nil {
logger.Fatalf("cannot get camera: %v", err)
}
visService, err := vision.from_robot(robot=robot, name='my_classifier')
if err != nil {
logger.Fatalf("Cannot get Vision Service: %v", err)
}
// gets the stream from a camera
camStream, err := myCam.Stream(context.Background())
// gets an image from the camera stream
img, release, err := camStream.Next(context.Background())
defer release()
// Apply the color classifier to the image from your camera (configured as "cam1")
classifications, err := visService.GetClassifications(context.Background(), img)
if err != nil {
logger.Fatalf("Could not get classifications: %v", err)
}
if len(classifications) > 0 {
logger.Info(classifications[0])
}
To learn more about how to use classification, see the Go SDK docs.
Tip
To see more code examples of how to use Viam’s Vision Service, see our example repo.
Have questions, or want to meet other people working on robots? Join our Community Discord.
Was this page helpful?
Glad to hear it! If there is anything we could be doing better, please create an issue.
We're sorry about that. If you'd like to talk to us for help, please join the Community Discord. To ensure we know what's wrong with this page, you can also open an issue.