Configure an mlmodel Detector or Classifier

Changed in RDK v0.2.36 and API v0.1.118

The mlmodel model of the Viam vision service supports machine learning detectors and classifiers that draw bounding boxes or return class labels based on a deployed TensorFlow Lite, TensorFlow, PyTorch, or ONNX ML model.

Prerequisites

Before configuring your mlmodel detector or classifier, you need to:

Configure your detector or classifier

Once you have deployed your ML model, configure your mlmodel detector or classifier:

"services": [
  {
    "name": "<service_name>",
    "type": "vision",
    "model": "mlmodel",
    "attributes": {
      "mlmodel_name": "<mlmodel-service-name>"
    }
  },
  ... // Other services
]

"services": [
  {
    "name": "person_detector",
    "type": "vision",
    "model": "mlmodel",
    "attributes": {
      "mlmodel_name": "my_mlmodel_service"
    }
  }
]

"services": [
  {
    "name": "fruit_classifier",
    "type": "vision",
    "model": "mlmodel",
    "attributes": {
      "mlmodel_name": "fruit_classifier"
    }
  }
]

Click the Save button in the top right corner of the page.

The following attributes are available for an mlmodel detector or classifier:

Tensor names

Both the mlmodel detector and classifier require that the input and output tensors defined by your ML model are named according to the following:

• For an mlmodel detector:
  • The input tensor must be named image
  • The output tensors must be named location, category, and score,
• For an mlmodel classifier:
  • The input tensor must be named image
  • The output tensor must be named probability

If you trained a TFlite ML model using the Viam app, your mlmodel tensors are already named in this fashion, and you can proceed to test your detector or classifier. However, if you uploaded your own ML model, or are using one from the Viam Registry, you may need to remap your tensor names to meet this requirement, and should follow the instructions to remap tensor names.

Remap tensor names

If you need to remap the tensor names defined by your ML model to meet the tensor name requirements of the mlmodel detector or classifier, you can use the remap_input_names and remap_output_names attributes:

{
  "type": "vision",
  "model": "mlmodel",
  "attributes": {
    "mlmodel_name": "my_model",
    "remap_output_names": {
      "my_model_output_tensor1": "category",
      "my_model_output_tensor2": "location",
      "my_model_output_tensor3": "score"
    },
    "remap_input_names": {
      "my_model_input_tensor1": "image"
    }
  },
  "name": "my-vision-service"
}

{
  "type": "vision",
  "model": "mlmodel",
  "attributes": {
    "mlmodel_name": "my_model",
    "remap_output_names": {
      "my_model_output_tensor1": "probability"
    },
    "remap_input_names": {
      "my_model_input_tensor1": "image"
    }
  },
  "name": "my-vision-service"
}

When done, click the Save button in the top right corner of the page, then proceed to test your detector or classifier.

Test your detector or classifier

You can test your detector or classifier with existing images in the Viam app or live camera footage. You can also test detectors and classifiers with existing images on a computer.

Existing images in the cloud

If you have images stored in the Viam Cloud, you can run your classifier against your images in the Viam app.

1. Navigate to the Data tab and click on the Images subtab.
2. Click on an image to open the side menu, and select the Actions tab under the Data tab.
3. In the Run model section, select your model and specify a confidence threshold.
4. Click Run model

If the classifier’s results exceed the confidence threshold, the Run model section shows a label and the responding confidence threshold.

Live camera footage

You can test your detector or classifier from the Control tab or with code using a camera that is part of your machine.

Test your vision service

1. Configure a camera component.

   Tip

   This is the camera whose name you need to pass to vision service methods.

2. After adding the camera, click the Save button in the top right corner of the page.

3. Click on the Test area on the vision service configuration panel, or navigate to the CONTROL tab, click on the vision service and select your camera and vision service and then click Refresh. The panel will show detections with confidence above the default_minimum_confidence with bounding boxes on the image.

{
  "pipeline": [
    {
      "type": "detections",
      "attributes": {
        "confidence_threshold": 0.5,
        "detector_name": "<vision-service-name>",
        "valid_labels": ["<label>"]
      }
    }
  ],
  "source": "<camera-name>"
}

{
  "pipeline": [
    {
      "type": "classifications",
      "attributes": {
        "confidence_threshold": 0.5,
        "classifier_name": "<vision-service-name>",
        "max_classifications": <int>,
        "valid_labels": [ "<label>" ]
      }
    }
  ],
  "source": "<camera-name>"
}

Test with code

The following code gets the machine’s vision service and then runs a detector or classifier vision model on an image from the machine’s camera "cam1".

from viam.services.vision import VisionClient

robot = await connect()
camera_name = "cam1"

cam1 = Camera.from_robot(robot, camera_name)
my_detector = VisionClient.from_robot(robot, "my_detector")

detections = await my_detector.get_detections_from_camera(camera_name)

# If you need to store the image, get the image first
# and then run detections on it. This process is slower:
img = await cam1.get_image()
detections_from_image = await my_detector.get_detections(img)

await robot.close()

import (
  "go.viam.com/rdk/config"
  "go.viam.com/rdk/services/vision"
  "go.viam.com/rdk/components/camera"
)

cameraName := "cam1"
myCam, err := camera.FromRobot(robot, cameraName)
if err != nil {
  logger.Fatalf("cannot get camera: %v", err)
}

myDetector, err := vision.from_robot(robot, "my_detector")
if err != nil {
    logger.Fatalf("Cannot get vision service: %v", err)
}

// Get detections from the camera output
detections, err := myDetector.DetectionsFromCamera(context.Background(), myCam, nil)
if err != nil {
    logger.Fatalf("Could not get detections: %v", err)
}
if len(directDetections) > 0 {
    logger.Info(detections[0])
}

// If you need to store the image, get the image first
// and then run detections on it. This process is slower:

// Get the stream from a camera
camStream, err := myCam.Stream(context.Background())

// Get an image from the camera stream
img, release, err := camStream.Next(context.Background())
defer release()

// Apply the color classifier to the image from your camera (configured as "cam1")
detectionsFromImage, err := myDetector.Detections(context.Background(), img, nil)
if err != nil {
    logger.Fatalf("Could not get detections: %v", err)
}
if len(detectionsFromImage) > 0 {
    logger.Info(detectionsFromImage[0])
}

from viam.services.vision import VisionClient

robot = await connect()
camera_name = "cam1"
cam1 = Camera.from_robot(robot, camera_name)
my_classifier = VisionClient.from_robot(robot, "my_classifier")

# Get the top 2 classifications with the highest confidence scores from the
# camera output
classifications = await my_classifier.get_classifications_from_camera(
    camera_name, 2)

# If you need to store the image, get the image first
# and then run classifications on it. This process is slower:
img = await cam1.get_image()
classifications_from_image = await my_classifier.get_classifications(img, 2)

await robot.close()

import (
  "go.viam.com/rdk/config"
  "go.viam.com/rdk/services/vision"
  "go.viam.com/rdk/components/camera"
)

cameraName := "cam1"
myCam, err := camera.FromRobot(robot, cameraName)
if err != nil {
  logger.Fatalf("cannot get camera: %v", err)
}

myClassifier, err := vision.from_robot(robot, "my_classifier")
if err != nil {
    logger.Fatalf("Cannot get vision service: %v", err)
}

// Get the top 2 classifications with the highest confidence scores from the camera output
classifications, err := visService.ClassificationsFromCamera(context.Background(), myCam, 2, nil)
if err != nil {
    logger.Fatalf("Could not get classifications: %v", err)
}
if len(directClassifications) > 0 {
    logger.Info(classifications[0])
}

// If you need to store the image, get the image first
// and then run classifications on it. This process is slower:

// Get the stream from a camera
camStream, err := myCam.Stream(context.Background())

// Get an image from the camera stream
img, release, err := camStream.Next(context.Background())
defer release()

// Apply the color classifier to the image from your camera (configured as "cam1")
// Get the top 2 classifications with the highest confidence scores
classificationsFromImage, err := visService.GetClassifications(context.Background(), img, 2, nil)
if err != nil {
    logger.Fatalf("Could not get classifications: %v", err)
}
if len(classificationsFromImage) > 0 {
    logger.Info(classificationsFromImage[0])
}

Existing images on your machine

If you would like to test your detector or classifier with existing images, load the images and pass them to the detector or classifier:

from viam.services.vision import VisionClient
from PIL import Image

robot = await connect()
# Grab Viam's vision service for the detector
my_detector = VisionClient.from_robot(robot, "my_detector")

# Load an image
img = Image.open('test-image.png')

# Apply the detector to the image
detections_from_image = await my_detector.get_detections(img)

await robot.close()

import (
  "go.viam.com/rdk/config"
  "go.viam.com/rdk/services/vision"
  "image/jpeg"
  "os"
)

myDetector, err := vision.from_robot(robot, "my_detector")
if err != nil {
    logger.Fatalf("Cannot get Vision Service: %v", err)
}

// Read image from existing file
file, err := os.Open("test-image.jpeg")
if err != nil {
    logger.Fatalf("Could not get image: %v", err)
}
defer file.Close()
img, err := jpeg.Decode(file)
if err != nil {
    logger.Fatalf("Could not decode image: %v", err)
}
defer img.Close()

// Apply the detector to the image
detectionsFromImage, err := myDetector.Detections(context.Background(), img, nil)
if err != nil {
    logger.Fatalf("Could not get detections: %v", err)
}
if len(detectionsFromImage) > 0 {
    logger.Info(detectionsFromImage[0])
}

from viam.services.vision import VisionClient
from PIL import Image

robot = await connect()
# Grab Viam's vision service for the classifier
my_classifier = VisionClient.from_robot(robot, "my_classifier")

# Load an image
img = Image.open('test-image.png')

# Apply the classifier to the image
classifications_from_image = await my_classifier.get_classifications(img)

await robot.close()

import (
  "go.viam.com/rdk/config"
  "go.viam.com/rdk/services/vision"
  "image"
  "image/png"
  "os"
)

myClassifier, err := vision.from_robot(robot, "my_classifier")
if err != nil {
    logger.Fatalf("Cannot get Vision Service: %v", err)
}

// Read image from existing file
file, err := os.Open("test-image.jpeg")
if err != nil {
    logger.Fatalf("Could not get image: %v", err)
}
defer file.Close()
img, err := jpeg.Decode(file)
if err != nil {
    logger.Fatalf("Could not decode image: %v", err)
}
defer img.Close()

// Apply the classifier to the image
classificationsFromImage, err := myClassifier.Classifications(context.Background(), img, nil)
if err != nil {
    logger.Fatalf("Could not get classifications: %v", err)
}
if len(classificationsFromImage) > 0 {
    logger.Info(classificationsFromImage[0])
}

Next Steps

For general configuration and development info, see: