How to train and deploy ML/computer vision models

1. Collect images

Start by collecting images from your cameras and syncing it to the Viam app. See Collect image data and sync it to the cloud for instructions.

When training machine learning models, it is important to supply a variety of different data about the subject in different situations, such as from different angles or in different lighting situations. The more varied the provided data set, the more accurate the resulting model becomes.

2. Label your images

Once you have enough images of the objects you’d like to identify captured and synced to the Viam app, use the interface on the DATA tab to label your data.

You can label your images to create:

• Detection models: Draw bounding boxes around distinct objects within captured images. The trained model will enable your machine to be able to detect those objects on its own.
• Classification models: Add tags to each of your images with class labels that describe it. The trained model will enable your machine to classify similar images on its own.

3. Create a dataset

A dataset allows you to conveniently view, work with, and train an ML model on a collection of images.

Use the interface on the DATA tab (or the viam data dataset add command) to add all images you want to train the model on to a dataset.

Click on an image you want to train your ML model. In the Actions pane on the right-hand side, enter a dataset name under Datasets, then press return. Repeat this with all images you want to add to your dataset.

viam dataset create --org-id=<org-id> --name=<name>
viam dataset data add filter --dataset-id=<dataset-id> --tags=red_star,blue_square

import asyncio

from viam.rpc.dial import DialOptions, Credentials
from viam.app.viam_client import ViamClient
from viam.utils import create_filter
from viam.proto.app.data import BinaryID


async def connect() -> ViamClient:
    dial_options = DialOptions(
      credentials=Credentials(
        type="api-key",
        # Replace "<API-KEY>" (including brackets) with your machine's API key
        payload='<API-KEY>',
      ),
      # Replace "<API-KEY-ID>" (including brackets) with your machine's
      # API key ID
      auth_entity='<API-KEY-ID>'
    )
    return await ViamClient.create_from_dial_options(dial_options)


async def main():
    # Make a ViamClient
    viam_client = await connect()
    # Instantiate a DataClient to run data client API methods on
    data_client = viam_client.data_client

    # Replace "<PART-ID>" (including brackets) with your machine's part id
    my_filter = create_filter(part_id="<PART-ID>")

    print("Getting data for part...")
    binary_metadata, _, _ = await data_client.binary_data_by_filter(
        my_filter,
        include_binary_data=False
    )
    my_binary_ids = []

    for obj in binary_metadata:
        my_binary_ids.append(
            BinaryID(
                file_id=obj.metadata.id,
                organization_id=obj.metadata.capture_metadata.organization_id,
                location_id=obj.metadata.capture_metadata.location_id
                )
            )
    print("Creating dataset...")
    # Create dataset
    try:
        dataset_id = await data_client.create_dataset(
            name="MyDataset",
            organization_id=ORG_ID
        )
        print("Created dataset: " + dataset_id)
    except Exception:
        print("Error. Check that the dataset name does not already exist.")
        print("See: https://app.viam.com/data/datasets")
        return 1

    print("Adding data to dataset...")
    await data_client.add_binary_data_to_dataset_by_ids(
        binary_ids=my_binary_ids,
        dataset_id=dataset_id
    )
    print("Added files to dataset.")
    print("See dataset: https://app.viam.com/data/datasets?id=" + dataset_id)

    viam_client.close()

if __name__ == '__main__':
    asyncio.run(main())

To remove an image from a dataset click on the x button next to the dataset name.

1. Train an ML model

In the Viam app, navigate to your list of DATASETS and select the one you want to train on.

Click Train model and follow the prompts.

Select to train a new model or update an existing model. You can train or update using Built-in training or using a training script from the Viam Registry.

Click Next steps.

2. Select the details for your ML model

• Enter a name or use the suggested name for your new model.
• For built-in training scripts, select a Model Type. Depending on the training script you’ve chose, you may have a number of these options:
  • Single Label Classification: The resulting model predicts one of the selected labels or UNKNOWN per image. Select this if you only have one label on each image. Ensure that the dataset you are training on also contains unlabeled images.
  • Multi Label Classification: The resulting model predicts one or more of the selected labels per image.
  • Object Detection: The resulting model predicts either no detected objects or any number of object labels alongside their locations per image.
• For built-in classification training, select the tags you want to train your model on from the Labels section. Unselected tags will be ignored, and will not be part of the resulting model.
• Click Train model.

3. Wait for your model to train

The model now starts training and you can follow its process in the Training section of the Models page.

Once the model has finished training, it becomes visible in the Models section of the page. You will receive an email when your model finishes training.

4. Debug your training job

If your training job failed you can check your job’s logs with the CLI.

You can obtain the job’s id by listing the jobs:

viam train list --org-id=<INSERT ORG ID> --job-status=unspecified

Then use the job id to get your training job’s logs:

viam train logs --job-id=<JOB ID>

1. Deploy your ML model

Navigate to the CONFIGURE tab of one of your machines in the Viam app. Here, add an ML model service that supports the ML model you just trained and add the model as the Model. For example use the TFLite CPU ML model service for TFlite ML models. This service will deploy and run the model.

2. Configure an mlmodel vision service

The vision service takes the ML model and applies it to the stream of images from your camera.

Add the vision / ML model service to your machine. Then, from the Select model dropdown, select the name of the ML model service you configured in the last step (for example, mlmodel-1).

Click Save to save your changes.

3. Use your detector or classifier

You can test your detector by clicking on the Test area of the vision service’s configuration panel or from the CONTROL tab.

The camera stream will show classification or detections when it identifies something, depending on your model. Try placing an object your ML model can recognize in front of the camera. If you are using a Viam rover, use the viam_base panel to move your rover, then click on the vision panel to check for classifications or detections.

"default_minimum_confidence": 0.82

{
  "mlmodel_name": "mlmodel-1",
  "default_minimum_confidence": 0.82
}

You can also test your detector or classifier with code.