Selectively Capture Data using filtered-camera

With the data management service, a Viam machine can capture data from a variety of components and sync that data to the Viam app. However, if your machine captures a large volume of data, especially image data such as pictures, you may wish to control which specific images are captured or uploaded.

For example, imagine that you have positioned your machine’s camera to view a busy city street. Your machine will happily capture as many pictures as its configured capture rate dictates, resulting in potentially a large number of images captured over the course of a day. If you wanted to see if any fancy sports cars might have driven by, you might have to search through many images to find out!

Instead, you can use the filtered-camera module to be able to selectively capture and sync only those images that meet the specific criteria you’ve outlined in a machine learning (ML) model. For example, you could train an ML model that is focused on sports cars, and only capture images from the camera feed when a sports car is detected in the frame.

In this tutorial, you will learn how to use the filtered-camera module to selectively capture images only when a specific object is detected within the camera feed: the Viam wooden figure. When this figure is not present in the camera frame, the filtered-camera module will stop capturing images until it detects the figure again.

You can train your own model to be able to identify any object you wish, or you can use a provided pre-trained model that is capable of identifying many common objects. This tutorial will provide instructions on both approaches.

The filtered-camera module is available from the Viam registry, and you can view the code on GitHub.

Prerequisites

Add a new machine in the Viam app. Then follow the setup instructions to install viam-server on the computer you’re using for your project and connect to the Viam app. Wait until your machine has successfully connected.

Add a camera and configure data capture

The filtered-camera module filters image data captured by a camera component, and so requires that your machine has a camera component and the data management service configured.

• A camera component, such as a webcam, allows your machine to see the world around it through an attached camera.
• The data management service enables your machine to capture images from an attached camera and sync them to the cloud.

Add a camera component

Add a camera component to your machine:

1. Navigate to your machine’s page on the Viam app and select the CONFIGURE tab.

2. Click the + (Create) button next to your main part in the left-hand menu and select Component.

3. Select camera and then select webcam.

4. Give the camera a name, such as my-webcam, and click Create.

5. If your machine is online and connected to the Viam app, your camera’s video path is automatically detected and configured. If your machine is not currently connected, you can manually select the video path for your camera, or bring your machine online to have this path automatically configured for you.

6. Click Save in the top right corner of the page to save your changes.

   

Add the data management service

Next, add the data management service to your machine to be able to capture and sync data:

1. On your machine’s CONFIGURE tab in the Viam app, navigate to the Services tab.

2. Click the + (Create) button next to your main part in the left-hand menu and select Service. Select data management.

3. Use the suggested name for your service or give it a name, like my-data-management, then click Create.

4. On the panel that appears, you can manage the capturing and syncing functions individually. By default, the data management service captures data to the ~/.viam/capture directory, and syncs captured data files to the Viam app every 6 seconds (0.1 minutes in the configuration). Leave the default settings as they are, and click Save in the top right corner of the page to save your changes.

   

For more information, see Add the data management service.

Configure data capture for a camera component

Once you have added the data management service, enable image data capture for your camera component:

1. Navigate to your machine’s CONFIGURE tab in the Viam app.

2. In the configuration pane for your configured camera component (named my-webcam in the above steps), find the Data capture section, and click the Add method button to enable data capture for this camera.

   • Set the Method to ReadImage and the Frequency to 0.333. This will capture an image from the camera roughly once every 3 seconds. You can adjust the capture frequency if you want the camera to capture more or less image data, but avoid configuring data capture to higher rates than your hardware can handle, as this could lead to performance degradation.

   • Set the Mime type to image/jpeg.

   • Ensure that the toggle on the right-hand side is set to On.

     

3. Click Save in the top right corner of the page to save your changes.

4. You can check that data is being captured and synced by clicking on the menu icon on the camera configuration pane and selecting View captured data.

   

   If you see images appear here, proceed to the next step. If you do not see images appear after a short time, see the troubleshooting section for further guidance.

For more information see Configure data capture and Configure cloud sync.

The machine learning (ML) model for filtering

The filtered-camera module that you will use in this tutorial supports two modes of filtering:

• Detection, where you use an ML model trained by drawing bounding boxes around distinct objects within captured images, which enables your machine to be able to detect those objects on its own.
• Classification, where you use an ML model trained by tagging images with a class label that best describes it, which enables your machine to classify similar images on its own.

In this tutorial, you will use filtering with a detection model to be able to detect certain objects in the camera feed and filter on the detected objects. To create a detection model, you can train a machine learning (ML) model on images captured by your camera.

Capture images and create a dataset

Position your machine to capture images of interesting objects that you want it to be able to identify on its own. This tutorial will use images of a wooden Viam figure from multiple angles and positions.

Once you have enough images captured and synced to the Viam app, add those images to a new dataset. A dataset allows you to conveniently view, work with, and train an ML model on a collection of images.

1. In the Viam app, navigate to the DATA tab to view your captured images.

2. Select an image you want to use in your ML model.

3. In the Actions pane on the right-hand side, enter a new dataset name under Datasets, then press return.

4. Repeat the steps above to add more images to your dataset, until you have enough to train an ML model on.

   

   For this tutorial, the dataset contains 20 images of the Viam wooden figure, and 5 images that do not contain the Viam figure. This meets the ML model requirements of having at least 80% of the images in the dataset labelled with the target object to detect, while still including a small number of unlabelled images.

Draw bounding boxes around matching objects

Once you have created a dataset containing the images you want to use, label and draw bounding boxes around the objects in those images you want your machine to be able to identify.

1. In the Viam app, navigate to the Datasets subtab of the DATA page and select the dataset you created.

2. Select an image you want to use in your ML model.

3. In the Actions pane on the right-hand side, select the Bounding box mode, enter a new label that describes the object in the image you want to detect, then draw a bounding box over the image on the right-hand side. The example below shows adding the viam-figure label to an image, with a rectangular bounding box drawn around it.

   

4. Repeat this process for the remaining images in your dataset. Once you have created a new label with your first image, you can select that label from the Current label dropdown for all future images in this dataset. Ensure that at least 80% of the images in your dataset are labelled, with the remaining images not containing objects to identify. If you want your machine to be able to identify multiple objects, you can add multiple labels per image as well.

Train a new ML model

Once your dataset is ready, train a new ML model on that dataset.

1. Click the Train model button in the upper-left corner of the Datasets subtab view for your dataset.

2. Select the Object detection model type, select the label or labels you added in the previous step, enter a name or use the suggested name for your new ML model, and click Train model.

   

   Your model will begin training on the images you have tagged, and should be ready after a short time. You can view your model’s training progress from the Models subtab under the DATA page.

   

   Models that are still being trained appear under Training, while models that have completed training and are ready for use appear under Models.

   

For more information, see Train a model.

Use an existing ML model

You can either use an existing model from the registry or upload an ML model trained outside the Viam platform.

Select an existing model from the Registry

The Viam registry hosts trained ML models that users have made public, which you can use to deploy classifiers or detectors for your use case onto your robot instead of training your own. You can see all available ML models in the Viam Registry.

Once you’ve decided on a model to use, continue to Add the ML model service.

Upload an existing ML model

If you want to use a pre-trained model instead of training your own, or already have a model you want to use, you can upload an existing model instead.

For example, you could use a pre-trained machine learning (ML) model from the Viam registry named EfficientDet-COCO. This model can detect a variety of objects, which you can find in the provided labels.txt file.

You can download that model here:

• effdet0.tflite: The TFLite model file containing the trained model.
• labels.txt: The corresponding labels file containing the labels to assign to matching detected objects. You can look through this file to see the full list of trained objects in the model.

To upload and use an existing ML model:

1. Navigate to the Models page in the Viam app and click the Upload model button.

2. Select New model and configure visibility for your model: public models are available to all Viam users while private models are only available to users in your organization.

3. If you haven’t already, you will be prompted to select an organization namespace.

4. Then, in the resulting Upload model screen, enter a name or use the suggested name for your model, select Object detection, and upload both the effdet0.tflite and labels.txt files. Add a brief description of your model, then click Upload model.

   

Your uploaded model is immediately available for use after upload.

If you are designing your own TensorFlow Lite model, see model requirements for guidance on structuring your own model.

For more information, see Upload an existing model.

Configure the ML model and vision services

The filtered-camera module filters image data based on your ML model, and so requires that your machine has an ML model service and vision service configured.

• The ML model service enables your machine to deploy a machine learning (ML) model to be used by other services.
• The vision service uses the deployed model together with your machine’s camera to detect objects defined in the ML model on its own.

Add the ML model service

Add the ML model service to your machine to be able to deploy and update ML models:

1. Navigate to your machine’s CONFIGURE tab in the Viam app.
2. Click the + (Create) button next to your main part in the left-hand menu and select Service, then select the built-in TFLite CPU model.
3. Use the suggested name for your service or give it a name, like my-mlmodel-service, then click Create.
4. On the panel that appears, select the Deploy model on machine toggle, then select your model from the Select model dropdown. If you don’t see your model name appear here, ensure that your model appears under the Models subtab of the DATA tab in the Viam app. If you trained your own model, ensure that the model has finished training and appears under the Models section of that page, and not the Training section.
5. Click Save in the top right corner of the page to save your changes.

For more information, see Create an ML model service.

Add the vision service

Add the vision service to your machine to be able to use the deployed ML model with your camera, and with the filtered-camera module.

1. Navigate to your machine’s CONFIGURE tab in the Viam app.

2. Click the + (Create) button next to your main part in the left-hand menu and select Service. Select Vision, then select the built-in ML model model.

3. Use the suggested name for your service or give it a name, like my-vision-service, then click Create.

4. On the panel that appears, select your ML model from the ML Model dropdown.

5. Click Save in the top right corner of the page to save your changes.

   

For more information, see Configure an mlmodel detector.

Test your ML model with a transform camera

Before filtering your data, you can create a transform camera to test that the ML model is working as expected with your camera. A transform camera will overlay a bounding box on your camera’s live feed when it detects objects that match its ML model. This step is optional, you can skip this step if you want to get right to filtering your data with the filtered-camera module.

Add a transform camera

To add a transform camera to your machine:

1. Navigate to your machine’s CONFIGURE tab in the Viam app.

2. Click the + (Create) button next to your main part in the left-hand menu and select Component. Select Camera, then select the built-in transform model.

3. Give the transform camera a name, like my-transform-camera, then click Create.

4. Find the card of the component you created on the CONFIGURE tab. Click the {} (Switch to advanced) button on the upper right of the card. On the panel that appears, copy and paste the following configuration:

   {
     "pipeline": [
       {
         "attributes": {
           "detector_name": "my-vision-service",
           "confidence_threshold": 0.5
         },
         "type": "detections"
       }
     ],
     "source": "my-webcam"
   }
   

   If you used different names for the vision service or the camera component, update this configuration with those names. You can adjust the confidence_threshold to suit your needs. A value of 0.5 is a relatively loose match, representing 50% confidence. To require that your machine match with more confidence, you can raise this value to something like 0.8, representing 80% confidence.

   

5. Click Save in the top right corner of the page to save your changes.

Test your ML model on a live camera feed

Now that you’ve configured a transform camera, you can see your ML model in action from the CONTROL tab:

1. On your machine’s CONTROL tab in the Viam app, enable the toggle for both your camera component (my-webcam) and your transform camera (my-transform-camera). You can find these toggles under their respective component: click a control pane to expand it if it is collapsed. The screenshot below shows a machine with a configured base component, so the two toggles appear under the base control pane, but you can always find them under their own control pane as well.

2. The camera component displays the raw camera feed, but the transform camera will additionally overlay a bounding box on the same feed if a matching object is detected. Try placing an object your ML model can recognize in front of the camera. The transform camera should draw a bounding box around that object in the live camera feed, and indicate a confidence threshold for the match.

   

3. When satisfied that your ML model is working well, you can disable both cameras. Alternatively, if the transform camera is not matching reliably, you will need to adjust your model. If you trained your model, consider adding and labelling more images in your dataset, or lowering the confidence_threshold of the transform camera. Ideally, you want your ML model to be able to identify objects with a high level of confidence, which usually is dependent on a robust source dataset.

Add and configure the filtered-camera module

With all the prerequisites in place, you are ready to add the filtered-camera module to your machine.

Navigate to your machine’s CONFIGURE tab in the Viam app.

1. Click the + (Create) button next to your main part in the left-hand menu and select Component. Select camera, then select the filtered-camera model. You can also search for filtered-camera directly. Click Add module.

2. Give the modular camera a name, like my-filtered-camera, then click Create.

3. On the panel that appears, copy and paste the following configuration into the attributes field:

   {
     "vision": "my-vision-service",
     "window_seconds": 10,
     "objects": {
       "viam-figure": 0.6
     },
     "camera": "my-webcam"
   }
   

   This example configures the filtered-camera module to require a 60% confidence (0.6) threshold match for the label viam-figure, meaning that it must be at least 60% confident based on your ML model that the image contains the labelled object in order to capture it. Images that do not meet this threshold do not trigger a successful match.

   Additionally, the example configures a window_seconds value of 10 seconds, which controls the duration of a buffer of images captured previous to a successful match. With this configuration, images captured up to 10 seconds before the successful match are included in the capture and sync process.

   If your model uses a different label, provide it here instead of viam-figure:

   • If you trained your own model, you assigned one or more labels when you drew bounding boxes around matching objects in your uploaded images.
   • If you uploaded our provided pre-trained model or are using your own, the labels can be found in the labels.txt file that you uploaded alongside your .tflite model file.

   If you used multiple labels in your ML model, you can specify them on multiple lines like so:

   {
     "vision": "my-vision-service",
     "window_seconds": 10,
     "objects": {
       "apple": 0.6,
       "orange": 0.6,
       "pear": 0.6
     },
     "camera": "my-webcam"
   }
   

4. Next, in the Data capture section on the filtered-camera module configuration pane, click the Add method button to enable data capture for this camera.

   • Set the Method to ReadImage and the Frequency to 0.333. This will capture an image from the camera roughly once every 3 seconds. You can adjust the capture frequency if you want the camera to capture more or less image data, but avoid configuring data capture to higher rates than your hardware can handle, as this could lead to performance degradation.

   • Set the Mime type to image/jpeg.

   • Ensure that the toggle on the right-hand side is set to On.

5. Click Save in the top right corner of the page to save your changes.

   

Test the filtered-camera module

With everything configured, the filtered-camera module is now ready to selectively capture only images that meet your ML model criteria.

1. Navigate to your machine’s CONFIGURE tab in the Viam app.

2. Find your camera component (my-webcam) and ensure that Data capture is disabled.

   

   We want to use the filtered-camera module to capture data instead, so that it can selectively capture image based on your ML model.

3. Position your machine’s camera so that no detectible objects are visible, then ensure that no new images are being synced to the Viam app by watching the DATA tab in the Viam app.

4. Then place an object that is part of your trained ML model within view of your camera, and watch images of that object begin to appear under the DATA tab automatically.

5. Remove the object from view of the camera, and images should stop being captured and synced.

Congratulations, you now have a smart filtered camera on your machine, and can fine tune the kind of image capture it performs.

Next steps

In this tutorial, you learned how to use the filtered-camera module to control the volume of data your machine writes and syncs, by using an ML model to detect objects in your camera feed and selectively capture and upload only those images that are matched by your model.

You could expand on this in many ways! For example, you can:

• Train an ML model on familiar faces, so that your machine can capture and upload images of any new faces it encounters, but ignore familiar faces entirely.
• Train an ML model on various common forms of delivery packaging, so that your machine can send you an image of a new delivery, but not clutter your inbox with images of other things, such as cars driving by.
• Train an ML model on a variety of domestic farm animals, so that your machine can alert you if a different kind of animal is detected in the vicinity, without capturing images of every animal.

You can also refine your existing ML model by adding and labelling new images that help the ML model better identify matching objects, and then upload the new version of your model using the ML model service.

If you trained a new model as part of this tutorial, try using the provided pre-trained model files instead, and then aiming your machine’s camera at objects listed in the labels.txt file to see how accurately it is able to detect those objects.

In this tutorial, you learned how to use the filtered-camera module with object detection, but you can also use it to perform object classification. See the filtered-camera module repository for the attributes to use to configure object classification.

Troubleshooting

No images appearing in the Viam app

Ensure that the camera component (my-webcam in this tutorial) has data capture enabled, and that the data management service (my-data-management in this tutorial) has both Capturing and Syncing enabled. Your machine can only sync images to the Viam app when it is online; if it is not presently online, it will sync captured images when next it connects to the internet.

Cannot train ML model

In order to train an ML model on images, you must label at least 10 images, and your dataset must include at least 80% labeled images. If you get an error on the Viam app models page similar to too few images when you go to train a model, try adding more images to your dataset and labelling them until you have at least 10 labelled images, and a dataset where at least 80% of the images it contains are labelled.