An Introduction to Using Viam's Data Management Service

In this tutorial, we will cover how to use Data Management, including capturing camera data, syncing that data from your robot to the cloud, viewing and filtering synced data, and downloading it to your computer.

Data Management

One key feature of Viam is Data Management, which helps you manage data on your robot every step of the way, from capturing component data on your robot, to managing your data securely in the cloud.

Viam’s Data Management Service has two distinct parts: data capture and cloud sync.

Data capture allows you to capture data from specific components on your robot running Viam. You can choose the components, corresponding methods, and the frequency of the data capture all within the Viam app.

Cloud sync runs in the background and uploads your robot’s captured data to Viam’s cloud at a defined frequency. Cloud sync is designed to be resilient and to preserve your data even during a network outage or if your robot has low network bandwidth. It also manages syncing data from your robot to the Viam app and deletes the captured data from your robot after a successful sync. Plus, your data is encrypted in transit and at rest in the cloud, so it is always protected.

Data capture is frequently used with data sync. However, if you want to manage your robot’s captured data yourself, it’s possible to enable data capture for components on your robot, but disable data sync.

Requirements

1. A rented or owned robot

   Viam’s Data Management features will work on any robot running Viam. To follow along with this tutorial, you can use one of your own robots, or you can rent a rover and use it to try out Viam’s Data Manager without having to configure your own hardware.

   To learn more about the Try Viam experience, see Try Viam.

   Note

   If you are using your own robot, be sure that you have viam-server installed on your robot. The data management setup process will be mostly the same, but you will need to substitute your robot’s components.

2. Go

   To use the data export feature, you need to have the Go binary installed on your local development computer. We suggest that you install this before your reservation starts to maximize your time with your rover.

Add the Data Management Service

First, you need to add and configure the Data Management Service to capture data and store it at a specified location. To enable the data capture on your robot, do the following:

1. On your robot’s Config page, navigate to the Services tab.

2. At the bottom of the page, create a service. Choose Data Management as the type and specify viam-data-manager as the name for your instance of the Data Management Service. This service syncs data from your robot to the Viam app in the cloud.

3. Then click Create Service.

4. On the panel that appears, you can manage the capturing and syncing functions individually. The Data Management Service captures data every 0.1 minutes in the ~/.viam/capture directory by default.

   You can leave the default settings as they are. Click Save Config at the bottom of the window.

For more detailed information see Add the Data Management Service.

Configure data capture for a component

With the Data Management service added, you can now configure data capture for specific components on your robot running Viam. You can choose the components, corresponding methods, and the frequency of the data capture all within the Viam app.

To enable image data capture for a camera component, follow these steps:

1. Navigate to the Components tab on your robot’s Config page.

2. Scroll down to the camera component. If you are using a Viam Rover, the camera is named cam. The camera component has a section labeled Data Capture Configuration.

3. Click Add Method to enable data capture for this camera.

   • Set the Type to “ReadImage” and the Frequency to 0.333. This will capture an image from the camera roughly once every 3 seconds. Feel free to adjust the frequency if you want the camera to capture more or less image data.

     Caution

     Avoid configuring data capture to higher rates than your hardware can handle, as this leads to performance degradation.

   • Select the MIME type you want to capture. Select image/jpeg data for this tutorial.

4. Click Save Config at the bottom of the window.

Now your image data will be saved locally on your robot in ~/.viam/capture, or whatever directory you configured the captured data from your Data Management Service to be saved to.

For more detailed information see Configure Data Capture and Configure Cloud Sync.

View and filter captured data

Now that you have data capture enabled for a camera on your Viam Rover, you can see the images from your robot in the Viam app.

• Head over to the DATA page.

• Select Rover Rental as the Location, and click SEARCH. If you have data capture set up correctly, you will see the captured images from your rover’s camera.

For more detailed information see View and Filter Data.

Export captured data

You’ve successfully saved data from your robot in the cloud. Now, let’s export that image data from the Viam app onto your local computer.

To export data from Viam:

1. First, install the Viam CLI and authenticate:

   go install go.viam.com/rdk/cli/viam@latest
   viam auth
   

2. Head back to the DATA page in the Viam app.

3. Below the SEARCH button in the Filtering panel, click Copy Export Command to copy the export command to the clipboard.

   

4. Run the copied command in a terminal:

   viam data export --org_ids=<org_id> --data_type=binary --mime_types=<mime_types> --destination=.
   

   This command uses the Viam CLI to download the data locally onto your computer based on the search criteria you select in the Viam app.

   Once the command has finished running and downloading the data, you can view and use the data locally in a new directory named data.

   Since images are downloaded in parallel, some may be out of order. Sort your folder by filename to see them in chronological order.

For more detailed information see Export Data.

Next steps

In this tutorial, you have learned how to use Data Management to capture camera data, sync that data from your robot to the cloud, and view and export the data.

You can now set up data management for your robots to track your robot’s usage over time, build dashboards to visualize the data from your robot, or train machine learning models to detect hand gestures or objects.

You can also ask questions in the Community Discord and we will be happy to help.