Configure Data Capture

The data management service captures data from one or more resources.

Get started with a quickstart guide or keep reading for more details.

Collect data (3 min)

Use Viam to gather images or sensor data from your machine.

You can change the frequency of data capture at any time for individual resources. If you use fragments, you can change the frequency of data capture in real time for some or all machines in a fleet at the resource or machine level.

For example, consider a tomato picking robot with a 3D camera and an arm. When you configure the robot, you may set the camera to capture point cloud data at a frequency of 30Hz. For the arm, you may want to capture joint positions at 1Hz. If your requirements change and you want to capture data from both components at 10Hz, you can change the configurations at any time by changing the number.

Data capture is frequently used with Cloud Sync. However, if you want to manage your machine’s captured data yourself, you can enable only data capture without cloud sync.

Add the data management service

To capture data from one or more machines, you must first add the data management service:

1. Navigate to the CONFIGURE tab of your machine’s page in the Viam app.
2. Click the + icon next to your machine part in the left-hand menu and select Service.
3. Select the data management type, then either use the suggested name or specify a name for your data management service, for example data-manager.
4. Click Create.
5. On the panel that appears, you can manage the capturing and syncing functions.
   • viam-server
   • viam-micro-server

   Specify the Directory, the sync Interval and any Tags to apply to captured data.

   If the sync Interval or the Directory is not specified, the data management service captures data at the default frequency every 0.1 minutes (after every 6 second interval) in the default ~/.viam/capture directory.

   Info

   If you change the directory for data capture only new data is stored in the new directory. Existing data remains in the directory where it was stored.

   Specify the sync Interval.

   Info

   With viam-micro-server, the capture_dir, tags, and additional_sync_paths attributes are ignored and should not be configured.

{
  "components": [],
  "services": [
    {
      "name": "your-data-manager",
      "type": "data_manager",
      "attributes": {
        "sync_interval_mins": 1,
        "capture_dir": "",
        "tags": [],
        "capture_disabled": false,
        "sync_disabled": true,
        "delete_every_nth_when_disk_full": 5
      }
    }
  ]
}

{
  "components": [],
  "services": [
    {
      "attributes": {
        "capture_dir": "",
        "tags": [],
        "additional_sync_paths": [],
        "sync_interval_mins": 3
      },
      "name": "my-data-manager",
      "namespace": "rdk",
      "type": "data_manager"
    }
  ]
}

6. Click the Save button in the top right corner of the page.

When your machine is capturing data, there is a Capturing indication in your machine’s status bar.

Configure data capture for individual resources

Once you have added the data capture service, you can specify the data you want to capture at a resource level.

Supported components and services

The following components and services support data capture:

To add data capture for a component or service, navigate to the CONFIGURE tab of your machine’s page in the Viam app.

For each resource you can capture data for, there is a Data capture section in its panel. Click Add Method and then select the Method type and the capture Frequency.

Click the Save button in the top right corner of the page.

Now, using viam-server, your data will be saved locally on your machine to the directory specified in the data management service. If you are using viam-micro-server, data will be captured at the configured capture frequency and saved in flash memory and synced to Viam app at the selected interval.

For example, a camera has the options ReadImage and NextPointCloud and a motor has the options Position and IsPowered.

{
  "services": [
    ...
    ,
    {
      "name": "data_manager",
      "type": "data_manager",
      "attributes": {
        "sync_interval_mins": 5,
        "capture_dir": "",
        "sync_disabled": false,
        "tags": []
      }
    }
  ],
  "remotes": [
    {
        ...
    }
  ],
  "components": [
        ...
    ,
    {
      "service_configs": [
        {
          "type": "data_manager",
          "attributes": {
            "capture_methods": [
              {
                "capture_frequency_hz": 0.333,
                "disabled": false,
                "method": "ReadImage",
                "additional_params": {
                  "reader_name": "cam1",
                  "mime_type": "image/jpeg"
                }
              }
            ]
          }
        }
      ],
      "model": "webcam",
      "name": "cam",
      "type": "camera",
      "attributes": {
        "video_path": "video0"
      },
      "depends_on": [
        "local"
      ]
    },
    ...
  ]
}

{
  "services": [
    {
      "attributes": {
        "capture_dir": "",
        "tags": [],
        "additional_sync_paths": [],
        "sync_interval_mins": 3
      },
      "name": "dm",
      "namespace": "rdk",
      "type": "data_manager"
    }
  ],
  "components": [
    {
      "type": "sensor",
      "model": "tmp36",
      "attributes": {
        "analog_reader": "temp",
        "num_readings": 15
      },
      "depends_on": [],
      "service_configs": [
        {
          "attributes": {
            "capture_methods": [
              {
                "capture_frequency_hz": 0.2,
                "additional_params": {},
                "method": "Readings"
              }
            ]
          },
          "type": "data_manager"
        }
      ],
      "name": "tmp36",
      "namespace": "rdk"
    },
    {
      "type": "sensor",
      "model": "wifi-rssi",
      "attributes": {},
      "service_configs": [
        {
          "type": "data_manager",
          "attributes": {
            "capture_methods": [
              {
                "additional_params": {},
                "method": "Readings",
                "capture_frequency_hz": 0.1
              }
            ]
          }
        }
      ],
      "name": "my-wifi-sensor",
      "namespace": "rdk"
    }
  ]
}

{
  "components": [
    {
      "name": "camera-1",
      "namespace": "rdk",
      "type": "camera",
      "model": "webcam",
      "attributes": {}
    }
  ],
  "services": [
    {
      "name": "vision-1",
      "namespace": "rdk",
      "type": "vision",
      "model": "mlmodel",
      "attributes": {},
      "service_configs": [
        {
          "type": "data_manager",
          "attributes": {
            "capture_methods": [
              {
                "method": "CaptureAllFromCamera",
                "capture_frequency_hz": 1,
                "additional_params": {
                  "mime_type": "image/jpeg",
                  "camera_name": "camera-1",
                  "min_confidence_score": "0.7"
                }
              }
            ]
          }
        }
      ]
    },
    {
      "name": "data_manager-1",
      "namespace": "rdk",
      "type": "data_manager",
      "attributes": {
        "sync_interval_mins": 0.1,
        "capture_dir": "",
        "tags": [],
        "additional_sync_paths": []
      }
    },
    {
      "name": "mlmodel-1",
      "namespace": "rdk",
      "type": "mlmodel",
      "model": "tflite_cpu",
      "attributes": {}
    }
  ]
}

You may capture data from one or more resource methods:

• To enable or disable data capture for a configured resource or method, use the on/off toggle on the resource’s configuration pane in the Viam app.
• To change the frequency of data capture for a method, enter the number of measurements you wish to capture per second in the frequency field on the resource’s configuration pane in the Viam app.

After adding configuration for the methods, click the Save button in the top right corner of the page.

If you want to remove a capture method from the configuration, click the delete icon.

Configure data capture for remote parts

Viam supports data capture from resources on remote parts. For example, if you use a part that does not have a Linux operating system or that does not have enough storage or processing power, you can still process and capture the data from that part’s resources by adding it as a remote part.

Currently, you can only configure data capture from remote resources in your JSON configuration. To add them to your JSON configuration you must explicitly add the remote resource’s type, model, name, and additional_params to the data_manager service configuration in the remotes configuration:

{
  "components": [
    {
      "name": "my-esp32",
      "model": "esp32",
      "type": "board",
      "namespace": "rdk",
      "attributes": {
        "pins": [27],
        "analogs": [
          {
            "pin": "34",
            "name": "A1"
          },
          {
            "pin": "35",
            "name": "A2"
          }
        ]
      },
      "service_configs": [
        {
          "type": "data_manager",
          "attributes": {
            "capture_methods": [
              {
                "method": "Analogs",
                "additional_params": {
                  "reader_name": "A1"
                },
                "capture_frequency_hz": 10
              },
              {
                "method": "Analogs",
                "additional_params": {
                  "reader_name": "A2"
                },
                "capture_frequency_hz": 10
              }
            ]
          }
        }
      ]
    }
  ]
}

{
  "services": [
    {
      "name": "data_manager",
      "type": "data_manager",
      "attributes": {
        "capture_dir": "",
        "sync_disabled": true,
        "sync_interval_mins": 5,
        "tags": ["tag1", "tag2"]
      },
      "name": "data_manager",
      "type": "data_manager"
    }
  ],
  "components": [],
  "remotes": [
    {
      "name": "esp-home",
      "address": "esp-home-main.33vvxnbbw9.viam.cloud:80",
      "service_configs": [
        {
          "type": "data_manager",
          "attributes": {
            "capture_methods": [
             // Captures data from two analog readers (A1 and A2)
             {
                "method": "Analogs",
                "capture_frequency_hz": 1,
                "name": "rdk:component:board/my-esp32",
                "additional_params": { "reader_name": "A1" },
                "disabled": false
             },
             {
                "method": "Analogs",
                "capture_frequency_hz": 1,
                "name": "rdk:component:board/my-esp32",
                "additional_params": { "reader_name": "A2" },
                "disabled": false
              },
              // Captures data from pin 27 of the board's GPIO
              {
                "method": "Gpios",
                "capture_frequency_hz": 1,
                "name": "rdk:component:board/my-esp32",
                "additional_params": {
                  "pin_name": “27”
                },
                "disabled": false
              }
            ]
          }
        }
      ],
      "secret": "REDACTED"
    }
  ]
}

{
  "services": [
    {
      "attributes": {
        "capture_dir": "",
        "sync_disabled": true,
        "sync_interval_mins": 5,
        "tags": []
      },
      "name": "data_manager",
      "type": "data_manager"
    }
  ],
  "components": [],
  "remotes": [
    {
      "name": "pi-test-main",
      "address": "pi-test-main.vw3iu72d8n.viam.cloud",
      "service_configs": [
        {
          "type": "data_manager",
          "attributes": {
            "capture_methods": [
              {
                "capture_frequency_hz": 1,
                "name": "rdk:component:camera/cam",
                "disabled": false,
                "method": "ReadImage",
                "additional_params": {
                  "mime_type": "image/jpeg",
                  "reader_name": "cam1"
                }
              }
            ]
          }
        }
      ],
      "secret": "REDACTED"
    }
  ]
}

Capture data selectively with filtering

See the Use filtering to collect and sync only certain images guide.

Automatic data deletion

If cloud sync is enabled, the data management service deletes captured data once it has successfully synced to the cloud.

With viam-server, the data management service will also automatically delete local data in the event your machine’s local storage fills up. Local data is automatically deleted when all of the following conditions are met:

• Data capture is enabled on the data manager service
• Local disk usage percentage is greater than or equal to 90%
• The Viam capture directory is at least 50% of the current local disk usage

If local disk usage is greater than or equal to 90%, but the Viam capture directory is not at least 50% of that usage, a warning log message will be emitted instead and no action will be taken.

Automatic file deletion only applies to files in the specified Viam capture directory, which is set to ~/.viam/capture by default. Data outside of this directory is not touched by automatic data deletion.

If your machine captures a large amount of data, or frequently goes offline for long periods of time while capturing data, consider moving the Viam capture directory to a larger, dedicated storage device on your machine if available. You can change the capture directory using the capture_dir attribute.

You can also control how local data is deleted if your machine’s local storage becomes full, using the delete_every_nth_when_disk_full attribute:

• The delete_every_nth_when_disk_full attribute controls how many files to delete when local storage meets the above fullness criteria. The data management service will delete every Nth file that has been captured upon reaching this threshold. The default value is 5, meaning that every fifth captured file will be deleted. This value should be suitable for most cases.

View captured data

To view captured data for a machine, click on the data icon next to the Save button on the top right of the app.

To view captured data for a machine part, click on the menu in the top right of its card or the menu in the machine resources list in the Builder menu and select View captured data.

To view captured data for a component or service, click on the menu in the top right of its card and select View captured data.

To view all the captured data you have access to, go to the DATA tab.

Troubleshooting

Images are dim on start up

If you are capturing camera data, it can happen that the camera captures and syncs miscolored or dark images upon start up. Wait for a few seconds and you should see correctly colored images.

Next steps

To sync your captured data with the cloud, configure cloud sync.

If you have synced data, such as sensor readings, you can query that data with SQL or MQL from the Viam app or a MQL-compatible client. If you have synced images, you can use those images to train machine learning models within the Viam app.

For comprehensive guides on using data capture and synchronization together with the ML model service, see:

Capture and sync image data

Capture images from a camera on your machine and selectively sync images to the cloud with filtering.

Train computer vision models

Use your image data to create and label a dataset and train a computer vision ML model.

To capture performance metric data for machines, filter, or visualize data, see:

Monitor Performance

Capture and sync data about your machines' performance.

Capture and sync image data

Capture images from a camera on your machine and selectively sync images to the cloud with filtering.

Air Quality Fleet

Configure a fleet of machines to capture air quality sensor data across different locations.