Data Capture and Sync
The data management service captures data from one or more resources locally, and syncs it to cloud storage when a connection to the cloud is available. You can configure which data you want to capture, as well as the capture rate and the sync frequency.
The data is captured locally on the machine’s storage and, by default, stored in the ~/.viam/capture directory.
If a machine restarts for any reason, capture automatically resumes and any data from already stored but not yet synced is synced.
The service can capture data from multiple resources at the same or different frequencies. The service does not impose a lower or upper limit on the frequency of data collection. However, in practice, your hardware may impose limits on the frequency of data collection. Avoid configuring data capture to higher rates than your hardware can handle, as this could lead to performance degradation.
Data capture is frequently used with cloud sync. However, if you want to manage your machine’s captured data yourself, you can enable only data capture without cloud sync.
The data is captured in the ESP32’s flash memory until it is uploaded to the Viam cloud.
If the machine restarts before all data is synced, all unsynced data captured since the last sync point is lost.
The service can capture data from multiple resources at the same or different frequencies. The service does not impose a lower or upper limit on the frequency of data collection. However, in practice, high frequency data collection (> 100Hz) requires special considerations on the ESP32.
Supported components and services
The following components and services support data capture, for the following methods:
| Type | Method |
|---|---|
| Arm | EndPosition, JointPositions |
| Board | Analogs, Gpios |
| Camera | GetImages, ReadImage, NextPointCloud |
| Encoder | TicksCount |
| Gantry | Lengths, Position |
| Motor | Position, IsPowered |
| Movement sensor | AngularVelocity, CompassHeading, LinearAcceleration, LinearVelocity, Orientation, Position |
| Sensor | Readings |
| Servo | Position |
| Vision service | CaptureAllFromCamera |
Security
The data management service uses gRPC calls to send and receive data, so your data is encrypted while in flight. When data is stored in the cloud, it is encrypted at rest by the cloud storage provider.
Data Integrity
Viam’s data management service is designed to safeguard against data loss, data duplication and otherwise compromised data.
If the internet becomes unavailable or the machine needs to restart during the sync process, the sync is interrupted. If the sync process is interrupted, the service will retry uploading the data at exponentially increasing intervals until the interval in between tries is at one hour, at which point the service retries the sync every hour. When the connection is restored and sync resumes, the service continues sync where it left off without duplicating data. If the interruption happens mid-file, sync resumes from the beginning of that file.
To avoid syncing files that are still being written to, the data management service only syncs files that haven’t been modified in the previous 10 seconds.
Storage
Data that is successfully synced to the cloud is automatically deleted from local storage.
When a machine loses its internet connection, it cannot resume cloud sync until it can reach the Viam cloud again.
To ensure that the machine can store all data captured while it has no connection, you need to provide enough local data storage.
Warning
If your machine’s disk fills up beyond a certain threshold, the data management service will delete captured data to free up additional space and maintain a working machine.
Configure data capture and sync
To capture data from one or more machines, you must first add the data management service: If you prefer to write raw JSON, here is an example: The following attributes are available for the data management service: With Now the data management service is enabled.
However, no data is being captured until you configure capture on one or more resources. You can capture data for any resource that supports it, including resources on remote parts. Once you have added the data capture service, you can specify the data you want to capture at a resource level. For each resource you can capture data for, there is a Data capture section in its panel.
Select a Method and specify a capture Frequency in hertz.
You can add multiple methods with different capture frequencies.
Some methods will prompt you to add additional parameters. The available methods, and corresponding additional parameters, will depend on the component or service type.
For example, a camera has the options Avoid configuring data capture to higher rates than your hardware can handle, as this leads to performance degradation. The following attributes are available for data capture configuration: Click the Save button in the top right corner of the page. Viam supports data capture from resources on remote parts.
For example, if you use a part that does not have a Linux operating system or that does not have enough storage or processing power, you can still process and capture the data from that part’s resources by adding it as a remote part. Currently, you can only configure data capture from remote resources in your JSON configuration.
To add them to your JSON configuration you must explicitly add the remote resource’s To view all the captured data you have access to, go to the DATA tab where you can also filter by location, type of data, and more. Or, you can navigate to the DATA tab from an individual resource, part, or machine’s configuration card to automatically apply relevant filters: To view captured data for a component or service, click on the menu in the top right of its card and select View captured data. To view captured data for a machine part, click on the menu in the top right of its card or the menu in the machine resources list in the Builder menu and select View captured data. To view captured data for a machine (including data from all parts of a multi-part machine), click on the data icon next to the Save button on the top right of the app.
Step 1: Configure the data management service
data management type, then either use the suggested name or specify a name for your data management service.
{
"components": [],
"services": [
{
"name": "my-data-manager",
"namespace": "rdk",
"type": "data_manager",
"attributes": {
"sync_interval_mins": 1,
"capture_dir": "",
"tags": [],
"capture_disabled": false,
"sync_disabled": true,
"delete_every_nth_when_disk_full": 5,
"maximum_num_sync_threads": 250
}
}
]
}
{
"components": [],
"services": [
{
"name": "my-data-manager",
"namespace": "rdk",
"type": "data_manager",
"attributes": {
"capture_dir": "",
"tags": [],
"additional_sync_paths": [],
"sync_interval_mins": 3
}
}
]
}
Name Type Required? Description capture_disabledbool Optional Toggle data capture on or off for the entire machine part. Note that even if capture is on for the whole part, but is not on for any individual components (see Step 2), data is not being captured.
Default: falsecapture_dirstring Optional Path to the directory on your machine where you want to store captured data. If you change the directory for data capture, only new data is stored in the new directory. Existing data remains in the directory where it was stored.
Default: ~/.viam/capturetagsarray of strings Optional Tags to apply to all images captured by this machine part. May include alphanumeric characters, underscores, and dashes. sync_disabledbool Optional Toggle cloud sync on or off for the entire machine part.
Default: falseadditional_sync_pathsstring array Optional Paths to any other directories on your machine from which you want to sync data to the cloud. Once data is synced from a directory, it is automatically deleted from your machine. sync_interval_minsfloat Optional Time interval in minutes between syncing to the cloud. Viam does not impose a minimum or maximum on the frequency of data syncing. However, in practice, your hardware or network speed may impose limits on the frequency of data syncing.
Default: 0.1, meaning once every 6 seconds.delete_every_nth_when_disk_fullint Optional How many files to delete when local storage meets the fullness criteria. The data management service will delete every Nth file that has been captured upon reaching this threshold. Use JSON mode to configure this attribute.
Default: 5, meaning that every fifth captured file will be deleted.maximum_num_sync_threadsint Optional Max number of CPU threads to use for syncing data to the Viam cloud.
Default: runtime.NumCPU/2 so half the number of logical CPUs available to viam-serverInfo
viam-micro-server, the capture_dir, tags, and additional_sync_paths, and delete_every_nth_when_disk_full attributes are ignored and should not be configured.Name Type Required? Description capture_disabledbool Optional Toggle data capture on or off for the entire machine part. Note that even if capture is on for the whole part, but is not on for any individual components (see Step 2), data is not being captured.
Default: falsesync_disabledbool Optional Toggle cloud sync on or off for the entire machine part.
Default: falsesync_interval_minutesfloat Optional Time interval in minutes between syncing to the cloud. Viam does not impose a minimum or maximum on the frequency of data syncing. However, in practice, your hardware or network speed may impose limits on the frequency of data syncing.
Default: 0.1, meaning once every 6 seconds.cache_size_kbfloat Optional The maximum amount of storage bytes (in kilobytes) allocated to a data collector.
Default: 1 KB.
Step 2: Configure data capture for your resources
ReadImage and NextPointCloud.
Keep in mind that some models do not support all options, for example webcams do not capture point clouds, and choose the method accordingly.
Caution
Name Type Required? Description capture_frequency_hzfloat Required Frequency in hertz at which to capture data. For example, to capture a reading every 2 seconds, enter 0.5.methodstring Required Depends on the type of component or service. See Supported components and services. additional_paramsdepends depends Varies based on the method. For example, ReadImage requires a MIME type.type, model, name, and additional_params to the data_manager service configuration in the remotes configuration:Key Description typeThe type tells your machine what the resource is. For example, a board. modelThe model is a colon-delimited-triplet that specifies the namespace, the type of the part, and the part itself. nameThe name specifies the fully qualified name of the part. additional_paramsThe additional parameters specify the data sources when you are using a board.
Step 4: (Optional) View captured data
Sync data conditionally
You can use a module to sync data only when a certain logic condition is met, instead of at a regular time interval. For example, if you rely on mobile data but have intermittent WiFi connection in certain locations or at certain times of the day, you may want to trigger sync to only occur when these conditions are met. To set up triggers for syncing see:
API
The data management service API supports the following methods:
| Method Name | Description | viam-micro-server Support |
|---|---|---|
Sync | Sync data stored on the machine to the cloud. | |
Reconfigure | Reconfigure this resource. | |
DoCommand | Execute model-specific commands that are not otherwise defined by the service API. | |
Close | Safely shut down the resource and prevent further use. |
The data client API supports a separate set of methods that allow you to upload and export data to and from the Viam app. For information about that API, see Data Client API.
Troubleshooting
Images are dim on start up
If you are capturing camera data, it can happen that the camera captures and syncs miscolored or dark images upon start up. Wait for a few seconds and you should see correctly colored images.
Next steps
If you have synced data, such as sensor readings, you can query that data with SQL or MQL from the Viam app or a MQL-compatible client. If you have synced images, you can use those images to train machine learning models within the Viam app.
Or check out the following guides and tutorials:
Was this page helpful?
Glad to hear it! If you have any other feedback please let us know:
We're sorry about that. To help us improve, please tell us what we can do better:
Thank you!