Next
Filter data
You can use the data management service to capture data from supported components and services, then sync it to the cloud. You can also sync data from arbitrary folders on your machine.
Data capture and sync in Viam involves two key pieces:
The data management service writes data from your configured Viam resources to local storage on your edge device and syncs data from the edge device to the cloud:
For more information, see How sync works.
From your machine’s CONFIGURE tab in the Viam app, add the data management
service.
On the panel that appears, configure data capture and sync attributes as applicable.
To both capture data and sync it to the cloud, keep both Capturing and Syncing switched on.
Click the Save button in the top right corner of the page to save your config.
{
"components": [],
"services": [
{
"name": "my-data-manager",
"namespace": "rdk",
"type": "data_manager",
"attributes": {
"sync_interval_mins": 1,
"capture_dir": "",
"tags": [],
"capture_disabled": false,
"sync_disabled": true,
"delete_data_on_part_deletion": true,
"delete_every_nth_when_disk_full": 5,
"maximum_num_sync_threads": 250
}
}
]
}
{
"components": [],
"services": [
{
"name": "my-data-manager",
"namespace": "rdk",
"type": "data_manager",
"attributes": {
"capture_dir": "",
"tags": [],
"additional_sync_paths": [],
"sync_interval_mins": 3
}
}
]
}
The following attributes are available for the data management service:
You can capture data for any resource that supports it, including resources on remote parts.
Configure data capture for individual resources in their configuration by:
For each resource you can capture data for, there is a Data capture section in its panel.
Select a Method and specify a capture Frequency in hertz, for example to 0.1
to capture an image every 10 seconds.
You can add multiple methods with different capture frequencies.
Some methods will prompt you to add additional parameters.
The available methods, and corresponding additional parameters, will depend on the component or service type.
For example, a camera has the options ReadImage
and NextPointCloud
.
Keep in mind that some models do not support all options, for example webcams do not capture point clouds, and choose the method accordingly.
Avoid configuring data capture to higher rates than your hardware can handle, as this leads to performance degradation.
This example configuration captures data from the ReadImage
method of a camera:
{
"services": [
...
,
{
"name": "data_manager",
"type": "data_manager",
"attributes": {
"sync_interval_mins": 5,
"capture_dir": "",
"sync_disabled": false,
"tags": []
}
}
],
"remotes": [
{
...
}
],
"components": [
...
,
{
"service_configs": [
{
"type": "data_manager",
"attributes": {
"capture_methods": [
{
"capture_frequency_hz": 0.333,
"disabled": false,
"method": "ReadImage",
"additional_params": {
"reader_name": "cam1",
"mime_type": "image/jpeg"
}
}
],
"retention_policy": {
"days": 5
}
}
}
],
"model": "webcam",
"name": "cam",
"type": "camera",
"attributes": {
"video_path": "video0"
},
"depends_on": [
"local"
]
},
...
]
}
This example configuration captures data from the GetReadings
method of a temperature sensor and wifi signal sensor:
{
"services": [
{
"attributes": {
"capture_dir": "",
"tags": [],
"additional_sync_paths": [],
"sync_interval_mins": 3
},
"name": "dm",
"namespace": "rdk",
"type": "data_manager"
}
],
"components": [
{
"type": "sensor",
"model": "tmp36",
"attributes": {
"analog_reader": "temp",
"num_readings": 15
},
"depends_on": [],
"service_configs": [
{
"attributes": {
"capture_methods": [
{
"capture_frequency_hz": 0.2,
"cache_size_kb": 10,
"additional_params": {},
"method": "Readings"
}
]
},
"type": "data_manager"
}
],
"name": "tmp36",
"namespace": "rdk"
},
{
"type": "sensor",
"model": "wifi-rssi",
"attributes": {},
"service_configs": [
{
"type": "data_manager",
"attributes": {
"capture_methods": [
{
"additional_params": {},
"method": "Readings",
"capture_frequency_hz": 0.1,
"cache_size_kb": 10
}
]
}
}
],
"name": "my-wifi-sensor",
"namespace": "rdk"
}
]
}
Example for a vision service:
This example configuration captures data from the CaptureAllFromCamera
method of the vision service:
{
"components": [
{
"name": "camera-1",
"namespace": "rdk",
"type": "camera",
"model": "webcam",
"attributes": {}
}
],
"services": [
{
"name": "vision-1",
"namespace": "rdk",
"type": "vision",
"model": "mlmodel",
"attributes": {},
"service_configs": [
{
"type": "data_manager",
"attributes": {
"capture_methods": [
{
"method": "CaptureAllFromCamera",
"capture_frequency_hz": 1,
"additional_params": {
"mime_type": "image/jpeg",
"camera_name": "camera-1",
"min_confidence_score": "0.7"
}
}
]
}
}
]
},
{
"name": "data_manager-1",
"namespace": "rdk",
"type": "data_manager",
"attributes": {
"sync_interval_mins": 0.1,
"capture_dir": "",
"tags": [],
"additional_sync_paths": []
}
},
{
"name": "mlmodel-1",
"namespace": "rdk",
"type": "mlmodel",
"model": "viam:mlmodel-tflite:tflite_cpu",
"attributes": {}
}
],
"modules": [
{
"type": "registry",
"name": "viam_tflite_cpu",
"module_id": "viam:tflite_cpu",
"version": "0.0.3"
}
]
}
Viam supports data capture from resources on remote parts.
For example, if you use a part that does not have a Linux operating system or does not have enough storage or processing power to run viam-server
, you can still process and capture the data from that part’s resources by adding it as a remote part.
Currently, you can only configure data capture from remote resources in your JSON configuration.
To add them to your JSON configuration you must explicitly add the remote resource’s type
, model
, name
, and additional_params
to the data_manager
service configuration in the remotes
configuration:
Key | Description |
---|---|
type | The type tells your machine what the resource is. For example, a board. |
model | The model is a colon-delimited-triplet that specifies the namespace, the type of the part, and the part itself. |
name | The name specifies the fully qualified name of the part. |
additional_params | The additional parameters specify the data sources when you are using a board. |
The following attributes are available for data capture configuration:
Click the Save button in the top right corner of the page to save your config.
If cloud sync is enabled, the data management service deletes captured data once it has successfully synced to the cloud.
If your robot is offline and can’t sync and your machine’s disk fills up beyond a certain threshold, the data management service will delete captured data to free up additional space and maintain a working machine.
The following components and services support data capture and cloud sync:
Type | Method |
---|---|
Arm | EndPosition , JointPositions |
Board | Analogs , Gpios |
Camera | GetImages , ReadImage , NextPointCloud |
Encoder | TicksCount |
Gantry | Lengths , Position |
Motor | Position , IsPowered |
Movement sensor | AngularVelocity , CompassHeading , LinearAcceleration , LinearVelocity , Orientation , Position |
Sensor | Readings |
Servo | Position |
Vision service | CaptureAllFromCamera |
To view all the captured data you have access to, go to the DATA tab where you can filter by location, type of data, and more.
You can also access data from a resource or machine part menu.
If you don’t need to capture data, for instance in a test scenario, you can turn off data capture to reduce unnecessary storage. Alternatively, see advanced data capture and sync configurations for other ways to control data usage, such as conditional sync or retention policies.
To turn off data capture for a specific resource’s capture method (for example, a camera component capturing through the GetImage
capture method) navigate to the Data capture section of your resource’s configuration card and toggle the configured capture method’s switch to Off.
You can also globally turn off data capture on the data_manager
service configuration card by toggling the Capturing switch to Off.
To turn off data sync, navigate to the data_manager
service configuration card and toggle the Syncing switch to Off.
Click the Save button in the top right corner of the page to save your config.
You can configure direct capture of tabular data to a MongoDB instance alongside disk storage on your edge device. This can be useful for powering real-time dashboards before data is synced from the edge to the cloud.
Configure using the mongo_capture_config
attributes in your data manager service.
Here is a sample configuration that will capture fake sensor readings both to the configured MongoDB URI as well as to the ~/.viam/capture
directory on disk:
When mongo_capture_config.uri
is configured, data capture will attempt to connect to the configured MongoDB server and write captured tabular data to the configured mongo_capture_config.database
and mongo_capture_config.collection
(or their defaults if unconfigured) after enqueuing that data to be written to disk.
If writes to MongoDB fail for any reason, data capture will log an error for each failed write and continue capturing.
Failing to write to MongoDB doesn’t affect capturing and syncing data to cloud storage other than adding capture latency.
viam-server
or writes timing out.By default, viam-server
checks for new data to sync at the configured interval (sync_interval_mins
).
You can additionally configure sync to only happen when certain conditions are met.
For example:
See Conditional cloud sync for how to implement conditional syncs.
Configure how long your synced data remains stored in the cloud:
retention_policies
at the resource level.
See the retention_policy
field in data capture configuration attributes.delete_data_on_part_deletion
field in the data management service configuration attributes.Configurable sync threads: You can control how many concurrent sync operations occur by adjusting the maximum_num_sync_threads
setting.
Higher values may improve throughput on more powerful hardware, but raising it too high may introduce instability on resource-constrained devices.
Wait time before syncing arbitrary files: If you choose to sync arbitrary files (beyond those captured by the data management service), the file_last_modified_millis
configuration attribute specifies how long a file must remain unmodified before the data manager considers it for syncing.
The default is 10 seconds.
Was this page helpful?
Glad to hear it! If you have any other feedback please let us know:
We're sorry about that. To help us improve, please tell us what we can do better:
Thank you!