Next
Filter data
You can use data management service to capture and sync data from your machine to the cloud. You can capture data from supported components and services or from arbitrary folders on your machines.
To start, configure the data management service to capture and sync data.
From your machine’s CONFIGURE tab in the Viam app, add the data management
service.
On the panel that appears, configure data capture and sync attributes as applicable.
To both capture data and sync it to the cloud, keep both Capturing and Syncing switched on.
Click the Save button in the top right corner of the page to save your config.
{
"components": [],
"services": [
{
"name": "my-data-manager",
"namespace": "rdk",
"type": "data_manager",
"attributes": {
"sync_interval_mins": 1,
"capture_dir": "",
"tags": [],
"capture_disabled": false,
"sync_disabled": true,
"delete_data_on_part_deletion": true,
"delete_every_nth_when_disk_full": 5,
"maximum_num_sync_threads": 250
}
}
]
}
{
"components": [],
"services": [
{
"name": "my-data-manager",
"namespace": "rdk",
"type": "data_manager",
"attributes": {
"capture_dir": "",
"tags": [],
"additional_sync_paths": [],
"sync_interval_mins": 3
}
}
]
}
The following attributes are available for the data management service:
Name | Type | Required? | Description | viam-micro-server Support |
---|---|---|---|---|
capture_disabled | bool | Optional | Toggle data capture on or off for the entire machine part. Note that even if capture is on for the whole part, but is not on for any individual components (see Step 2), data is not being captured. Default: false | |
capture_dir | string | Optional | Path to the directory on your machine where you want to store captured data. If you change the directory for data capture, only new data is stored in the new directory. Existing data remains in the directory where it was stored. Default: ~/.viam/capture | |
tags | array of strings | Optional | Tags to apply to all images captured by this machine part. May include alphanumeric characters, underscores, and dashes. | |
sync_disabled | bool | Optional | Toggle cloud sync on or off for the entire machine part. Default: false | |
additional_sync_paths | string array | Optional | Paths to any other directories on your machine from which you want to sync data to the cloud. Once data is synced from a directory, it is automatically deleted from your machine. | |
sync_interval_mins | float | Optional | Time interval in minutes between syncing to the cloud. Viam does not impose a minimum or maximum on the frequency of data syncing. However, in practice, your hardware or network speed may impose limits on the frequency of data syncing. Default: 0.1 , meaning once every 6 seconds. | |
delete_data_on_part_deletion | bool | Optional | Whether deleting this machine or machine part should result in deleting all the data captured by that machine part. Default: false | |
delete_every_nth_when_disk_full | int | Optional | How many files to delete when local storage meets the fullness criteria. The data management service will delete every Nth file that has been captured upon reaching this threshold. Use JSON mode to configure this attribute. Default: 5 , meaning that every fifth captured file will be deleted. | |
maximum_num_sync_threads | int | Optional | Max number of CPU threads to use for syncing data to the Viam Cloud. Default: runtime.NumCPU/2 so half the number of logical CPUs available to viam-server | |
mongo_capture_config.uri | string | Optional | The MongoDB URI data capture will attempt to write tabular data to after it is enqueued to be written to disk. When non-empty, data capture will capture tabular data to the configured MongoDB database and collection at that URI. See mongo_capture_config.database and mongo_capture_config.collection below for database and collection defaults.See Data capture directly to MongoDB for an example config. | |
mongo_capture_config.database | string | Optional | When mongo_capture_config.uri is non empty, changes the database data capture will write tabular data to.Default: "sensorData" | |
mongo_capture_config.collection | string | Optional | When mongo_capture_config.uri is non empty, changes the collection data capture will write tabular data to.Default: "readings" | |
cache_size_kb | float | Optional | viam-micro-server only. The maximum amount of storage bytes (in kilobytes) allocated to a data collector.Default: 1 KB. | |
file_last_modified_millis | float | Optional | The amount of time to pass since arbitrary files were last modified until they are synced. Normal Default: 10000 milliseconds. |
Data capture supports capturing tabular data directly to MongoDB in addition to capturing to disk.
This feature is intended to support use cases like offline dashboards which don’t require strong data delivery or consistency guarantees.
Here is a sample configuration that will capture fake sensor readings both to the configured MongoDB URI as well as to the ~/.viam/capture
directory on disk:
{
"components": [
{
"name": "sensor-1",
"namespace": "rdk",
"type": "sensor",
"model": "fake",
"attributes": {},
"service_configs": [
{
"type": "data_manager",
"attributes": {
"capture_methods": [
{
"method": "Readings",
"capture_frequency_hz": 0.5,
"additional_params": {}
}
]
}
}
]
}
],
"services": [
{
"name": "data_manager-1",
"namespace": "rdk",
"type": "data_manager",
"attributes": {
"mongo_capture_config": {
"uri": "mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000"
}
}
}
]
}
When mongo_capture_config.uri
is configured, data capture will attempt to connect to the configured MongoDB server and write captured tabular data to the configured mongo_capture_config.database
and mongo_capture_config.collection
(or their defaults if unconfigured) after enqueuing that data to be written to disk.
If writes to MongoDB fail for any reason, data capture will log an error for each failed write and continue capturing.
Failing to write to MongoDB doesn’t affect capturing and syncing data to cloud storage other than adding capture latency.
viam-server
or writes timing out.You can capture data for any resource that supports it, including resources on remote parts. Scroll to the resource card you wish to configure data capture and sync on.
For each resource you can capture data for, there is a Data capture section in its panel.
Select a Method and specify a capture Frequency in hertz, for example to 0.1
to capture an image every 10 seconds.
You can add multiple methods with different capture frequencies.
Some methods will prompt you to add additional parameters.
The available methods, and corresponding additional parameters, will depend on the component or service type.
For example, a camera has the options ReadImage
and NextPointCloud
.
Keep in mind that some models do not support all options, for example webcams do not capture point clouds, and choose the method accordingly.
Avoid configuring data capture to higher rates than your hardware can handle, as this leads to performance degradation.
This example configuration captures data from the ReadImage
method of a camera:
{
"services": [
...
,
{
"name": "data_manager",
"type": "data_manager",
"attributes": {
"sync_interval_mins": 5,
"capture_dir": "",
"sync_disabled": false,
"tags": []
}
}
],
"remotes": [
{
...
}
],
"components": [
...
,
{
"service_configs": [
{
"type": "data_manager",
"attributes": {
"capture_methods": [
{
"capture_frequency_hz": 0.333,
"disabled": false,
"method": "ReadImage",
"additional_params": {
"reader_name": "cam1",
"mime_type": "image/jpeg"
}
}
],
"retention_policy": {
"days": 5
}
}
}
],
"model": "webcam",
"name": "cam",
"type": "camera",
"attributes": {
"video_path": "video0"
},
"depends_on": [
"local"
]
},
...
]
}
This example configuration captures data from the GetReadings
method of a temperature sensor and wifi signal sensor:
{
"services": [
{
"attributes": {
"capture_dir": "",
"tags": [],
"additional_sync_paths": [],
"sync_interval_mins": 3
},
"name": "dm",
"namespace": "rdk",
"type": "data_manager"
}
],
"components": [
{
"type": "sensor",
"model": "tmp36",
"attributes": {
"analog_reader": "temp",
"num_readings": 15
},
"depends_on": [],
"service_configs": [
{
"attributes": {
"capture_methods": [
{
"capture_frequency_hz": 0.2,
"cache_size_kb": 10,
"additional_params": {},
"method": "Readings"
}
]
},
"type": "data_manager"
}
],
"name": "tmp36",
"namespace": "rdk"
},
{
"type": "sensor",
"model": "wifi-rssi",
"attributes": {},
"service_configs": [
{
"type": "data_manager",
"attributes": {
"capture_methods": [
{
"additional_params": {},
"method": "Readings",
"capture_frequency_hz": 0.1,
"cache_size_kb": 10
}
]
}
}
],
"name": "my-wifi-sensor",
"namespace": "rdk"
}
]
}
Example for a vision service:
This example configuration captures data from the CaptureAllFromCamera
method of the vision service:
{
"components": [
{
"name": "camera-1",
"namespace": "rdk",
"type": "camera",
"model": "webcam",
"attributes": {}
}
],
"services": [
{
"name": "vision-1",
"namespace": "rdk",
"type": "vision",
"model": "mlmodel",
"attributes": {},
"service_configs": [
{
"type": "data_manager",
"attributes": {
"capture_methods": [
{
"method": "CaptureAllFromCamera",
"capture_frequency_hz": 1,
"additional_params": {
"mime_type": "image/jpeg",
"camera_name": "camera-1",
"min_confidence_score": "0.7"
}
}
]
}
}
]
},
{
"name": "data_manager-1",
"namespace": "rdk",
"type": "data_manager",
"attributes": {
"sync_interval_mins": 0.1,
"capture_dir": "",
"tags": [],
"additional_sync_paths": []
}
},
{
"name": "mlmodel-1",
"namespace": "rdk",
"type": "mlmodel",
"model": "viam:mlmodel-tflite:tflite_cpu",
"attributes": {}
}
],
"modules": [
{
"type": "registry",
"name": "viam_tflite_cpu",
"module_id": "viam:tflite_cpu",
"version": "0.0.3"
}
]
}
Viam supports data capture from resources on remote parts. For example, if you use a part that does not have a Linux operating system or that does not have enough storage or processing power, you can still process and capture the data from that part’s resources by adding it as a remote part.
Currently, you can only configure data capture from remote resources in your JSON configuration.
To add them to your JSON configuration you must explicitly add the remote resource’s type
, model
, name
, and additional_params
to the data_manager
service configuration in the remotes
configuration:
Key | Description |
---|---|
type | The type tells your machine what the resource is. For example, a board. |
model | The model is a colon-delimited-triplet that specifies the namespace, the type of the part, and the part itself. |
name | The name specifies the fully qualified name of the part. |
additional_params | The additional parameters specify the data sources when you are using a board. |
The following attributes are available for data capture configuration:
Name | Type | Required? | Description |
---|---|---|---|
capture_frequency_hz | float | Required | Frequency in hertz at which to capture data. For example, to capture a reading every 2 seconds, enter 0.5 . |
method | string | Required | Depends on the type of component or service. See Supported components and services. |
retention_policy | object | Optional | Option to configure how long data collected by this component or service should remain stored in the Viam Cloud. You must set this in JSON mode. See the JSON example for a camera component. Options: "days": <int> , "binary_limit_gb": <int> , "tabular_limit_gb": <int> .Days are in UTC time. Setting a retention policy of 1 day means that data stored now will be deleted the following day in UTC time. You can set either or both of the size limit options and size is in gigabytes. |
additional_params | depends | depends | Varies based on the method. For example, ReadImage requires a MIME type. |
Click the Save button in the top right corner of the page to save your config.
If cloud sync is enabled, the data management service deletes captured data once it has successfully synced to the cloud.
If your robot is offline and can’t sync and your machine’s disk fills up beyond a certain threshold, the data management service will delete captured data to free up additional space and maintain a working machine.
If this is a test project, make sure you stop data capture to avoid charges for a large amount of unwanted data.
In the Data capture section of your resource’s configuration card, toggle the switch to Off.
Click the Save button in the top right corner of the page to save your config.
To view all the captured data you have access to, go to the DATA tab where you can filter by location, type of data, and more.
You can also access data from a resource or machine part menu.
The following components and services support data capture and cloud sync:
Type | Method |
---|---|
Arm | EndPosition , JointPositions |
Board | Analogs , Gpios |
Camera | GetImages , ReadImage , NextPointCloud |
Encoder | TicksCount |
Gantry | Lengths , Position |
Motor | Position , IsPowered |
Movement sensor | AngularVelocity , CompassHeading , LinearAcceleration , LinearVelocity , Orientation , Position |
Sensor | Readings |
Servo | Position |
Vision service | CaptureAllFromCamera |
Capturing too much data: You can use filtering to collect and sync only certain images to capture data selectively.
Rentention policy: Set a retention_policy
attribute in your data capture configuration to avoid keeping data stored in the Viam Cloud longer than a specified number of days.
Pausing sync: You can pause cloud sync at any time by navigating to your machine’s CONFIGURE tab and disabling Syncing for your data management service.
If you have captured data that you do not want to sync, delete the data on the machine before resuming cloud sync.
To delete the data locally, ssh
into your machine and delete the data in the directory where you capture data.
Sync data conditionally: You can use a module to sync data only when a certain logic condition is met, instead of at a regular time interval. For example, if you rely on mobile data but have intermittent WiFi connection in certain locations or at certain times of the day, you may want to trigger sync to only occur when these conditions are met. To set up triggers for syncing see Conditional cloud sync.
Now that you have captured data, you could create a dataset and use this data to train your own Machine Learning model with the Viam platform.
Was this page helpful?
Glad to hear it! If you have any other feedback please let us know:
We're sorry about that. To help us improve, please tell us what we can do better:
Thank you!