Retrain a model
A model loses accuracy when the conditions it runs in drift away from the data it was trained on. The fix is not a one-time deployment. It is a loop: capture the cases the model gets wrong, label them, train a new version, and deploy it.
Viam’s data, training, and fleet tools are the pieces of that loop. This guide walks through the end-to-end cycle from production machine back to production machine.
The guide uses a vision model as its example. For a vision model, drift looks like this: lighting shifts, new object variations appear, someone moves a camera, or a new SKU shows up on the line. This training-to-production gap is a common reason a vision system fails after launch.
The cycle is the same for any model you train on Viam, but evaluation and auto-predictions support only classification and detection models. For how the stages fit together, see How training works.
The cycle
Production ───► Capture ───► Label ───► Train ───► Deploy
detections fail failing images failing images
▲ │
└──────────────────────────────── monitor ────────────────────────────────┘
You do not have to complete every step in a single sitting, and you do not have to do it manually every time. Scheduled data capture and training jobs can automate most of the loop once you have configured it.
1. Detect that your model needs retraining
Signs that a model needs new training data:
- Detections are missing on objects you know are in the frame.
- False positives on classes the model was never trained on.
- The label distribution in production does not match the training set (for example, a model trained in daylight is running at night).
- Confidence scores have drifted downward even on correct detections.
A simple approach: log detection results alongside camera frames for a day, then review a sample. Any class with precision or recall below your target threshold is a candidate for retraining.
To measure precision and recall instead of estimating them, label a sample of those frames in a dataset and evaluate the model on it. The per-class breakdown shows which classes fall below your target.
2. Capture failing images
The image that made your model fail is the single most valuable training example you can add. Use the data management service to save camera frames to the Viam Cloud so they become available for labeling.
Two capture strategies, depending on volume:
- Time-based capture saves a frame every N seconds. Good for building a broad dataset across a day or week. Adjust the interval to control storage cost.
- Conditional capture saves frames only when a trigger condition fires (for example, “when the vision service returns a confidence below 0.6”). This keeps only the hard cases, which is what you need for retraining.
Both strategies are configured on the data management service. The images sync to the DATA tab where you label them in the next step.
3. Label the new images
Open the DATA tab in the Viam app and filter to your captured images. For each image, draw bounding boxes and apply class labels. See Annotate data for the UI walkthrough.
If you have a lot of images, use automatic annotation to generate initial labels from an existing model, then review and correct them. Correction is usually faster than labeling from scratch.
Add the newly labeled images to an existing dataset or create a new one. Using the same dataset as the original training keeps a single lineage: the next training job sees both the old and new images.
4. Train a new model version
With the updated dataset, start a new training job. See built-in training or custom training for the full flow. Key choices:
- Same architecture, same hyperparameters if you are just correcting drift or adding examples. Retraining with a known-good setup is predictable and easy to compare.
- New architecture if the task has changed (for example, you are adding a new class that the previous model could not detect). Expect to tune more.
- Same or new model name. If you train under the same model name, the result is a new version of that model, and machines that track its latest version pick it up. If you train under a new name, the result is a separate model, and you switch machines to it by changing their configuration. A new version must use the same framework as the model’s earlier versions.
Viam runs training jobs on cloud infrastructure. You do not need to provision a GPU machine locally. Logs stay available for seven days after the job completes.
Schedule automatic retraining
For a model trained with built-in training, you can have Viam start a new training job on a schedule instead of starting each one yourself. See Schedule retraining.
5. Deploy the new version
Test on one machine first
Pick a non-critical machine (or a local test machine) and update it to the new version. No application restart is needed.
Let the new version run for a few minutes against real camera input. Watch the vision service’s Control tab overlay. Confirm:
- Detections appear on the objects you labeled.
- The confidence scores are reasonable.
- No new false positives appeared on unrelated objects.
Roll out to the fleet
Once the new version passes single-machine validation, promote it to the rest of the fleet. See Deploy ML models across your fleet for updating a shared fragment and staging the rollout. Whether machines pick up the new version automatically depends on whether they track the latest version or pin one.
6. Monitor and repeat
Accuracy monitoring is not automatic on Viam today. There is no dashboard that reports “your model’s precision has dropped 5% this week.” Practical approaches:
- Log confidence scores from your detector to a time-series store and watch for trends.
- Sample review: periodically label a sample of production images in a test dataset and evaluate the model on it. Compare precision and recall with the previous evaluation to see whether accuracy has dropped.
- Alert on low confidence: configure a trigger that fires when detections below a threshold exceed a rate, so you know to pull more training data.
When drift shows up again, the cycle repeats. The first retraining cycle is the slowest; subsequent cycles reuse the same dataset, labeling workflow, and deployment procedure.
Next steps
- Data capture and sync: set up production image capture
- Annotate data: label captured data
- Train a model: produce a new model version
- Deploy a model: choose which version a machine runs, and update or roll it back
- Act on detections and alert on detections: use the new version in production logic
Was this page helpful?
Glad to hear it! If you have any other feedback please let us know:
We're sorry about that. To help us improve, please tell us what we can do better:
Thank you!