Retrain a model

A model loses accuracy when the conditions it runs in drift away from the data it was trained on. The fix is not a one-time deployment. It is a loop: capture the cases the model gets wrong, label them, train a new version, and deploy it.

Viam’s data, training, and fleet tools are the pieces of that loop. This guide walks through the end-to-end cycle from production machine back to production machine.

The guide uses a vision model as its example. For a vision model, drift looks like this: lighting shifts, new object variations appear, someone moves a camera, or a new SKU shows up on the line. This training-to-production gap is a common reason a vision system fails after launch.

The cycle is the same for any model you train on Viam, but evaluation and auto-predictions support only classification and detection models. For how the stages fit together, see How training works.

The cycle

Production      ───► Capture        ───► Label          ───► Train ───► Deploy
detections fail      failing images      failing images
    ▲                                                                         │
    └──────────────────────────────── monitor ────────────────────────────────┘

You do not have to complete every step in a single sitting, and you do not have to do it manually every time. Scheduled data capture and training jobs can automate most of the loop once you have configured it.

1. Detect that your model needs retraining

Signs that a model needs new training data:

  • Detections are missing on objects you know are in the frame.
  • False positives on classes the model was never trained on.
  • The label distribution in production does not match the training set (for example, a model trained in daylight is running at night).
  • Confidence scores have drifted downward even on correct detections.

A simple approach: log detection results alongside camera frames for a day, then review a sample. Any class with precision or recall below your target threshold is a candidate for retraining.

To measure precision and recall instead of estimating them, label a sample of those frames in a dataset and evaluate the model on it. The per-class breakdown shows which classes fall below your target.

2. Capture failing images

The image that made your model fail is the single most valuable training example you can add. Use the data management service to save camera frames to the Viam Cloud so they become available for labeling.

Two capture strategies, depending on volume:

  • Time-based capture saves a frame every N seconds. Good for building a broad dataset across a day or week. Adjust the interval to control storage cost.
  • Conditional capture saves frames only when a trigger condition fires (for example, “when the vision service returns a confidence below 0.6”). This keeps only the hard cases, which is what you need for retraining.

Both strategies are configured on the data management service. The images sync to the DATA tab where you label them in the next step.

3. Label the new images

Open the DATA tab in the Viam app and filter to your captured images. For each image, draw bounding boxes and apply class labels. See Annotate data for the UI walkthrough.

If you have a lot of images, use automatic annotation to generate initial labels from an existing model, then review and correct them. Correction is usually faster than labeling from scratch.

Add the newly labeled images to an existing dataset or create a new one. Using the same dataset as the original training keeps a single lineage: the next training job sees both the old and new images.

4. Train a new model version

With the updated dataset, start a new training job. See built-in training or custom training for the full flow. Key choices:

  • Same architecture, same hyperparameters if you are just correcting drift or adding examples. Retraining with a known-good setup is predictable and easy to compare.
  • New architecture if the task has changed (for example, you are adding a new class that the previous model could not detect). Expect to tune more.
  • Same or new model name. If you train under the same model name, the result is a new version of that model, and machines that track its latest version pick it up. If you train under a new name, the result is a separate model, and you switch machines to it by changing their configuration. A new version must use the same framework as the model’s earlier versions.

Viam runs training jobs on cloud infrastructure. You do not need to provision a GPU machine locally. Logs stay available for seven days after the job completes.

Schedule automatic retraining

For a model trained with built-in training, you can have Viam start a new training job on a schedule instead of starting each one yourself. See Schedule retraining.

5. Deploy the new version

Test on one machine first

Pick a non-critical machine (or a local test machine) and update it to the new version. No application restart is needed.

Let the new version run for a few minutes against real camera input. Watch the vision service’s Control tab overlay. Confirm:

  • Detections appear on the objects you labeled.
  • The confidence scores are reasonable.
  • No new false positives appeared on unrelated objects.

Roll out to the fleet

Once the new version passes single-machine validation, promote it to the rest of the fleet. See Deploy ML models across your fleet for updating a shared fragment and staging the rollout. Whether machines pick up the new version automatically depends on whether they track the latest version or pin one.

6. Monitor and repeat

Accuracy monitoring is not automatic on Viam today. There is no dashboard that reports “your model’s precision has dropped 5% this week.” Practical approaches:

  • Log confidence scores from your detector to a time-series store and watch for trends.
  • Sample review: periodically label a sample of production images in a test dataset and evaluate the model on it. Compare precision and recall with the previous evaluation to see whether accuracy has dropped.
  • Alert on low confidence: configure a trigger that fires when detections below a threshold exceed a rate, so you know to pull more training data.

When drift shows up again, the cycle repeats. The first retraining cycle is the slowest; subsequent cycles reuse the same dataset, labeling workflow, and deployment procedure.

Next steps