> For AI agents: guidance on navigating Viam documentation is available at https://docs.viam.com/llms.txt.

# Retrain a model

Close the loop when a model's accuracy drops in production: capture the cases it gets wrong, label them, retrain, and redeploy a new model version to one or many machines.
> Source: https://docs.viam.com/train/retrain-a-model/


A model loses accuracy when the conditions it runs in drift away from the data it was trained on. The fix is not a one-time deployment. It is a loop: capture the cases the model gets wrong, label them, train a new version, and deploy it.

Viam's data, training, and fleet tools are the pieces of that loop. This guide walks through the end-to-end cycle from production machine back to production machine.

The guide uses a vision model as its example. For a vision model, drift looks like this: lighting shifts, new object variations appear, someone moves a camera, or a new SKU shows up on the line. This [training-to-production gap](https://medium.com/sciforce/why-your-computer-vision-model-struggles-in-the-real-world-3a6a999cdf8f) is a common reason a vision system fails after launch.

The cycle is the same for any model you train on Viam, but [evaluation](/train/evaluate-a-model/) and [auto-predictions](/train/automate-annotation/) support only classification and detection models. For how the stages fit together, see [How training works](/train/overview/#how-training-works).

## The cycle

<div style="max-width: fit-content;">

```text
Production      ───► Capture        ───► Label          ───► Train ───► Deploy
detections fail      failing images      failing images
    ▲                                                                         │
    └──────────────────────────────── monitor ────────────────────────────────┘
```

</div>

You do not have to complete every step in a single sitting, and you do not have to do it manually every time. Scheduled data capture and training jobs can automate most of the loop once you have configured it.

## 1. Detect that your model needs retraining

Signs that a model needs new training data:

- Detections are missing on objects you know are in the frame.
- False positives on classes the model was never trained on.
- The label distribution in production does not match the training set (for example, a model trained in daylight is running at night).
- Confidence scores have drifted downward even on correct detections.

A simple approach: log detection results alongside camera frames for a day, then review a sample. Any class with precision or recall below your target threshold is a candidate for retraining.

To measure precision and recall instead of estimating them, label a sample of those frames in a dataset and [evaluate the model](/train/evaluate-a-model/) on it.
The per-class breakdown shows which classes fall below your target.

## 2. Capture failing images

The image that made your model fail is the single most valuable training example you can add. Use the [data management service](/data/capture-sync/capture-and-sync-data/) to save camera frames to the Viam Cloud so they become available for labeling.

Two capture strategies, depending on volume:

- **Time-based capture** saves a frame every N seconds. Good for building a broad dataset across a day or week. Adjust the interval to control storage cost.
- **Conditional capture** saves frames only when a trigger condition fires (for example, "when the vision service returns a confidence below 0.6"). This keeps only the hard cases, which is what you need for retraining.

Both strategies are configured on the data management service. The images sync to the [**DATA** tab](https://app.viam.com/data/all) where you label them in the next step.

## 3. Label the new images

Open the [**DATA** tab](https://app.viam.com/data/all) in the Viam app and filter to your captured images. For each image, draw bounding boxes and apply class labels. See [Annotate data](/train/annotate-data/) for the UI walkthrough.

If you have a lot of images, use [automatic annotation](/train/automate-annotation/) to generate initial labels from an existing model, then review and correct them. Correction is usually faster than labeling from scratch.

Add the newly labeled images to an existing dataset or [create a new one](/train/create-a-dataset/). Using the same dataset as the original training keeps a single lineage: the next training job sees both the old and new images.

## 4. Train a new model version

With the updated dataset, start a new training job. See [built-in training](/train/train-a-model/) or [custom training](/train/custom-training-scripts/) for the full flow. Key choices:

- **Same architecture, same hyperparameters** if you are just correcting drift or adding examples. Retraining with a known-good setup is predictable and easy to compare.
- **New architecture** if the task has changed (for example, you are adding a new class that the previous model could not detect). Expect to tune more.
- **Same or new model name.** If you train under the same model name, the result is a new version of that model, and machines that track its latest version pick it up. If you train under a new name, the result is a separate model, and you switch machines to it by changing their configuration. A new version must use the same framework as the model's earlier versions.

Viam runs training jobs on cloud infrastructure. You do not need to provision a GPU machine locally. Logs stay available for seven days after the job completes.

### Schedule automatic retraining

For a model trained with built-in training, you can have Viam start a new training job on a schedule instead of starting each one yourself. See [Schedule retraining](/train/train-a-model/#schedule-retraining).

## 5. Deploy the new version

### Test on one machine first

Pick a non-critical machine (or a local test machine) and [update it to the new version](/train/deploy-a-model/#update-a-deployed-model). No application restart is needed.

Let the new version run for a few minutes against real camera input. Watch the vision service's Control tab overlay. Confirm:

- Detections appear on the objects you labeled.
- The confidence scores are reasonable.
- No new false positives appeared on unrelated objects.

### Roll out to the fleet

Once the new version passes single-machine validation, promote it to the rest of the fleet. See [Deploy ML models across your fleet](/fleet/deploy-ml-models/) for updating a shared fragment and staging the rollout. Whether machines pick up the new version automatically depends on whether they [track the latest version or pin one](/train/deploy-a-model/#choose-a-model-version).

## 6. Monitor and repeat

Accuracy monitoring is not automatic on Viam today. There is no dashboard that reports "your model's precision has dropped 5% this week." Practical approaches:

- **Log confidence scores** from your detector to a time-series store and watch for trends.
- **Sample review**: periodically label a sample of production images in a test dataset and [evaluate the model](/train/evaluate-a-model/) on it.
  Compare precision and recall with the previous evaluation to see whether accuracy has dropped.
- **Alert on low confidence**: configure a [trigger](/vision/object-detection/alert-on-detections/) that fires when detections below a threshold exceed a rate, so you know to pull more training data.

When drift shows up again, the cycle repeats. The first retraining cycle is the slowest; subsequent cycles reuse the same dataset, labeling workflow, and deployment procedure.

## Next steps

- [Data capture and sync](/data/capture-sync/capture-and-sync-data/): set up production image capture
- [Annotate data](/train/annotate-data/): label captured data
- [Train a model](/train/train-a-model/): produce a new model version
- [Deploy a model](/train/deploy-a-model/): choose which version a machine runs, and update or roll it back
- [Act on detections](/vision/object-detection/act-on-detections/) and [alert on detections](/vision/object-detection/alert-on-detections/): use the new version in production logic

