Alert on inferences

At this point, you should have already set up and tested computer vision functionality. On this page, you’ll learn how to use triggers to send alerts in the form of email notifications or webhook requests when certain detections or classifications are made.

You will build a system that can monitor camera feeds and detect situations that require review. In other words, this system performs anomaly detection. Whenever the system detects an anomaly, it will send an email notification.

First, you’ll set up data capture and sync to record images with the anomaly and upload them to the cloud. Next, you’ll configure a trigger to send email notifications or webhook requests when the anomaly is detected.

Prerequisites

Configure a filtered camera

Your physical camera is working and your vision service is set up. Now you will pull them together to filter out only images where an inference is made with the filtered-camera module. This camera module takes the vision service and applies it to your webcam feed, filtering the output so that later, when you configure data management, you can save only the images that contain people inferred to match the filtering criteria rather than all images the camera captures.

Configure the camera module with classification or object labels according to the labels your ML model provides that you want to alert on. Follow the instructions in the filtered-camera module readme. For example, if using the YOLOv8 model (named yolo) for hardhat detection, you would configure the module like the following:

Configure data capture and sync

Viam’s built-in data management service allows you to, among other things, capture images and sync them to the cloud.

Configure data capture on the filtered-camera camera to capture images of detections or classifications:

1. First, you need to add the data management service to your machine to make it available to capture data on your camera.

   Navigate to your machine’s CONFIGURE tab.

   Click the + (Create) button next to your main part in the left-hand menu and select Service. Type “data” and click data management / RDK. Name your data management service data-manager and click Create.

   Leave all the default data service attributes as they are and click Save in the top right corner of the screen to save your changes.

2. Now you’re ready to enable data capture on your detector camera. Locate the objectfilter-cam panel.

3. Click Add method. Click the Type dropdown and select ReadImage. Set the capture frequency to 0.2 images per second (equivalent to one image every 5 seconds). You can always change the frequency to suit your use case. Set the MIME type to image/jpeg.

Set up alerts

Triggers allow you to send webhook requests or email notifications when certain events happen.

You can use the Data has been synced to the cloud (part_data_ingested) trigger to send alerts whenever an image with an anomaly detection is synced to the cloud from your object filter camera.

Set up the trigger with a webhook or with Viam’s built-in email alerts which sends a generic email letting you know that data has been synced.

Configure a trigger on your machine

Now it’s time to configure a trigger so that you get an email when a person is not wearing a hard hat.

Go to the CONFIGURE tab of your machine on the Viam app. Click the + (Create) button in the left side menu and select Trigger.

Name the trigger and click Create.

Select trigger Type as Data has been synced to the cloud and Data Types as Binary (image).

To configure notifications, either

• add a webhook and enter the URL of your custom cloud function
• add an email address to use Viam’s built-in email notifications

For both options also configure the time between notifications.

Click Save in the top right corner of the screen to save your changes.

Webhooks

Example cloud function

If you are using a cloud function or lambda to process the request from viam-server, you can use this template.

The following example function prints the received headers:

from flask import Flask, request

app = Flask(__name__)


@app.route("/", methods=['GET', 'POST'])
def trigger():
    headers = request.headers
    data = {}
    if request.data:
        data = request.json
    payload = {
        "Org-Id": headers.get('org-id', 'no value'),
        "Organization-Name": headers.get('organization-name', '') or
        data.get('org_name', 'no value'),
        "Location-Id": headers.get('location-id', 'no value'),
        "Location-Name": headers.get('location-name', '') or
        data.get('location_name', 'no value'),
        "Part-Id": headers.get('part-id', 'no value'),
        "Part-Name": headers.get('part-name', 'no value'),
        "Robot-Id": headers.get('robot-id', 'no value'),
        "Machine-Name": headers.get('machine-name', '') or
        data.get('machine_name', 'no value'),
        "Component-Type": data.get('component_type', 'no value'),
        "Component-Name": data.get('component_name', 'no value'),
        "Method-Name": data.get('method_name', 'no value'),
        "Min-Time-Received": data.get('min_time_received', 'no value'),
        "Max-Time-Received": data.get('max_time_received', 'no value'),
        "Data-Type": data.get('data_type', 'no value'),
        "File-Id": data.get('file_id', 'no value'),
        "Trigger-Condition": data.get("trigger_condition", 'no value'),
        "Data": data.get('data', 'no value')
    }
    print(payload)

    return payload


if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8080)

import functions_framework
import requests
import time


@functions_framework.http
def hello_http(request):
    headers = request.headers
    data = {}
    if request.data:
        data = request.json
    payload = {
        "Org-Id": headers.get("org-id", "no value"),
        "Organization-Name": headers.get("organization-name", "")
        or data.get("org_name", "no value"),
        "Location-Id": headers.get("location-id", "no value"),
        "Location-Name": headers.get("location-name", "")
        or data.get("location_name", "no value"),
        "Part-Id": headers.get("part-id", "no value"),
        "Part-Name": headers.get("part-name", "no value"),
        "Robot-Id": headers.get("robot-id", "no value"),
        "Machine-Name": headers.get("machine-name", "")
        or data.get("machine_name", "no value"),
        "Component-Type": data.get("component_type", "no value"),
        "Component-Name": data.get("component_name", "no value"),
        "Method-Name": data.get("method_name", "no value"),
        "Min-Time-Received": data.get("min_time_received", "no value"),
        "Max-Time-Received": data.get("max_time_received", "no value"),
        "Data-Type": data.get("data_type", "no value"),
        "File-Id": data.get('file_id', "no value"),
        "Trigger-Condition": data.get("trigger_condition", "no value"),
        "Data": data.get('data', "no value")
    }
    print(payload)

    return 'Received headers: {}'.format(payload)

Returned headers

When an event occurs, Viam sends a HTTP request to the URL you specified for the trigger:

The request includes the following headers:

Returned data

The request body includes the following data:

Test the whole system

You’ve built all the pieces of the system and connected them together. Now it’s time to test the whole thing.

Make sure viam-server is running on your machine. Run your camera in front of what you’re detecting and wait for an anomaly to appear. Wait a couple of minutes for the email to arrive in your inbox. Congratulations, you’ve successfully built your anomaly detection monitor!

Troubleshooting

Test the vision service

To see the detections or classifications occurring in real time and verify if their confidence level reaches the threshold you have set, you can navigate to the vision service card and expand the TEST panel.