Create a Facial Verification System

With the machine learning (ML) service, a Viam machine can use an ML model together with its vision service to detect the presence of certain objects or patterns in the world around it. In this tutorial, you will learn how to build a facial verification system using Viam which can detect when a person appears in view of a camera, and either enter an alarm state if the detected person is not a valid approved person, or enter a disarm state if the detected person is approved. While the verification system itself is a classifier vision service, to accomplish this you configure it with dependencies on a variety of resources:

You will use two vision detectors, each powered by its own ML model:

1. A people-detect ML model detector, which can identify whether an object detected in your camera feed is a person or not. You will train this model by capturing images of a variety of people using your camera and the data management service, and classifying matching pictures with labels when a person is present in the frame.
2. A face-detect ML model detector, which can identify the face of a specific person. You will use a pre-existing facial recognition model that uses the DeepFace library, and provide photos of each person you want your security system to recognize.

Layering these two detectors, your verification system will trigger a countdown when it detects a person in its feed and disarm the alarm if it detects an approved face within the countdown period.

Here you can see the detector waiting in TRIGGER_1 state, its default state, until a person appears in front of the camera. As soon as the person is detected, the detector transitions to the COUNTDOWN state, where a countdown of 10 seconds begins. After a few seconds, the detector recognizes the person’s face, and enters the DISARMED state.

Had the person’s face not matched an approved face, the detector would instead have transitioned to the ALARM state. For more information on the various states used by the verification system, see Configure a verification system.

To keep this tutorial simple, you will use a transform camera to overlay the current state of the verification system on your live camera feed. If you wanted to take this tutorial further, you could use these state transitions to power other services or functions of your machine, such as emitting an audio warning on ALARM state, or updating an LED display during COUNTDOWN with the remaining time until alarm.

Prerequisites

Before following this tutorial, you should:

1. Add a new machine in the Viam app.
2. Install viam-server on your new machine.

Your machine must have a camera component, such as a webcam. Make sure to connect your camera to your machine’s computer (if it isn’t built-in) before starting the project and power it on.

Configure a camera

Navigate to the CONFIGURE tab of your machine’s page on the Viam app. Configure the camera you want to use for your security system. We configured ours as a webcam, but you can use whatever model of camera you’d like. Reference these available models.

To configure a webcam:

1. Click the + icon next to your machine part in the left-hand menu and select Component.
2. Select the camera type, then select the webcam model.
3. Enter the name my_webcam for your camera and click Create.
4. If your machine is online and connected to the Viam app, your camera’s video path is automatically detected and configured. If your machine is not currently connected, you can manually select the video path for your camera, or bring your machine online to have this path automatically configured for you.

Position your camera somewhere where it can easily see the people it will be configured to detect.

Next, configure the person detector, or, the coarser layer of the security system that verifies that there’s a person moving.

Configure an mlmodel person detector

In order for your machine’s camera to be able to detect the presence of a person in its field of vision, you can either use an existing ML Model from the registry capable of detecting people or train your own.

Use an existing ML model

The ML model service allows you to deploy a machine learning model to your robot. For your machine to be able to detect people, you will use a Machine Learning model from the Viam Registry called EfficientDet-COCO. The model can detect a variety of things which you can see in labels.txt file including persons.

1. Navigate to your machine’s CONFIGURE tab on the Viam app.
2. Click the + icon next to your machine part in the left-hand menu and select Service.
3. Select type ML model, then select model TFLite CPU.
4. Enter persondetect as the name for your ML model service, then click Create.
5. Select Deploy model on machine for the Deployment field.
6. Click Select model, then select the EfficientDet-COCO model by viam-labs from the Registry tab of the modal that appears.

Finally, configure an mlmodel detector vision service to use your new "persondetect" ML model:

1. Navigate to your machine’s CONFIGURE tab on the Viam app.
2. Click the + icon next to your machine part in the left-hand menu and select Service.
3. Select the vision type, then select the ML model model.
4. Give the detector the name people-detect and click Create.
5. Select the persondetect ML model service your model is deployed on from the ML Model dropdown.
6. Click Save.

For more information, see Configure an mlmodel detector

Continue to Configure a facial detector.

Train your own model

To train your own model, you will need to capture images of a variety of people using your camera, and upload them to the Viam app using the data management service.

To add the data management service and configure data capture:

1. Navigate to your machine’s page on the Viam app and select the CONFIGURE tab.
2. Click the + icon next to your machine part in the left-hand menu and select Service.
3. Choose data management as the type and then either use the suggested name or specify a name for your data management service, such as data-manager. Click Create.
4. On the panel that appears, you can manage the capturing and syncing functions individually. By default, the data management service captures data to the ~/.viam/capture directory, and syncs captured data files to the Viam app every 6 seconds (0.1 minutes in the configuration). Leave the default settings as they are, and click Save in the top right of the screen to save your changes.
5. Scroll to the panel of the camera you just configured. Find the Data capture section. Click Add Method. If you’re using a webcam, select the Method type ReadImage. Set the Frequency to 0.333. This will capture an image from the camera once every 3 seconds. Set the MIME type to image/jpeg. Click Save.
6. Toggle the Data capture on. Now, your camera is taking pictures. Walk in front of it a number of times, perhaps with a friend or two, letting the camera capture many images of you. For best results, try capturing a variety of angles and use different lighting.
7. Select the DATA page from the top of the screen. Here you can view the images captured so far from the camera on your machine. You should see new images appearing steadily as cloud sync uploads them from your machine.

For more information, see configure data capture for individual components.

Next, position your camera to capture a variety of images of people. Consider the lighting conditions, and angle of vision of the position where you intend to place your camera when you deploy it for actual use. For example, if you will be using your facial detection machine to look out your front window at your entrance way, you will want to be sure to include many images of people at about window-height, and perhaps in different lighting conditions or different stages of walking or standing at the door.

Then, create a new dataset using your uploaded images and train a new model using that model:

1. Create a new dataset and add the images you captured. Remember that you must add at least 10 images that contain people, as well as a few (but no more than 20% of the total images) that do not contain people.
2. Label the images that contain people with bounding boxes, and add the label person. You only want this model to be able to distinguish between what is and isn’t a person, so you can conduct this training step with anyone, not necessarily the specific people you intend to approve later.
3. Train a model on your dataset. Give it the name "persondetect", and select Object Detection as the Model Type.
4. Deploy the model to your machine so it can be used by other services, such as the vision service.

Finally, configure an mlmodel detector to use your new "persondetect" ML model:

1. Navigate to your machine’s CONFIGURE tab on the Viam app.
2. Click the + icon next to your machine part in the left-hand menu and select Service.
3. Select the vision type, then select the ML model model.
4. Give the detector the name people-detect and click Create.
5. Select the persondetect ML model service your model is deployed on from the ML Model dropdown.
6. Click Save.

For more information, see Configure an mlmodel detector

Now you are ready to configure the more fine-grained layer: the facial recognition detector.

Configure a facial detector

We now have a machine capable of detecting people in its camera feed, but we also want to be able to identify specific people in order to decide to either trigger an alarm if the specific person is not an approved person, or to disarm entirely if the detected person is allowed. First, select a profile picture of at least one face that you want your detector to be able to identify. A good profile picture clearly shows the face of the person in good lighting, with all facial features visible. Continue this process for each additional person you want your detector to be able to identify. Remember that a person who walks in front of your machine’s camera who is not able to be identified will trigger the ALARM state!

Once you have one or more pictures selected, copy them to your machine’s filesystem in your preferred fashion. For example, you could use the scp command to transfer an image to your machine like so:

scp /path/to/my-photo.jpg username@my-machine.local:/home/me/my-photo.jpg

After you have copied at least one image of a person to your machine, you are ready to configure the second detection layer: the facial recognition detector. For this tutorial, you will use Viam Labs’s facial-detector module, available from the Viam Registry. The facial-detector module provides a modular vision service that uses Facebook’s DeepFace library to perform facial detections.

To add the facial-detector module to your machine:

1. Navigate to your machine’s CONFIGURE page in the Viam app.

2. Click the + icon next to your machine part in the left-hand menu and select Service. Select vision, then select the detector:facial-detector model. You can also search for facial-detector directly.

3. Click Add module.

4. Name your modular vision service face-detect, then click Create.

5. On the panel that appears, enter the following configuration into the attributes field:

   {
     "face_labels": {
       "my_name": "/home/me/my-photo.jpg"
     },
     "recognition_model": "ArcFace",
     "detection_framework": "ssd"
   }
   

   Edit the attributes as applicable according to the configuration information on GitHub:

   • "face_labels": Label a photo of the face of each person you want your security system to recognize with the name you want for the label paired with the image path on your machine running viam-server. You can use scp to transfer your pictures from your development machine to that machine.
   • "recognition_model": The model to use for facial recognition. "ArcFace" is chosen as the default for a good balance of speed and accuracy.
   • "detection_framework": The detection framework to use for facial detection. "ssd" is chosen as the default for a good balance of speed and accuracy.

See the facial-detector module documentation for more information on the available attributes.

Configure a verification system

Now that you have configured both the coarser people-detect object detector and the more fine-grained face-detect facial detector, you are ready to add the alarm logic that uses these detectors to either trigger an alarm or disarm, based on the detected person. For this, add and configure the verification-system module from the Viam Registry following the steps below:

1. Navigate to your machine’s CONFIGURE page in the Viam app.

2. Click the + icon next to your machine part in the left-hand menu and select Service. Select vision, then select the classifier:verification-system model. You can also search for verification-system directly.

3. Click Add module.

4. Name your modular vision service security, then click Create.

5. On the panel that appears, enter the following configuration into the attributes field:

   {
     "trigger_1_confidence": 0.35,
     "verification_detector": "face-detect",
     "camera_name": "my-webcam",
     "trigger_2_confidence": 0.5,
     "trigger_1_labels": ["Person"],
     "trigger_2_labels": ["Person"],
     "disable_alarm": false,
     "trigger_2_detector": "people-detect",
     "verification_labels": ["my_name"],
     "trigger_1_detector": "people-detect",
     "disarmed_time_s": 10,
     "countdown_time_s": 10
   }
   

   In the configuration above:

   • "trigger_1_detector" and "trigger_2_detector" both use the people-detect ML model you created to determine if a person is present in the camera frame. For this tutorial, you are configuring both of these triggers identically to use the person detection ML model.
   • "trigger_1_labels" and "trigger_2_labels" similarly both use the "person" label you added to images when training the people-detect model. For this tutorial, you are configuring both of these labels identically to use the person detection ML model.
   • "verification_detector" uses the face-detect ML model you configured when you added images of faces to approved and labelled them in the configuration.
   • "verification_labels" contains an array of approved names that match each name you assigned to an image in the facial-detector modules’ "face_labels" configuration attribute.
   • "camera_name" is the name of the camera to use to detect people and faces. If you used a different name for your camera, update this parameter with your camera’s name.
   • Edit the other attributes to reflect your desired confidence thresholds and times between states.

See the verification-system module documentation for more information about the trigger states and their various configuration options.

Configure a transform camera

At this point, your machine is fully capable of detecting people in its camera feed, and of identifying whether a specific detected person is “approved” (defined under "face_labels") or not. To easily see this in action, you can add a transform camera to your machine to overlay the current state of the on top of the camera feed.

To add a transform camera to your machine:

1. Navigate to your machine’s CONFIGURE page in the Viam app.

2. lick the + icon next to your machine part in the left-hand menu and select Component. Select camera, then select the built-in transform model.

3. Give the transform camera a name, like my-transform-camera, then click Create.

4. Click the {} (Switch to Advanced) button in the top right of the component panel to edit the camera’s attributes directly with JSON.

5. Copy and paste the following configuration into the attributes field:

   {
     "pipeline": [
       {
         "attributes": {
           "classifier_name": "security",
           "confidence_threshold": 0.5
         },
         "type": "classification"
       }
     ],
     "source": "my-webcam"
   }
   

   If you used different names for the vision service or the camera component, update this configuration with those names. You can adjust the confidence_threshold to suit your needs. A value of 0.5 is a relatively loose match, representing 50% confidence.

6. Click Save at the top right of the window to save your changes.

View your verification system in action

With everything configured, you are now ready to see your facial recognition machine in action by watching the transform camera as a person passes in front of the camera.

To view your machine’s transform camera overlay:

1. On your machine’s CONTROL page in the Viam app, select the transform camera pane, which is listed by the name you gave it in the previous session, such as my-transform-camera.

2. Enable the view toggle to see a live camera feed from your camera, overlaid by the current state of the verification-system module, which should be TRIGGER_1 if no people are present in-frame.

3. Have one or more people walk in front of the camera and look directly into it. Watch the state change to COUNTDOWN and then DISARMED when an approved person is detected, or to ALARM if no approved person appears within 10 seconds!

   

Next steps

Now that you’ve got the verification aspect of your system working, you can use this as a launch point for customizing your own DIY home security system. For example:

• Write a program using one of the Viam SDK to poll the facial-verification module for its current state, and take action when a particular state is reached. For example, you could use GetClassificationsFromCamera() to capture when a transition into the ALARM state occurs, and then send you an email with the captured image of the trespasser!
• Try changing the type of detectors, using different detectors for the TRIGGER_1 and TRIGGER_2 states.
• Add the filtered camera module to your machine, and use it as the source camera in your verification system in order to save images to the Viam Cloud only when the system enters into specific states. This way, you could limit the images captured and synced to only those you are interested in reviewing later, for example.
• If you don’t want the ALARM capabilities, and would like to just use it as a notification system when a detector gets triggered, set disable_alarm: true in the config, which prevents TRIGGER_2 from entering into the COUNTDOWN state, meaning the system will only cycle between the states of TRIGGER_1 and TRIGGER_2.
• Use entering into the state TRIGGER_2 as a way to send notifications.