Use Object Detection to Turn Your Lights On

This tutorial uses the Viam vision service with your computer’s built-in webcam to detect the presence of a person and turn on a lamp when you sit down at your desk.

You can turn it into a night light for reading books, a security robot that alerts you when a person is close by, or a bathroom light that only activates when people enter; the opportunities are endless.

This project is a great place to start if you are new to building robots because the only hardware it requires in addition to your computer is a smart plug or smart bulb.

Hardware requirements

You need the following hardware for this tutorial:

Software requirements

You will use the following software in this tutorial:

Install viam-server and connect to your robot

Go to the Viam app and create a new machine.

Go to the Setup tab of your new machine’s page and follow the steps to install viam-server on your computer.

Configure the camera component

On your new machine’s page, go to the Config tab.

Navigate to the Components subtab and click Create component in the lower-left corner.

Select camera for type and webcam for model.

Enter my-camera as the name for your camera, then click Create.

Click the video path field to reveal a dropdown populated with camera paths that have been identified on your machine.

Select the path to the camera you want to use.

Click Save Config in the bottom left corner of the screen.

Navigate to the Control tab where you can see your camera working.

Configure your services

This tutorial uses a pre-trained machine learning (ML) model from the Viam registry named EfficientDet-COCO. This model can detect a variety of objects, which you can find in the provided labels.txt file.

If you want to train your own model instead, follow the instructions to train a model.

Navigate to the Services subtab of your machine’s Config tab.

Configure the ML model service

The ML model service allows you to deploy a machine learning model to your robot.

Click Create service in the lower-left corner of the page. Select ML Model for the type, then select TFLite CPU for the model. Enter people as the name for your mlmodel, then click Create.

In the new ML Model service panel, configure your service.

mlmodel service panel with empty sections for Model Path, and Optional Settings such as Label Path and Number of threads.

Select the Deploy model on robot for the Deployment field. Then select the viam-labs:EfficientDet-COCO model from the Models dropdown.

Configure an mlmodel detector

Click Create service in the lower-left corner of the page. For your vision service, select type vision and model mlmodel. Enter myPeopleDetector for the name, then click Create.

In the new vision service panel, configure your service.

From the Select model dropdown, select the name of the TFLite model (people).

Configure the detection camera

To be able to test that the vision service is working, add a transform camera which will add bounding boxes and labels around the objects the service detects.

Click the Components subtab and click the Create component button in the lower-left corner. Create a transform camera by selecting type camera and model transform. Name it detectionCam and click Create.

detectionCam component panel with type camera and model transform, Attributes section has source and pipeline but they are empty.

In the new transform camera panel, replace the attributes JSON object with the following object which specifies the camera source that the transform camera will be using and defines a pipeline that adds the defined myPeopleDetector:

{
  "source": "my-camera",
  "pipeline": [
    {
      "type": "detections",
      "attributes": {
        "detector_name": "myPeopleDetector",
        "confidence_threshold": 0.5
      }
    }
  ]
}

Click Save config in the bottom left corner of the screen.

detectionCam component panel with type camera and model transform, Attributes section filled with source and pipeline information.

Set up the Kasa smart plug

  1. Plug your smart plug into any power outlet and turn it on by pressing the white button on the smart plug. To connect the plug to your wifi, download the Kasa Smart app from the App Store or Google Play to your mobile phone. When you first open the app, you will be prompted to create an account. As you do this, you will receive an email with the subject line “TP-Link ID: Activation Required” to complete your account registration.

  2. Follow the steps in Kasa’s setup guide to add your device and connect it to your wifi. Once it is connected, you will no longer need to use the mobile app.

  3. Open a terminal on your computer and run the following command to install the smart plug Python API:

    pip3 install python-kasa
    
  4. Run the following command to return information about your smart device:

    kasa discover
    

    You should see this command output something like this:

    Terminal output with information about the smart plug including the host, device state (on), timestamp, hardware and software versions, MAC address, location (latitude and longitude), whether the LED is currently on, and the timestamp of when it last turned on. There is also a list of modules (schedule, usage, antitheft, time, and cloud).

    Write down or save the host address (for example, 10.1.11.221).

    You will need to include it in your Python code in a later step.

Write Python code to control your object detection robot

Now that you have your machine configured and your Kasa plug set up, you are ready to set up the code for the logic of the robot. The files used in this section can all be found in the GitHub repo for this project.

Create the main script file

On your computer, navigate to the directory where you want to put the code for this project. Create a file there called lightupbot.py. This will be the main script for the machine. Copy the entirety of this file and paste it into your lightupbot.py file. Save lightupbot.py.

Connect the code to the robot

You need to tell the code how to access your specific robot (which in this case represents your computer and its webcam).

  1. Navigate to the Code sample tab on the Viam app. Make sure Python is selected in the Language selector.

  2. Get the robot address and API key from the code sample and set them as environment variables or add them at the top of lightupbot.py.

    You also need to tell the code how to access your smart plug.

  3. Add the host address (for example, 10.1.11.221) of your smart plug that you found in the kasa discover step to line 55 of lightupbot.py.

Run the code

Now you are ready to test your robot!

From a command line on your computer, navigate to the project directory and run the code with this command:

python3 lightupbot.py

If the camera detects a person, it will print to the terminal “This is a person!” and turn on the smart plug. If it does not find a person, it will write “There’s nobody here” and will turn off the plug.

Try moving in and out of your webcam’s field of view. You will see your light turn on and off as the robot detects you!

Your terminal output should look like this as your project runs:

python3 lightupbot.py
This is a person!
turning on
There's nobody here
turning off

Next steps

In this tutorial, you learned how to build an object detection robot that turns your lights on using Viam. You could use this same concept to build a smart fan that only turns on if you are sitting at your desk working, turn on the lights in your bathroom mirror only when you are in front of the sink, or activate a pet feeder every time your cat looks at the camera.

To turn this robot into a security alert system, try the other tutorial in this series: Build a Person Detection Security Robot That Sends You a Photo of the Person Stealing Your Chocolates.

For more robotics projects, check out our other tutorials.

You can also ask questions in the Community Discord and we will be happy to help.