Camera Component

A camera component is a source of 2D and/or 3D images. You can use the component to configure a webcam, lidar, time-of-flight sensor, or another type of camera.

The API for camera components allows you to:

  • Request single images or a stream in 2D color, or display z-depth.

  • Request a point cloud. Each 3D point cloud image consists of a set of coordinates (x,y,z) representing depth in mm.

The configuration of your camera component depends on your camera model. You can use different models to:

  • Configure physical cameras that generate images or point clouds.
  • Combine streams from multiple cameras into one.
  • Transform and process images.

Configuration

For configuration information, click on one of the following models:

ModelDescription
ffmpegUses a camera, a video file, or a stream as a camera.
image_fileGets color and depth images frames from a file path.
velodyneUses velodyne lidar.
webcamA standard camera that streams camera data.
rtspA streaming camera with an MJPEG track.
fakeA camera model for testing.
single_streamA HTTP client camera that streams image data from an HTTP endpoint.
dual_streamA HTTP client camera that combines the streams of two camera servers to create colorful point clouds.
join_color_depthJoins the outputs of a color and depth camera already registered in your config to create a third “camera” that outputs the combined and aligned image.
align_color_depth_extrinsicsUses the intrinsics of the color and depth camera, as well as the extrinsic pose between them, to align two images.
align_color_depth_homographyUses a homography matrix to align the color and depth images.
join_pointcloudsCombines the point clouds from multiple camera sources and projects them to be from the point of view of target_frame.
transformA pipeline for applying transformations to an input image source.

Control your camera with Viam’s client SDK libraries

To get started using Viam’s SDKs to connect to and control your robot, go to your robot’s page on the Viam app, navigate to the Code Sample tab, select your preferred programming language, and copy the sample code generated.

When executed, this sample code will create a connection to your robot as a client. Then control your robot programmatically by adding API method calls as shown in the following examples.

These examples assume you have a camera called "my_camera" configured as a component of your robot. If your camera has a different name, change the name in the code.

Be sure to import the camera package for the SDK you are using:

from viam.components.camera import Camera
import (
  "go.viam.com/rdk/components/camera"
)

API

The camera component supports the following methods:

Method NameDescription
GetImageReturn an image from the camera.
GetPointCloudReturn a point cloud from the camera.
GetPropertiesReturn the camera intrinsic and camera distortion parameters, as well as whether the camera supports returning point clouds.
DoCommandSend or receive model-specific commands.

GetImage

Returns an image from the camera. You can request a specific MIME type but the returned MIME type is not guaranteed. If the server does not know how to return the specified MIME type, the server returns the image in another format instead.

Parameters:

  • mime_type (str): The MIME type of the image. The returned MIME type is not guaranteed to match the image output type.

Returns:

my_camera = Camera.from_robot(robot=robot, name="my_camera")

frame = await my_cam.get_image()

Be sure to close the image when finished.

For more information, see the Python SDK Docs.

Parameters:

  • ctx (Context): A Context carries a deadline, a cancellation signal, and other values across API boundaries.
  • errHandlers (ErrorHandler): A handler for errors allowing for logic based on consecutively retrieved errors).

Returns:

myCamera, err := camera.FromRobot(robot, "my_camera")

// gets the stream from a camera
stream, err := myCamera.Stream(context.Background())

// gets an image from the camera stream
img, release, err := stream.Next(context.Background())
defer release()

For more information, see the Go SDK Docs.

GetPointCloud

Get a point cloud from the camera as bytes with a MIME type describing the structure of the data. The consumer of this call should decode the bytes into the format suggested by the MIME type.

Parameters:

  • None.

Returns:

  • (Tuple[bytes,str]): The pointcloud data as bytes paired with a string representing the mimetype of the pointcloud (for example, PCD).

To deserialize the returned information into a numpy array, use the Open3D library:

import numpy as np
import open3d as o3d

my_camera= Camera.from_robot(robot=robot, name="my_camera")

data, _ = await my_camera.get_point_cloud()

# write the point cloud into a temporary file
with open("/tmp/pointcloud_data.pcd", "wb") as f:
    f.write(data)
pcd = o3d.io.read_point_cloud("/tmp/pointcloud_data.pcd")
points = np.asarray(pcd.points)

For more information, see the Python SDK Docs.

Parameters:

  • ctx (Context): A Context carries a deadline, a cancellation signal, and other values across API boundaries.

Returns:

  • (pointcloud.PointCloud): A general purpose container of points. It does not dictate whether or not the cloud is sparse or dense.
  • (error): An error, if one occurred.
myCamera, err := camera.FromRobot(robot, "my_camera")

pointCloud, err := myCamera.NextPointCloud(context.Background())

For more information, see the Go SDK Docs.

GetProperties

Get the camera intrinsic parameters and camera distortion, as well as whether the camera supports returning point clouds.

Parameters:

  • None.

Returns:

my_camera= Camera.from_robot(robot=robot, name="my_camera")

properties = await my_camera.get_properties()

For more information, see the Python SDK Docs.

Parameters:

  • ctx (Context): A Context carries a deadline, a cancellation signal, and other values across API boundaries.

Returns:

  • (Properties): Properties of the particular implementation of a camera.
  • (error): An error, if one occurred.
myCamera, err := camera.FromRobot(robot, "my_camera")

// gets the properties from a camera
properties, err := myCamera.Properties(context.Background())

For more information, see the Go SDK Docs.

DoCommand

Execute model-specific commands that are not otherwise defined by the component API. For native models, model-specific commands are covered with each model’s documentation. If you are implementing your own camera and add features that have no native API method, you can access them with DoCommand.

Parameters:

Returns:

my_camera= Camera.from_robot(robot, "my_camera")

command = {"cmd": "test", "data1": 500}
result = my_camera.do(command)

For more information, see the Python SDK Docs.

Parameters:

Returns:

  myCamera, err := camera.FromRobot(robot, "my_camera")

  command := map[string]interface{}{"cmd": "test", "data1": 500}
  result, err := myCamera.DoCommand(context.Background(), command)

For more information, see the Go SDK Code.

Troubleshooting

You can find additional assistance in the Troubleshooting section.

You can also ask questions in the Community Discord and we will be happy to help.

Next Steps