Camera API
The camera API allows you to give commands to your camera components for getting images or point clouds.
The API for camera components allows you to:
- Request single images or a stream in 2D color, or display z-depth.
- Request a point cloud. Each 3D point cloud image consists of a set of coordinates (x,y,z) representing depth in mm.
The camera component supports the following methods:
Method Name | Description |
---|---|
GetImage | Return an image from the camera. |
GetImages | Get simultaneous images from different imagers, along with associated metadata. |
GetPointCloud | Get a point cloud from the camera as bytes with a MIME type describing the structure of the data. |
GetProperties | Get the camera intrinsic parameters and camera distortion, as well as whether the camera supports returning point clouds. |
DoCommand | Execute model-specific commands that are not otherwise defined by the component API. |
GetGeometries | Get all the geometries associated with the camera in its current configuration, in the frame of the camera. |
GetResourceName | Get the ResourceName for this camera with the given name. |
Close | Safely shut down the resource and prevent further use. |
Establish a connection
To get started using Viam’s SDKs to connect to and control your machine, go to your machine’s page on the Viam app, navigate to the CONNECT tab’s Code sample page, select your preferred programming language, and copy the sample code.
API key and API key ID
By default, the sample code does not include your machine API key and API key ID. We strongly recommend that you add your API key and API key ID as an environment variable and import this variable into your development environment as needed.
To show your machine’s API key and API key ID in the sample code, toggle Include API key on the CONNECT tab’s Code sample page.
Caution
Do not share your API key or machine address publicly. Sharing this information could compromise your system security by allowing unauthorized access to your machine, or to the computer running your machine.
When executed, this sample code creates a connection to your machine as a client.
The following examples assume you have a camera called "my_camera"
configured as a component of your machine.
If your camera has a different name, change the name
in the code.
Be sure to import the camera package for the SDK you are using:
from viam.components.camera import Camera
import (
"go.viam.com/rdk/components/camera"
)
API
GetImage
Return an image from the camera. You can request a specific MIME type but the returned MIME type is not guaranteed. If the server does not know how to return the specified MIME type, the server returns the image in another format instead.
Parameters:
mime_type
(str) (required): The desired mime type of the image. This does not guarantee output type.extra
(Mapping[str, Any]) (optional): Extra options to pass to the underlying RPC call.timeout
(float) (optional): An option to set how long to wait (in seconds) before calling a time-out and closing the underlying RPC call.
Returns:
- (viam.media.video.ViamImage): The frame.
Example:
from viam.media.video import CameraMimeType
my_camera = Camera.from_robot(robot=machine, name="my_camera")
# Assume "frame" has a mime_type of "image/vnd.viam.dep"
frame = await my_camera.get_image(mime_type = CameraMimeType.VIAM_RAW_DEPTH)
# Convert "frame" to a standard 2D image representation.
# Remove the 1st 3x8 bytes and reshape the raw bytes to List[List[Int]].
standard_frame = frame.bytes_to_depth_array()
If the mime_type
of your image is image/vnd.viam.dep
, pass the returned image data to the Viam Python SDK’s ViamImage.bytes_to_depth_array()
method to decode the raw image data to a standard 2D image representation.
In addition, the Python SDK provides the helper functions viam_to_pil_image
and pil_to_viam_image
to decode the ViamImage
into a PIL Image
and vice versa.
For example:
from viam.media.utils.pil import pil_to_viam_image, viam_to_pil_image
# Get the ViamImage from your camera.
frame = await my_camera.get_image()
# Convert "frame" to a PIL Image representation.
pil_frame = viam_to_pil_image(frame)
# Use methods from the PIL Image class to get size.
x, y = pil_frame.size[0], pil_frame.size[1]
# Crop image to get only the left two fifths of the original image.
cropped_pil_frame = pil_frame.crop((0, 0, x / 2.5, y))
# Convert back to ViamImage.
cropped_frame = pil_to_viam_image(cropped_pil_frame, frame.mime_type)
Tip
Be sure to close the image when finished.
For more information, see the Python SDK Docs.
Info
Unlike most Viam component APIs, the methods of the Go camera client do not map exactly to the names of the other SDK’s camera methods.
To get an image in the Go SDK, you first need to construct a Stream
and then you can get the next image from that stream.
Parameters:
ctx
(Context): A Context carries a deadline, a cancellation signal, and other values across API boundaries.errHandlers
(…gostream.ErrorHandler): A handler for errors allowing for logic based on consecutively retrieved errors.
Returns:
- (gostream.VideoStream): A
VideoStream
that streams video until closed. - (error): An error, if one occurred.
Example:
myCamera, err := camera.FromRobot(machine, "my_camera")
// gets the stream from a camera
stream, err := myCamera.Stream(context.Background())
// gets an image from the camera stream
img, release, err := stream.Next(context.Background())
defer release()
For more information, see the Go SDK Docs.
GetImages
Usage
Intended specifically for use with cameras that support simultaneous depth and color image streams, like the Intel RealSense or the Luxonis OAK-D. If your camera does not have multiple imagers, this method will work without capturing multiple images simultaneously.
You can use the rgb-d-overlay
module to view and compare the camera streams returned by this method.
See the module readme for further instructions.
Get simultaneous images from different imagers, along with associated metadata.
The multiple images returned from GetImages()
do not represent a time series of images.
Parameters:
timeout
(float) (optional): An option to set how long to wait (in seconds) before calling a time-out and closing the underlying RPC call.
Returns:
- (Tuple[List[video.NamedImage], common.ResponseMetadata]): A tuple containing two values; the first [0] a list of images returned from the camera system, and the second [1] the metadata associated with this response.
Example:
my_camera = Camera.from_robot(robot=machine, name="my_camera")
images, metadata = await my_camera.get_images()
first_image = images[0]
timestamp = metadata.captured_at
For more information, see the Python SDK Docs.
Parameters:
ctx
(Context): A Context carries a deadline, a cancellation signal, and other values across API boundaries.
Returns:
- ([]NamedImage): The list of images returned from the camera system, with the name of the imager associated with the image.
- (resource.ResponseMetadata): The metadata, which holds the timestamp of when the data was captured.
- (error): An error, if one occurred.
Example:
myCamera, err := camera.FromRobot(machine, "my_camera")
images, metadata, err := myCamera.Images(context.Background())
For more information, see the Go SDK Docs.
GetPointCloud
Get a point cloud from the camera as bytes with a MIME type describing the structure of the data. The consumer of this call should decode the bytes into the format suggested by the MIME type.
Parameters:
extra
(Mapping[str, Any]) (optional): Extra options to pass to the underlying RPC call.timeout
(float) (optional): An option to set how long to wait (in seconds) before calling a time-out and closing the underlying RPC call.
Returns:
- (Tuple[bytes, str]): A tuple containing two values; the first [0] the pointcloud data, and the second [1] the mimetype of the pointcloud (for example, PCD).
Example:
import numpy as np
import open3d as o3d
data, _ = await my_camera.get_point_cloud()
# write the point cloud into a temporary file
with open("/tmp/pointcloud_data.pcd", "wb") as f:
f.write(data)
pcd = o3d.io.read_point_cloud("/tmp/pointcloud_data.pcd")
points = np.asarray(pcd.points)
For more information, see the Python SDK Docs.
Parameters:
ctx
(Context): A Context carries a deadline, a cancellation signal, and other values across API boundaries.
Returns:
- (pointcloud.PointCloud): A general purpose container of points. It does not dictate whether or not the cloud is sparse or dense.
- (error): An error, if one occurred.
Example:
myCamera, err := camera.FromRobot(machine, "my_camera")
// gets the next point cloud from a camera
pointCloud, err := myCamera.NextPointCloud(context.Background())
For more information, see the Go SDK Docs.
Parameters:
Returns:
Example:
var nextPointCloud = await myCamera.pointCloud();
For more information, see the Flutter SDK Docs.
GetProperties
Get the camera intrinsic parameters and camera distortion, as well as whether the camera supports returning point clouds.
Parameters:
timeout
(float) (optional): An option to set how long to wait (in seconds) before calling a time-out and closing the underlying RPC call.
Returns:
- (viam.components.camera.Camera.Properties): The properties of the camera.
Example:
my_camera = Camera.from_robot(robot=machine, name="my_camera")
properties = await my_camera.get_properties()
For more information, see the Python SDK Docs.
Parameters:
ctx
(Context): A Context carries a deadline, a cancellation signal, and other values across API boundaries.
Returns:
- (Properties): Properties of the particular implementation of a camera.
- (error): An error, if one occurred.
For more information, see the Go SDK Docs.
Parameters:
- None.
Returns:
Example:
var cameraProperties = await myCamera.properties();
For more information, see the Flutter SDK Docs.
DoCommand
Execute model-specific commands that are not otherwise defined by the component API.
For native models, model-specific commands are covered with each model’s documentation.
If you are implementing your own camera and adding features that have no native API method, you can access them with DoCommand
.
Parameters:
command
(Mapping[str, ValueTypes]) (required): The command to execute.timeout
(float) (optional): An option to set how long to wait (in seconds) before calling a time-out and closing the underlying RPC call.
Returns:
- (Mapping[str, viam.utils.ValueTypes]): Result of the executed command.
Raises:
- (NotImplementedError): Raised if the Resource does not support arbitrary commands.
Example:
command = {"cmd": "test", "data1": 500}
result = await my_camera.do_command(command)
For more information, see the Python SDK Docs.
GetGeometries
Get all the geometries associated with the camera in its current configuration, in the frame of the camera. The motion and navigation services use the relative position of inherent geometries to configured geometries representing obstacles for collision detection and obstacle avoidance while motion planning.
Parameters:
extra
(Mapping[str, Any]) (optional): Extra options to pass to the underlying RPC call.timeout
(float) (optional): An option to set how long to wait (in seconds) before calling a time-out and closing the underlying RPC call.
Returns:
- (List[viam.proto.common.Geometry]): The geometries associated with the Component.
Example:
geometries = await my_camera.get_geometries()
if geometries:
# Get the center of the first geometry
print(f"Pose of the first geometry's centerpoint: {geometries[0].center}")
For more information, see the Python SDK Docs.
GetResourceName
Get the ResourceName
for this camera with the given name.
Parameters:
name
(str) (required): The name of the Resource.
Returns:
- (viam.proto.common.ResourceName): The ResourceName of this Resource.
Example:
my_camera_name = Camera.get_resource_name("my_camera")
For more information, see the Python SDK Docs.
Close
Safely shut down the resource and prevent further use.
Parameters:
- None.
Returns:
- None.
Example:
await my_camera.close()
For more information, see the Python SDK Docs.
Parameters:
ctx
(Context): A Context carries a deadline, a cancellation signal, and other values across API boundaries.
Returns:
- (error): An error, if one occurred.
Example:
myCamera, err := camera.FromRobot(machine, "my_camera")
err = myCamera.Close(context.Background())
For more information, see the Go SDK Docs.
Have questions, or want to meet other people working on robots? Join our Community Discord.
If you notice any issues with the documentation, feel free to file an issue or edit this file.
Was this page helpful?
Glad to hear it! If you have any other feedback please let us know:
We're sorry about that. To help us improve, please tell us what we can do better:
Thank you!