Troubleshoot problems

If your machine is not working as expected, there are several steps you can take for debugging. Start by checking the machine logs. If that doesn’t help, you can enable debug logs or ssh into the machine.

For common errors see Common Errors.

Check logs

Machine shows as offline

If your machine shows as offline in the Viam app, restart viam-server by running the command to start viam-server and adding the -debug option.

To do this, you will need to know if you installed viam-server with viam-agent (most common) or manually. You can check this by seeing if viam-agent is running. Run the following command:

ps aux | grep viam-agent
root      566431  0.5  0.2 1247148 20336 ?       Ssl  11:24   0:00 /opt/viam/bin/viam-agent --config /etc/viam.json

If you see a process running viam-agent, then you used viam-agent to install viam-server. If not follow the steps for the standalone version.

Check for errors on the CONFIGURE page

Go to your machine’s CONFIGURE page and check whether any configured components have a red exclamation symbol on their configuration card. If so click on the symbol or expand the ERROR LOGS panel. The expanded panel shows you errors produced by that resource.

Check logs on the LOGS tab

If your machine shows as online in the Viam app, go to the LOGS tab and check for errors or other information relevant to the issue.

You can filter your logs by keyword, log levels and time.

The default log level for viam-server and any running resources is "Info". If you are not seeing helpful logs, you can try changing the log level to "Debug".

You can enable debug logs for all resources on a machine by adding "debug": true to the machine’s JSON configuration:

{
  "debug": true,
  "components": [{ ... }]
}

If this produces too many logs, you can instead enable debug logs for individual resources using the … menu on each resource. You can also set the log level by configuring the log attribute in the machine configuration to match on patterns that capture the resources you are interested in. For example:

"components": [ ... ]
"log": [
    {
    "pattern": "rdk.components.arm",
    "level": "debug",
    }, {
    "pattern": "rdk.services.*",
    "level": "debug",
    }, {
    "pattern": "<module-name>",
    "level": "debug",
    }
]

For more information on setting log levels see, Logging.

You may also find that not all logs you are expecting are displayed. By default, viam-server deduplicates log messages that are deemed noisy. To disable this behavior, see Disable log deduplication.

To access logs from the commandline, use viam machines logs on the command line or the Machines API.

Advanced debugging for Go modules

If you have a Go module that doesn’t shut down properly or hangs during operation, you can get more information by sending a SIGQUIT signal to the process. Run the following command, replacing <PID> with your module’s process identifier:

kill -3 <PID>

The process will dump a stack trace, visible in the viam-server logs, that shows:

• All currently running goroutines and their states
• Where execution is blocked or deadlocked
• Internal state information that might not appear in regular logs

Remote shell on the machine

To remotely access your machine from your terminal:

1. Add the ViamShellDanger fragment to your machine.

2. Once you have added the fragment, you can use the Viam CLI to open a shell on the machine.

   viam machines part shell --organization=<org name> --location=<location name> --machine=<machine id>
   

3. You can access the local log file on your machine if needed.

4. If you need to copy files from your machine, use the viam machine part cp command.

Restart your machine

1. Navigate to the app’s CONFIGURE tab in Builder mode.
2. Click the … menu on the right side of the machine part’s card, and select Restart part.

It takes a few minutes for the server to shut down and restart.

Revert to earlier configuration

The Viam app keeps a record of your configuration changes, allowing you to revert to earlier configurations if needed. To see the history of the configuration of a machine part, click on History on the CONFIGURE tab.

To restore to an earlier version of your configuration, click the Restore version button next to the desired configuration.

Share a location with Viam support

If you have additional questions on debugging your machine, reach out to us on our Community Discord. Please post the error message you received along with the steps you took that caused the issue and we’ll see if we can help.

If you request support, you may be asked to share your location with the Viam Support team. To do so, navigate to the location you need support with and click, Add Viam support.

Once you have received support, you can remove Viam Support from your location by clicking Remove Viam support.