Remote operations of vehicles
In various examples, at least partial control of a vehicle may be transferred to a control system remote from the vehicle. Sensor data may be received from a sensor(s) of the vehicle and the sensor data may be encoded to generate encoded sensor data. The encoded sensor data may be transmitted to the control system for display on a virtual reality headset of the control system. Control data may be received by the vehicle and from the control system that may be representative of a control input(s) from the control system, and actuation by an actuation component(s) of the vehicle may be caused based on the control input.
1 . A method comprising:
sending, using an autonomous machine and to a remote system, sensor data obtained using one or more sensors of the autonomous machine as the autonomous machine operates in an environment;
receiving, using the autonomous machine and from the remote system, location data indicating a location in the environment for the autonomous machine to navigate toward, the location data determined based at least on one or more waypoints that indicate the location in the environment;
determining, based at least on the location data indicating the location in the environment, one or more controls for navigating the autonomous machine toward the location; and
controlling, based at least on the one or more controls, the autonomous machine to navigate toward the location in the environment.
2 . The method of claim 1 , wherein the one or more waypoints include one or more virtual waypoints within a visualization generated based at least on the sensor data.
3 . The method of claim 1 , wherein the one or more waypoints are identified by a remote operator of the remote system during display of a visualization generated based at least on the sensor data.
4 . The method of claim 3 , wherein the visualization includes a virtual representation of the environment.
5 . The method of claim 4 , wherein the virtual representation is a three-dimensional (3D) virtual representation.
6 . The method of claim 4 , wherein the one or more waypoints are identified by the remote operator based at least on the remote operator pointing to one or more virtual locations within the virtual representation of the environment.
7 . The method of claim 4 , wherein the one or more waypoints are associated with the virtual representation, and the one or more controls are determined based at least on converting the one or more waypoints to one or more real-world waypoints.
8 . The method of claim 4 , wherein the virtual representation of the environment is capable of being displayed from different vantage points.
9 . The method of claim 3 , further comprising:
determining, based at least on the sensor data, that the autonomous machine has encountered a situation that requires at least partial control from the remote system,
wherein the display of the visualization includes the situation.
10 . The method of claim 3 , wherein the display by the remote system further includes a display of one or more video streams represented by the sensor data.
11 . The method of claim 1 , wherein during the navigate toward the location in the environment, the autonomous machine is capable of executing a safety procedure and disregarding the location data indicating the location in the environment received from the remote system.
12 . The method of claim 1 , further comprising:
determining, based at least on the one or more waypoints, a path through the environment,
wherein the one or more controls are determined based at least on the path.
13 . The method of claim 1 , wherein the method is executed using at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing deep learning operations;
a system implemented using a machine;
a system for generating synthetic data;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
14 . A method comprising:
receiving, at a remote system and from an autonomous machine, sensor data obtained using one or more sensors of the autonomous machine as the autonomous machine operates in an environment;
displaying a visualization generated based at least on the sensor data;
receiving one or more inputs indicating, within the visualization, one or more waypoints for the autonomous machine to navigate to within the environment; and
sending data indicating the one or more waypoints to the autonomous machine to control the autonomous machine to navigate toward a location in the environment.
15 . The method of claim 14 , wherein the one or more waypoints include one or more virtual waypoint locations within the visualization.
16 . The method of claim 14 , wherein the visualization includes a virtual representation of the environment.
17 . The method of claim 16 , wherein the virtual representation is a three-dimensional (3D) virtual representation.
18 . The method of claim 16 , wherein the one or more waypoints are identified by a remote operator based at least on the remote operator pointing to one or more virtual locations within the virtual representation of the environment.
19 . The method of claim 14 , wherein the visualization is capable of being displayed from different vantage points.
20 . The method of claim 14 , wherein the visualization includes a presentation of one or more video streams represented by the sensor data.
21 . A method comprising:
sending, from an autonomous machine and to a remote system, sensor data obtained using one or more sensors of the autonomous machine as the autonomous machine operates in an environment;
receiving, at the remote system and from the autonomous machine, the sensor data;
displaying, at the remote system, a visualization generated based at least on the sensor data;
receiving, at the remote system, one or more inputs indicating, within the visualization, one or more waypoints for the autonomous machine to navigate toward a location within the environment;
sending, from the remote system and to the autonomous machine, data indicating the one or more waypoints;
receiving, at the autonomous machine and from the remote system, the data indicating the one or more waypoints; and
based at least on the data indicating the one or more waypoints, the autonomous machine to navigate toward the location in the environment.
22 . The method of claim 21 , wherein the visualization includes a virtual representation of the environment.
23 . The method of claim 22 , wherein the virtual representation is a three-dimensional (3D) virtual representation.
24 . The method of claim 22 , wherein the one or more waypoints are identified by a remote operator based at least on the remote operator pointing to one or more virtual locations within the virtual representation of the environment.
25 . The method of claim 21 , wherein the displaying by the remote system further includes a display of one or more video streams represented by the sensor data.