Providing remote assistance to an autonomous vehicle
View Patent ↗An event is detected that impairs a confidence level of the autonomous vehicle in progressing through a current route. In response to detecting the event, the autonomous vehicle communicates information about the event to a remote source of guidance. The autonomous vehicle can receive instructions from the remote source of guidance on how to handle the event. The autonomous vehicle can then implement the instructions to handle the event while it operates.
1. A method for remotely guiding an autonomous vehicle, the method being implemented by one or more processors and comprising:
processing an alert from the autonomous vehicle that includes sensor information, the alert indicating that the autonomous vehicle is uncertain in traversing a segment of roadway while on a trip;
displaying an interface for a human operator who is remote from the autonomous vehicle, the interface including a visual representation of the autonomous vehicle and a set of interactive features that is generated from the sensor information;
detecting a first selection input from the remote human operator on a selected region of the interface;
identifying a specific set of sensors based on the selected region of the interface and the visual representation of the autonomous vehicle;
converting the first selection input into a request for additional sensor data from the specific set of sensors of the autonomous vehicle;
communicating the request to the autonomous vehicle, the request causing the autonomous vehicle to obtain the additional sensor data collected from the specific set of sensors;
receiving the additional sensor data from the autonomous vehicle;
displaying the additional sensor data on the interface;
detecting a second selection input from the remote human operator of one or more of the interactive features in the set of interactive features, the second selection input specifying a response strategy comprising a condition and a plurality of actions for the autonomous vehicle to perform without additional input from the remote human operator; and
communicating the response strategy to the autonomous vehicle, the response strategy causing the autonomous vehicle to perform at least some of the plurality of actions, to monitor for the condition while approaching or traversing the segment of roadway, and in response to detecting the condition, to perform one or more actions to change a direction, speed, or driving lane of the autonomous vehicle without additional input from the remote human operator.
2. The method of claim 1 , wherein displaying the interface includes superimposing one or more interactive features of the interface over content that is generated from the sensor information.
3. The method of claim 1 , wherein displaying the interface includes displaying a time remaining for the remote human operator to provide the second selection input before the autonomous vehicle performs a default action.
4. A non-transitory computer-readable medium that stores instructions for remotely guiding an autonomous vehicle, the instructions, when executed by one or more processors, causing the one or more processors to perform operations comprising:
processing an alert from the autonomous vehicle that includes sensor information, the alert indicating that the autonomous vehicle is uncertain in traversing a segment of roadway while on a trip;
displaying an interface for a human operator who is remote from the autonomous vehicle, the interface including a visual representation of the autonomous vehicle and a set of interactive features that is generated from the sensor information;
detecting a first selection input from the remote human operator on a selected region of the interface;
identifying a specific set of sensors based on the selected region of the interface and the visual representation of the autonomous vehicle;
converting the first selection input into a request for additional sensor data from the specific set of sensors of the autonomous vehicle;
communicating the request to the autonomous vehicle, the request causing the autonomous vehicle to obtain the additional sensor data collected from the specific set of sensors;
receiving the additional sensor data from the autonomous vehicle;
displaying the additional sensor data on the interface;
detecting a second selection input from the remote human operator of one or more of the interactive features in the set of interactive features, the second selection input specifying a response strategy comprising a condition and a plurality of actions for the autonomous vehicle to perform without additional input from the remote human operator; and
communicating the response strategy to the autonomous vehicle, the response strategy causing the autonomous vehicle to perform at least some of the plurality of actions, to monitor for the condition while approaching or traversing the segment of roadway, and in response to detecting the condition, to perform one or more actions to change a direction, speed, or driving lane of the autonomous vehicle without additional input from the remote human operator.
5. The method of claim 1 , wherein displaying the interface includes displaying image content captured by one or more cameras of the autonomous vehicle as the autonomous vehicle approaches the segment of roadway.
6. The method of claim 1 , wherein displaying the interface includes providing an interactive feature to enable the remote human operator to request additional information from the autonomous vehicle.
7. The method of claim 6 , wherein the interactive feature enables the remote human operator to request additional images from any one of multiple cameras on the autonomous vehicle.
8. The method of claim 6 , wherein the interactive feature enables the remote human operator to directionally control one or more sensors of the autonomous vehicle to obtain additional information from the autonomous vehicle.
9. The method of claim 1 , wherein displaying the interface includes providing an interactive feature to enable the remote human operator to communicate an instruction to the autonomous vehicle to take no action to avoid a detected object or event that is causing uncertainty for the autonomous vehicle.
10. The method of claim 1 , wherein displaying the interface includes providing multiple interactive features to enable the remote human operator to communicate one or more instructions that correspond to a steering or braking operation that is to be performed by the autonomous vehicle.
11. The method of claim 10 , wherein the multiple interactive features enable the remote human operator to communicate one or more instructions that specify a magnitude of the steering or braking operation.
12. The method of claim 1 , wherein displaying the interface for the remote human operator includes providing one or more features that enable the remote human operator to view images that are above the autonomous vehicle or next to the segment of roadway.
13. The non-transitory computer readable medium of claim 4 , wherein one of the interactive features in the set of interactive features is to enable the remote human operator to instruct a null direction or speed adjustment.
14. The non-transitory computer readable medium of claim 4 , wherein displaying the interface includes superimposing one or more interactive features of the interface over content that is generated from the sensor information.
15. The non-transitory computer readable medium of claim 4 , wherein displaying the interface includes displaying a time remaining for the remote human operator to provide the second selection input before the autonomous vehicle performs a default action.
16. The non-transitory computer readable medium of claim 4 , wherein displaying the interface includes displaying image content captured by one or more cameras of the autonomous vehicle as the autonomous vehicle approaches the segment of roadway.
17. The non-transitory computer readable medium of claim 4 , wherein displaying the interface includes providing an interactive feature to enable the remote human operator to request additional information from the autonomous vehicle.
18. The non-transitory computer readable medium of claim 4 , wherein displaying the interface includes providing an interactive feature to enable the remote human operator to communicate an instruction to the autonomous vehicle to take no action to avoid a detected object or event that is causing uncertainty for the autonomous vehicle.
19. A computer system comprising:
a memory to store a set of instructions;
a network interface; and
one or more processors that access instructions from the memory, to remotely guide an autonomous vehicle using the network interface, wherein the one or more processors:
process an alert from the autonomous vehicle that includes sensor information, the alert indicating that the autonomous vehicle is uncertain in traversing a segment of roadway while on a trip;
display an interface for a human operator who is remote from the autonomous vehicle, the interface including a visual representation of the autonomous vehicle and a set of interactive features that is generated from the sensor information;
detect a first selection input from the remote human operator on a selected region of the interface;
identify a specific set of sensors based on the selected region of the interface and the visual representation of the autonomous vehicle;
convert the first selection input into a request for additional sensor data from the specific set of sensors of the autonomous vehicle;
communicate the request to the autonomous vehicle, the request causing the autonomous vehicle to obtain the additional sensor data collected from the specific set of sensors;
receive the additional sensor data from the autonomous vehicle;
display the additional sensor data on the interface;
detect a second selection input from the remote human operator of one or more of the interactive features in the set of interactive features, the second selection input specifying a response strategy comprising a condition and a plurality of actions for the autonomous vehicle to perform without additional input from the remote human operator; and
communicate the response strategy to the autonomous vehicle, the response strategy causing the autonomous vehicle to perform at least some of the plurality of actions, to monitor for the condition while approaching or traversing the segment of roadway, and in response to detecting the condition, to perform one or more actions to change a direction, speed, or driving lane of the autonomous vehicle without additional input from the remote human operator.
20. The method of claim 1 , wherein the autonomous vehicle is uncertain in traversing the segment of roadway when an uncertainly level exceeds a first threshold, or a confidence value is less than a second threshold, with respect to the autonomous vehicle's capability to safely respond to an event or condition on the segment of roadway.