Safety of autonomous vehicles using a virtual augmented support environment

1. A method for providing remote assistance to guide a vehicle through a real-world physical environment, the method comprising:

receiving, from the vehicle at a remote support system, a request for support and telemetry data representing characteristics of the real-world physical environment sensed by the vehicle;

generating at the remote support system based on the telemetry data, a representation of a computer-simulated environment that simulates features of the real-world physical environment;

instantiating, by the remote support system, a virtual support element superimposed in the computer-simulated environment that simulates the real-world physical environment;

causing the vehicle to operate in a follow mode in which the autonomous drive system of the vehicle controls the vehicle to follow a path through the real-world physical environment corresponding to a path of the virtual support element through the computer-simulated environment;

generating, at the remote support system, a virtual view of the computer-simulated environment from a perspective of a virtual driver of the virtual support element;

presenting the virtual view on a display for assisting a remote teleoperator in controlling the virtual support element;

obtaining, at the remote support system, navigation controls for controlling the virtual support element;

controlling, by the remote support system, virtual navigation of the virtual support element through the computer-simulated environment based on the navigation controls;

prior to detecting an updated position of the vehicle responsive to the navigation controls, updating the virtual view of the computer-simulated environment based on the virtual navigation; and

transmitting to the vehicle, position information associated with the virtual navigation of the virtual support element through the computer-simulated environment to guide the vehicle operating in the follow mode through the real-world physical environment.

2. The method of claim 1 , further comprising:

detecting completion of the vehicle traversing through the real-world physical environment;

causing the vehicle to cease recognition of the virtual support element; and

causing the vehicle to return to a normal mode in which the autonomous drive system controls navigation of the vehicle in absence of the virtual support element.

3. The method of claim 1 , wherein obtaining the navigation controls comprises:

receiving, by the remote support system, manual steering and acceleration inputs from a teleoperator; and

generating the navigation controls based on the manual steering and acceleration inputs.

4. The method of claim 1 , wherein controlling the virtual navigation of the virtual support element through the computer-simulated environment comprises:

receiving, by the remote support system, locations of markers in the computer-simulated environment corresponding to locations of hazards in the real-world physical environment; and

automatically generating the navigation controls to traverse through the simulated environment while avoiding the markers.

5. The method of claim 1 , wherein controlling the virtual navigation of the virtual support element through the computer-simulated environment comprises:

identifying a location of the real-world physical environment of the vehicle;

obtaining, based on the identified location, a pre-recorded route through the real-world physical environment that avoids hazards in the real-world physical environment; and

automatically generating the navigation controls to control the virtual support element based on the pre-recorded route.

6. The method of claim 1 , wherein generating the virtual view comprises:

generating a virtual guideline superimposed on a position of the vehicle as a visual cue to assist the remote support system in navigating the virtual support element.

7. The method of claim 1 , wherein instantiating the virtual support element comprises:

generating the virtual support element as a virtual vehicle having a width at least as wide as a width of the vehicle.

8. A non-transitory computer-readable storage medium storing instructions for providing remote assistance to guide a vehicle through a physical environment, the instructions when executed by a processor causing the processor to perform steps including:

receiving, from the vehicle at a remote support system, a request for support and telemetry data representing characteristics of the real-world physical environment sensed by the vehicle;

generating at the remote support system based on the telemetry data, a representation of a computer-simulated environment that simulates features of the real-world physical environment;

instantiating, by the remote support system, a virtual support element superimposed in the computer-simulated environment that simulates the real-world physical environment;

causing the vehicle to operate in a follow mode in which the autonomous drive system of the vehicle controls the vehicle to follow a path through the real-world physical environment corresponding to a path of the virtual support element through the computer-simulated environment;

generating, at the remote support system, a virtual view of the computer-simulated environment from a perspective of a virtual driver of the virtual support element;

presenting the virtual view on a display for assisting a remote teleoperator in controlling the virtual support element;

obtaining, at the remote support system, navigation controls for controlling the virtual support element;

controlling, by the remote support system, virtual navigation of the virtual support element through the computer-simulated environment based on the navigation controls;

prior to detecting an updated position of the vehicle responsive to the navigation controls, updating the virtual view of the computer-simulated environment based on the virtual navigation; and

transmitting to the vehicle, position information associated with the virtual navigation of the virtual support element through the computer-simulated environment to guide the vehicle operating in the follow mode through the real-world physical environment.

9. The non-transitory computer-readable storage medium of claim 8 , the instructions when executed further causing the processor to perform steps including:

detecting completion of the vehicle traversing through the real-world physical environment;

causing the vehicle to cease recognition of the virtual support element; and

causing the vehicle to return to a normal mode in which the autonomous drive system controls navigation of the vehicle in absence of the virtual support element.

10. The non-transitory computer-readable storage medium of claim 8 , wherein obtaining the navigation controls comprises:

receiving, by the remote support system, manual steering and acceleration inputs from a teleoperator; and

generating the navigation controls based on the manual steering and acceleration inputs.

11. The non-transitory computer-readable storage medium of claim 8 , wherein controlling the virtual navigation of the virtual support element through the computer-simulated environment comprises:

receiving, by the remote support system, locations of markers in the computer-simulated environment corresponding to locations of hazards in the real-world physical environment; and

automatically generating the navigation controls to traverse through the simulated environment while avoiding the markers.

12. The non-transitory computer-readable storage medium of claim 8 , wherein controlling the virtual navigation of the virtual support element through the computer-simulated environment comprises:

identifying a location of the real-world physical environment of the vehicle;

obtaining, based on the identified location, a pre-recorded route through the real-world physical environment that avoids hazards in the real-world physical environment; and

automatically generating the navigation controls to control the virtual support element based on the pre-recorded route.

13. The non-transitory computer-readable storage medium of claim 8 , wherein generating the virtual view comprises:

generating a virtual guideline superimposed on a position of the vehicle as a visual cue to assist the remote support system in navigating the virtual support element.

14. The non-transitory computer-readable storage medium of claim 8 , wherein instantiating the virtual support element comprises:

generating the virtual support element as a virtual vehicle having a width at least as wide as the width of the vehicle.

15. A computing system comprising:

one or more processors; and

a non-transitory computer-readable storage medium storing instructions for providing remote assistance to guide a vehicle through a physical environment, the instructions when executed by the one or more processors causing the one or more processors to perform steps including:

receiving, from the vehicle at a remote support system, a request for support and telemetry data representing characteristics of the real-world physical environment sensed by the vehicle;

generating at the remote support system based on the telemetry data, a representation of a computer-simulated environment that simulates features of the real-world physical environment;

instantiating, by the remote support system, a virtual support element superimposed in the computer-simulated environment that simulates the real-world physical environment;

causing the vehicle to operate in a follow mode in which the autonomous drive system of the vehicle controls the vehicle to follow a path through the real-world physical environment corresponding to a path of the virtual support element through the computer-simulated environment;

generating, at the remote support system, a virtual view of the computer-simulated environment from a perspective of a virtual driver of the virtual support element;

presenting the virtual view on a display for assisting a remote teleoperator in controlling the virtual support element;

obtaining, at the remote support system, navigation controls for controlling the virtual support element;

controlling, by the remote support system, virtual navigation of the virtual support element through the computer-simulated environment based on the navigation controls;

prior to detecting an updated position of the vehicle responsive to the navigation controls, updating the virtual view of the computer-simulated environment based on the virtual navigation; and

transmitting to the vehicle, position information associated with the virtual navigation of the virtual support element through the computer-simulated environment to guide the vehicle operating in the follow mode through the real-world physical environment.

16. The computing system of claim 15 , the instructions when executed further causing the processor to perform steps including:

detecting completion of the vehicle traversing through the real-world physical environment;

causing the vehicle to cease recognition of the virtual support element; and

causing the vehicle to return to a normal mode in which the autonomous drive system controls navigation of the vehicle in absence of the virtual support element.

17. The computing system of claim 15 , wherein obtaining the navigation controls comprises:

receiving, by the remote support system, manual steering and acceleration inputs from a teleoperator; and

generating the navigation controls based on the manual steering and acceleration inputs.

18. The computing system of claim 15 , wherein controlling the virtual navigation of the virtual support element through the computer-simulated environment comprises:

receiving, by the remote support system, locations of markers in the computer-simulated environment corresponding to locations of hazards in the real-world physical environment; and

automatically generating the navigation controls to traverse through the simulated environment while avoiding the markers.

19. The computing system of claim 15 , wherein controlling the virtual navigation of the virtual support element through the computer-simulated environment comprises:

identifying a location of the real-world physical environment of the vehicle;

obtaining, based on the identified location, a pre-recorded route through the real-world physical environment that avoids hazards in the real-world physical environment; and

automatically generating the navigation controls to control the virtual support element based on the pre-recorded route.

20. The computing system of claim 15 , wherein generating the virtual view comprises:

generating a virtual guideline superimposed on a position of the vehicle as a visual cue to assist the remote support system in navigating the virtual support element.