Validation of teleoperations system for autonomous vehicles using interactive validation scene playback

1 . A method comprising:

retrieving a first driving scene from a storage of a validation system, wherein the first driving scene is a recorded feed that corresponds to a real-world driving scenario previously captured by a first vehicle, the first driving scene being retrieved from the storage by a teleoperations system that includes a teleoperations console and a teleoperations human interface system;

presenting a playback of the first driving scene during a simulation process, the playback of the first driving scene being presented on the teleoperations console to enable the teleoperations console to simulate remotely monitoring the first vehicle using the teleoperations console during the real-world driving scenario previously captured by the first vehicle;

during presentation of the playback of the first driving scene by the teleoperations console during the simulation process, monitoring one or more controls on the teleoperations console using the teleoperations human interface system, wherein the one or more controls on the teleoperations console includes a control to simulate causing the first vehicle being remotely monitored via the teleoperations console to exit an autonomous mode;

determining, based on monitoring the one or more controls on the teleoperations console using the teleoperations human interface system, a first evaluation outcome for an operator for the first driving scene; and

based at least in part on the first evaluation outcome for the operator for the first driving scene, certifying the operator for performing teleoperations functions for a fleet of vehicles using the validation system.

2 . The method of claim 1 , wherein presenting the playback of the first driving scene during the simulation process includes presenting, on the teleoperations console, one or more video streams recorded by the first vehicle during the real-world driving scenario.

3 . The method of claim 1 , wherein the first driving scene is captured by the first vehicle while the first vehicle is operating in the autonomous mode.

4 . The method of claim 3 , wherein presenting the playback of the first driving scene during the simulation process includes presenting, on the teleoperations console, information relating to an autonomy system of the first vehicle, the autonomy system configured to operate the first vehicle in the autonomous mode.

5 . The method of claim 4 , wherein the playback of the first driving scene is presented by the teleoperations console to simulate the operator remotely monitoring the first vehicle using the teleoperations console while the first vehicle is operating in the autonomous mode.

6 . The method of claim 1 , wherein determining the first evaluation outcome for the operator for the first driving scene includes:

determining that the first driving scene includes an input requirement; and

based on monitoring one or more controls on the teleoperations console, determining whether the operator satisfied the input requirement of the first driving scene.

7 . The method of claim 1 , wherein determining the first evaluation outcome for the operator for the first driving scene includes monitoring one or more controls on the teleoperations console to determine that the operator provides an input via the control to simulate causing the first vehicle to exit the autonomous mode.

8 . The method of claim 1 , wherein the one or more controls on the teleoperations console includes an input to cause the vehicle being monitored to be remotely monitored via the teleoperations console to simulate performing an emergency stop maneuver.

9 . The method of claim 8 , wherein determining the first evaluation outcome for the operator for the first driving scene includes monitoring the one or more controls on the teleoperations console to determine that the operator provides an input via the control to simulate causing the first vehicle to simulate performing the emergency stop maneuver.

10 . The method of claim 1 , further comprising:

validating a teleoperations system for monitoring a fleet of vehicles based at least in part first evaluation outcome for the operator for the first driving scene, wherein the teleoperations system includes a plurality of teleoperations consoles.

11 . The method of claim 1 , wherein the teleoperations console presenting the playback of the first driving scene during the simulation process is further configurable to perform teleoperations monitoring of a second vehicle during real-world operations of the second vehicle.

12 . The method of claim 1 , further comprising:

retrieving a second driving scene during the simulation process, wherein the second driving scene is a simulated driving scenario of a virtual vehicle;

presenting the second driving scene on the teleoperations console during the simulation process, the second driving scene being presented on the teleoperations console to simulate remotely controlling the virtual vehicle during the simulated driving scenario;

determining a second evaluation outcome for the second driving scene; and

wherein certifying the operator for performing teleoperations functions for a fleet of vehicles is based further on the second evaluation outcome for the operator for the second driving scene.

13 . The method of claim 12 , further comprising selecting the first driving scene based on the real-world scenario associated with the first driving scene.

14 . The method of claim 12 , further comprising:

presenting playbacks of other driving scenes to generate evaluation outcomes for the other driving scenes; and

wherein validating the teleoperations system is further based on aggregate measures of evaluation outcomes for the other driving scenes.

15 . The method of claim 1 further including:

recording the first driving scene during operation of the first vehicle in the real-world driving scenario;

analyzing the first driving scene to determine whether the first driving scene is suitable for the playback during the simulation process; and

storing the first driving scene in the storage of the validation system when it is determined that the first driving scene is suitable for the playback during the simulation process.

16 . The method of claim 15 wherein analyzing the first driving scene to determine whether the first driving scene is suitable for the playback during the simulation process includes categorizing the first driving scene as being suitable for the playback during the simulation process for a first validation requirement, the method further including:

determining, before retrieving the first scene from the storage of the validation system, that the operator is to be evaluated based on the first validation requirement.

17 . A method comprising:

obtaining, from a first vehicle, a set of data captured by the first vehicle during a real-world driving scenario of the first vehicle, wherein the set of data is obtained by a validation system that includes a validation scene processor;

generating a first driving scene based on the set of data captured by the first vehicle during the real-world driving scenario, wherein the first driving scene is generated by the validation scene processor;

presenting playbacks of the first driving scene on a teleoperations system that includes a plurality of teleoperations consoles during a simulation process, each of the playbacks of the first driving scene being presented on each teleoperations console included in the plurality of teleoperations consoles to simulate remotely monitoring the first vehicle driving in the first driving scene based on data captured by the first vehicle during the real-world driving scenario, wherein each of the teleoperations consoles included in the plurality of teleoperations consoles includes a control configured to simulate the first vehicle being remotely monitored via each of the teleoperations consoles exiting an autonomous mode during the simulation process;

generating respective evaluation outcomes for the first driving scene for each of the teleoperations consoles of the plurality of teleoperations consoles; and

validating the teleoperations system based at least in part on an aggregate measure of the respective evaluation outcomes for the first driving scene generated for each of the teleoperations consoles of the plurality of teleoperations consoles.

18 . The method of claim 17 , further comprising:

presenting a second driving scene, wherein the second driving scene is a simulated driving scenario of a virtual vehicle; and

validating the teleoperations system based further on an aggregate measure of evaluation outcomes for the second driving scene.

19 . A teleoperations system comprising:

one or more processors;

one or more teleoperations consoles including a first teleoperations console, the first teleoperations console including one or more displays configured to present one or more live video feeds from a first vehicle that is remotely monitored via the first teleoperations console;

one or more controls for receiving inputs to generate remote commands to the first vehicle that is remotely monitored by the first teleoperations console, wherein the remote commands cause the first vehicle to take an action; and

one or more memory resources storing instructions that, when executed by the one or more processors of the first teleoperations console, cause the first teleoperations console to operate in a validation mode to:

present, on the one or more displays, a playback of a driving scene corresponding to a real-world driving scenario captured by a second vehicle during a simulation process, the playback of the driving scene being presented by the first teleoperations console to simulate monitoring the second vehicle using the first teleoperations console during the real-world driving scenario;

during presentation of the playback of the driving scene by the first teleoperations console during the simulation process, monitor the one or more controls on the first teleoperations console;

wherein the one or more controls on the first teleoperations console includes a control to simulate causing a vehicle being remotely monitored via the teleoperations console to exit an autonomous mode; and

determine, based on monitoring the one or more controls on the first teleoperations console, an evaluation outcome for the driving scene.