Exception situation playback for tele-operators

1. A method for resolving an exception situation in autonomous driving, comprising:

receiving an assistance request to resolve the exception situation from an autonomous vehicle (AV);

identifying a solution to the exception situation;

forwarding the solution to a tele-operator;

receiving a request for playback data from the tele-operator;

receiving, from the AV, the playback data, wherein the playback data comprises snapshots n i of data related to autonomous driving stored at the AV at respective consecutive times t i , wherein i=1, . . . , N;

displaying, to the tele-operator, a time-warp of the playback data such that a first portion of the playback data including activity related to the exception situation is played at a first speed and a second portion of the playback data not including any activity related to the exception situation is played at a second speed that is faster than the first speed;

highlighting, in a playback of the playback data and for display to the tele-operator, only real world objects causing the exception situation by:

starting from a last snapshot in the playback data, identifying an object that caused the exception situation;

identifying the object in every previous snapshot of the snapshots; and

highlighting the object during the playback;

obtaining, from the tele-operator, a validated solution based on the tele-operator using the playback data; and

autonomously operating the AV according to the validated solution.

2. The method of claim 1 , further comprising:

prior to obtaining the validated solution from the tele-operator, sequentially displaying, to the tele-operator, images from a front facing camera of the AV.

3. The method of claim 1 , further comprising:

receiving, from the tele-operator, a perspective-shift command including a perspective; and

sequentially displaying, to the tele-operator, images from the perspective.

4. The method of claim 1 , further comprising:

receiving, from the tele-operator, an annotation of the playback data, the annotation describing the exception situation and a performance of the AV with respect to the exception situation.

5. The method of claim 1 , further comprising:

transmitting the validated solution to the AV.

6. A method for recording exception situations by an autonomous vehicle (AV), comprising:

storing, in a memory, at consecutive times t i respective snapshots n i of data related to autonomous driving, wherein i=1, . . . , N;

identifying an exception situation occurring at time t n+1 ;

in response to determining that an assistance of a tele-operator is not required, replacing a first snapshot n 1 with a new snapshot corresponding to the time t n+1 ; and

in response to determining that the assistance of the tele-operator is required:

freezing the snapshots n i , for i=1, . . . , N;

transmitting playback data comprising the snapshots n i to an autonomous system;

notifying, via a notification, an agent proximal to the AV of the exception situation, wherein the notification comprises an identifier of the AV and an identification of the exception situation;

displaying, to the tele-operator, at least a portion of the playback data;

highlighting, while displaying the at least the portion of the playback data, real world objects causing the exception situation, wherein highlighting the real world objects causing the exception situation comprises:

starting from a last snapshot in the playback data, identifying a real world object that caused the exception situation;

identifying the real world object in every previous snapshot of the playback data; and

highlighting the real world object while displaying the at least the portion of the playback data;

receiving a validated solution from the tele-operator, wherein the tele-operator uses the snapshots n i to generate the validated solution; and

operating the AV according to the validated solution.

7. The method of claim 6 , wherein transmitting the snapshots n i to the autonomous system comprises:

transmitting the snapshots n i to the autonomous system in response to receiving a request from the autonomous system for the snapshots n i .

8. The method of claim 6 , wherein the memory is a circular buffer.

9. The method of claim 8 ,

wherein the first snapshot n 1 is stored at a location L of the circular buffer, and

wherein replacing the first snapshot n 1 with the new snapshot corresponding to the time t n+1 comprises:

replacing the first snapshot n 1 at the location L with the new snapshot.

10. The method of claim 6 , wherein the data related to autonomous driving comprises raw sensor data.

11. The method of claim 6 , wherein the data related to autonomous driving comprises processed data of raw sensor data.

12. The method of claim 6 , wherein freezing the snapshots n i , for i=1, . . . , N further comprises:

storing the new snapshot corresponding to the time t n+1 in another memory that is different than the memory.

13. A system for resolving an exception situation in autonomous driving, comprising:

a memory; and

a processor, the memory includes instructions executable by the processor to:

receive an assistance request to resolve the exception situation from an autonomous vehicle (AV);

receive a request for playback data from a tele-operator;

receive from the AV the playback data, wherein the playback data comprises snapshots n i of data related to autonomous driving stored at the AV at respective consecutive times wherein i=1, . . . , N;

notify an agent proximal to the AV of the exception situation;

receive, from the agent, agent playback data, wherein the agent freezes the agent playback data in response to being notified of the exception situation;

starting from a last snapshot in the playback data, identify a real world object that caused the exception situation;

identify the real world object in every previous snapshot of the playback data; and

highlight the real world object while displaying the playback data to the tele-operator;

obtain, from the tele-operator, a validated solution based on the tele-operator using the playback data and the agent playback data; and

operating the AV according to the validated solution.

14. The system of claim 13 , wherein the instructions further comprise instructions to:

sequentially display, to the tele-operator, images from a front facing camera of the AV.

15. The system of claim 13 , wherein the instructions further comprise instructions to:

receive, from the tele-operator, a perspective-shift command including a perspective; and

sequentially display, to the tele-operator, images from the perspective.

16. The system of claim 13 , wherein the instructions further comprise instructions to:

receive, from the tele-operator, an annotation of the playback data.

17. The system of claim 13 , wherein the instructions further comprise instructions to:

transmit the validated solution to the AV.

18. The system of claim 13 , wherein the instructions further comprise instructions to:

receive, from the tele-operator, an annotation of the playback data, subsequent to transmitting the validated solution to the AV.