Dynamic Scene Representation

1 . A computer-implemented method comprising:

receiving sensor data associated with a period of operation in an environment by at least one sensor of a vehicle, wherein the sensor data includes (i) trajectory data associated with the vehicle during the period of operation, and (ii) at least one of trajectory data associated with one or more agents in the environment during the period of operation or data associated with one or more static objects in the environment during the period of operation;

determining, at each of a series of times during the period of operation, that at least one of (i) the one or more agents or (ii) the one or more static objects is relevant to the vehicle, wherein determining that at least one of (i) the one or more agents or (ii) the one or more static objects is relevant to the vehicle is based on a likelihood that at least one of (i) the one or more agents or (ii) the one or more static objects is predicted to affect a planned future trajectory of the vehicle;

identifying, from the series of times, one or more times during the period of operation when there is a change to at least one of (i) the one or more agents or (ii) the one or more static objects determined to be relevant to the vehicle;

designating each of the one or more identified times as a boundary point that separates the period of operation into one or more scenes; and

generating a representation of the one or more scenes based on the designated boundary points, wherein each of the one or more scenes includes (i) a portion of the trajectory data associated with the vehicle, and (ii) at least one of a portion of the trajectory data associated with the one or more agents or a portion of the data associated with the one or more static objects.

2 . The computer-implemented method of claim 1 , wherein generating a representation of the one or more scenes comprises:

generating a respective representation of each of the one or more scenes that includes (i) the trajectory data for the vehicle during the scene and (ii) one or both of (a) trajectory data for at least one agent that is determined to be relevant to the planned future trajectory of the vehicle during the scene, or (b) data associated with at least one static object that is determined to be relevant to the planned future trajectory of the vehicle during the scene.

3 . The computer-implemented method of claim 2 , wherein one or both of (i) the trajectory data for the vehicle during the scene or (ii) the trajectory data for the at least one agent that is determined to be relevant to the planned future trajectory of the vehicle during the scene comprises confidence information indicating an estimated accuracy of the trajectory data.

4 . The computer-implemented method of claim 1 , wherein identifying, from the series of times, one or more times during the period of operation when there is a change to at least one of (i) the one or more agents or (ii) the one or more static objects determined to be relevant to the vehicle comprises:

determining that at least one of the one or more agents that was determined to be relevant to the vehicle is no longer relevant to the vehicle.

5 . The computer-implemented method of claim 1 , further comprising:

based on the received sensor data, deriving past trajectory data for (i) the vehicle and (ii) the one or more agents in the environment during the period of operation; and

based on the received sensor data, generating future trajectory data for (i) the vehicle and (ii) the one or more agents in the environment during the period of operation.

6 . The computer implemented method of claim 1 , wherein determining that at least one of (i) the one or more agents or (ii) the one or more static objects is relevant to the vehicle comprises predicting at least one of: (a) a likelihood that the planned future trajectory of the vehicle will intersect a predicted trajectory for the one or more agents, or (b) a likelihood that at least one of (i) the one or more agents or (ii) the one or more static objects will be located within a predetermined zone of proximity to the vehicle.

7 . The computer-implemented method of claim 1 , further comprising:

based on a selected scene included in the one or more scenes, predicting one or more alternative versions of the selected scene.

8 . The computer-implemented method of claim 7 , wherein predicting one or more alternative versions of the selected scene comprises:

generating, for the selected scene, one or more alternative versions of one or both of (i) the trajectory data for the vehicle during the scene or (ii) the trajectory data for at least one agent in the environment during the scene.

9 . The computer-implemented method of claim 1 , further comprising:

based on (i) a first scene included in the one or more scenes and (ii) a second scene included in the one or more scenes, generating a representation of a new scene comprising:

at least one of (i) trajectory data for the vehicle during the first scene or (ii) trajectory data for at least one agent in the environment during the first scene; and

at least one of (i) trajectory data for the vehicle during the second scene or (ii) trajectory data for at least one agent in the environment during the second scene.

10 . The computer-implemented method of claim 1 , wherein determining that at least one of (i) the one or more agents or (ii) the one or more static objects is relevant to the vehicle comprises determining that a probability that at least one of (i) the one or more agents or (ii) the one or more static objects will affect the planned future trajectory of the vehicle during a future time horizon exceeds a predetermined threshold probability.

11 . A non-transitory computer-readable medium comprising program instructions stored thereon that are executable to cause a computing system to:

receive sensor data associated with a period of operation in an environment by at least one sensor of a vehicle, wherein the sensor data includes (i) trajectory data associated with the vehicle during the period of operation, and (ii) at least one of trajectory data associated with one or more agents in the environment during the period of operation or data associated with one or more static objects in the environment during the period of operation;

determine, at each of a series of times during the period of operation, that at least one of (i) the one or more agents or (ii) the one or more static objects is relevant to the vehicle, wherein determining that at least one of (i) the one or more agents or (ii) the one or more static objects is relevant to the vehicle is based on a likelihood that at least one of (i) the one or more agents or (ii) the one or more static objects is predicted to affect a planned future trajectory of the vehicle;

identify, from the series of times, one or more times during the period of operation when there is a change to at least one of (i) the one or more agents or (ii) the one or more static objects determined to be relevant to the vehicle;

designate each of the one or more identified times as a boundary point that separates the period of operation into one or more scenes; and

generate a representation of the one or more scenes based on the designated boundary points, wherein each of the one or more scenes includes (i) a portion of the trajectory data associated with the vehicle, and (ii) at least one of a portion of the trajectory data associated with the one or more agents or a portion of the data associated with the one or more static objects.

12 . The computer-readable medium of claim 11 , wherein generating a representation of the one or more scenes comprises:

generating a respective representation of each of the one or more scenes that includes (i) the trajectory data for the vehicle during the scene and (ii) one or both of (a) trajectory data for at least one agent that is determined to be relevant to the planned future trajectory of the vehicle during the scene, or (b) data associated with at least one static object that is determined to be relevant to the planned future trajectory of the vehicle during the scene.

13 . The computer-readable medium of claim 12 , wherein one or both of (i) the trajectory data for the vehicle during the scene or (ii) the trajectory data for the at least one agent that is determined to be relevant to the planned future trajectory of the vehicle during the scene comprises confidence information indicating an estimated accuracy of the trajectory data.

14 . The computer-readable medium of claim 11 , wherein identifying, from the series of times, one or more times during the period of operation when there is a change to at least one of (i) the one or more agents or (ii) the one or more static objects determined to be relevant to the vehicle comprises:

determining that at least one of the one or more agents that was determined to be relevant to the vehicle is no longer relevant to the vehicle.

15 . The computer-readable medium of claim 11 , wherein the computer-readable medium further comprises program instructions stored thereon that are executable to cause the computing system to:

based on the received sensor data, deriving past trajectory data for (i) the vehicle and (ii) the one or more agents in the environment during the period of operation; and

based on the received sensor data, generating future trajectory data for (i) the vehicle and (ii) the one or more agents in the environment during the period of operation.

16 . The computer-readable medium of claim 11 , wherein determining that at least one of (i) the one or more agents or (ii) the one or more static objects is relevant to the vehicle comprises predicting at least one of: (a) a likelihood that the planned future trajectory of the vehicle will intersect a predicted trajectory for the one or more agents, or (b) a likelihood that at least one of (i) the one or more agents or (ii) the one or more static objects will be located within a predetermined zone of proximity to the vehicle.

17 . The computer-readable medium of claim 11 , wherein the computer-readable medium further comprises program instructions stored thereon that are executable to cause the computing system to:

based on a selected scene included in the one or more scenes, predicting one or more alternative versions of the selected scene.

18 . The computer-readable medium of claim 17 , wherein predicting one or more alternative versions of the selected scene comprises:

generating, for the selected scene, one or more alternative versions of one or both of (i) the trajectory data for the vehicle during the scene or (ii) the trajectory data for at least one agent in the environment during the scene.

19 . The computer-readable medium of claim 11 , wherein the computer-readable medium further comprises program instructions stored thereon that are executable to cause the computing system to:

based on (i) a first scene included in the one or more scenes and (ii) a second scene included in the one or more scenes, generating a representation of a new scene comprising:

at least one of (i) trajectory data for the vehicle during the first scene or (ii) trajectory data for at least one agent in the environment during the first scene; and

at least one of (i) trajectory data for the vehicle during the second scene or (ii) trajectory data for at least one agent in the environment during the second scene.

20 . A computing system comprising:

at least one processor;

a non-transitory computer-readable medium; and

program instructions stored on the non-transitory computer-readable medium that are executable by the at least one processor such that the computing system is capable of:

receiving sensor data associated with a period of operation in an environment by at least one sensor of a vehicle, wherein the sensor data includes (i) trajectory data associated with the vehicle during the period of operation, and (ii) at least one of trajectory data associated with one or more agents in the environment during the period of operation or data associated with one or more static objects in the environment during the period of operation;

determining, at each of a series of times during the period of operation, that at least one of (i) the one or more agents or (ii) the one or more static objects is relevant to the vehicle, wherein determining that at least one of (i) the one or more agents or (ii) the one or more static objects is relevant to the vehicle is based on a likelihood that at least one of (i) the one or more agents or (ii) the one or more static objects is predicted to affect a planned future trajectory of the vehicle;

identifying, from the series of times, one or more times during the period of operation when there is a change to at least one of (i) the one or more agents or (ii) the one or more static objects determined to be relevant to the vehicle;

designating each of the one or more identified times as a boundary point that separates the period of operation into one or more scenes; and

generating a representation of the one or more scenes based on the designated boundary points, wherein each of the one or more scenes includes (i) a portion of the trajectory data associated with the vehicle, and (ii) at least one of a portion of the trajectory data associated with the one or more agents or a portion of the data associated with the one or more static objects.