Agent prioritization for autonomous vehicles

1. A method performed by one or more data processing apparatus, the method comprising:

processing an input that characterizes a trajectory of a vehicle in an environment using an importance scoring model to generate an output that defines a respective importance score for each of a plurality of agents currently located at respective locations in the environment in a vicinity of the vehicle,

wherein the importance score for an agent characterizes an estimated impact of the agent on planning decisions generated by a planning system of the vehicle which plans a future trajectory of the vehicle;

selecting, for one or more of the agents, a respective prediction model for use in generating data characterizing the agent based on the importance score for the agent, comprising:

identifying, as high-priority agents, a proper subset of the plurality of agents with the highest importance scores; and

selecting, for only those agents of the plurality of agents that are identified as high-priority agents, a first prediction model for use in generating data characterizing the agent; and

generating, for each of the high-priority agents, data characterizing the agent using the first prediction model selected for the agent; and

providing the data characterizing the high-priority agents generated using the first prediction model to the planning system of the vehicle to generate the planning decisions which plan the future trajectory of the vehicle.

2. The method of claim 1 , further comprising:

obtaining historical data characterizing the trajectory of the vehicle in the environment, the historical data comprising, for each of a plurality of previous time points, data defining: (i) a spatial position in the environment occupied by the vehicle at the previous time point, and (ii) respective values of each motion parameter in a predetermined set of motion parameters, wherein the value of each motion parameter characterizes a respective feature of a motion of the vehicle at the previous time point;

generating a representation of the trajectory of the vehicle in the environment, wherein:

the representation of the trajectory of the vehicle in the environment is a concatenation of a plurality of channels;

each channel is represented as a two-dimensional array of data values;

each position in each channel corresponds to a respective spatial position in the environment;

corresponding positions in different channels correspond to the same spatial position in the environment;

the channels comprise a time channel and a respective motion channel corresponding to each motion parameter in the predetermined set of motion parameters; and

for each particular spatial position in the environment occupied by the vehicle at a particular previous time point:

the position in the time channel which corresponds to the particular spatial position defines the particular previous time point; and

for each motion channel, the position in the motion channel which corresponds to the particular spatial position defines the value of the motion parameter corresponding to the motion channel at the particular previous time point;

wherein processing an input that characterizes a trajectory of the vehicle in the environment comprises processing an input that includes the representation of the trajectory of the vehicle in the environment.

3. The method of claim 2 , wherein:

the output of the importance scoring model comprises an output channel that is represented as a two-dimensional array of data values;

each position in the output channel corresponds to a respective spatial position in the environment; and

for each spatial position in the environment that is occupied by an agent of the plurality of agents at a current time point, the position in the output channel that corresponds to the spatial position defines an importance score of the agent.

4. The method of claim 3 , further comprising, for each agent of the plurality of agents:

generating a respective feature representation of the agent, comprising:

generating one or more importance score features of the agent from the output channel;

generating one or more additional features of the agent based on sensor data captured by one or more sensors of the vehicle; and

generating the feature representation of the agent from the importance score features of the agent and the additional features of the agent;

processing the feature representation of the agent using an importance score refining model to generate a refined importance score for the agent that characterizes an estimated impact of the agent on planning decisions generated by a planning system of the vehicle which plans a future trajectory of the vehicle.

5. The method of claim 2 , wherein obtaining the respective values of each motion parameter in the predetermined set of motion parameters for a previous time point comprises one or more of:

obtaining the value of a velocity motion parameter which characterizes a velocity of the vehicle at the previous time point;

obtaining the value of an acceleration motion parameter which characterizes an acceleration of the vehicle at the previous time point; and

obtaining the value of a heading motion parameter which characterizes a heading of the vehicle at the previous time point.

6. The method of claim 2 , wherein the input processed by the importance scoring model further comprises one or more of: (i) a road-graph channel representing a known geometry of the environment, (ii) a vehicle localization channel which represents a spatial position of the vehicle in the environment at a current time point by a vehicle bounding box, and (iii) an auxiliary localization channel which represents respective spatial positions of the plurality of agents in the environment at a current time point by respective bounding boxes.

7. The method of claim 2 , further comprising generating a joint representation of trajectories of the plurality of agents in the environment in the vicinity of the vehicle, wherein the input processed by the importance scoring model further comprises the joint representation of the trajectories of the plurality of agents.

8. The method of claim 7 , wherein:

the joint representation of the trajectories of the plurality of agents in the environment comprises an auxiliary time channel and a respective auxiliary motion channel corresponding to each motion parameter in the predetermined set of motion parameters;

each channel is represented as a two-dimensional array of data values and each data value in each channel corresponds to a respective spatial position in the environment; and

for each particular spatial position in the environment occupied by a particular agent of the plurality of agents at a particular previous time point:

the data value in the auxiliary time channel which corresponds to the particular spatial position defines the particular previous time point; and

for each auxiliary motion channel, the data value in the auxiliary motion channel which corresponds to the particular spatial position defines a value of the motion parameter corresponding to the auxiliary motion channel which characterizes a respective feature of a motion of the particular agent at the particular previous time point.

9. The method of claim 1 , wherein identifying, as high-priority agents, a proper subset of the plurality of agents with the highest importance scores comprises:

identifying, as high-priority agents, a predetermined number of the plurality of agents with the highest importance scores.

10. The method of claim 1 , wherein selecting, for one or more of the agents, a respective prediction model for use in generating data characterizing the agent based on the importance score for the agent further comprises:

identifying, as low-priority agents, one or more of the agents that are not identified as high-priority agents;

selecting, for each of the low-priority agents, a second prediction model for use in generating data characterizing the agent;

wherein the first prediction model has a first number of model parameters, the second prediction model has a second number of model parameters, and the first number of model parameters is greater than the second number of model parameters; and

further comprising:

generating, for each of the low-priority agents, data characterizing the agent using the second prediction model selected for the agent; and

providing the data characterizing the low-priority agents generated using the second prediction model to the planning system in addition to the data characterizing the high-priority agents generated using the first prediction model.

11. A system comprising:

one or more computers; and

one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

processing an input that characterizes a trajectory of a vehicle in an environment using an importance scoring model to generate an output that defines a respective importance score for each of a plurality of agents currently located at respective locations in the environment in a vicinity of the vehicle,

wherein the importance score for an agent characterizes an estimated impact of the agent on planning decisions generated by a planning system of the vehicle which plans a future trajectory of the vehicle;

selecting, for one or more of the agents, a respective prediction model for use in generating data characterizing the agent based on the importance score for the agent, comprising:

identifying, as high-priority agents, a proper subset of the plurality of agents with the highest importance scores; and

selecting, for only those agents of the plurality of agents that are identified as high-priority agents, a first prediction model for use in generating data characterizing the agent; and

generating, for each of the high-priority agents, data characterizing the agent using the first prediction model selected for the agent; and

providing the data characterizing the high-priority agents generated using the first prediction model to the planning system of the vehicle to generate the planning decisions which plan the future trajectory of the vehicle.

12. The system of claim 11 , wherein the operations further comprise:

obtaining historical data characterizing the trajectory of the vehicle in the environment, the historical data comprising, for each of a plurality of previous time points, data defining: (i) a spatial position in the environment occupied by the vehicle at the previous time point, and (ii) respective values of each motion parameter in a predetermined set of motion parameters, wherein the value of each motion parameter characterizes a respective feature of a motion of the vehicle at the previous time point;

generating a representation of the trajectory of the vehicle in the environment, wherein:

the representation of the trajectory of the vehicle in the environment is a concatenation of a plurality of channels;

each channel is represented as a two-dimensional array of data values;

each position in each channel corresponds to a respective spatial position in the environment;

corresponding positions in different channels correspond to the same spatial position in the environment;

the channels comprise a time channel and a respective motion channel corresponding to each motion parameter in the predetermined set of motion parameters; and

for each particular spatial position in the environment occupied by the vehicle at a particular previous time point:

the position in the time channel which corresponds to the particular spatial position defines the particular previous time point; and

for each motion channel, the position in the motion channel which corresponds to the particular spatial position defines the value of the motion parameter corresponding to the motion channel at the particular previous time point;

wherein processing an input that characterizes a trajectory of the vehicle in the environment comprises processing an input that includes the representation of the trajectory of the vehicle in the environment.

13. The system of claim 12 , wherein:

the output of the importance scoring model comprises an output channel that is represented as a two-dimensional array of data values;

each position in the output channel corresponds to a respective spatial position in the environment; and

for each spatial position in the environment that is occupied by an agent of the plurality of agents at a current time point, the position in the output channel that corresponds to the spatial position defines an importance score of the agent.

14. The system of claim 13 , wherein the operations further comprise, for each agent of the plurality of agents:

generating a respective feature representation of each agent, comprising:

generating one or more importance score features of the agent from the output channel;

generating one or more additional features of the agent based on sensor data captured by one or more sensors of the vehicle; and

generating the feature representation of the agent from the importance score features of the agent and the additional features of the agent;

processing the feature representation of the agent using an importance score refining model to generate a refined importance score for the agent that characterizes an estimated impact of the agent on planning decisions generated by a planning system of the vehicle which plans a future trajectory of the vehicle.

15. The system of claim 12 , wherein obtaining the respective values of each motion parameter in the predetermined set of motion parameters for a previous time point comprises one or more of:

obtaining the value of a velocity motion parameter which characterizes a velocity of the vehicle at the previous time point;

obtaining the value of an acceleration motion parameter which characterizes an acceleration of the vehicle at the previous time point; and

obtaining the value of a heading motion parameter which characterizes a heading of the vehicle at the previous time point.

16. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

processing an input that characterizes a trajectory of a vehicle in an environment using an importance scoring model to generate an output that defines a respective importance score for each of a plurality of agents currently located at respective locations in the environment in a vicinity of the vehicle,

wherein the importance score for an agent characterizes an estimated impact of the agent on planning decisions generated by a planning system of the vehicle which plans a future trajectory of the vehicle;

selecting, for one or more of the agents, a respective prediction model for use in generating data characterizing the agent based on the importance score for the agent, comprising:

identifying, as high-priority agents, a proper subset of the plurality of agents with the highest importance scores; and

selecting, for only those agents of the plurality of agents that are identified as high-priority agents, a first prediction model for use in generating data characterizing the agent; and

generating, for each of the high-priority agents, data characterizing the agent using the first prediction model selected for the agent; and

providing the data characterizing the high-priority agents generated using the first prediction model to the planning system of the vehicle to generate the planning decisions which plan the future trajectory of the vehicle.

17. The one or more non-transitory computer storage media of claim 16 , wherein the operations further comprise:

obtaining historical data characterizing the trajectory of the vehicle in the environment, the historical data comprising, for each of a plurality of previous time points, data defining: (i) a spatial position in the environment occupied by the vehicle at the previous time point, and (ii) respective values of each motion parameter in a predetermined set of motion parameters, wherein the value of each motion parameter characterizes a respective feature of a motion of the vehicle at the previous time point;

generating a representation of the trajectory of the vehicle in the environment, wherein:

the representation of the trajectory of the vehicle in the environment is a concatenation of a plurality of channels;

each channel is represented as a two-dimensional array of data values;

each position in each channel corresponds to a respective spatial position in the environment;

corresponding positions in different channels correspond to the same spatial position in the environment;

the channels comprise a time channel and a respective motion channel corresponding to each motion parameter in the predetermined set of motion parameters; and

for each particular spatial position in the environment occupied by the vehicle at a particular previous time point:

the position in the time channel which corresponds to the particular spatial position defines the particular previous time point; and

for each motion channel, the position in the motion channel which corresponds to the particular spatial position defines the value of the motion parameter corresponding to the motion channel at the particular previous time point;

wherein processing an input that characterizes a trajectory of the vehicle in the environment comprises processing an input that includes the representation of the trajectory of the vehicle in the environment.

18. The one or more non-transitory computer storage media of claim 17 , wherein:

the output of the importance scoring model comprises an output channel that is represented as a two-dimensional array of data values;

each position in the output channel corresponds to a respective spatial position in the environment; and

for each spatial position in the environment that is occupied by an agent of the plurality of agents at a current time point, the position in the output channel that corresponds to the spatial position defines an importance score of the agent.

19. The one or more non-transitory computer storage media of claim 18 , wherein the operations further comprise, for each agent of the plurality of agents:

generating a respective feature representation of each agent, comprising:

generating one or more importance score features of the agent from the output channel;

generating one or more additional features of the agent based on sensor data captured by one or more sensors of the vehicle; and

generating the feature representation of the agent from the importance score features of the agent and the additional features of the agent;

processing the feature representation of the agent using an importance score refining model to generate a refined importance score for the agent that characterizes an estimated impact of the agent on planning decisions generated by a planning system of the vehicle which plans a future trajectory of the vehicle.

20. The one or more non-transitory computer storage media of claim 17 , wherein obtaining the respective values of each motion parameter in the predetermined set of motion parameters for a previous time point comprises one or more of:

obtaining the value of a velocity motion parameter which characterizes a velocity of the vehicle at the previous time point;

obtaining the value of an acceleration motion parameter which characterizes an acceleration of the vehicle at the previous time point; and

obtaining the value of a heading motion parameter which characterizes a heading of the vehicle at the previous time point.