Agent trajectory prediction using target locations

1. A method performed by one or more computers, the method comprising:

obtaining scene context data characterizing an environment, the scene context data comprising data that characterizes a trajectory of an agent in a vicinity of a vehicle in an environment up to a current time point;

identifying a plurality of initial target locations in the environment;

generating, for each of a plurality of target locations that each corresponds to one of the initial target locations, (i) a respective predicted coordinate offset between the target location and the corresponding initial target location and (ii) a respective predicted likelihood score that represents a likelihood that the target location will be an intended final location for a future trajectory of the agent starting from the current time point;

selecting a first subset of the target locations based on the respective predicted likelihood scores of the target locations;

for each target location in the first subset of the target locations, generating a predicted future trajectory for the agent that is a prediction of the future trajectory of the agent given that the target location is the intended final location for the future trajectory;

selecting, as likely future trajectories of the agent starting from the current time point, one or more of the predicted future trajectories; and

controlling the vehicle using the one or more selected predicted future trajectories.

2. The method of claim 1 , wherein the scene context data further comprises data that characterizes respective trajectories of one or more other agents in the environment up to the current time point.

3. The method of claim 1 , wherein selecting the first subset of the target locations comprises selecting a threshold number of target locations having the highest respective predicted likelihood scores.

4. The method of claim 1 , wherein identifying the plurality of initial target locations in the environment comprises:

dividing a region of the environment that is in a vicinity of the agent into a grid having a plurality of grid points; and

selecting at least some of the grid points in the grid as initial target locations.

5. The method of claim 1 , wherein identifying the plurality of initial target locations in the environment comprises:

obtaining road graph data identifying lanes on one or more roads in the environment; and

sampling, as initial target locations, points from the identified lanes.

6. The method of claim 5 , wherein sampling points from the identified lanes comprises:

for each identified lane, sampling points on a lane center line of the identified lane.

7. The method of claim 1 , further comprising:

processing the scene context data using a context encoding neural network to generate an encoded representation of the scene context data.

8. The method of claim 7 , wherein generating the predicted future trajectory for the agent that is a prediction of the future trajectory of the agent given that the target location is the intended final location for the future trajectory comprises:

processing the encoded representation and data specifying the target location using a motion estimation neural network to generate the predicted future trajectory.

9. The method of claim 7 , wherein generating, for each of a plurality of target locations that each corresponds to one of the initial target locations, (i) the respective predicted coordinate offset between the target location and the corresponding initial target location (ii) the respective predicted likelihood score comprises:

processing the encoded representation and data specifying the corresponding initial target location using a target prediction neural network to generate a target prediction output that comprises the respective predicted likelihood score.

10. The method of claim 1 , wherein the data defining the offset comprises, for each coordinate of the initial target location, parameters of respective probability distributions over offsets for the coordinate, and wherein the method further comprises generating the offset by sampling from the respective probability distributions.

11. The method of claim 1 , wherein the data defining the offset comprises, for each coordinate of the initial target location, a respective offset for the coordinate.

12. The method of claim 7 , wherein selecting, as likely future trajectories of the agent starting from the current time point, one or more of the predicted future trajectories comprises:

generating a respective refined score for each of the predicted future trajectories that represents a likelihood that the predicted future trajectory will be the future trajectory of the agent after the current time point given the scene context data; and

selecting one or more of the predicted future trajectories based on the refined scores.

13. The method of claim 12 , wherein generating a respective refined score for each of the predicted future trajectories that represents a likelihood that the predicted future trajectory will be the future trajectory of the agent after the current time point given the scene context data comprises:

processing the encoded representation and the predicted future trajectory using a trajectory scoring neural network to generate the respective refined score for the predicted trajectory.

14. The method of claim 13 , wherein selecting one or more of the predicted future trajectories comprises:

ordering the predicted future trajectories according to the respective refined scores; and

for each predicted future trajectory starting from a first predicted trajectory in the order and until a specified number of trajectories have been selected:

determining whether the predicted trajectory is more than a threshold distance from all trajectories that have already been selected according to a distance metric; and

selecting the predicted trajectory only if the predicted trajectory is more than a threshold distance from all trajectories that have already been selected according to a distance metric.

15. A system comprising:

one or more computers; and

one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform:

obtaining scene context data characterizing an environment, the scene context data comprising data that characterizes a trajectory of an agent in a vicinity of a vehicle in an environment up to a current time point;

identifying a plurality of initial target locations in the environment;

identifying a plurality of initial target locations in the environment;

generating, for each of a plurality of target locations that each corresponds to one of the initial target locations, (i) a respective predicted coordinate offset between the target location and the corresponding initial target location and (ii) a respective predicted likelihood score that represents a likelihood that the target location will be an intended final location for a future trajectory of the agent starting from the current time point;

selecting a first subset of the target locations based on the respective predicted likelihood scores of the target locations;

for each target location in the first subset of the target locations, generating a predicted future trajectory for the agent that is a prediction of the future trajectory of the agent given that the target location is the intended final location for the future trajectory;

selecting, as likely future trajectories of the agent starting from the current time point, one or more of the predicted future trajectories; and

controlling the vehicle using the one or more selected predicted future trajectories.

16. The system of claim 15 , wherein the stored instructions cause the one or more computers to perform:

processing the scene context data using a context encoding neural network to generate an encoded representation of the scene context data;

processing the encoded representation and data specifying the corresponding initial target location using a target prediction neural network to generate a target prediction output that comprises the respective coordinate offset and the respective predicted likelihood score; and

processing the encoded representation and data specifying the target location using a motion estimation neural network to generate the predicted future trajectory.

17. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform:

obtaining scene context data characterizing an environment, the scene context data comprising data that characterizes a trajectory of an agent in a vicinity of a vehicle in an environment up to a current time point;

identifying a plurality of initial target locations in the environment;

generating, for each of a plurality of target locations that each corresponds to one of the initial target locations, (i) a respective predicted coordinate offset between the target location and the corresponding initial target location and (ii) a respective predicted likelihood score that represents a likelihood that the target location will be an intended final location for a future trajectory of the agent starting from the current time point;

selecting a first subset of the target locations based on the respective predicted likelihood scores of the target locations;

for each target location in the first subset of the target locations, generating a predicted future trajectory for the agent that is a prediction of the future trajectory of the agent given that the target location is the intended final location for the future trajectory;

selecting, as likely future trajectories of the agent starting from the current time point, one or more of the predicted future trajectories;

and controlling the vehicle using the one or more selected predicted future trajectories.

18. The one or more computer-readable storage media of claim 17 , wherein the stored instructions cause the one or more computers to perform:

processing the scene context data using a context encoding neural network to generate an encoded representation of the scene context data;

processing the encoded representation and data specifying the corresponding initial target location using a target prediction neural network to generate a target prediction output that comprises the respective coordinate offset and the respective predicted likelihood score; and

processing the encoded representation and data specifying the target location using a motion estimation neural network to generate the predicted future trajectory.