Neural networks with attention al bottlenecks for trajectory planning

1. A method comprising:

obtaining input data comprising an intended route for a vehicle and data characterizing an environment in a vicinity of the vehicle, wherein the data characterizing the environment comprises dense data characterizing the environment and sparse data characterizing the environment;

processing the input data including the intended route, the dense data, and the sparse data using an input encoder neural network to generate feature data that includes a respective feature representation for each of a plurality of locations in the environment;

applying spatial attention to the feature representations to generate a respective attention weight for each of the plurality of locations;

generating a respective attended feature representation for each of the plurality of locations by applying the attention weight for the location to the feature representation for the location;

generating a bottlenecked representation of the attended feature representations that has a lower dimensionality than the feature data;

generating an unbottlenecked dense feature representation by processing only the dense data using a dense encoder neural network, wherein the sparse data comprises data characterizing dynamic objects in the vicinity of the vehicle and the dense data includes a roadmap of a road being traversed by the vehicle; and

processing, using a motion generator neural network, a motion input that includes (i) the bottlenecked representation generated from the intended route, the dense data that includes the roadmap, and the sparse data that includes the data characterizing the dynamic objects, and (ii) the unbottlenecked dense feature representation generated from only the dense data that includes the roadmap, to generate a planned future trajectory defined by a sequence of poses of the vehicle with each pose corresponding to a different future time.

2. The method of claim 1 , wherein the dense data further includes one or more of: speed limits for different areas of the road, past poses of the vehicle, or a current vehicle position.

3. The method of claim 1 , wherein the sparse data further includes data characterizing a state of traffic lights in the vicinity of the vehicle.

4. The method of claim 1 , wherein applying spatial attention to the feature representations to generate the respective attention weight for each of the plurality of locations comprises:

processing the feature representations using an atrous spatial attention layer to generate the attention weights.

5. The method of claim 4 , wherein the atrous spatial attention layer is configured to:

apply a plurality of different atrous convolutions to a feature map of the feature representations to generate a plurality of convolved feature maps, wherein each of the plurality of different atrous convolutions has a different atrous rate;

combine the convolved feature maps to generate a combined convolved feature map; and

generate the attention weights from the combined convolved feature map.

6. The method of claim 1 , wherein generating the bottlenecked representation of the attended feature representations that has a lower dimensionality than the feature data comprises:

generating a respective position-dependent representation for each of the plurality of locations by concatenating the feature representation for the location with a positional encoding for the location; and

processing the position-dependent representations using a feedforward neural network that is configured to map the position-dependent representations to a lower-dimensional latent space to generate the bottlenecked representation.

7. The method of claim 1 , further comprising:

providing a visual representation of the attention weights for presentation to a user.

8. The method of claim 1 , wherein processing, using the motion generator neural network, the motion input that includes (i) the bottlenecked representation generated from the intended route, the dense data, and the sparse data, and (ii) the unbottlenecked dense feature representation generated from only the dense data to generate the planned future trajectory comprises:

at each of a plurality of planning iterations:

processing the motion input for the planning iteration using the motion generator neural network to generate a motion output that characterizes a next point on the planned future trajectory.

9. The method of claim 8 , wherein for each of the plurality of planning iterations other than the first planning iteration, the motion input for the planning iteration further comprises:

data identifying points on the planned future trajectory generated by the motion generator neural network at previous planning iterations.

10. The method of claim 8 , wherein the motion output further comprises data identifying a predicted future pose of the vehicle, and wherein the motion input further comprises the data identifying the predicted future pose of the vehicle generated at a preceding planning iteration.

11. The method of claim 1 ,

wherein the method further comprises:

generating, from the data characterizing the dynamic objects in the vicinity of the vehicle, a respective dynamic object feature representation for each of the plurality of locations in the environment, and

wherein the feature data includes the feature representations and the dynamic object feature representations.

12. The method of claim 11 , wherein the feature data includes, for each of the plurality of locations, a concatenation of the feature representation of the location and the dynamic object feature representation.

13. The method of claim 11 , further comprising:

at each of a plurality of planning iterations, processing a dynamic input comprising the dynamic object feature representation using a perception neural network to generate a prediction characterizing future positions of the dynamic objects.

14. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

obtaining input data comprising an intended route for a vehicle and data characterizing an environment in a vicinity of the vehicle, wherein the data characterizing the environment comprises dense data characterizing the environment and sparse data characterizing the environment;

processing the input data including the intended route, the dense data, and the sparse data using an input encoder neural network to generate feature data that includes a respective feature representation for each of a plurality of locations in the environment;

applying spatial attention to the feature representations to generate a respective attention weight for each of the plurality of locations;

generating a respective attended feature representation for each of the plurality of locations by applying the attention weight for the location to the feature representation for the location;

generating a bottlenecked representation of the attended feature representations that has a lower dimensionality than the feature data;

generating an unbottlenecked dense feature representation by processing only the dense data using a dense encoder neural network, wherein the sparse data comprises data characterizing dynamic objects in the vicinity of the vehicle and the dense data includes a roadmap of a road being traversed by the vehicle; and

processing, using a motion generator neural network, a motion input that includes (i) the bottlenecked representation generated from the intended route, the dense data that includes the roadmap, and the sparse data that includes the data characterizing the dynamic objects, and (ii) the unbottlenecked dense feature representation generated from only the dense data that includes the roadmap, to generate a planned future trajectory defined by a sequence of poses of the vehicle with each pose corresponding to a different future time.

15. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

obtaining input data comprising an intended route for a vehicle and data characterizing an environment in a vicinity of the vehicle, wherein the data characterizing the environment comprises dense data characterizing the environment and sparse data characterizing the environment;

processing the input data including the intended route, the dense data, and the sparse data using an input encoder neural network to generate feature data that includes a respective feature representation for each of a plurality of locations in the environment;

applying spatial attention to the feature representations to generate a respective attention weight for each of the plurality of locations;

generating a respective attended feature representation for each of the plurality of locations by applying the attention weight for the location to the feature representation for the location;

generating a bottlenecked representation of the attended feature representations that has a lower dimensionality than the feature data;

generating an unbottlenecked dense feature representation by processing only the dense data using a dense encoder neural network, wherein the sparse data comprises data characterizing dynamic objects in the vicinity of the vehicle and the dense data includes a roadmap of a road being traversed by the vehicle; and

processing, using a motion generator neural network, a motion input that includes (i) the bottlenecked representation generated from the intended route, the dense data that includes the roadmap, and the sparse data that includes the data characterizing the dynamic objects, and (ii) the unbottlenecked dense feature representation generated from only the dense data that includes the roadmap, to generate a planned future trajectory defined by a sequence of poses of the vehicle with each pose corresponding to a different future time.

16. The system of claim 15 , wherein applying spatial attention to the feature representations to generate the respective attention weight for each of the plurality of locations comprises:

processing the feature representations using an atrous spatial attention layer to generate the attention weights.

17. The system of claim 15 , the operations further comprising:

providing a visual representation of the attention weights for presentation to a user.