Multi-task machine-learned models for object intention determination in autonomous driving
Generally, the disclosed systems and methods utilize multi-task machine-learned models for object intention determination in autonomous driving applications. For example, a computing system can receive sensor data obtained relative to an autonomous vehicle and map data associated with a surrounding geographic environment of the autonomous vehicle. The sensor data and map data can be provided as input to a machine-learned intent model. The computing system can receive a jointly determined prediction from the machine-learned intent model for multiple outputs including at least one detection output indicative of one or more objects detected within the surrounding environment of the autonomous vehicle, a first corresponding forecasting output descriptive of a trajectory indicative of an expected path of the one or more objects towards a goal location, and/or a second corresponding forecasting output descriptive of a discrete behavior intention determined from a predefined group of possible behavior intentions.
1. A method for detecting and forecasting an actor in an environment of an autonomous vehicle, comprising:
obtaining sensor data descriptive of the environment of the autonomous vehicle, the environment containing the actor;
obtaining map data associated with the environment; and
processing the sensor data and the map data with a machine-learned model comprising (i) one or more shared layers that are trained to generate intermediate features and (ii) one or more detection layers and one or more forecasting layers, wherein:
the one or more detection layers process the intermediate features to generate a detection output descriptive of the actor in the environment, and
the one or more forecasting layers process the intermediate features to generate:
a first forecasting output descriptive of a trajectory indicative of an expected path of the actor towards a goal location, and
a second forecasting output descriptive of a discrete behavior intention determined from a predefined group of possible behavior intentions for the actor.
2. The method of claim 1 , wherein the one or more detection layers and one or more forecasting layers are positioned structurally after the one or more shared layers.
3. The method of claim 1 , wherein:
the machine-learned model comprises a convolutional neural network; and
at least one of the (i) one or more shared layers that are trained to generate intermediate features and (ii) one or more detection layers and one or more forecasting layers comprises a convolutional layer of the convolutional neural network.
4. The method of claim 1 , wherein the detection output comprises a detection score indicative of a likelihood of the actor being in one of a plurality of predetermined classes.
5. The method of claim 1 , wherein the detection output comprises a bounding shape associated with the actor.
6. The method of claim 1 , wherein the first forecasting output is represented by trajectory data comprising a sequence of bounding shapes at a plurality of time stamps.
7. The method of claim 1 , wherein the predefined group of possible behavior intentions for the actor comprises at least one of keep lane, turn left, turn right, left change lane, right change lane, stopped, parked, or reverse driving.
8. The method of claim 1 , wherein processing the sensor data and the map data with the machine-learned model comprises processing a fused representation of a given view of the sensor data obtained relative to the autonomous vehicle and the map data in the given view.
9. The method of claim 8 , wherein the given view of the sensor data and the map data comprises a birds-eye view.
10. The method of claim 8 , wherein the given view of the sensor data is represented as a multi-dimensional tensor having at least one of a height dimension or a time dimension stacked into a channel dimension with the multi-dimensional tensor.
11. An autonomous vehicle (AV) control system, comprising:
one or more processors; and
one or more non-transitory computer-readable media that store instructions for execution by the one or more processors that cause the AV control system to perform operations, the operations comprising:
obtaining sensor data descriptive of an environment of the autonomous vehicle, the environment containing an actor;
obtaining map data associated with the environment; and
processing the sensor data and the map data with a machine-learned model comprising (i) one or more shared layers that are trained to generate intermediate features and (ii) one or more detection layers and one or more forecasting layers, wherein:
the one or more detection layers process the intermediate features to generate a detection output descriptive of the actor in the environment, and
the one or more forecasting layers process the intermediate features to generate:
a first forecasting output descriptive of a trajectory indicative of an expected path of the actor towards a goal location, and
a second forecasting output descriptive of a discrete behavior intention determined from a predefined group of possible behavior intentions for the actor.
12. The AV control system of claim 11 , wherein the one or more detection layers and one or more forecasting layers are positioned structurally after the one or more shared layers.
13. The AV control system of claim 11 , wherein:
the machine-learned model comprises a convolutional neural network; and
at least one of the (i) one or more shared layers that are trained to generate intermediate features and (ii) one or more detection layers and one or more forecasting layers comprises a convolutional layer of the convolutional neural network.
14. The AV control system of claim 11 , wherein processing the sensor data and the map data with the machine-learned model comprises processing a fused representation of a given view of the sensor data obtained relative to the autonomous vehicle and the map data in the given view.
15. The AV control system of claim 11 , wherein the detection output comprises a detection score indicative of a likelihood of the actor being in one of a plurality of predetermined classes.
16. The AV control system of claim 11 , wherein the first forecasting output is represented by trajectory data comprising a sequence of bounding shapes at a plurality of time stamps.
17. The AV control system of claim 11 , wherein the predefined group of possible behavior intentions for the actor comprises at least one of keep lane, turn left, turn right, left change lane, right change lane, stopped, parked, or reverse driving.
18. An autonomous vehicle, comprising:
one or more sensors that generate sensor data descriptive of an environment of the autonomous vehicle;
one or more processors; and
one or more non-transitory computer-readable media that store instructions for execution by the one or more processors that cause the one or more processors to perform operations, the operations comprising:
processing the sensor data and map data associated with the environment with a machine-learned model comprising (i) one or more shared layers that are trained to generate intermediate features and (ii) one or more detection layers and one or more forecasting layers, wherein:
the one or more detection layers process the intermediate features to generate a detection output descriptive of an actor in the environment, and
the one or more forecasting layers process the intermediate features to generate:
a first forecasting output descriptive of a trajectory indicative of an expected path of the actor towards a goal location, and
a second forecasting output descriptive of a discrete behavior intention determined from a predefined group of possible behavior intentions for the actor.
19. The autonomous vehicle of claim 18 , wherein the one or more detection layers and one or more forecasting layers are positioned structurally after the one or more shared layers.
20. The autonomous vehicle of claim 18 , wherein:
the machine-learned model comprises a convolutional neural network; and
at least one of the (i) one or more shared layers that are trained to generate intermediate features and (ii) one or more detection layers and one or more forecasting layers comprises a convolutional layer of the convolutional neural network.