Multi-task machine-learned models for object intention determination in autonomous driving
Generally, the disclosed systems and methods utilize multi-task machine-learned models for object intention determination in autonomous driving applications. For example, a computing system can receive sensor data obtained relative to an autonomous vehicle and map data associated with a surrounding geographic environment of the autonomous vehicle. The sensor data and map data can be provided as input to a machine-learned intent model. The computing system can receive a jointly determined prediction from the machine-learned intent model for multiple outputs including at least one detection output indicative of one or more objects detected within the surrounding environment of the autonomous vehicle, a first corresponding forecasting output descriptive of a trajectory indicative of an expected path of the one or more objects towards a goal location, and/or a second corresponding forecasting output descriptive of a discrete behavior intention determined from a predefined group of possible behavior intentions.
1. A computing system, comprising:
one or more processors;
a machine-learned intent model configured to function as a perception system to detect and track objects near an autonomous vehicle in a single forward pass through the machine-learned intent model;
one or more non-transitory computer-readable media that store instructions for execution by the one or more processors that cause the computing system to perform operations, the operations comprising:
obtaining sensor data descriptive of an environment of an autonomous vehicle;
obtaining map data associated with a surrounding geographic environment of the autonomous vehicle;
providing, as input to the machine-learned intent model, the sensor data and the map data; and
receiving, in response to providing the sensor data and the map data as input to the machine-learned intent model, a jointly determined prediction from the machine-learned intent model for multiple outputs comprising:
at least one detection output indicative of one or more objects detected within the surrounding geographic environment of the autonomous vehicle;
a first forecasting output descriptive of a trajectory indicative of an expected path of the one or more objects towards a goal location; and
a second forecasting output descriptive of a discrete behavior intention determined from a predefined group of possible behavior intentions for the one or more objects.
2. The computing system of claim 1 , wherein the at least one detection output comprises a detection score indicative of a likelihood that a portion of the sensor data includes an object in one or more predetermined classes.
3. The computing system of claim 1 , wherein the first forecasting output descriptive of a trajectory indicative of an expected path of the one or more objects towards a goal location is represented by trajectory data comprising a sequence of bounding shapes at a plurality of time stamps.
4. The computing system of claim 3 , wherein the trajectory data comprising a sequence of bounding shapes is provided at a plurality of time stamps representative of a current location and future locations for the one or more objects.
5. The computing system of claim 1 , wherein the predefined group of possible behavior intentions for the one or more objects comprises one or more of keep lane, turn left, turn right, left change lane, right change lane, stopped, parked, and reverse driving.
6. The computing system of claim 1 , wherein the machine-learned intent model comprises a plurality of shared layers that are used at least in part for determining the at least one detection output the first forecasting output, and the second forecasting output.
7. The computing system of claim 1 , wherein the machine-learned intent model comprises a fully-convolutional deep neural network.
8. The computing system of claim 1 , wherein the sensor data comprises a birds-eye view representation of LIDAR data obtained relative to the autonomous vehicle, and wherein the map data comprises a birds-eye view representation of a surrounding geographic environment of the autonomous vehicle.
9. The computing system of claim 8 , wherein the birds-eye view representation of the LIDAR data is represented as a multi-dimensional tensor having a height dimension and a time dimension stacked into a channel dimension associated with the multi-dimensional tensor.
10. An autonomous vehicle, comprising:
one or more sensors that generate sensor data relative to the autonomous vehicle;
one or more processors;
a machine-learned convolutional neural network comprising a plurality of shared layers that determine features used to jointly determine multiple outputs of the machine-learned convolutional neural network; and
one or more non-transitory computer-readable media that store instructions for execution by the one or more processors that cause the autonomous vehicle to perform operations, the operations comprising:
obtaining the sensor data descriptive of an environment of the autonomous vehicle;
obtaining map data associated with a surrounding geographic environment of the autonomous vehicle;
providing, as input to the machine-learned convolutional neural network, the sensor data and the map data; and
receiving, in response to providing the sensor data and map data as input to the machine-learned convolutional neural network, a jointly determined prediction from the machine-learned convolutional neural network for multiple outputs comprising:
at least one detection output indicative of one or more objects detected within the surrounding environment of the autonomous vehicle;
a trajectory output associated with the one or more objects; and
an intent output associated with the one or more objects.
11. The autonomous vehicle of claim 10 , wherein the at least one detection output comprises a detection score indicative of a likelihood that a portion of the sensor data includes an object in one or more predetermined classes.
12. The autonomous vehicle of claim 10 , wherein the intent output comprises at least one discrete variable and wherein the trajectory output comprises at least one continuous variable representative of a forecasted motion of the one or more objects.
13. The autonomous vehicle of claim 10 , wherein the trajectory output associated with the one or more objects comprises a trajectory indicative of an expected path of the object towards a goal location.
14. The autonomous vehicle of claim 10 , wherein the trajectory output is represented by trajectory data comprising a sequence of bounding shapes at a plurality of time stamps.
15. The autonomous vehicle of claim 10 , wherein the intent output associated with the one or more objects comprises a discrete behavior intention determined from a predefined group of possible behavior intentions.
16. The autonomous vehicle of claim 10 , wherein the sensor data comprises a birds-eye view representation of LIDAR data obtained relative to the autonomous vehicle, and wherein the map data comprises a birds-eye view representation of a surrounding geographic environment of the autonomous vehicle.
17. The autonomous vehicle of claim 16 , wherein the birds-eye view representation of the LIDAR data is represented as a multi-dimensional tensor having a height dimension and a time dimension stacked into a channel dimension associated with the multi-dimensional tensor.
18. A computer-implemented method, comprising:
receiving sensor data descriptive of an environment of an autonomous vehicle;
receiving map data associated with a surrounding geographic environment of the autonomous vehicle;
providing as input to a machine-learned intent model, the sensor data and the map data; and
receiving in response to providing the sensor data and map data as input to the machine-learned intent model, a jointly determined prediction from the machine-learned intent model for multiple outputs comprising at least one detection output indicative of one or more objects detected within the surrounding environment of the autonomous vehicle, a first corresponding forecasting output descriptive of a trajectory indicative of an expected path of the one or more objects towards a goal location, and a second corresponding forecasting output descriptive of a discrete behavior intention determined from a predefined group of possible behavior intentions for the one or more objects.
19. The computer-implemented method of claim 18 , wherein the machine-learned intent model comprises a fully-convolutional deep neural network comprising a plurality of shared layers that are used at least in part for jointly determining the multiple outputs comprising at least one detection output indicative of one or more objects detected within the surrounding environment of the autonomous vehicle, the first corresponding forecasting output descriptive of the trajectory indicative of the expected path of the one or more objects towards the goal location, and the second corresponding forecasting output descriptive of the discrete behavior intention determined from the predefined group of possible behavior intentions for the one or more objects.
20. The computer-implemented method of claim 18 , wherein the sensor data comprises a birds-eye view representation of LIDAR data obtained relative to the autonomous vehicle, and wherein the map data comprises a birds-eye view representation of a surrounding geographic environment of the autonomous vehicle.