Three-dimensional bounding box from two-dimensional image and point cloud data

1. A computer-implemented method comprising:

receiving sensor data comprising a plurality of measurements of an environment;

inputting at least a portion of the sensor data into a machine learned model;

determining, as a first feature vector and based at least in part on a first portion of the machine learned model, a first set of values associated with a measurement of the plurality of measurements;

determining, as a second feature vector and based at least in part on a second portion of the machine learned model, a second set of values associated with the plurality of measurements;

receiving image data from an image sensor;

determining, as a third feature vector and based at least in part on a third portion of the machine learned model, a third set of values associated with a portion of the image data;

combining, as a combined feature vector, the first feature vector, the second feature vector, and the third feature vector;

inputting the combined feature vector into a fourth portion of the machine learned model; and

receiving, from the fourth portion of the machine learned model, information associated with an object represented in the sensor data.

2. The computer-implemented method of claim 1 , wherein determining, as the third feature vector, the third set of values associated with the portion of the image data comprises:

determining a portion of the image data associated with the object;

determining, based at least in part on the portion of the image data associated with the object, a subset of the sensor data associated with the portion of the image data;

inputting the portion of the image data into a fifth fourth portion of the machine learned model; and

receiving, from the fifth portion of the machine learned model, an appearance feature vector;

wherein combining the first feature vector, the second feature vector, and the third feature vector comprises combining the appearance feature vector with the first feature vector and the second feature vector,

wherein inputting the sensor data into the machine learned model comprises inputting the subset of sensor data into the machine learned model, and

wherein the information associated with the object is further based on the appearance feature vector.

3. The computer-implemented method of claim 1 , wherein:

combining the first feature vector and the second feature vector comprises concatenating the first feature vector and the second feature vector.

4. The computer-implemented method of claim 1 , wherein the plurality of measurements comprises a plurality of Light Detection and Ranging (“LiDAR”) measurements.

5. The computer-implemented method of claim 1 , wherein the information associated with the object comprises a plurality of points that define a three-dimensional bounding box associated with the object.

6. The computer-implemented method of claim 5 , wherein the sensor data comprises point cloud data, the computer-implemented method further comprising:

determining, for a first point in the point cloud data, a first set of offsets corresponding to first estimated positions of corners of a first candidate three-dimensional bounding box relative to the first point; and

determining a confidence value associated with the first candidate three-dimensional bounding box.

7. The computer-implemented method of claim 6 , further comprising:

determining, for a second point of the plurality of points, a second set off offsets and a second confidence score, the second set of offsets corresponding to second estimated positions of the corners of a second candidate three-dimensional bounding box relative to the second point,

wherein the plurality of points that define the three-dimensional bounding box correspond to the first estimated positions based on the first confidence score being higher than the second confidence score.

8. The computer-implemented method of claim 1 , wherein the third portion of the machine learned model is trained using a regression loss.

9. The computer-implemented method of claim 1 , further comprising:

controlling an autonomous vehicle to navigate relative to the object.

10. The computer-implemented method of claim 1 , wherein:

the first feature vector comprises a local feature vector extracted from a first processing algorithm; and

the second feature vector comprises a global feature vector extracted from a second processing algorithm.

11. A system comprising:

one or more processors; and

non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the system to:

input sensor data into a machine learned model, the sensor data including a plurality of measurements;

determine, based on a first portion of the machine learned model, a first feature vector, the first feature comprising a first set of values associated with a first measurement of the plurality of measurements;

determine, based on a second portion of the machine learned model, a second feature vector, the second feature vector comprising a second set of values associated with the plurality of measurements;

receive image data from an image sensor;

determine, based on a third portion of the machine learned model, a third feature vector, the third feature vector comprising a third set of values associated with a portion of the image data;

combine the first feature vector, the second feature vector, and the third feature vector as a combined feature vector;

input the combined feature vector into a fourth portion of the machine learned model; and

receive, from the fourth portion of the machine learned model, information associated with an object represented in the sensor data.

12. The system of claim 11 , wherein the instructions to determine the third feature vector comprise further instructions to further cause the system to:

determine a portion of the image data associated with the object;

determine a subset of the sensor data associated with the portion of the image data;

input the portion of the image data into a fifth portion of the machine learned model; and

receive, from the fourth portion of the machine learned model, an appearance feature vector comprising a third set of values;

wherein combining the first feature vector, the second feature vector, and the third feature vector comprises combining the appearance feature vector with the first feature vector and the second feature vector, and

wherein the information associated with the object is further based on the appearance feature vector.

13. The system of claim 11 , wherein the information associated with the object comprises a plurality of points that define a three-dimensional bounding box associated with the object.

14. The system of claim 13 , wherein the sensor data comprises point cloud data, the instructions further causing the system to:

determine, for a first point in the point cloud data, a first set of offsets corresponding to estimated positions of corners of a candidate three-dimensional bounding box relative to the first point; and

determine a confidence value associated with the first candidate three-dimensional bounding box.

15. The system of claim 14 , wherein the plurality of points that define the three-dimensional bounding box is based at least in part on the estimated positions and the confidence value.

16. A system comprising:

an autonomous vehicle configured to operate in an environment;

a sensor configured to generate point cloud data corresponding to the environment;

an image sensor configured to capture image data of the environment;

one or more processors: and

non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the system to perform acts comprising:

inputting sensor data into a machine learned model, the sensor data including a plurality of measurements;

determining, based on a first portion of the machine learned model, a first feature vector, the first feature comprising a first set of values associated with a first measurement of the plurality of measurements;

determining, based on a second portion of the machine learned model, a second feature vector, the second feature vector comprising a second set of values associated with the plurality of measurements;

receiving image data from the image sensor;

determining, as a third feature vector and based at least in part on a third portion of the machine learned model, a third set of values associated with a portion of the image data;

combining the first feature vector, the second feature vector, and the third feature vector as a combined feature vector;

inputting the combined feature vector into a fourth portion of the machine learned model; and

receiving, from the fourth portion of the machine learned model, information associated with an object represented in the sensor data.

17. The system of claim 16 , wherein determining the third set of values comprises:

determining a portion of the image data associated with the object;

determining a subset of the sensor data associated with the portion of the image data;

inputting the portion of the image data into a fifth portion of the machine learned model; and

receiving, from the fifth portion of the machine learned model, an appearance feature vector comprising a third set of values associated with the image data;

wherein combining the first feature vector, the second feature vector, and the third feature vector comprises combining the appearance feature vector with the first feature vector and the second feature vector,

wherein inputting the sensor data into the machine learned model comprises inputting the subset of sensor data into the machine learned model, and

wherein the information associated with the object is further based on the appearance feature vector.

18. The system of claim 16 , wherein:

the first portion of the machine learned model comprises a first processing algorithm; and

the second portion of the machine learned model comprises a second processing algorithm.

19. The system of claim 16 , wherein the information associated with the object comprises a three-dimensional bounding box.

20. The system of claim 19 , wherein:

the sensor data comprises a point cloud generated by a LiDAR sensor; and

the information associated with the object comprises a plurality of offsets associated with a point in the point cloud and a confidence value associated with the plurality of offsets, the plurality of offsets corresponding to corners of the three-dimensional bounding box.