Systems and methods for object detection including pose and size estimation
The present disclosure is directed to systems and methods for performing object detection and pose estimation in 3D from 2D images. Object detection can be performed by a machine-learned model configured to determine various object properties. Implementations according to the disclosure can use these properties to estimate object pose and size.
1 . A computer-implemented method for three-dimensional (3D) object detection in two-dimensional (2D) images, the method comprising:
obtaining, by one or more computing devices, a 2D image that includes an object, wherein the 2D image comprises a plurality of pixels;
inputting, by the one or more computing devices, the 2D image into a machine-learned object detection model comprising a plurality of heads, wherein the plurality of heads comprise:
a first head configured to generate a centroid heat map, wherein the centroid heat map provides a respective heat value for each of the plurality of pixels, and wherein the respective heat value for each pixel describes a probability that a centroid of the object is depicted by such pixel; and
a second head configured to generate a plurality of distance fields, wherein each distance field provides a respective distance value for each of contributing pixels of the plurality of pixels, and wherein the respective distance value for each pixel of the contributing pixels of the plurality of pixels indicates a distance to one of a plurality of vertices associated with a bounding box for the object, wherein each distance field provides the respective distance value for each of the contributing pixels of the plurality of pixels that has a heat value above a predetermined value;
generating, by the one or more computing devices, the centroid heat map with the first head of the machine-learned object detection model;
generating, by the one or more computing devices, the plurality of distance fields with the second head of the machine-learned object detection model; and
determining, by the one or more computing devices and based at least in part on the centroid heat map and the plurality of distance fields, a dataset comprising 3D coordinates for the bounding box associated with the object in the 2D image.
2 . The computer-implemented method of claim 1 , wherein
determining, by the one or more computing devices and based at least in part on the centroid heat map and the plurality of distance fields, the dataset comprising the 3D coordinates for the bounding box comprises:
identifying, by the one or more computing devices, the centroid of the object based at least in part on the centroid heat map;
applying, by the one or more computing devices, a voting scheme on the plurality of distance fields to identify a plurality of 2D projections of the vertices of the bounding box in the 2D image; and
reconstructing, by the one or more computing devices, the 3D coordinates for defining the bounding box from the plurality of 2D projections of the vertices.
3 . The computer-implemented method of claim 1 , wherein the plurality of heads further comprise:
a third head configured to generate a segmentation mask the object, wherein the segmentation mask provides a binary response for each pixel in the plurality of pixels indicating each pixel as associated with either background or the object.
4 . The computer-implemented method of claim 1 , wherein the respective heat value for each of the plurality of pixels is determined using a bivariate normal distribution.
5 . The computer-implemented method of claim 1 , wherein generating the plurality of distance fields is based at least in part on the respective heat value for each of the plurality of pixels.
6 . The computer-implemented method of claim 1 , wherein the dataset comprises x, y, z-coordinates for each vertex in the plurality of vertices.
7 . The computer-implemented method of claim 6 , wherein the x, y, z-coordinates for each vertex in the plurality of vertices define one or more of: a 3D location of the object, an object orientation, and an object size.
8 . The computer-implemented method of claim 1 , further comprising:
acquiring, by the one or more computing devices, one or more camera intrinsics associated with the 2D image, augmented reality data associated with the 2D image, or a combination, and wherein determining the 3-D coordinates comprises generating a physical scale based at least in part on the augmented reality data associated with the 2D data, the one or more camera intrinsics, or both.
9 . The computer-implemented method of claim 1 , wherein the machine-learned object detection model is trained using the dataset comprising:
one or more augmented images, wherein, optionally, at least one augmented image in the one or more augmented images includes a segmentation label defining each pixel in the one augmented image as either background or associated with the object, and wherein each augmented image is generated using an augmented reality engine.
10 . The computer-implemented method of claim 1 , wherein generating each augmented image comprises:
obtaining, by one or more computing devices, a three-dimensional model of a virtual object;
obtaining, by the one or more computing devices, data comprising one or more image frames that depict a scene;
determining, by the one or more computing devices, a position and an orientation for the virtual object within the scene;
generating, by the one or more computing devices and using an augmented reality rendering engine, an augmented image that depicts the virtual object within the scene at the position and the orientation;
associating, by the one or more computing devices, a label with the augmented image, wherein the label at least one of: identifies the virtual object, indicates the position of the virtual object within the scene, indicates the orientation of the virtual object within the scene, indicates the segmentation label.
11 . The computer-implemented method of claim 1 , wherein the second head is trained based at least in part on a masked mean squared error loss between prediction and ground truth, and wherein a mask is determined based on the heat values defined by the centroid heatmap.
12 . The computer-implemented method of claim 1 , wherein the method is performed substantially in real-time.
13 . The computer-implemented method of claim 1 , wherein the machine-learned object detection model comprises an encoder-decoder backbone, and wherein the encoder-decoder backbone is configured to map the 2D image into a lower dimensional space prior to performing one or more learning tasks and map the results of the one or more learning tasks to a higher dimensional space to extract output of the plurality of heads.
14 . The computer-implemented method of claim 1 , wherein the machine-learned object detection model is a fully convolutional network.
15 . The computer-implemented method of claim 1 , wherein inputting the 2D image further comprises:
inputting, by the one or more computing devices, the 2D image into the machine-learned object detection model that comprises a multi-headed model.
16 . The computer-implemented method of claim 1 , wherein each pixel of the contributing pixels of the plurality of pixels contributes to computing coordinates of one of vertices associated with the bounding box for the object based on the predetermined value that is a value of a response of the pixel in the heat map.
17 . A computing system comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising:
obtaining a 2D image that includes an object, wherein the 2D image comprises a plurality of pixels;
inputting the 2D image into a machine-learned object detection model comprising a plurality of heads, wherein the plurality of heads comprise:
a first head configured to generate a centroid heat map, wherein the centroid heat map provides a respective heat value for each of the plurality of pixels, and wherein the respective heat value for each pixel describes a probability that a centroid of the object is depicted by such pixel; and
a second head configured to generate a plurality of distance fields, wherein each distance field provides a respective distance value for each of contributing pixels of the plurality of pixels, wherein the respective distance value for each pixel of the contributing pixels of the plurality of pixels indicates a distance to one of a plurality of vertices associated with a bounding box for the object, wherein each distance field provides the respective distance value for each of the contributing pixels of the plurality of pixels that has a heat value above a predetermined value;
generating the centroid heat map with the first head of the machine-learned object detection model;
generating the plurality of distance fields with the second head of the machine-learned object detection model; and
determining, based at least in part on the centroid heat map and the plurality of distance fields, a dataset comprising 3D coordinates for the bounding box associated with the object in the 2D image.
18 . The computing system of claim 17 , wherein
determining, by the one or more computing devices and based at least in part on the centroid heat map and the plurality of distance fields, the dataset comprising the 3D coordinates for the bounding box comprises:
identifying, by the one or more computing devices, the centroid of the object based at least in part on the centroid heat map;
applying, by the one or more computing devices, a voting scheme on the plurality of distance fields to identify a plurality of 2D projections of the vertices of the bounding box in the 2D image; and
reconstructing, by the one or more computing devices, the 3D coordinates for defining the bounding box from the plurality of 2D projections of the vertices.
19 . The computing system of claim 17 , wherein the first head comprises a heat map head, and the second head comprises a distance field head.
20 . The computing system of claim 17 , wherein each pixel of the contributing pixels of the plurality of pixels contributes to computing coordinates of one of vertices associated with the bounding box for the object based on the predetermined value that is a value of a response of the pixel in the heat map.