6D object pose estimation with 2D and 3D pointwise features
Methods and systems for determining a 6D pose of an object in an image are disclosed. In embodiments, an input image is received from a sensor, wherein the input image includes an object in the image. A trained image encoder transforms the input image into a normal map and an instance segmentation map. The normal map is encoded with pointwise 2D features. A 3D CAD model is selected from memory that resembles the object in the image. The 3D CAD model is encoded with pointwise 3D features. The pointwise 2D features are matched with the pointwise 3D features to obtain correspondences between the 2D features and the 3D features. The 6D pose of the object is then determined based on the correspondences.
1 . A computer-implemented method for determining a 6D pose of an object in an image, the computer-implemented method comprising:
receiving an input image from a sensor, wherein the input image includes an object;
via a trained image encoder, transforming the input image into a normal map and an instance segmentation map, wherein the image encoder is configured to generate both the normal map and the instance segmentation map from the same input image;
encoding the normal map with per-pixel 2D features;
supplying both the normal map and the instance segmentation map, as generated by the image encoder, as joint inputs to a deep neural network;
executing the deep neural network to select a three-dimensional CAD model from memory that resembles the object in the image;
encoding the three-dimensional CAD model with per-pixel 3D features by:
estimating by the deep neural network based on the normal map and the instance segmentation man, a scale variable (S) indicative of a density of per-pixel 2D features of the object in the input image, and
providing the scale variable (S) as an input to a three-dimensional graphical neural network (GNN 3D) that encodes the three-dimensional CAD model with per-pixel 3D features, the GNN 3D being configured to adapt a granularity of feature aggregation in the three-dimensional CAD model based on the scale variable (S);
matching the per-pixel 2D features with the per-pixel 3D features to obtain correspondences between the per-pixel 2D features and the per-pixel 3D features; and
determining a 6D pose of the object based on the correspondences.
2 . The computer-implemented method of claim 1 , wherein the encoding of the normal map is performed utilizing a two-dimensional graphical neural network (GNN 2D).
3 . The computer-implemented method of claim 1 , wherein the selecting the three-dimensional CAD model from memory is performed using a deep neural network (DNN).
4 . The computer-implemented method of claim 3 , wherein the DNN includes both a classification function and a scale estimation function.
5 . The computer-implemented method of claim 3 , wherein the normal map and the instance segmentation map are inputs to the DNN such that the DNN selects the three-dimensional CAD model from memory based on the inputs.
6 . The computer-implemented method of claim 1 , wherein the encoding the three-dimensional CAD model with the per-pixel 3D features includes placing weights on the per-pixel 3D features based on an observed distance of the object from the sensor.
7 . The computer-implemented method of claim 1 , wherein the sensor is a camera, a radar sensor, a lidar sensor, or a sonar sensor.
8 . A system for determining a 6D pose of an object in an image utilizing image processing, the system comprising:
an input interface configured to receive an input image from a sensor;
a processor in communication with the input interface and programmed to:
utilize an image encoder to convert the input into a normal map and an instance segmentation map,
encode the normal map with per-pixel 2D features,
select a three-dimensional CAD model from memory associated with an object in the input image,
encode the three-dimensional CAD model with per-pixel 3D features by:
estimating, by the deep neural network based on the normal map and the instance e segmentation map, a scale variable (S) indicative of a density of per-pixel 2D features of the object in the input image, and
providing the scale variable (S) as an input to a three-dimensional graphical neural network (GNN 3D) that encodes the three-dimensional CAD model with per-pixel 3D features, the GNN 3D being configured to adapt a granularity of feature aggregation in the three-dimensional CAD model based on the scale variable (S),
match the per-pixel 3D features with the per-pixel 3D features to obtain correspondences therebetween, and
determine a 6D pose of the object based on the correspondences.
9 . The system of claim 8 , wherein the processor is further programmed to encode the normal map utilizing a two-dimensional graphical neural network (GNN 2D).
10 . The system of claim 8 , wherein the processor is further programmed to select the three-dimensional CAD model from memory using a deep neural network (DNN).
11 . The system of claim 10 , wherein the normal map and the instance segmentation map are inputs to the DNN such that the processor is programmed to select, via the DNN, the three-dimensional CAD model from memory based on the normal map and the instance segmentation map.
12 . The system of claim 8 , wherein the processor is further programmed to place weights on the per-pixel 3D features when encoding the three-dimensional CAD model with the per-pixel 3D features, wherein the weights are based on an observed distance of the object from the sensor.
13 . A computer-implemented method for determining a 6D pose of an object in an image, the computer-implemented method comprising:
receiving an input image from a sensor, wherein the input image includes an object in the image;
via a trained image encoder, producing a normal map corresponding to the image;
via the trained image encoder, producing an instance segmentation map corresponding to the image, wherein the image encoder is configured to generate both the normal map and the instance segmentation map from the same input image;
determining local per-pixel features of the object in the normal map;
selecting a three-dimensional CAD model from memory that resembles the object in the image;
estimating by a deep neural network based on the normal map and the instance segmentation map, a scale variable (S) indicative of a density of local per-pixel features of the object in the input image;
determining local per-pixel features of the three-dimensional CAD model by providing the scale variable (S) as an input to a three-dimensional graphical neural network (GNN 3D) that encodes the three-dimensional CAD model with per-pixel 3D features, the GNN 3D being configured to adapt a granularity of feature aggregation in the three-dimensional CAD model based on the scale variable (S);
matching the local per-pixel features of the object in the normal map with the local per-pixel features of the three-dimensional CAD model to find correspondences; and
sending control signals to control a robot based on the correspondences.
14 . The computer-implemented method of claim 13 , wherein the step of matching includes placing weights on the local per-pixel features in the normal map or the local per-pixel features in the three-dimensional CAD model based on a density of those local per-pixel features.
15 . The computer-implemented method of claim 13 , wherein:
the step of determining the local per-pixel features of the object in the normal map is performed utilizing a first machine-learning model;
the step of selecting the three-dimension CAD from memory is performed utilizing a second machine-learning model; and
the step of determining local per-pixel features of the three-dimensional CAD model is performed utilizing a third machine-learning model.
16 . The computer-implemented method of claim 15 , wherein:
the third machine-learning model determines at least some of the local per-pixel features of the three-dimensional CAD model based on a density of the local per-pixel features of the object in the normal map.
17 . The computer-implemented method of claim 1 , wherein the GNN 3D is configured to adapt the granularity of feature aggregation in the three-dimensional CAD model based on the scale variable (S) such that, for denser per-pixel 2D features, higher weights are assigned to closer neighboring nodes in the 3D graph, and for sparser per-pixel 2D features, higher weights are assigned to further neighboring nodes.
18 . The system of claim 8 , wherein the GNN 3D is configured to adapt the granularity of feature aggregation in the three-dimensional CAD model based on the scale variable (S) such that, for denser per-pixel 2D features, higher weights are assigned to closer neighboring nodes in the 3D graph, and for sparser per-pixel 2D features, higher weights are assigned to further neighboring nodes.