Systems and methods for single-shot multi-object 3D shape reconstruction and categorical 6D pose and size estimation
System, methods, and other embodiments described herein relate to single-shot multi-object three-dimensional (3D) shape reconstruction and categorical six-dimensional (6D) pose and size estimation. In one embodiment, a method includes inferring a heatmap based upon a feature pyramid, where the feature pyramid is generated based upon a red green blue depth (RGB-D) image that includes objects. The method further includes sampling a 3D parameter map at locations corresponding to peaks in the heatmap, where the 3D parameter map is inferred based upon the feature pyramid, and where the locations include latent shape codes, 6D poses, and one-dimensional (1D) scales. The method further includes generating point clouds based upon the latent shape codes, the 6D poses, and the 1D scales.
1 . A computing system for single-shot multi-object three-dimensional (3D) shape reconstruction and categorical six-dimensional (6D) pose and size estimation, the computing system comprising:
a processor; and
memory communicably coupled to the processor that stores instructions that, when executed by the processor, cause the processor to:
infer a heatmap based upon a feature pyramid, wherein the feature pyramid is generated based upon a red green blue depth (RGB-D) image that includes objects;
infer a three-dimensional (3D) parameter map based upon the feature pyramid;
sample a 3D parameter map at locations corresponding to peaks in the heatmap, wherein the locations include latent shape codes, 6D poses, and one-dimensional (1D) scales; and
generate point clouds based upon the latent shape codes, the 6D poses, and the 1D scales.
2 . The computing system of claim 1 , wherein the point clouds are indicative of shapes of the objects, sizes of the objects, and poses of the objects.
3 . The computing system of claim 1 , wherein the instructions further cause the processor to:
utilize the point clouds in a computer simulation; and
train a machine learning model based upon the computer simulation or real-world data, wherein a real-world vehicle utilizes the machine learning model to navigate about a real-world environment.
4 . The computing system of claim 1 , wherein the instructions further cause the processor to:
generate, by way of a residual neural network and based upon the RGB-D image, RGB features, and depth features;
concatenate the RGB features and the depth features to generate concatenated features;
provide the concatenated features as input to a feature pyramid network (FPN) backbone; and
obtain the feature pyramid as an output of the FPN backbone.
5 . The computing system of claim 1 , wherein the instructions further cause the processor to:
train an encoder-decoder network based upon computer-aided design (CAD) models that are indicative of different shapes.
6 . The computing system of claim 5 , wherein the instructions further cause the processor to:
prior to training the encoder-decoder network, sample points from a CAD model in the CAD models, wherein the points are indicative of a shape; and
apply a scaling transform to the points such that the points are centered at an origin and unit normalized.
7 . The computing system of claim 6 , wherein the instructions for generating the point clouds based upon the latent shape codes, the 6D poses, and the 1D scales comprise further instructions that cause the processor to:
provide the latent shape codes as input to a decoder of the encoder-decoder network;
obtain initial point clouds as output of the decoder; and
transform the initial point clouds using the 6D poses and the 1D scales to generate the point clouds.
8 . A non-transitory computer-readable medium for single-shot multi-object three-dimensional (3D) shape reconstruction and categorical six-dimensional (6D) pose and size estimation and including instructions that, when executed by a processor, cause the processor to:
infer a heatmap based upon a feature pyramid, wherein the feature pyramid is generated based upon a red green blue depth (RGB-D) image that includes objects;
infer a three-dimensional (3D) parameter map based upon the feature pyramid;
sample a 3D parameter map at locations corresponding to peaks in the heatmap, wherein the locations include latent shape codes, 6D poses, and one-dimensional (1D) scales; and
generate point clouds based upon the latent shape codes, the 6D poses, and the 1D scales.
9 . The non-transitory computer-readable medium of claim 8 , wherein the instructions for generating the point clouds based upon the latent shape codes, the 6D poses, and the 1D scales comprise further instructions that cause the processor to:
provide the latent shape codes as input to a decoder of an encoder-decoder network;
obtain initial point clouds as output of the decoder; and
transform the initial point clouds using the 6D poses and the 1D scales to generate the point clouds.
10 . The non-transitory computer-readable medium of claim 9 , wherein the 6D poses comprise translation vectors and rotation matrices, and wherein the instructions for transforming the initial point clouds using the 6D poses and the 1D scales comprise further instructions that cause the processor to:
multiply points in the initial point clouds by respective translation vectors, respective rotation matrices, and respective 1D scales to generate the point clouds.
11 . The non-transitory computer-readable medium of claim 8 , wherein the peaks in the heatmap correspond to centers of the objects in the RGB-D image.
12 . The non-transitory computer-readable medium of claim 8 , wherein the instructions further cause the processor to:
utilize the point clouds in a computer simulation; and
train a machine learning model based upon the computer simulation, wherein a real-world robotic device utilizes the machine learning model to navigate about a real-world environment.
13 . A method comprising:
inferring a heatmap based upon a feature pyramid, wherein the feature pyramid is generated based upon a red green blue depth (RGB-D) image that includes objects;
inferring a three-dimensional (3D) parameter map based upon the feature pyramid;
sampling a 3D parameter map at locations corresponding to peaks in the heatmap, wherein the locations include latent shape codes, 6D poses, and one-dimensional (1D) scales; and
generating point clouds based upon the latent shape codes, the 6D poses, and the 1D scales.
14 . The method of claim 13 , wherein the point clouds are indicative of shapes of the objects, sizes of the objects, and poses of the objects.
15 . The method of claim 13 , further comprising:
determining the peaks in the heatmap, wherein the peaks correspond to centers of the objects in the RGB-D image.
16 . The method of claim 13 , further comprising:
capturing the RGB-D image using a camera of a robotic device; and
controlling the robotic device based upon the point clouds.
17 . The method of claim 13 , further comprising:
utilizing the point clouds in a computer simulation; and
training a machine learning model based upon the computer simulation, wherein a real-world robotic device utilizes the machine learning model to navigate about a real-world environment.
18 . The method of claim 13 , further comprising:
applying textures to the point clouds, wherein the textures correspond to types of the objects.