Granted Patent B2

US 12,632,621 · App. 17/868,614 · Granted May 19, 2026

Inventors: Muhammad Zubair Irshad (Atlanta, GA); Sergey Zakharov (San Francisco, CA); Rares A. Ambrus (San Francisco, CA); Adrien D. Gaidon (Los Altos, CA)

Assignees: TOYOTA RESEARCH INSTITUTE, INC.; TOYOTA JIDOSHA KABUSHIKI KAISHA

G06F30/23G06F30/27G06T7/50

Abstract

Claims (61)

1 . A computer-implemented method, comprising:

outputting sampled shape and appearance codes from respective heatmap heads into respective shape and appearance multi-layer perceptrons (MLPs), outputting sampled 6D pose and size codes from a respective heatmap head into a differential iso-surface projection engine, and outputting 2D masks from a respective heatmap head into an optimizer, wherein the sampled codes and 2D masks serve as initial estimates of each code and 2D mask;

training the respective shape and appearance MLPs using shape and appearance priors from an implicit joint differentiable database, to obtain further shape and appearance codes;

inputting the further shape and appearance codes into the differential iso-surface projection engine; during training of the MLPs, outputting from the differential iso-surface projection engine into the optimizer latest shape, appearance, 6D pose and size codes, and 2D masks;

using an Octree-based coarse-to-fine differentiable optimization that refines from a coarse resolution to a higher resolution by iteratively increasing an Octree resolution level until a desired resolution level is reached to jointly optimize the latest shape, appearance, 6D pose and size codes, and 2D masks, using the database, and return the optimized codes and 2D masks to the respective MLPs; and

recovering an object surface from the latest shape codes to the desired resolution level using the differential iso-surface projection engine.

2 . The computer-implemented method of claim 1 , further comprising, before the outputting step:

inputting, into the heatmap heads, a center location of each object based on peak values detected from heatmaps; and sampling, from an output of each heatmap head, the shape, appearance, and 6D pose and size codes, and the 2D masks.

3 . The computer-implemented method of claim 2 , wherein the inputting step further comprises:

inputting an RGB-D image; extracting RGB features and Depth features from the inputted RGB-D image;

concatenating the extracted RGB features and the extracted Depth features along a convolutional channel to generate concatenated features;

generating a feature pyramid based upon the concatenated RGB features and the concatenated Depth features;

predicting object-centric heatmaps based upon the feature pyramid; determining peak values in the heatmaps corresponding to centers of the objects in the RGB-D image; and

detecting and outputting the center location of each object based on the peak values in the heatmaps.

4 . The computer-implemented method of claim 1 , further comprising:

training the respective shape and appearance MLPs using an RGB loss minimizing function.

5 . The computer-implemented method of claim 1 , wherein the database represents shapes as Signed Distance Fields (SDF) and appearance as Texture Fields (TF).

6 . The computer-implemented method of claim 1 , wherein the optimizing regresses 3D rotation, 3D translation, and 1D scales.

7 . The computer-implemented method of claim 1 , wherein the optimizing uses a combined loss minimizing function.

8 . A system comprising:

a processor; and a memory coupled to the processor to store instructions which, when executed by the processor, cause the processor to perform operations, the operations comprising:

inputting, into heatmap heads, a center location of each object based on peak values detected from heatmaps;

sampling, from an output of each heatmap head, shape, appearance, and 6D pose and size codes, and 2D masks;

outputting sampled shape and appearance codes from respective heatmap heads into respective shape and appearance multi-layer perceptrons (MLPs), outputting 6D pose and size codes into a differential iso-surface projection engine, and outputting 2D masks into an optimizer, wherein the sampled codes and 2D masks serve as initial estimates of each code and 2D mask;

training the respective shape and appearance MLPs using shape and appearance priors from an implicit joint differentiable database, to obtain further shape and appearance codes;

inputting the further shape and appearance codes into the differential iso-surface projection engine;

during training of the MLPs, outputting from the differential iso-surface projection engine into the optimizer latest shape, appearance, 6D pose and size codes, and 2D masks;

using an Octree-based coarse-to-fine differentiable optimization that refines from a coarse resolution to a higher resolution by iteratively increasing an Octree resolution level until a desired resolution level is reached to jointly optimize the latest shape, appearance, 6D pose and size codes, and 2D masks, using the database, and return the optimized codes and 2D masks to the respective MLPs; and

recovering an object surface from the latest shape codes to the desired resolution level using the differential iso-surface projection engine.

9 . The system of claim 8 , the operations further comprising:

training the respective shape and appearance MLPs using an RGB loss minimizing function.

10 . The system of claim 8 , wherein the database represents shapes as Signed Distance Fields (SDF) and appearance as Texture Fields (TF).

11 . The system of claim 8 , wherein the first inputting step of the operations further comprises:

inputting an RGB-D image; extracting RGB features and Depth features from the inputted RGB-D image;

concatenating the extracted RGB features and the extracted Depth features along a convolutional channel to generate concatenated features;

generating a feature pyramid based upon the concatenated RGB features and the concatenated Depth features;

predicting object-centric heatmaps based upon the feature pyramid;

determining peak values in the heatmaps corresponding to centers of the objects in the RGB-D image; and

detecting and outputting the center location of each object based on the peak values in the heatmaps.

12 . The system of claim 8 , wherein the optimizing regresses 3D rotation, 3D translation, and 1D scales.

13 . The system of claim 8 , wherein the optimizing uses a combined loss minimizing function.

14 . A non-transitory machine-readable medium having instructions stored therein, which, when executed by a processor, cause the processor to perform operations, the operations comprising:

outputting sampled shape and appearance codes from respective heatmap heads into respective shape and appearance multi-layer perceptrons (MLPs), outputting sampled 6D pose and size codes from a respective heatmap head into a differential iso-surface projection engine, and outputting 2D masks from a respective heatmap head into an optimizer, wherein the sampled codes and 2D masks serve as initial estimates of each code and 2D mask;

training the respective shape and appearance MLPs using shape and appearance priors from an implicit joint differentiable database, to obtain further shape and appearance codes;

inputting the further shape and appearance codes into the differential iso-surface projection engine;

during training of the MLPs, outputting from the differential iso-surface projection engine into the optimizer latest shape, appearance, 6D pose and size codes, and 2D masks;

using an Octree-based coarse-to-fine differentiable optimization that refines from a coarse resolution to a higher resolution by iteratively increasing an Octree resolution level until a desired resolution level is reached to jointly optimize the latest shape, appearance, 6D pose and size codes, and 2D masks, using the database, and return the optimized codes and 2D masks to the respective MLPs; and

recovering an object surface from the latest shape codes to the desired resolution level using the differential iso-surface projection engine.

15 . The non-transitory machine-readable medium of claim 14 , the operations further comprising, before the outputting step:

inputting, into the heatmap heads, a center location of each object based on peak values detected from heatmaps; and sampling, from an output of each heatmap head, the shape, appearance, and 6D pose and size codes, and the 2D masks.

16 . The non-transitory machine-readable medium of claim 14 , wherein the first inputting step of the operations further comprises:

inputting an RGB-D image; extracting RGB features and Depth features from the inputted RGB-D image;

concatenating the extracted RGB features and the extracted Depth features along a convolutional channel to generate concatenated features;

generating a feature pyramid based upon the concatenated RGB features and the concatenated Depth features;

predicting object-centric heatmaps based upon the feature pyramid; determining peak values in the heatmaps corresponding to centers of the objects in the RGB-D image; and

detecting and outputting the center location of each object based on the peak values in the heatmaps.

17 . The non-transitory machine-readable medium of claim 14 , the operations further comprising:

training the respective shape and appearance MLPs using an RGB loss minimizing function.

18 . The non-transitory machine-readable medium of claim 14 , wherein the database represents shapes as Signed Distance Fields (SDF) and appearance as Texture Fields (TF).

19 . The system of claim 8 , wherein the system is a vehicle control system.

20 . The system of claim 8 , wherein the system is a robotics system.

Assignments (2)

Continuity (1)

References Cited (36)