Systems and methods for object dimensioning based on partial visual information
A method for estimating tightly enclosing bounding boxes by a computing system includes: controlling a scanning system including one or more depth cameras to capture visual information of the scene including one or more objects; detecting the one or more objects of the scene based on the visual information; singulating each the one or more objects from the frame of the scene to generate one or more 3D models corresponding to the one or more objects, the one or more 3D models including a partial 3D model of a corresponding one of the one or more objects; extrapolating a more complete 3D model of the corresponding one of the one or more objects based on the partial 3D model; and estimating a tightly enclosing bounding box of the corresponding one of the one or more objects based on the more complete 3D model.
1. A method for estimating tightly enclosing bounding boxes comprising:
controlling, by a computing system, a scanning system comprising one or more depth cameras to capture visual information of a scene comprising one or more objects;
detecting, by the computing system, the one or more objects of the scene based on the visual information;
singulating, by the computing system, each of the one or more objects from a frame of the scene to generate one or more 3D models corresponding to the one or more objects, the one or more 3D models comprising a partial 3D model of a corresponding one of the one or more objects;
extrapolating, by the computing system, a more complete 3D model of the corresponding one of the one or more objects based on the partial 3D model, wherein:
the extrapolating the more complete 3D model comprises searching an inventory of 3D models for a model matching the partial 3D model and to output the model matching the partial 3D model as the more complete 3D model, wherein each of the 3D models of the inventory is associated with a corresponding descriptor in feature space, and wherein the searching the inventory of 3D model comprises:
extracting a descriptor of the partial 3D model by supplying the partial 3D model to a feature extractor comprising a convolutional neural network, and
identifying the model matching the partial 3D model in accordance with a highest similarity between the descriptor of the partial 3D model and the descriptors of the 3D models of the inventory; and
estimating, by the computing system, a tightly enclosing bounding box of the corresponding one of the one or more objects based on the more complete 3D model.
2. The method of claim 1 , wherein the scanning system further comprises one or more color cameras separate from the one or more depth cameras.
3. The method of claim 1 , wherein the one or more depth cameras comprises:
a time-of-flight depth camera;
a structured light depth camera;
a stereo depth camera comprising at least two color cameras;
a stereo depth camera comprising:
at least two color cameras; and
a color projector;
a stereo depth camera comprising at least two infrared cameras; or
a stereo depth camera comprising:
a color camera;
a plurality of infrared cameras; and
an infrared projector configured to emit light in a wavelength interval that is detectable by the plurality of infrared cameras.
4. The method of claim 1 , wherein the detecting the one or more objects in the scene comprises separating the one or more objects from depictions of background and ground plane in the visual information.
5. The method of claim 1 , wherein the extrapolating the more complete 3D model comprises supplying the partial 3D model to a generative model trained to predict a generated 3D model based on an input partial 3D model, the more complete 3D model comprising the generated 3D model.
6. The method of claim 5 , wherein the generative model comprises a conditional generative adversarial network.
7. The method of claim 1 , wherein the extracting the descriptor of the partial 3D model comprises:
defining a bounding box around the 3D model;
voxelizing the partial 3D model to compute a plurality of voxels; and
supplying the voxels to the convolutional neural network.
8. The method of claim 1 , wherein the extracting the descriptor of the partial 3D model comprises:
rendering a plurality of 2D views of the partial 3D model;
extracting feature vectors from the 2D views of the partial 3D model by supplying the plurality of 2D views to a first stage convolutional neural network; and
combining the feature vectors to generate the descriptor.
9. The method of claim 8 , wherein the feature vectors are combined using max pooling.
10. The method of claim 1 , wherein the extrapolating the complete 3D model comprises:
classifying the partial 3D model to compute a matching classification;
loading one or more heuristic rules for generating more complete 3D models for the matching classification; and
generating the more complete 3D model from the partial 3D model in accordance with the one or more heuristic rules.
11. The method of claim 10 , wherein the one or more heuristic rules comprise one or more assumed axes of symmetry of the more complete 3D model based on the matching classification, or a canonical general shape of the more complete 3D model based on the matching classification.
12. The method of claim 1 , wherein the one or more objects comprise a plurality of objects, and
wherein the singulating each the one or more objects from the frame of the scene comprises singulating the plurality of objects by applying an appearance-based segmentation to the visual information.
13. The method of claim 1 , wherein the one or more objects comprise a plurality of objects, and
wherein the singulating each the one or more objects from the frame of the scene comprises singulating the plurality of objects by applying semantic segmentation to the visual information.
14. The method of claim 13 , wherein the applying semantic segmentation comprises supplying the visual information to a trained fully convolutional neural network to compute a segmentation map, and
wherein each partial 3D model corresponds to one segment of the segmentation map.
15. The method of claim 1 , further comprising associating the tightly enclosing bounding box with an item descriptor.
16. A system for estimating tightly enclosing bounding boxes comprising:
a scanning system comprising one or more depth cameras; a processor configured to control the scanning system; and
memory storing instructions that, when executed by the processor, cause the processor to:
control the one or more depth cameras to capture visual information of a scene comprising one or more objects;
detect the one or more objects of the scene based on the visual information;
singulate each the one or more objects from a frame of the scene to generate one or more 3D models corresponding to the one or more objects, the one or more 3D models comprising a partial 3D model of a corresponding one of the one or more objects;
extrapolate a more complete 3D model of the corresponding one of the one or more objects based on the partial 3D model, wherein:
the extrapolating the more complete 3D model comprises searching an inventory of 3D models for a model matching the partial 3D model and to output the model matching the partial 3D model as the more complete 3D model, wherein each of the 3D models of the inventory is associated with a corresponding descriptor in feature space, and wherein the searching the inventory of 3D model comprises:
extracting a descriptor of the partial 3D model by supplying the partial 3D model to a feature extractor comprising a convolutional neural network, and
identifying the model matching the partial 3D model in accordance with a highest similarity between the descriptor of the partial 3D model and the descriptors of the 3D models of the inventory; and
estimate a tightly enclosing bounding box of the corresponding one of the one or more objects based on the more complete 3D model.
17. The system of claim 16 , wherein the instructions to extrapolate the more complete 3D model comprise instructions that, when executed by the processor, cause the processor to supply the partial 3D model to a generative model trained to predict a generated 3D model based on an input partial 3D model, the more complete 3D model comprising the generated 3D model.
18. The system of claim 17 , wherein the generative model comprises a conditional generative adversarial network.
19. The system of claim 16 , wherein the instructions to extrapolate the more complete 3D model comprise instructions that, when executed by the processor, cause the processor to search an inventory of 3D models for a model matching the partial 3D model and to output the model matching the partial 3D model as the more complete 3D model.