Systems and methods for object dimensioning based on partial visual information
A method for estimating tightly enclosing bounding boxes by a computing system includes: controlling a scanning system including one or more depth cameras to capture visual information of the scene including one or more objects; detecting the one or more objects of the scene based on the visual information; singulating each the one or more objects from the frame of the scene to generate one or more 3D models corresponding to the one or more objects, the one or more 3D models including a partial 3D model of a corresponding one of the one or more objects; extrapolating a more complete 3D model of the corresponding one of the one or more objects based on the partial 3D model; and estimating a tightly enclosing bounding box of the corresponding one of the one or more objects based on the more complete 3D model.
1. A computing system for estimating tightly enclosing bounding boxes comprising:
one or more processors; and
one or more computer-readable media having stored thereon executable instructions that when executed by the one or more processors configure the computing system to:
detect, by the computing system, one or more objects of a scene based on visual information received at the computing system;
extract a descriptor of a partial 3D model of the one or more objects by supplying the partial 3D model to a feature extractor comprising a convolutional neural network;
identify, within an inventory of 3D models, a more complete 3D model matching the partial 3D model in accordance with a highest similarity between the descriptor of the partial 3D model and descriptors of 3D models within the inventory; and
estimate, by the computing system, a tightly enclosing bounding box of a corresponding one of the one or more objects based on the more complete 3D model.
2. The computing system of claim 1 , further comprising a scanning system comprising one or more depth cameras to capture visual information of a scene comprising one or more objects.
3. The computing system of claim 2 , wherein the scanning system further comprises one or more color cameras separate from the one or more depth cameras.
4. The computing system of claim 2 , wherein the one or more depth cameras comprises:
a time-of-flight depth camera;
a structured light depth camera;
a stereo depth camera comprising at least two color cameras;
a stereo depth camera comprising:
at least two color cameras, and
a color projector;
a stereo depth camera comprising at least two infrared cameras; or
a stereo depth camera comprising:
a color camera,
a plurality of infrared cameras, and
an infrared projector configured to emit light in a wavelength interval that is detectable by the plurality of infrared cameras.
5. The computing system of claim 1 , wherein the detecting the one or more objects in the scene comprises separating the one or more objects from depictions of background and ground plane in the visual information.
6. The computing system of claim 1 , wherein the extracting the descriptor of the partial 3D model comprises:
defining a bounding box around the more complete 3D model;
voxelizing the partial 3D model to compute a plurality of voxels; and
supplying the voxels to the convolutional neural network.
7. The computing system of claim 6 , wherein the extracting the descriptor of the partial 3D model comprises:
rendering a plurality of 2D views of the partial 3D model;
extracting feature vectors from the 2D views of the partial 3D model by supplying the plurality of 2D views to a first stage convolutional neural network; and
combining the feature vectors to generate the descriptor.
8. The computing system of claim 7 , wherein the feature vectors are combined using max pooling.
9. The computing system of claim 1 , wherein the executable instructions include instructions that are executable to configure the computing system to associate the tightly enclosing bounding box with an item descriptor.
10. A computer-implemented method for estimating tightly enclosing bounding boxes comprising:
detecting, by a computing system, one or more objects of a scene based on visual information received at the computing system;
extracting a descriptor of a partial 3D model of the one or more objects by supplying the partial 3D model to a feature extractor comprising a convolutional neural network;
identifying, within an inventory of 3D models, a more complete 3D model matching the partial 3D model in accordance with a highest similarity between the descriptor of the partial 3D model and descriptors of 3D models within the inventory; and
estimating, by the computing system, a tightly enclosing bounding box of a corresponding one of the one or more objects based on the more complete 3D model.
11. The computer-implemented method of claim 10 , further comprising a scanning system comprising one or more depth cameras to capture visual information of a scene comprising one or more objects.
12. The computer-implemented method of claim 11 , wherein the scanning system further comprises one or more color cameras separate from the one or more depth cameras.
13. The computer-implemented method of claim 11 , wherein the one or more depth cameras comprises:
a time-of-flight depth camera;
a structured light depth camera;
a stereo depth camera comprising at least two color cameras;
a stereo depth camera comprising:
at least two color cameras, and
a color projector;
a stereo depth camera comprising at least two infrared cameras; or
a stereo depth camera comprising:
a color camera,
a plurality of infrared cameras, and
an infrared projector configured to emit light in a wavelength interval that is detectable by the plurality of infrared cameras.
14. The computer-implemented method of claim 10 , wherein the detecting the one or more objects in the scene comprises separating the one or more objects from depictions of background and ground plane in the visual information.
15. The computer-implemented method of claim 10 , wherein the extracting the descriptor of the partial 3D model comprises:
defining a bounding box around the more complete 3D model;
voxelizing the partial 3D model to compute a plurality of voxels; and
supplying the voxels to the convolutional neural network.
16. The computer-implemented method of claim 15 , wherein the extracting the descriptor of the partial 3D model comprises:
rendering a plurality of 2D views of the partial 3D model;
extracting feature vectors from the 2D views of the partial 3D model by supplying the plurality of 2D views to a first stage convolutional neural network; and
combining the feature vectors to generate the descriptor.
17. The computer-implemented method of claim 16 , wherein the feature vectors are combined using max pooling.
18. The computer-implemented method of claim 10 , further comprising configuring the computing system to associate the tightly enclosing bounding box with an item descriptor.
19. A computer-readable media comprising one or more physical computer-readable storage media having stored thereon computer-executable instructions that, when executed at a processor, cause a computer system to perform a method for estimating tightly enclosing bounding boxes, the method comprising:
detecting, by a computing system, one or more objects of a scene based on visual information received at the computing system;
extracting a descriptor of a partial 3D model of the one or more objects by supplying the partial 3D model to a feature extractor comprising a convolutional neural network;
identifying, within an inventory of 3D models, a more complete 3D model matching the partial 3D model in accordance with a highest similarity between the descriptor of the partial 3D model and descriptors of 3D models within the inventory; and
estimating, by the computing system, a tightly enclosing bounding box of a corresponding one of the one or more objects based on the more complete 3D model.
20. The computer-readable media of claim 19 , further comprising a scanning system comprising one or more depth cameras to capture visual information of a scene comprising one or more objects.