Machine learning image-based estimation of container fullness

1 . A method performed by a computing system, the method comprising:

receiving, from an image-capturing device, a two-dimensional image of a container including a first compressible item;

generating a depth map using a monocular depth estimation (MDE) model based at least in part on the two-dimensional image, the depth map including depth information of a surface profile of the first compressible item;

generating a three-dimensional representation comprising points associated with the container based at least in part on the depth map;

generating a voxel grid with volumes associated with the points based at least in part on the three-dimensional representation;

determining a dimension for a voxel of the voxel grid;

determining a dimension of the container with respect to the image-capturing device;

determining a number of voxels included in the voxel grid that are associated with the first compressible item based at least in part on the surface profile and the dimension of the container;

determining a volume of the first compressible item based at least in part on the dimension of the voxel, the number of voxels describing the first compressible item, and the surface profile of the first compressible item;

determining a fullness of the container based at least in part on a ratio of the volume of the first compressible item and a reference volume of the container;

comparing the fullness of the container in the container to a target fullness; and

determining whether the fullness of the container exceeds the target fullness based at least in part on the comparison.

2 . The method of claim 1 , wherein an arrangement of the first compressible item forms a cavity, and wherein determining the volume of the first compressible item comprises determining a volume of the first compressible item and the cavity based at least in part on the surface profile of the first compressible item and the dimension of the container.

3 . The method of claim 1 , wherein the container is arranged on material handling equipment moving from a source location to a target location, wherein the method further comprises:

causing the image-capturing device to capture the two-dimensional image of the container between the source location and the target location, wherein the fullness of the container is determined between the source location and the targe location.

4 . A computing system comprising:

one or more processors; and

one or more computer-readable media including instructions that, when executed by the one or more processors, cause the one or more processors to:

cause an image-capturing device to capture a first two-dimensional image of a container in motion between a source location and a target location;

receive, from the image-capturing device, the first two-dimensional image of the container including a first compressible item;

determine, using a machine learning model, three-dimensional information based at least in part on the first two-dimensional image;

determine a volume of the first compressible item based at least in part on the three-dimensional information;

determine a fullness of the container based at least in part on the volume of the first compressible item;

compare the fullness of the container to a target fullness of the container; and

determine whether the fullness of the container exceeds the target fullness based at least in part on the comparison, wherein the fullness of the container is determined while the container is between the source location and the target location.

5 . The computing system of claim 4 , wherein the machine learning model is monocular depth estimation (MDE) model, and wherein the one or more computer-readable media including instructions that, when executed by the one or more processors, further cause the one or more processors to:

generate a depth map for the container including the first compressible item using the MDE model; and

generate a three-dimensional representation of the container including the first compressible item based at least in part on the depth map, wherein the three-dimensional representation comprises the three-dimensional information.

6 . The computing system of claim 4 , wherein the machine learning model comprises a transformer model, and wherein the one or more computer-readable media including instructions that, when executed by the one or more processors, further cause the one or more processors to:

determine, using an encoder and decoder of the transformer model, a fullness of the container without explicit depth estimation and volume calculation.

7 . The computing system of claim 5 , wherein the one or more computer-readable media including instructions that, when executed by the one or more processors, further cause the one or more processors to:

receive, from the image-capturing device, a second two-dimensional image of the container including the first compressible item and a second compressible item in physical contact with the first compressible item and compressing the first compressible item;

generate a depth map for the container including the first compressible item and the second compressible item using the MDE model;

generate a point cloud of the container including the first compressible item and the second compressible item;

determine an aggregate volume of the first compressible item and the second compressible item based at least in part on the point cloud;

determine the fullness of the container based at least in part on a ratio of the aggregate volume and the volume of the container while empty;

compare the fullness of the container in the container to a target fullness; and

determine whether the fullness of the container exceeds the target fullness based at least in part on the comparison.

8 . The computing system of claim 5 , wherein the one or more computer-readable media including instructions that, when executed by the one or more processors, further cause the one or more processors to:

receive, from the image-capturing device, a video comprising a first two-dimensional images of the container including the first and a second two-dimensional image of the container including the first compressible item;

determine a first visibility of the container and the first compressible item and a second visibility of the container and the first compressible item; and

select the first and a second two-dimensional image to use to generate the depth map based at least in part on the first visibility of the first compressible item.

9 . The computing system of claim 5 , wherein the depth map includes depth information of a surface profile of the first compressible item, and wherein the one or more computer-readable media including instructions that, when executed by the one or more processors, further cause the one or more processors to:

generate a voxel grid of the container including the first compressible item based at least in part on the three-dimensional representation;

determine a dimension of the container with respect to the image-capturing device; and

determine a number of voxels of the voxel grid associated with the first compressible item based at least in part on the surface profile of the first compressible item and the dimension of the container, wherein the fullness of the container is based at least in a part on the number of voxels.

10 . The computing system of claim 5 , wherein the one or more computer-readable media including instructions that, when executed by the one or more processors, further cause the one or more processors to:

determine a container type of the container; and

access a database of container types and identify a reference volume of the container is based at least in part on the container type, wherein the fullness of the container is based at least part on the reference volume.

11 . The computing system of claim 5 , wherein the one or more computer-readable media including instructions that, when executed by the one or more processors, further cause the one or more processors to:

determine a usable volume based at least in part the volume of the first compressible item and a reference volume of the container; and

determine that the usable volume comprises a first sub-usable volume and a second sub-usable volume of usable space based at least in part on a relationship between a first pixel of the depth map associated with the first sub-usable volume and a second pixel associated with the second sub-usable volume.

12 . The computing system of claim 11 , wherein the one or more computer-readable media including instructions that, when executed by the one or more processors, further cause the one or more processors to:

determine whether a path from the first pixel of the depth map associated with the first sub-usable volume to second pixel associated with the second sub-usable volume without traverses a third pixel associated with the first compressible item, wherein determine that the usable volume comprises a first sub-usable volume and a second sub-usable volume of usable space is based at least in part on the determination.

13 . One or more non-transitory computer-readable media including stored thereon a sequence of instructions that, when executed by one or more processors, causes the one or more processors to:

cause an image-capturing device to capture a first two-dimensional information of a container in motion between a source location and a target location:

receive the first two-dimensional information of the container including a first compressible item from the image-capturing device, the two-dimensional image-capturing device arranged at a first location;

generate a machine learning model input based at least in part on the first two-dimensional information; generate, using a machine learning model, a three-dimensional representation of the container based at least in part on the machine learning model input;

determine, using the machine learning model, a volume of the first compressible item based at least in part on the three-dimensional representation; and

determine, using the machine learning model, a first fullness of the container based at least in part on the volume of the first compressible item, wherein the first fullness of the container is determined while the container is between the source location and the target location.

14 . The one or more non-transitory computer-readable media of claim 13 , wherein the instructions that, when executed by one or more processors, further cause the one or more processors to:

generate, using the machine learning model, a depth map of the container including the first compressible item based at least in part on the machine learning model input, wherein the depth map includes depth information of a surface profile of the first compressible item, and wherein the three-dimensional representation is based at least in part on the depth map; and

determine the volume of the first compressible item based at least in part on the three-dimensional representation, wherein the first fullness of the container is based at least in a part on the volume.

15 . The one or more non-transitory computer-readable media of claim 13 , wherein the instructions that, when executed by one or more processors, further cause the one or more processors to:

generate a voxel grid of the container including the first compressible item based in part on the three-dimensional representation;

determine a dimension of the container with respect to the image capturing device; and

determine a number of voxels of the voxel grid associated with the first compressible item based at least in part on a surface profile of the first compressible item and the dimension of the container with respect to the image-capturing device, wherein the volume of the first compressible item is based at least in a part on the number of voxels, wherein the first fullness of the container is based at least in a part on the number of voxels.

16 . The one or more non-transitory computer-readable media of claim 13 , wherein the machine learning model comprises a transformer model, and wherein the instructions that, when executed by one or more processors, further cause the one or more processors to:

determine, using an encoder and decoder of the transformer model, to determine the first fullness of the container without explicit depth estimation or volume calculation.

17 . The one or more non-transitory computer-readable media of claim 13 , wherein the instructions that, when executed by one or more processors, further cause the one or more processors to:

receive a second two-dimensional information from the image-capturing device at a second location, the image-capturing device at the first location having a first field of view, and the image-capturing device at the second location having a second field of view;

generate a second machine learning model input based at least in part on the second two-dimensional information;

determine a second fullness of the container based at least in part on using the second machine learning model input;

receive a ground truth fullness of the container based at least in part on a three-dimensional image from a second image-capturing device; and

evaluate the first fullness and the second fullness based at least in part on the ground truth fullness.

18 . The one or more non-transitory computer-readable media of claim 17 , wherein the instructions that, when executed by one or more processors, further cause the one or more processors to:

determine a usable volume based at least in part the volume of the first compressible item and the volume of the container while empty; and

determine that the usable volume comprises a first sub-usable volume and a second sub-usable volume of usable space based at least in part on a relationship between a first pixel of a depth map associated with the first sub-usable volume and a second pixel associated with the second sub-usable volume.

19 . The one or more non-transitory computer-readable media of claim 18 , wherein the instructions that, when executed by one or more processors, further cause the one or more processors to:

determine whether a path from the first pixel of the depth map associated with the first sub-usable volume to second pixel associated with the second sub-usable volume without traverses a third pixel associated with the first compressible item, wherein determine that the usable volume comprises a first sub-usable volume and a second sub-usable volume of usable space is based at least in part on the determination.