Granted Patent B2

US 12,228,388 · App. 18/330,248 · Granted Feb 18, 2025

Inventors: Francesco Peruch (Sunnyvale, CA); Giuliano Pasqualotto (Mountain View, CA); Giridhar Murali (Sunnyvale, CA); Ethan Cheng (Cupertino, CA); Kinh Tieu (Sunnyvale, CA)

Assignee: PACKSIZE, LLC

G01B11/00G06T7/10G06T7/11G06T7/12G06T7/187G06T7/50G06T7/62G06V20/64G06T2207/10028

Abstract

Claims (145)

1. A computer system for volumetric sizing, comprising:

one or more processors; and

one or more non-transitory computer-readable media having stored thereon executable instructions that when executed by the one or more processors configure the computer system to:

receive at least a frame of a scene, the frame comprising a color image and a depth image arranged in a plurality of pixels;

detect an object in the frame;

segment the object from the scene by:

identifying one or more initial pixels of the object, and

performing an iterative flood fill operation, starting with the initial pixels of the object, each iteration of the flood fill operation comprising adding a plurality of neighboring pixels of the frame to pixels of the object when distances between the neighboring pixels and the pixels of the object are within a threshold distance;

determine a ground plane in the frame, the object resting on the ground plane;

compute a rectangular outline bounding a projection of a plurality of pixels of the object onto the ground plane;

compute a height of the object above the ground plane;

output computed dimensions of the object in accordance with a length and a width of the rectangular outline and the height, and

wherein the executable instructions include instructions that are executable to configure the computer system to cause a display device to:

display a view of the color image of the scene captured by a depth camera;

overlay a reticle on the view in the display device; and

wherein the one or more initial pixels of the object correspond to the pixels under the reticle.

2. The computer system of claim 1 , wherein the executable instructions include instructions that are executable to configure the computer system to cause a depth camera system to capture the frame in response to detecting an activation of a trigger, wherein the one or more initial pixels of the object correspond to the pixels under the reticle when the trigger is activated.

3. The computer system of claim 1 , wherein the executable instructions include instructions that are executable to configure the computer system to segment the object from the scene by:

defining a graph, wherein each vertex of the graph corresponds to a vertical projection of the pixels of the scene onto the ground plane and wherein two vertices are connected by an edge when their distance is smaller than threshold;

detecting connected components of the vertical projection of the pixels; and

identifying the largest connected component of the graph as the pixels of the object.

4. The computer system of claim 1 , wherein the executable instructions include instructions that are executable to configure the computer system to output the computed dimensions on a display device.

5. The computer system of claim 1 , wherein the executable instructions include instructions that are executable to configure the computer system to further determine the computed dimensions in accordance with a box mode, the object comprising two vertical faces, the instructions corresponding to the box mode comprising executable instructions that, when executed by the one or more processors, cause the computer system to:

identify a corner of the object, wherein the corner of the object is located at an intersection of two lines formed in the projection of the pixels of the vertical faces of the object onto the ground plane;

compute the height of the object above the ground plane by computing heights of top edges of the two vertical faces; and

compute dimensions of vertical planes of the object.

6. The computer system of claim 5 , wherein the executable instructions include instructions that are executable to configure the computer system to compute the heights of the top edges of the two vertical faces by:

dividing the ground plane into a plurality of cells;

selecting a plurality of cells comprising the lines;

computing a maximum height of each cell based on the pixels of the object in each of the cells; and

computing a height of a box based on the maximum heights of the cells.

7. The computer system of claim 5 , wherein the executable instructions include instructions that are executable to configure the computer system to activate or deactivate the box mode based on a user interface switch.

8. The computer system of claim 5 , wherein the executable instructions include instructions that are executable to configure the computer system to activate the box mode in response to detecting that the object comprises two vertical planes arranged at right angles to the ground plane and at right angles to each other.

9. The computer system of claim 1 , wherein the computed dimensions of the object correspond to dimensions of a box tightly fitting the object.

10. A computer system for volumetric sizing, comprising:

one or more processors; and

one or more non-transitory computer-readable media having stored thereon executable instructions that when executed by the one or more processors configure the computer system to:

cause a depth camera system to capture at least a frame of a scene, the frame comprising a color image and a depth image arranged in a plurality of pixels;

detect an object in the frame;

segment the object from the scene by:

defining a graph, wherein each vertex of the graph corresponds to a vertical projection of the pixels of the scene onto a ground plane and wherein two vertices are connected by an edge when their distance is smaller than threshold,

detecting connected components of the vertical projection of the pixels, and

identifying the largest connected component of the graph as the pixels of the object,

determine a ground plane in the frame, the object resting on the ground plane;

compute a height of the object above the ground plane;

compute a rectangular outline bounding a projection of a plurality of pixels of the object onto the ground plane; and

output computed dimensions of the object in accordance with a length and a width of the rectangular outline and the height.

11. The computer system of claim 10 , wherein the executable instructions include instructions that are executable to:

identify, when capturing the frame, an orientation of the depth camera system based on data from an inertial measurement unit rigidly connected to the depth camera system;

identify a plurality of bottom pixels of the frame based on the orientation;

compute a partial plane from the bottom pixels of the frame; and

extend the partial plane in the depth image to define the ground plane.

12. The computer system of claim 11 , wherein a width of a strip of the bottom pixels is calculated in accordance with noise characteristics of the depth camera system.

13. A computer system for volumetric sizing, comprising:

one or more processors; and

one or more non-transitory computer-readable media having stored thereon executable instructions that when executed by the one or more processors configure the computer system to:

cause a depth camera system to capture at least a frame of a scene, the frame comprising a color image and a depth image arranged in a plurality of pixels;

detect an object in the frame;

determine a ground plane in the frame, the object resting on the ground plane;

compute a height of the object above the ground plane;

compute a rectangular outline bounding a projection of a plurality of pixels of the object onto the ground plane;

determine the computed dimensions in accordance with a box mode, the object comprising two vertical faces, by:

identifying a corner of the object, wherein the corner of the object is located at an intersection of two lines formed in the projection of the pixels of the vertical faces of the object onto the ground plane,

computing the height of the object above the ground plane by computing heights of top edges of the two vertical faces, and

computing dimensions of vertical planes of the object; and

output computed dimensions of the object in accordance with a length and a width of the rectangular outline and the height.

14. The computer system of claim 13 , wherein the executable instructions include instructions that are executable to determine the computed dimensions in accordance with a box mode, the object comprising two vertical faces, by:

identifying a corner of the object, wherein the corner of the object is located at an intersection of two lines formed in the projection of the pixels of the vertical faces of the object onto the ground plane;

computing the height of the object above the ground plane by computing heights of top edges of the two vertical faces; and

computing dimensions of vertical planes of the object.

15. The computer system of claim 13 , wherein the executable instructions include instructions that are executable to compute the heights of the top edges of the two vertical faces by:

dividing the ground plane into a plurality of cells;

selecting a plurality of cells comprising the lines;

computing a maximum height of each cell based on the pixels of the object in each of the cells; and

computing a height of the box based on the maximum heights of the cells.

16. The computer system of claim 13 , wherein the executable instructions include instructions that are executable to activate or deactivate the box mode based on a user interface switch.

17. The computer system of claim 13 , wherein the executable instructions include instructions that are executable to activate the box mode in response to detecting that the object comprises two vertical planes arranged at right angles to the ground plane and at right angles to each other.

18. A computer system for volumetric sizing, comprising:

one or more processors; and

one or more non-transitory computer-readable media having stored thereon executable instructions that when executed by the one or more processors configure the computer system to:

receive at least a frame of a scene, the frame comprising a color image and a depth image arranged in a plurality of pixels;

detect an object in the frame;

segment the object from the scene by:

identifying one or more initial pixels of the object, and

performing an iterative flood fill operation, starting with the initial pixels of the object, each iteration of the flood fill operation comprising adding a plurality of neighboring pixels of the frame to pixels of the object when distances between the neighboring pixels and the pixels of the object are within a threshold distance;

determine a ground plane in the frame, the object resting on the ground plane;

compute a rectangular outline bounding a projection of a plurality of pixels of the object onto the ground plane;

compute a height of the object above the ground plane;

output computed dimensions of the object in accordance with a length and a width of the rectangular outline and the height, and

wherein the executable instructions include instructions that are executable to configure the computer system to segment the object from the scene by:

defining a graph, wherein each vertex of the graph corresponds to a vertical projection of the pixels of the scene onto the ground plane and wherein two vertices are connected by an edge when their distance is smaller than threshold;

detecting connected components of the vertical projection of the pixels; and

identifying the largest connected component of the graph as the pixels of the object.

19. The computer system of claim 18 , wherein the executable instructions include instructions that are executable to configure the computer system to cause a display device to:

display a view of the color image of the scene captured by a depth camera;

overlay a reticle on the view in the display device; and

wherein the one or more initial pixels of the object correspond to the pixels under the reticle.

20. The computer system of claim 19 , wherein the executable instructions include instructions that are executable to configure the computer system to cause a depth camera system to capture the frame in response to detecting an activation of a trigger, wherein the one or more initial pixels of the object correspond to the pixels under the reticle when the trigger is activated.

21. The computer system of claim 18 , wherein the executable instructions include instructions that are executable to configure the computer system to output the computed dimensions on a display device.

22. The computer system of claim 18 , wherein the executable instructions include instructions that are executable to configure the computer system to further determine the computed dimensions in accordance with a box mode, the object comprising two vertical faces, the instructions corresponding to the box mode comprising executable instructions that, when executed by the one or more processors, cause the computer system to:

identify a corner of the object, wherein the corner of the object is located at an intersection of two lines formed in the projection of the pixels of the vertical faces of the object onto the ground plane;

compute the height of the object above the ground plane by computing heights of top edges of the two vertical faces; and

compute dimensions of vertical planes of the object.

23. The computer system of claim 22 , wherein the executable instructions include instructions that are executable to configure the computer system to compute the heights of the top edges of the two vertical faces by:

dividing the ground plane into a plurality of cells;

selecting a plurality of cells comprising the lines;

computing a maximum height of each cell based on the pixels of the object in each of the cells; and

computing a height of a box based on the maximum heights of the cells.

24. The computer system of claim 22 , wherein the executable instructions include instructions that are executable to configure the computer system to activate or deactivate the box mode based on a user interface switch.

25. The computer system of claim 22 , wherein the executable instructions include instructions that are executable to configure the computer system to activate the box mode in response to detecting that the object comprises two vertical planes arranged at right angles to the ground plane and at right angles to each other.

26. The computer system of claim 18 , wherein the computed dimensions of the object correspond to dimensions of a box tightly fitting the object.

27. A computer system for volumetric sizing, comprising:

one or more processors; and

one or more non-transitory computer-readable media having stored thereon executable instructions that when executed by the one or more processors configure the computer system to:

receive at least a frame of a scene, the frame comprising a color image and a depth image arranged in a plurality of pixels;

detect an object in the frame;

segment the object from the scene by:

identifying one or more initial pixels of the object, and

performing an iterative flood fill operation, starting with the initial pixels of the object, each iteration of the flood fill operation comprising adding a plurality of neighboring pixels of the frame to pixels of the object when distances between the neighboring pixels and the pixels of the object are within a threshold distance;

determine a ground plane in the frame, the object resting on the ground plane;

compute a rectangular outline bounding a projection of a plurality of pixels of the object onto the ground plane;

compute a height of the object above the ground plane;

output computed dimensions of the object in accordance with a length and a width of the rectangular outline and the height, and

wherein the executable instructions include instructions that are executable to configure the computer system to further determine the computed dimensions in accordance with a box mode, the object comprising two vertical faces, the instructions corresponding to the box mode comprising executable instructions that, when executed by the one or more processors, cause the computer system to:

identify a corner of the object, wherein the corner of the object is located at an intersection of two lines formed in the projection of the pixels of the vertical faces of the object onto the ground plane;

compute the height of the object above the ground plane by computing heights of top edges of the two vertical faces; and

compute dimensions of vertical planes of the object.

28. The computer system of claim 27 , wherein the executable instructions include instructions that are executable to configure the computer system to cause a display device to:

display a view of the color image of the scene captured by a depth camera;

overlay a reticle on the view in the display device; and

wherein the one or more initial pixels of the object correspond to the pixels under the reticle.

29. The computer system of claim 28 , wherein the executable instructions include instructions that are executable to configure the computer system to cause a depth camera system to capture the frame in response to detecting an activation of a trigger, wherein the one or more initial pixels of the object correspond to the pixels under the reticle when the trigger is activated.

30. The computer system of claim 27 , wherein the executable instructions include instructions that are executable to configure the computer system to segment the object from the scene by:

defining a graph, wherein each vertex of the graph corresponds to a vertical projection of the pixels of the scene onto the ground plane and wherein two vertices are connected by an edge when their distance is smaller than threshold;

detecting connected components of the vertical projection of the pixels; and

identifying the largest connected component of the graph as the pixels of the object.

31. The computer system of claim 27 , wherein the executable instructions include instructions that are executable to configure the computer system to output the computed dimensions on a display device.

32. The computer system of claim 27 , wherein the executable instructions include instructions that are executable to configure the computer system to compute the heights of the top edges of the two vertical faces by:

dividing the ground plane into a plurality of cells;

selecting a plurality of cells comprising the lines;

computing a maximum height of each cell based on the pixels of the object in each of the cells; and

computing a height of a box based on the maximum heights of the cells.

33. The computer system of claim 27 , wherein the executable instructions include instructions that are executable to configure the computer system to activate or deactivate the box mode based on a user interface switch.

34. The computer system of claim 27 , wherein the executable instructions include instructions that are executable to configure the computer system to activate the box mode in response to detecting that the object comprises two vertical planes arranged at right angles to the ground plane and at right angles to each other.

35. The computer system of claim 27 , wherein the computed dimensions of the object correspond to dimensions of a box tightly fitting the object.

Assignments (5)

Continuity (4)

References Cited (29)