IP Library Granted Patent US 10,783,656
Granted Patent B2
US 10,783,656 · App. 15/983,325 · Granted Sep 22, 2020

System and method of determining a location for placement of a package

Inventors: David Korobov (Wilmette, IL); Jay J. Williams (Glenview, IL); Cuneyt M. Taskiran (Chicago, IL); Yan Zhang (Buffalo Grove, IL); Sheida Gohari (Chicago, IL)
Assignee: Zebra Technologies Corporation
G06T7/70G06Q10/083G06T7/11G08B7/06G06T2207/10028G06T2207/20132
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,783,656
App. No.
15/983,325
Granted
Sep 22, 2020
Kind
B2
Abstract

A method of determining a location for placement of a package in an imaging controller includes: obtaining depth data of a package space; detecting an occluded portion of the depth data representing an area of the package space obstructed by an occlusion; retrieving stored unobstructed depth data representing the area of the package space in the absence of the occlusion; replacing the occluded portion with the unobstructed depth data to generate patched depth data; obtaining, based on the patched depth data, a location within the package space for placement of a package; and presenting an indication of the location.

Claims (74)

1. A method of determining a location for placement of a package in an imaging controller comprising:

obtaining a plurality of prior sets of depth data;

obtaining intermediate depth data representing a package space;

responsive to no occlusion being detected for the intermediate depth data, and no occlusion being detected for a threshold number of the plurality of prior sets of depth data, storing the intermediate depth data as an unobstructed depth data set;

obtaining depth data representing the package space;

detecting an occluded portion of the depth data representing an area of the package space obstructed by an occlusion;

retrieving from the stored unobstructed depth data set an unobstructed depth data representing the area of the package space in the absence of the occlusion;

replacing the occluded portion with the unobstructed depth data to generate patched depth data;

obtaining, based on the patched depth data, a location within the package space for placement of a package; and

presenting an indication of the location.

2. The method of claim 1 , further comprising:

detecting a package clutter portion of the depth data, the package clutter portion representing one or more staged or queued packages in the package space; and

cropping the package clutter portion out of the depth data.

3. The method of claim 2 , wherein detecting the package clutter portion of the depth data comprises:

detecting a package wall of the package space;

setting a distance threshold; and

identifying one or more subsets of the depth data having a smaller distance than the distance threshold, the subsets defining the package clutter portion.

4. The method of claim 1 , wherein the detecting comprises:

obtaining image data of the package space;

identifying a preliminary region containing the occlusion using the image data;

retrieving a subset of the depth data corresponding to the image data within the preliminary region; and

detecting the occluded portion within the subset of the depth data.

5. The method of claim 4 , wherein identifying the preliminary region comprises:

cropping the image data; and

providing the cropped image data to an occlusion detector to generate the preliminary region.

6. The method of claim 4 , wherein detecting the occluded portion comprises:

selecting clusters of the depth data by comparing the depth data to predefined occlusion identifiers defining structural properties of an expected occlusion.

7. The method of claim 1 , further comprising storing the patched depth data as the unobstructed depth data.

8. The method of claim 1 , further comprising:

prior to detecting the occluded portion, determining a proportion of missing depth data;

when the proportion is below a predefined threshold, detecting the occluded portion; and

when the proportion is above the predefined threshold, discarding the depth data.

9. The method of claim 1 , wherein presenting the indication comprises at least one of: displaying an augmented reality image, projecting the indication via a steerable pointing device, and providing an audio communication.

10. The method of claim 1 , further comprising: prior to obtaining the location, receiving a barcode scan of the package.

11. A computing device for determining a location for placement of a package, the computing device comprising:

a communications interface coupled to a data capture device;

a memory; and

an imaging controller connected to the memory and the communications interface, the imaging controller configured to:

obtain a plurality of prior sets of depth data;

obtain intermediate depth data representing the package space;

responsive to no occlusion being detected for the intermediate depth data, and no occlusion being detected for a threshold number of the plurality of prior sets of depth data, storing the intermediate depth data as an unobstructed depth data set;

obtain depth data representing a package space;

detect an occluded portion of the depth data representing an area of the package space obstructed by an occlusion;

retrieve from the stored unobstructed depth data set an unobstructed depth data representing the area of the package space in the absence of the occlusion;

replace the occluded portion with the unobstructed depth data to generate patched depth data;

obtain, based on the patched depth data, a location within the package space for placement of a package; and

present an indication of the location.

12. The computing device of claim 11 , wherein the imaging controller is further configured to:

detect a package clutter portion of the depth data, the package clutter portion representing one or more staged or queued packages in the package space; and

crop the package clutter portion out of the depth data.

13. The computing device of claim 12 , wherein the imaging controller is further configured to detect the package clutter portion of the depth data by:

detecting a package wall of the package space;

setting a distance threshold; and

identifying one or more subsets of the depth data having a smaller distance than the distance threshold, the subsets defining the package clutter portion.

14. The computing device of claim 11 , wherein the imaging controller is further configured to occluded portion by:

obtaining image data of the package space;

identifying a preliminary region containing the occlusion using the image data;

retrieving a subset of the depth data corresponding to the image data within the preliminary region; and

detecting the occluded portion within the subset of the depth data.

15. The computing device of claim 14 , wherein the imaging controller is further configured to identify the preliminary region by:

cropping the image data; and

providing the cropped image data to an occlusion detector to generate the preliminary region.

16. The computing device of claim 14 , wherein the imaging controller is further configured to detect the occluded portion by:

selecting clusters of the depth data by comparing the depth data to predefined occlusion identifiers defining structural properties of an expected occlusion.

17. The computing device of claim 11 , wherein the imaging controller is further configured to store the patched depth data in the memory as the unobstructed depth data.

18. The computing device of claim 11 , wherein the imaging controller is further configured to:

prior to detecting the occluded portion, determine a proportion of missing depth data;

when the proportion is below a predefined threshold, detect the occluded portion; and

when the proportion is above the predefined threshold, discard the depth data.

19. The computing device of claim 11 , wherein the imaging controller is further configured to present the indication by controlling at least one of:

a display to display an augmented reality image containing the indication;

a steerable pointing device to project the indication; and

a speaker to generate an audible indication.

20. The computing device of claim 11 , wherein the imaging controller is further configured, prior to obtaining the location, to receive a barcode scan of the package.

Assignments (5)
RELEASE OF SECURITY INTEREST - 364 - DAY Recorded Mar 5, 2021
From: JPMORGAN CHASE BANK, N.A.
To: ZEBRA TECHNOLOGIES CORPORATION; LASER BAND, LLC; TEMPTIME CORPORATION
Reel/Frame 056036/0590 →
SECURITY INTEREST Recorded Sep 1, 2020
From: ZEBRA TECHNOLOGIES CORPORATION; LASER BAND, LLC; TEMPTIME CORPORATION
To: JPMORGAN CHASE BANK, N.A.
Reel/Frame 053841/0212 →
SECURITY INTEREST Recorded Jul 3, 2019
From: ZEBRA TECHNOLOGIES CORPORATION
To: JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT
Reel/Frame 049674/0916 →
MERGER Recorded Jan 16, 2019
From: ZIH CORP.
To: ZEBRA TECHNOLOGIES CORPORATION
Reel/Frame 048470/0848 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 22, 2018
From: KOROBOV, DAVID; WILLIAMS, JAY J.; TASKIRAN, CUNEYT M.; ZHANG, YAN; GOHARI, SHEIDA
To: ZIH CORP.
Reel/Frame 045868/0982 →
Continuity (1)
Related Publication 20190355144A1 · Nov 21, 2019