IP Library Granted Patent US 11,334,739
Granted Patent B2
US 11,334,739 · App. 16/888,273 · Granted May 17, 2022

Barcode readers with 3D camera(s)

Inventors: Yuri Astvatsaturov (Lake Forest, IL); Darran Michael Handshaw (Sound Beach, NY); Edward Barkan (Miller Place, NY); Mark Drzymala (Saint James, NY)
Assignee: Zebra Technologies Corporation
G06K7/1417G06K9/6257G06V10/42G06V10/56G06V20/647G06V40/107G08B13/19613
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,334,739
App. No.
16/888,273
Granted
May 17, 2022
Kind
B2
Abstract

Methods and systems include using three-dimensional imaging apparatuses to capture three-dimensional images and analyze resulting three-dimensional image data to enhance captured two-dimensional images for scanning related processes such as object identification, symbology detection, object recognition model training, and identifying improper scan attempts or other actions performed by an operator. Imaging systems such as bi-optic readers, handheld scanners, and machine vision systems are described using three-dimensional imaging apparatuses and described capturing three-dimensional images and using with captured two-dimensional images.

Claims (60)

1. A method of barcode scanning using a barcode reader, the method comprising:

capturing, using a two-dimensional (2D) imaging apparatus within the barcode reader and having a first field of view (FOV), a 2D image of a first environment appearing within the first FOV and storing 2D image data corresponding to the 2D image;

capturing, using a three-dimensional (3D) imaging apparatus associated with the barcode reader and having a second FOV that overlaps at least partially with the first FOV, a 3D image of a second environment appearing within the second FOV and storing 3D image data corresponding to the 3D image;

identifying one or more 3D image features within the second environment from the 3D image data;

enhancing the 2D image data, resulting in enhanced 2D image data, by correlating the one or more 3D image features to at least one or more 2D image features in the 2D image data; and

processing the enhanced 2D image data to at least one of (a) decode a barcode captured within the enhanced 2D image data, (b) train an object recognition model with the enhanced 2D image data, (c) recognize an object within the enhanced 2D image data, (d) identify an action performed by an operator of the barcode reader, and (e) change at least one parameter associated with the barcode reader operation.

2. The method of claim 1 , wherein the 3D image data comprises 3D point cloud data and the one or more 3D image features includes one or more geometric features of an object presented within the second FOV.

3. The method of claim 1 , wherein the 3D image data comprises 3D point cloud data and the one or more 3D image features includes a color or a color gradation corresponding an object presented within the second FOV.

4. The method of claim 1 , wherein the identifying the one or more 3D image features includes identifying the one or more 3D image features that are positioned within a predetermined distance range away from the 3D imaging apparatus.

5. The method of claim 4 , wherein the enhancing the 2D image data further includes filtering the at least one or more 2D image features such that the processing the enhanced 2D image data excludes processing of image data associated with the at least one or more 2D image features.

6. The method of claim 4 , wherein the enhancing the 2D image data further includes filtering the at least one or more 2D image features such that the processing the enhanced 2D image data is limited to processing of image data associated with the at least one or more 2D image features.

7. The method of claim 4 , wherein the barcode reader is a stationary barcode reader configured to be positioned within a workstation and operated by the operator, and

wherein the predetermined distance range away from the 3D imaging apparatus extends from the 3D imaging apparatus to an edge of the workstation proximate the operator.

8. The method of claim 4 , wherein the barcode reader is a bi-optic barcode reader having a product-scanning region, and

wherein the predetermined distance range away from the 3D imaging apparatus extends from the 3D imaging apparatus to a distal boundary of the product-scanning region.

9. The method of claim 1 , wherein the one or more 3D image features includes at least one of (i) at least a portion of an operator's hand, and (ii) an object grasped by the operator's hand.

10. The method of claim 1 , wherein the processing the enhanced 2D image data includes the identifying the action performed by the operator, and

responsive to the action performed by the operator being identified as one of presenting an object within a product-scanning region and presenting the object proximate to the product-scanning region, and further responsive to no barcode being detected within at least one of the 2D image data and the enhanced 2D image data, generating an alert suitable for signaling a potential theft event.

11. The method of claim 1 , wherein the processing the enhanced 2D image data includes the identifying the action performed by the operator, and

responsive to the action performed by the operator being identified as one of presenting an object within a product-scanning region and presenting the object proximate to the product-scanning region, and detecting a partially covered barcode or fully covered barcode on the object within at least one of the 2D image data and the enhanced 2D image data, generating an alert suitable for signaling a potential theft event.

12. The method of claim 1 , wherein the at least one parameter associated with the barcode reader operation is an exposure time of the barcode reader, an illumination pulse duration of the barcode reader, a focus position of the barcode reader, an imaging zoom level of the barcode reader, and an illumination source of the barcode reader.

13. The method of claim 12 , wherein the illumination source is a diffuse illumination source or a direct illumination source.

14. The method of claim 1 , wherein, in response to identifying the one or more 3D image features within the second environment from the 3D image data, adjusting an illumination brightness of the barcode reader prior to capturing the 2D image of the first environment.

15. A method processing data using a barcode reader, the method comprising:

capturing, using a two-dimensional (2D) imaging apparatus within the barcode reader and having a first field of view (FOV), a 2D image of a first environment appearing within the first FOV and storing 2D image data corresponding to the 2D image;

capturing, using a three-dimensional (3D) imaging apparatus associated with the barcode reader and having a second FOV that overlaps at least partially with the first FOV, a 3D image of a second environment appearing within the second FOV and storing 3D image data corresponding to the 3D image;

identifying one or more 2D image features within the first environment from the 2D image data;

enhancing the 3D image data, resulting in enhanced 3D image data, by correlating the one or more 2D image features to at least one or more 3D image features in the 3D image data; and

processing the enhanced 3D image data to at least one of (a) train an object recognition model with the enhanced 3D image data, (b) recognize an object within the enhanced 3D image data, (c) identify an action performed by a user of the barcode reader, and (d) change at least one parameter associated with the 3D imaging apparatus.

16. The method of claim 15 , wherein the 2D image data comprises one of monochrome image data, grayscale image data, and polychrome image data, and

wherein the one or more 2D image features includes at least one of a barcode and one or more geometric features of an object presented within the first FOV.

17. The method of claim 16 ,

wherein the one or more 2D image features includes the barcode, and

wherein the enhancing the 3D image data includes mapping a location of the barcode from the 2D image data to the 3D image data.

18. The method of claim 16 ,

wherein the 2D image data comprises polychrome image data,

wherein the one or more 2D image features includes the one or more geometric features of the object presented within the first FOV, and

wherein the enhancing the 3D image data includes mapping at least a portion of the polychrome image data to the 3D image data based at least in part on the one or more geometric features of the object presented within the first FOV.

19. The method of claim 16 , wherein the enhancing the 3D image data further includes filtering the at least one or more 3D image features such that the processing the enhanced 3D image data excludes processing of image data associated with the at least one or more 3D image features.

20. The method of claim 16 , wherein the enhancing the 3D image data further includes filtering the at least one or more 3D image features such that the processing the enhanced 3D image data is limited to processing of image data associated with the at least one or more 3D image features.

21. The method of claim 16 , wherein the enhancing the 3D image data further includes filtering the 3D image data based on a predetermined distance range away from the 3D imaging apparatus.

22. The method of claim 21 , wherein the barcode reader is a stationary barcode reader configured to be positioned within a workstation and operated by the operator, and

wherein the predetermined distance range away from the 3D imaging apparatus extends from the 3D imaging apparatus to an edge of the workstation proximate the operator.

23. The method of claim 21 , wherein the barcode reader is a bi-optic barcode reader having a product-scanning region, and

wherein the predetermined distance range away from the 3D imaging apparatus extends from the 3D imaging apparatus to a distal boundary of the product-scanning region.

24. The method of claim 15 , wherein the one or more 2D image features includes at least one of (i) at least a portion of an operator's hand, and (ii) an object grasped by the operator's hand.

25. The method of claim 15 , wherein the processing the enhanced 3D image data includes the identifying the action performed by the operator, and

responsive to the action performed by the operator being identified as one of presenting an object within a product-scanning region and presenting the object proximate to the product-scanning region, and further responsive to no barcode being detected within the 2D image data generating an alert suitable for signaling a potential theft event.

26. The method of claim 15 , wherein identifying the one or more 2D image features comprises:

identifying environmental features on the 2D image, wherein the environmental features are features in the image outside of an object presented within the first FOV;

converting the environmental features into masking features configured to cover the identified environmental features in the 2D image; and

identifying the masking features as the one or more 2D image features.

27. The method of claim 15 , wherein identifying one or more 2D image features comprises:

identifying a barcode for the object in the 2D image data;

decoding the barcode to generate barcode payload data and determining object identification from the barcode payload data; and

determining, from the objecting identification, the one or more 2D image features.

28. The method of claim 15 , wherein processing the enhanced 3D image data to train the object recognition model with the enhanced 3D image data comprises:

identifying a barcode in the 2D image data, determining a barcode detection event timeframe, and training the object recognition model with the enhanced 3D image data corresponding to the barcode detection event timeframe; or

identifying a barcode in the 2D image data, identifying the object within the enhanced 3D image data corresponding to the barcode in the 2D image data, and upon identifying other object in the 3D image not corresponding to the barcode, removing the other objects from the enhanced 3D image data before training the object recognition model with the enhanced 3D image data.

29. The method of claim 15 , wherein the at least one parameter associated with the 3D imaging apparatus comprises a projected amount of illumination of the 3D imaging apparatus, a projected direction of illumination of the 3D imaging apparatus, or a source of illumination for the 3D imaging apparatus.

Assignments (4)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 26, 2021
From: ASTVATSATUROV, YURI; HANDSHAW, DARRAN MICHAEL; BARKAN, EDWARD; DRZYMALA, MARK
To: ZEBRA TECHNOLOGIES CORPORATION
Reel/Frame 056360/0818 →
SECURITY INTEREST Recorded Apr 12, 2021
From: ZEBRA TECHNOLOGIES CORPORATION
To: JPMORGAN CHASE BANK, N.A.
Reel/Frame 056471/0868 →
RELEASE OF SECURITY INTEREST - 364 - DAY Recorded Mar 5, 2021
From: JPMORGAN CHASE BANK, N.A.
To: ZEBRA TECHNOLOGIES CORPORATION; LASER BAND, LLC; TEMPTIME CORPORATION
Reel/Frame 056036/0590 →
SECURITY INTEREST Recorded Sep 1, 2020
From: ZEBRA TECHNOLOGIES CORPORATION; LASER BAND, LLC; TEMPTIME CORPORATION
To: JPMORGAN CHASE BANK, N.A.
Reel/Frame 053841/0212 →