IP Library Granted Patent US 11,430,222
Granted Patent B2
US 11,430,222 · App. 16/941,415 · Granted Aug 30, 2022

Sensor mapping to a global coordinate system using a marker grid

Inventors: Shahmeer Ali Mirza (Celina, TX); Sailesh Bharathwaaj Krishnamurthy (Irving, TX); Crystal Maung (Dallas, TX)
Assignee: 7-ELEVEN, INC.
G06V20/52G06T7/292G06V10/255G06V20/41G06T2207/30208G06V20/44G06V2201/07
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,430,222
App. No.
16/941,415
Granted
Aug 30, 2022
Kind
B2
Abstract

An object tracking system includes a sensor and a tracking system. The sensor is configured to capture a first frame of a global plane for at least a portion of a marker grid in a space. The tracking system is configured to receive a first coordinate in the global plane for a first corner of a marker grid, to determine a second coordinate in the global plane for the first marker on the marker grid, and to determine a third coordinate in the global plane where the second marker on the marker grid. The tracking system is further configured to determine a first pixel location for the first marker, to determine a second pixel location for the second marker, and to generate a homography based on the second coordinate for the first marker, the third coordinate for the second marker, the first pixel location, and the second pixel location.

Claims (116)

1. An object tracking system, comprising:

a first sensor configured to capture a first frame of a global plane for at least a portion of a marker grid in a space, wherein:

the global plane represents (x,y) coordinates for the at least a portion of the space;

the marker grid comprises a portable surface that comprises a first marker and a second marker visible on the portable surface;

the first marker is a first object that identifies a first location on the marker grid;

the second marker is a second object that identifies a second location on the marker grid;

the first frame comprises a plurality of pixels; and

each pixel from the plurality of pixels is associated with a pixel location comprising a pixel row and a pixel column; and

a tracking system operably coupled to the first sensor, comprising:

one or more processors configured to:

determine a first (x,y) coordinate identifying a first x-value and a first y-value in the global plane where the first marker is located based at least in part upon its position on the marker grid;

determine a second (x,y) coordinate identifying a second x-value and a second y-value in the global plane where the second marker is located based at least in part upon its position on the marker grid;

receive the first frame;

identify a first pixel within the first frame corresponding with the first marker;

identify a second pixel within the first frame corresponding with the second marker;

determine a first pixel location for the first pixel, wherein the first pixel location comprises a first pixel row and a first pixel column of the first frame;

determine a second pixel location for the second pixel, wherein the second pixel location comprises a second pixel row and a second pixel column of the first frame; and

generate a first homography based on the first (x,y) coordinate for the first marker, the second (x,y) coordinate for the second marker, the first pixel location, and the second pixel location, wherein the first homography maps between pixel locations in the first frame and (x,y) coordinates in the global plane.

2. The system of claim 1 , wherein identifying the first pixel within the first frame comprises identifying a first bounding box for the first marker within the first frame, wherein the first bounding box comprises a first plurality of pixels that contain at least a portion of the first marker.

3. The system of claim 1 , wherein the one or more processors are further configured to:

determine a number of identified markers within the first frame; and

determine that the number of identified markers exceeds a predetermined threshold value; and

wherein identifying the first marker is in response to determining that the number of identified markers exceeds the predetermined threshold value.

4. The system of claim 1 , wherein the tracking system further comprises:

one or more memories operable to store a marker dictionary comprising words;

the first marker comprises a word; and

identifying the first marker comprises:

identifying a word within the first frame;

comparing the identified word to the words in the marker dictionary; and

determining the identified word matches the word associated with the first marker in the marker dictionary.

5. The system of claim 1 , wherein the tracking system further comprises:

one or more memories operable to store a marker dictionary comprising symbols;

the first marker comprises a symbol; and

identifying the first marker comprises:

identifying a symbol within the first frame;

comparing the identified symbol to the symbols in the marker dictionary; and

determining the identified symbol matches the symbol associated with the first marker in the marker dictionary.

6. The system of claim 1 , wherein the one or more processors are further configured to:

receive a third (x,y) coordinate identifying a third x-value and a third y-value in the global plane where a first corner of the marker grid is located in the space;

receive a fourth (x,y) coordinate identifying a fourth x-value and a fourth y-value in the global plane where a second corner of the marker grid is located in the space;

determine a rotation angle within the global plane based on the third (x,y) coordinate for the first corner of the marker grid and the fourth (x,y) coordinate for the second corner of the marker grid; and

wherein determining the first (x,y) coordinate for the first marker comprises:

applying a translation using a first offset from the third (x,y) coordinate for the first corner of the marker grid; and

applying a rotation using the rotation angle about the third (x,y) coordinate for the first corner of the marker grid.

7. The system of claim 1 , wherein the first homography comprises coefficients that translate between the pixel locations in the first frame and (x,y) coordinates in the global plane.

8. The system of claim 1 , further comprising a second sensor operably coupled to the tracking system, configured to capture a second frame of the at least a portion of the marker grid in the space; and

wherein the one or more processors are further configured to:

determine a third pixel location in the second frame for the first marker;

determine a fourth pixel location in the second frame for the second marker; and

generate a second homography based on the third pixel location, the fourth pixel location, the first (x,y) coordinate for the first marker, and the second (x,y) marker for the second marker, wherein:

the second homography comprises coefficients that translate between the pixel locations in the second frame and (x,y) coordinates in the global plane; and

coefficients of the second homography are different from coefficients of the first homography.

9. The system of claim 1 , further comprising a second sensor operably coupled to the tracking system, configured to capture a second frame of at least a second portion of the space; and wherein:

the first sensor and the second sensor are members of a plurality of sensors configured as a sensor array; and

the sensor array is positioned parallel with the global plane.

10. The system of claim 1 , wherein:

each pixel in the first frame is associated with a pixel value; and

the first homography is further configured to translate between pixel values in the first frame and z-coordinates in the global plane.

11. The system of claim 1 , wherein the one or more processors are further configured to:

identify a first pixel marker within the first frame, wherein the first pixel marker is a first light source; and

identify a second pixel marker within the first frame, wherein the second pixel marker is a second light source.

12. The system of claim 1 , wherein the one or more processors are further configured to:

identify a first pixel marker within the first frame, wherein the first pixel marker is a first feature of the first marker; and

identify a second pixel marker within the first frame, wherein the second pixel marker is a second feature of the second marker.

13. A sensor mapping method, comprising:

capturing a first frame of a global plane for at least a portion of a marker grid in a space, wherein:

the global plane represents (x,y) coordinates for the at least a portion of the space;

the marker grid comprises a portable surface that comprises a first marker and a second marker visible on the portable surface;

the first marker is a first object that identifies a first location on the marker grid;

the second marker is a second object that identifies a second location on the marker grid;

the first frame comprises a plurality of pixels; and

each pixel from the plurality of pixels is associated with a pixel location comprising a pixel row and a pixel column;

determining a first (x,y) coordinate identifying a first x-value and a first y-value in the global plane where the first marker is located based at least in part upon its position on the marker grid;

determining a second (x,y) coordinate identifying a second x-value and a second y-value in the global plane where the second marker is located based at least in part upon its position on the marker grid;

identifying a first pixel within the first frame corresponding with the first marker;

identifying a second pixel within the first frame corresponding with the second marker;

determining a first pixel location for the first pixel, wherein the first pixel location comprises a first pixel row and a first pixel column of the first frame;

determining a second pixel location for the second pixel, wherein the second pixel location comprises a second pixel row and a second pixel column of the first frame; and

generating a first homography based on the first (x,y) coordinate for the first marker, the second (x,y) coordinate for the second marker, the first pixel location, and the second pixel location, wherein the first homography comprises maps between pixel locations in the first frame and (x,y) coordinates in the global plane.

14. The method of claim 13 , wherein identifying the first pixel within the first frame comprises identifying a first bounding box for the first marker within the first frame, wherein the first bounding box comprises a first plurality of pixels that contain at least a portion of the first marker.

15. The method of claim 13 , further comprising:

determining a number of identified markers within the first frame;

determining that the number of identified markers exceeds a predetermined threshold value; and

identifying the first marker is in response to determining that the number of identified markers exceeds the predetermined threshold value.

16. The method of claim 13 , wherein identifying the first marker comprises:

identifying a word within the first frame;

comparing the identified word to words in a marker dictionary; and

determining the identified word matches a word associated with the first marker in the marker dictionary.

17. The method of claim 13 , wherein identifying the first marker comprises:

identifying a symbol within the first frame;

comparing the identified symbol to the symbols in a marker dictionary; and

determining the identified symbol matches a symbol associated with the first marker in the marker dictionary.

18. The method of claim 13 , further comprising:

receiving a third (x,y) coordinate identifying a third x-value and a third y-value in the global plane where a first corner of the marker grid is located in the space;

receiving a fourth (x,y) coordinate identifying a fourth x-value and a fourth y-value in the global plane where a second corner of the marker grid is located in the space;

determining a rotation angle within the global plane based on the third (x,y) coordinate for the first corner of the marker grid and the fourth (x,y) coordinate for the second corner of the marker grid; and

wherein determining the first (x,y) coordinate for the first marker comprises:

applying a translation using a first offset from the third (x,y) coordinate for the first corner of the marker grid; and

applying a rotation using the rotation angle about the third (x,y) coordinate for the first corner of the marker grid.

19. The method of claim 13 , wherein the first homography comprises coefficients that translate between the pixel locations in the first frame and (x,y) coordinates in the global plane.

20. A computer program comprising executable instructions stored in a non-transitory computer readable medium that when executed by a processor causes the processor to:

capture a first frame of a global plane for at least a portion of a marker grid in a space, wherein:

the global plane represents (x,y) coordinates for the at least a portion of the space;

the marker grid comprises a portable surface that comprises a first marker and a second marker visible on the portable surface;

the first marker is a first object that identifies a first location on the marker grid;

the second marker is a second object that identifies a second location on the marker grid;

the first frame comprises a plurality of pixels; and

each pixel from the plurality of pixels is associated with a pixel location comprising a pixel row and a pixel column;

determine a first (x,y) coordinate identifying a first x-value and a first y-value in the global plane where the first marker is located based at least in part upon its position on the marker grid;

determine a second (x,y) coordinate identifying a second x-value and a second y-value in the global plane where the second marker is located based at least in part upon its position on the marker grid;

identify a first pixel within the first frame corresponding with the first marker;

identify a second pixel within the first frame corresponding with the second marker;

determine a first pixel location for the first pixel, wherein the first pixel location comprises a first pixel row and a first pixel column of the first frame;

determine a second pixel location for the second pixel, wherein the second pixel location comprises a second pixel row and a second pixel column of the first frame; and

generate a first homography based on the first (x,y) coordinate for the first marker, the second (x,y) coordinate for the second marker, the first pixel location, and the second pixel location, wherein the first homography comprises maps between pixel locations in the first frame and (x,y) coordinates in the global plane.

21. The computer program of claim 20 , wherein the first homography comprises coefficients that translate between the pixel locations in the first frame and (x,y) coordinates in the global plane.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 28, 2020
From: MIRZA, SHAHMEER ALI; KRISHNAMURTHY, SAILESH BHARATHWAAJ; MAUNG, CRYSTAL
To: 7-ELEVEN, INC.
Reel/Frame 053333/0892 →
Continuity (3)
Continuation 16794057 · Feb 18, 2020
Continuation 16663472 · Oct 25, 2019
Related Publication 20210124937A1 · Apr 29, 2021