Granted Patent B2

US 11,557,124 · App. 17/104,959 · Granted Jan 17, 2023

Inventors: Shahmeer Ali Mirza (Celina, TX); Sailesh Bharathwaaj Krishnamurthy (Irving, TX); Madan Mohan Chinnam (Irving, TX); Crystal Maung (Dallas, TX)

Assignee: 7-ELEVEN, INC.

G06V20/52G06T7/292G06V10/255G06V20/41G06T7/80G06T2207/30208G06V20/44G06V2201/07

Abstract

Claims (85)

1. An object tracking system, comprising:

a sensor configured to capture frames of a global plane for at least a portion of a space, wherein:

the global plane represents (x,y) coordinates for the at least a portion of the space;

each frame comprises a plurality of pixels; and

each pixel from the plurality of pixels is associated with a pixel location comprising a pixel row and a pixel column; and

a tracking system operably coupled to the sensor, comprising:

one or more memories operable to store a homography associated with the sensor, wherein the homography comprises coefficients that translate between pixel locations in a frame and (x,y) coordinates in the global plane; and

one or more processors operably coupled to the one or more memories, configured to:

receive a first frame from the sensor;

identify a pixel location within the first frame;

determine an estimated sensor location for the sensor by applying the homography to the pixel location;

determine an actual sensor location for the sensor;

determine a location difference between the estimated sensor location and the actual sensor location;

compare the location difference to a difference threshold level; and

recompute the homography in response to determining that the location difference exceeds the difference threshold level.

2. The system of claim 1 , wherein the pixel location corresponds with a pixel in the center of the frame.

3. The system of claim 1 , further comprising a position sensor operably coupled to the sensor, wherein the position sensor is configured to output the actual sensor location for the sensor.

4. The system of claim 1 , wherein:

identifying the pixel location within the first frame comprises detecting a first marker within the first frame; and

the pixel location corresponds with a location of the first marker within the first frame.

5. The system of claim 1 , wherein the sensor is further configured to capture depth information.

6. The system of claim 1 , wherein the global plane is parallel with a floor of the space.

7. The system of claim 1 , wherein recomputing the homography comprises:

receive a first (x,y) coordinate identifying a first x-value and a first y-value in the global plane where a first marker is located in the space, wherein the first marker is a first object identifying a first location in the space;

receive a second (x,y) coordinate identifying a second x-value and a second y-value in the global plane where a second marker is located in the space, wherein the second marker is a second object identifying a second location in the space;

receive a second frame from the sensor;

identify the first marker and the second marker within the second frame;

determine a first pixel location in the second frame for the first marker, wherein the first pixel location comprises a first pixel row and a first pixel column of the second frame;

determine a second pixel location in the second frame for the second marker, wherein the second pixel location comprises a second pixel row and a second pixel column of the second frame; and

recompute the homography based on the first (x,y) coordinate, the second (x,y) coordinate, the first pixel location, and the second pixel location.

8. A homography error correction method, comprising:

receiving a first frame from a sensor, wherein the sensor configured to capture frames of a global plane for at least a portion of a space, wherein:

the global plane represents (x,y) coordinates for the at least a portion of the space;

each frame comprises a plurality of pixels; and

each pixel from the plurality of pixels is associated with a pixel location comprising a pixel row and a pixel column;

identifying a pixel location within the first frame;

determining an estimated sensor location for the sensor by applying a homography to the pixel location, wherein:

the homography is associated with the sensor; and

the homography comprises coefficients that translate between pixel locations in a frame and (x,y) coordinates in the global plane;

determining an actual sensor location for the sensor;

determining a location difference between the estimated sensor location and the actual sensor location;

comparing the location difference to a difference threshold level; and

recomputing the homography in response to determining that the location difference exceeds the difference threshold level.

9. The method of claim 8 , wherein the pixel location corresponds with a pixel in the center of the frame.

10. The method of claim 8 , wherein determining the actual sensor location for the sensor comprises receiving location information from a position sensor.

11. The method of claim 8 , wherein:

identifying the pixel location within the first frame comprises detecting a first marker within the first frame; and

the pixel location corresponds with a location of the first marker within the first frame.

12. The method of claim 8 , wherein the sensor is further configured to capture depth information.

13. The method of claim 8 , wherein the global plane is parallel with a floor of the space.

14. The method of claim 8 , wherein recomputing the homography comprises:

receiving a first (x,y) coordinate identifying a first x-value and a first y-value in the global plane where a first marker is located in the space, wherein the first marker is a first object identifying a first location in the space;

receiving a second (x,y) coordinate identifying a second x-value and a second y-value in the global plane where a second marker is located in the space, wherein the second marker is a second object identifying a second location in the space;

receiving a second frame from the sensor;

identifying the first marker and the second marker within the second frame;

determining a first pixel location in the second frame for the first marker, wherein the first pixel location comprises a first pixel row and a first pixel column of the second frame;

determining a second pixel location in the second frame for the second marker, wherein the second pixel location comprises a second pixel row and a second pixel column of the second frame; and

recomputing the homography based on the first (x,y) coordinate, the second (x,y) coordinate, the first pixel location, and the second pixel location.

15. A non-transitory computer readable medium storing computer instructions, wherein the computer instructions when executed by a processor causes the processor to:

receive a first frame from a sensor, wherein the sensor configured to capture frames of a global plane for at least a portion of a space, wherein:

the global plane represents (x,y) coordinates for the at least a portion of the space;

each frame comprises a plurality of pixels; and

each pixel from the plurality of pixels is associated with a pixel location comprising a pixel row and a pixel column;

identify a pixel location within the first frame;

determine an estimated sensor location for the sensor by applying a homography to the pixel location, wherein:

the homography is associated with the sensor; and

the homography comprises coefficients that translate between pixel locations in a frame and (x,y) coordinates in the global plane;

determine an actual sensor location for the sensor;

determine a location difference between the estimated sensor location and the actual sensor location;

compare the location difference to a difference threshold level; and

recompute the homography in response to determining that the location difference exceeds the difference threshold level.

16. The non-transitory computer readable medium of claim 15 , wherein the pixel location corresponds with a pixel in the center of the frame.

17. The non-transitory computer readable medium of claim 15 , wherein determining the actual sensor location for the sensor comprises receiving location information from a position sensor.

18. The non-transitory computer readable medium of claim 15 , wherein:

identifying the pixel location within the first frame comprises detecting a first marker within the first frame; and

the pixel location corresponds with a location of the first marker within the first frame.

19. The non-transitory computer readable medium of claim 15 , wherein the sensor is further configured to capture depth information.

20. The non-transitory computer readable medium of claim 15 , wherein recomputing the homography comprises:

receiving a first (x,y) coordinate identifying a first x-value and a first y-value in the global plane where a first marker is located in the space, wherein the first marker is a first object identifying a first location in the space;

receiving a second (x,y) coordinate identifying a second x-value and a second y-value in the global plane where a second marker is located in the space, wherein the second marker is a second object identifying a second location in the space;

receiving a second frame from the sensor;

identifying the first marker and the second marker within the second frame;

determining a first pixel location in the second frame for the first marker, wherein the first pixel location comprises a first pixel row and a first pixel column of the second frame;

determining a second pixel location in the second frame for the second marker, wherein the second pixel location comprises a second pixel row and a second pixel column of the second frame; and

recomputing the homography based on the first (x,y) coordinate, the second (x,y) coordinate, the first pixel location, and the second pixel location.

Assignments (2)

Continuity (32)

Cited By (2)

US 12,272,091 US 12,277,732