Granted Patent B2

US 11,893,759 · App. 17/105,167 · Granted Feb 6, 2024

Inventors: Shahmeer Ali Mirza (Celina, TX); Sailesh Bharathwaaj Krishnamurthy (Irving, TX); Kyle Dalal (Coppell, TX)

Assignee: 7-ELEVEN, INC.

G06T7/73G06Q30/0641G06T7/292G06V10/44G06V10/764G06V20/41G06V20/52G06V40/103G06V40/107G06T2207/30208G06V20/44G06V2201/07

Abstract

Claims (89)

1. An object tracking system, comprising:

a first sensor configured to capture frames of a global plane for at least a first portion of a space, wherein:

the global plane represents (x,y) coordinates for the at least a portion of the space; and

each frame comprises a plurality of pixels;

a second sensor configured to capture frames of the global plane for at least a second portion of the space; and

a tracking system operably coupled to the first sensor and the second sensor, comprising:

one or more memories operable to store:

a disparity mapping between the first sensor and the second sensor, wherein the disparity mapping maps pixel locations from the first sensor to pixel locations from the second sensor;

a first homography associated with the first sensor, wherein the first homography comprises coefficients that translate between pixel locations in a frame from the first sensor and (x,y) coordinates in the global plane; and

a second homography associated with the second sensor, wherein the second homography comprises coefficients that translate between pixel locations in a frame from the second sensor and (x,y) coordinates in the global plane; and

one or more processors operably coupled to the one or more memories, configured to:

receive a first frame from the first sensor;

identify a first pixel location for a marker within the first frame;

determine an (x,y) coordinate for the marker in the global plane by applying the first homography to the first pixel location;

identify a second pixel location for the marker by applying the second homography to the (x,y) coordinate;

identify a third pixel location by applying the disparity mapping to the first pixel location;

determine a distance difference between the second pixel location and the third pixel location;

compare the distance difference to a difference threshold level; and

recompute at least one of the first homography and the second homography in response to determining that the distance difference exceeds the difference threshold level.

2. The system of claim 1 , wherein the disparity mapping is a function that maps between pixel locations from the first sensor and pixel locations from the second sensor.

3. The system of claim 1 , wherein the disparity mapping is a table that maps between pixel locations from the first sensor and pixel locations from the second sensor.

4. The system of claim 1 , wherein the first sensor and the second sensor are each further configured to capture depth information.

5. The system of claim 1 , wherein a field of view for the first sensor and field of view for the second sensor at least partially overlap.

6. The system of claim 1 , wherein the global plane is parallel with a floor of the space.

7. The system of claim 1 , wherein recomputing the homography comprises:

receiving a first (x,y) coordinate identifying a first x-value and a first y-value in the global plane where a first marker is located in the space, wherein the first marker is a first object identifying a first location in the space;

receiving a second (x,y) coordinate identifying a second x-value and a second y-value in the global plane where a second marker is located in the space, wherein the second marker is a second object identifying a second location in the space;

receiving a second frame from the first sensor;

identifying the first marker and the second marker within the second frame;

determining a third pixel location in the second frame for the first marker, wherein the third pixel location comprises a first pixel row and a first pixel column of the second frame;

determining a fourth pixel location in the second frame for the second marker, wherein the fourth pixel location comprises a second pixel row and a second pixel column of the second frame; and

recomputing the first homography based on the first (x,y) coordinate, the second (x,y) coordinate, the third pixel location, and the fourth pixel location.

8. A homography error correction method, comprising:

receiving a first frame from a first sensor, wherein the first sensor is configured to capture frames of a global plane for at least a first portion of a space, wherein:

the global plane represents (x,y) coordinates for the at least a portion of the space; and

each frame comprises a plurality of pixels;

identifying a first pixel location for a marker within the first frame;

determining an (x,y) coordinate for the marker in the global plane by applying a first homography to the first pixel location, wherein:

the first homography is associated with the first sensor; and

the first homography comprises coefficients that translate between pixel locations in a frame from the first sensor and (x,y) coordinates in the global plane;

identifying a second pixel location for the marker by applying a second homography to the (x,y) coordinate, wherein:

a second sensor is configured to capture frames of the global plane for at least a second portion of the space;

the second homography is associated with the second sensor; and

the second homography comprises coefficients that translate between pixel locations in a frame from the second sensor and (x,y) coordinates in the global plane;

identifying a third pixel location by applying a disparity mapping to the first pixel location, wherein the disparity mapping maps pixel locations from the first sensor to pixel locations from the second sensor;

determining a distance difference between the second pixel location and the third pixel location;

comparing the distance difference to a difference threshold level; and

recomputing at least one of the first homography and the second homography in response to determining that the distance difference exceeds the difference threshold level.

9. The method of claim 8 , wherein the disparity mapping is a function that maps between pixel locations from the first sensor and pixel locations from the second sensor.

10. The method of claim 8 , wherein the disparity mapping is a table that maps between pixel locations from the first sensor and pixel locations from the second sensor.

11. The method of claim 8 , wherein the first sensor and the second sensor are each further configured to capture depth information.

12. The method of claim 8 , wherein a field of view for the first sensor and field of view for the second sensor at least partially overlap.

13. The method of claim 8 , wherein the global plane is parallel with a floor of the space.

14. The method of claim 8 , wherein recomputing the homography comprises:

receiving a first (x,y) coordinate identifying a first x-value and a first y-value in the global plane where a first marker is located in the space, wherein the first marker is a first object identifying a first location in the space;

receiving a second (x,y) coordinate identifying a second x-value and a second y-value in the global plane where a second marker is located in the space, wherein the second marker is a second object identifying a second location in the space;

receiving a second frame from the first sensor;

identifying the first marker and the second marker within the second frame;

determining a third pixel location in the second frame for the first marker, wherein the third pixel location comprises a first pixel row and a first pixel column of the second frame;

determining a fourth pixel location in the second frame for the second marker, wherein the fourth pixel location comprises a second pixel row and a second pixel column of the second frame; and

recomputing the first homography based on the first (x,y) coordinate, the second (x,y) coordinate, the third pixel location, and the fourth pixel location.

15. A non-transitory computer readable medium storing instructions that when executed by a processor cause the processor to:

receive a first frame from a first sensor, wherein the first sensor is configured to capture frames of a global plane for at least a first portion of a space, wherein:

the global plane represents (x,y) coordinates for the at least a portion of the space; and

each frame comprises a plurality of pixels;

identify a first pixel location for a marker within the first frame;

determine an (x,y) coordinate for the marker in the global plane by applying a first homography to the first pixel location, wherein:

the first homography is associated with the first sensor; and

the first homography comprises coefficients that translate between pixel locations in a frame from the first sensor and (x,y) coordinates in the global plane;

identify a second pixel location for the marker by applying a second homography to the (x,y) coordinate, wherein:

the second sensor is configured to capture frames of the global plane for at least a second portion of the space;

the second homography is associated with the second sensor; and

the second homography comprises coefficients that translate between pixel locations in a frame from the second sensor and (x,y) coordinates in the global plane;

identify a third pixel location by applying a disparity mapping to the first pixel location, wherein the disparity mapping maps pixel locations from the first sensor to pixel locations from the second sensor;

determine a distance difference between the second pixel location and the third pixel location;

compare the distance difference to a difference threshold level; and

recompute at least one of the first homography and the second homography in response to determining that the distance difference exceeds the difference threshold level.

16. The non-transitory computer-readable medium of claim 15 , wherein the disparity mapping is a function that maps between pixel locations from the first sensor and pixel locations from the second sensor.

17. The non-transitory computer-readable medium of claim 15 , wherein the disparity mapping is a table that maps between pixel locations from the first sensor and pixel locations from the second sensor.

18. The non-transitory computer-readable medium of claim 15 , wherein the first sensor and the second sensor are each further configured to capture depth information.

19. The non-transitory computer-readable medium of claim 15 , wherein a field of view for the first sensor and field of view for the second sensor at least partially overlap.

20. The non-transitory computer-readable medium of claim 15 , wherein recomputing the homography comprises:

receiving a first (x,y) coordinate identifying a first x-value and a first y-value in the global plane where a first marker is located in the space, wherein the first marker is a first object identifying a first location in the space;

receiving a second (x,y) coordinate identifying a second x-value and a second y-value in the global plane where a second marker is located in the space, wherein the second marker is a second object identifying a second location in the space;

receiving a second frame from the first sensor;

identifying the first marker and the second marker within the second frame;

determining a third pixel location in the second frame for the first marker, wherein the third pixel location comprises a first pixel row and a first pixel column of the second frame;

determining a fourth pixel location in the second frame for the second marker, wherein the fourth pixel location comprises a second pixel row and a second pixel column of the second frame; and

recomputing the first homography based on the first (x,y) coordinate, the second (x,y) coordinate, the third pixel location, and the fourth pixel location.

Assignments (1)

Continuity (30)

Cited By (3)

US 12,254,650 US 12,272,091 US 12,380,726