IP Library Granted Patent US 11,188,763
Granted Patent B2
US 11,188,763 · App. 16/663,710 · Granted Nov 30, 2021

Topview object tracking using a sensor array

Inventors: Sailesh Bharathwaaj Krishnamurthy (Irving, TX); Shahmeer Ali Mirza (Celina, TX)
Assignee: 7-ELEVEN, INC.
G06K9/00771G06T7/215G06T7/223G06T7/246G06T7/292G06T2207/10021G06T2207/30232
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,188,763
App. No.
16/663,710
Granted
Nov 30, 2021
Kind
B2
Abstract

An object tracking system includes a first sensor, a second sensor, and a tracking system. The first sensor is configured to capture a first frame of a global plane for at least a first portion of a space. The second sensor is configured to capture a second frame of at least a second portion of the space. The tracking system is configured to determine the object is within an overlap region with the second sensor based on a first pixel location. The tracking system is further configured to determine a first coordinate in the global plane for the object, to determine a second pixel location in the second frame for the object based on the first coordinate, and to store the second pixel location with an object identifier a tracking list associated with the second sensor.

Claims (118)

1. An object tracking system, comprising:

a first sensor configured to capture a first frame of a global plane for at least a first portion of a space, wherein:

the global plane represents (x,y) coordinates for the at least a portion of the space;

the first frame comprises a plurality of pixels;

each pixel from the plurality of pixels is associated with a pixel location comprising a pixel row and a pixel column; and

each pixel in the first frame is associated with a pixel value that indicates a distance between the first sensor and a surface in the space;

a second sensor configured to capture a second frame of at least a second portion of the space, wherein the second portion of the space at least partially overlaps with the first portion of the space to define an overlap region; and

a tracking system operably coupled to the first sensor and the second sensor, comprising:

one or more memories operable to store:

a first homography associated with the first sensor, wherein_the first homography is configured to:

translate between pixel locations in the first frame and (x,y) coordinates in the global plane; and

translate between pixel values in the first frame and z-coordinates in the global plane;

a second homography associated with the second sensor, wherein the second homography is configured to translate between pixel locations in the second frame and (x,y) coordinates in the global plane;

a first tracking list associated with the first sensor, wherein the first tracking list identifies:

an object identifier for an object being tracked by the first sensor; and

pixel location information corresponding with a location of the object in the first frame; and

a second tracking list associated with the second sensor; and

one or more processors operably coupled to the one or more memories, configured to:

receive the first frame;

identify the object within the first frame;

determine a first pixel location in the first frame for the object, wherein the first pixel location comprises a first pixel row and a first pixel column of the first frame;

determine a first pixel value at the first pixel location;

determine the object is within the overlap region with the second sensor based on the first pixel location;

apply the first homography to the first pixel location to determine a first (x,y) coordinate identifying an x-value and a y-value in the global plane where the object is located;

apply the first homography to the first pixel value to determine a first height for the object;

identify the object identifier for the object from the first tracking list;

store the object identifier for the object in the second tracking list;

apply the second homography to the first (x,y) coordinate to determine a second pixel location in the second frame for the object, wherein the second pixel location comprises a second pixel row and a second pixel column of the second frame;

store the second pixel location with the object identifier for the object in the second tracking list; and

store the first height for the object in the second tracking list.

2. The system of claim 1 , wherein:

the pixel location information in the first tracking list identifies previous pixel locations for the object; and

the one or more processors are further configured to:

determine a travel direction for the object based on the previous pixel locations for the object; and

store the travel direction for the object with the object identifier for the object in the second tracking list.

3. The system of claim 1 , wherein:

the one or more memories are further operable to store an adjacency list that identifies a first range of pixels in the first frame that corresponds with the overlap region between the first frame and the second frame; and

determining the object is within the overlap region comprises determining that the pixel location for the object is within the first range of pixels in the first frame.

4. The system of claim 1 , further comprising a third sensor configured to capture a third frame of at least a third portion of the space, wherein the third portion of the space at least partially overlaps with the first portion of the space to define a second overlap region; and

wherein the one or more memories are further operable to store an adjacency list that:

identifies a first range of pixels in the first frame that corresponds with the overlap region between the first frame and the second frame; and

identifies a second range of pixels in the first frame that corresponds with the second overlap region between the first frame and the third frame.

5. The system of claim 1 , wherein:

the first sensor and the second sensor are members of a plurality of sensors configured as a sensor array; and

the sensor array is positioned parallel with the global plane.

6. The system of claim 1 , wherein the one or more processors are further configured to:

receive a third frame from the first sensor;

determine the object is not present in the third frame; and

discard information associated with the object from the first tracking list in response to determining that the object is not present in the third frame.

7. An object tracking method, comprising:

receiving a first frame of a global plane for at least a first portion of a space from a first sensor, wherein:

the global plane represents (x,y) coordinates for the at least a portion of the space;

the first frame comprises a plurality of pixels;

each pixel from the plurality of pixels is associated with a pixel location comprising a pixel row and a pixel column;

each pixel in the first frame is associated with a pixel value that indicates a distance between the first sensor and a surface in the space;

identifying the object within the first frame;

determining a first pixel location in the first frame for the object, wherein the first pixel location comprises a first pixel row and a first pixel column of the first frame;

determining a first pixel value at the first pixel location;

determining the object is within the overlap region with a second sensor based on the first pixel location, wherein the second sensor configured to capture a second frame of at least a second portion of the space;

applying a first homography to the first pixel location to determine a first (x,y) coordinate identifying an x-value and a y-value in the global plane where the object is located, wherein:

the first homography is configured to translate between pixel locations in the first frame and (x,y) coordinates in the global plane; and

the first homography is configured to translate between pixel values in the first frame and z-coordinates in the global plane;

applying the first homography to the first pixel value to determine a first height for the object;

identifying an object identifier for the object from a first tracking list, wherein the first tracking list identifies:

an object identifier for an object being tracked by the first sensor; and

pixel location information corresponding with a location of the object in the first frame;

storing the object identifier for the object in a second tracking list associated with the second sensor;

applying a second homography to the first (x,y) coordinate to determine a second pixel location in the second frame for the object, wherein:

the second pixel location comprises a second pixel row and a second pixel column of the second frame; and

the second homography is configured to translate between pixel locations in the second frame and (x,y) coordinates in the global plane;

storing the second pixel location with the object identifier for the object in the second tracking list; and

storing the first height for the object in the second tracking list.

8. The method of claim 7 , further comprising:

determining a travel direction for the object based on previous pixel locations for the object; and

storing the travel direction for the object with the object identifier for the object in the second tracking list.

9. The method of claim 7 , wherein determining the object is within the overlap region comprises determining that the pixel location for the object is within a first range of pixels that corresponds with the overlap region between the first frame and the second frame in the first frame.

10. The method of claim 7 , further comprising:

receiving a third frame from the first sensor;

determining the object is not present in the third frame; and

discarding information associated with the object from the first tracking list in response to determining that the object is not present in the third frame.

11. The method of claim 7 , wherein:

the first sensor and the second sensor are members of a plurality of sensors configured as a sensor array; and

the sensor array is positioned parallel with the global plane.

12. A computer program comprising executable instructions stored in a non-transitory computer readable medium that when executed by a processor causes the processor to:

receive a first frame of a global plane for at least a first portion of a space from a first sensor, wherein:

the global plane represents (x,y) coordinates for the at least a portion of the space;

the first frame comprises a plurality of pixels;

each pixel from the plurality of pixels is associated with a pixel location comprising a pixel row and a pixel column; and

each pixel in the first frame is associated with a pixel value that indicates a distance between the first sensor and a surface in the space;

identify the object within the first frame;

determine a first pixel location in the first frame for the object, wherein the first pixel location comprises a first pixel row and a first pixel column of the first frame;

determine a first pixel value at the first pixel location;

determine the object is within the overlap region with a second sensor based on the first pixel location, wherein the second sensor configured to capture a second frame of at least a second portion of the space;

apply a first homography to the first pixel location to determine a first (x,y) coordinate identifying an x-value and a y-value in the global plane where the object is located, wherein:

the first homography is configured to translate between pixel locations in the first frame and (x,y) coordinates in the global plane; and

the first homography is configured to translate between pixel values in the first frame and z-coordinates in the global plane;

apply the first homography to the first pixel value to determine a first height for the object;

identify an object identifier for the object from a first tracking list, wherein the first tracking list identifies:

the object identifier for an object being tracked by the first sensor; and

pixel location information corresponding with a location of the object in the first frame;

store the object identifier for the object in a second tracking list associated with the second sensor; and

store the first height for the object in the second tracking list.

13. The computer program of claim 12 , further comprising instructions that when executed by the processor causes the processor to:

apply a second homography to the first (x,y) coordinate to determine a second pixel location in the second frame for the object, wherein:

the second pixel location comprises a second pixel row and a second pixel column of the second frame; and

the second homography is configured to translate between pixel locations in the second frame and (x,y) coordinates in the global plane; and

store the second pixel location with the object identifier for the object in the second tracking list.

14. The computer program of claim 12 , further comprising instructions that when executed by the processor causes the processor to:

determine a travel direction for the object based on previous pixel locations for the object; and

store the travel direction for the object with the object identifier for the object in the second tracking list.

15. The computer program of claim 12 , wherein determining the object is within the overlap region comprises determining that the pixel location for the object is within a first range of pixels that corresponds with the overlap region between the first frame and the second frame in the first frame.

16. The computer program of claim 12 , further comprising instructions that when executed by the processor causes the processor to:

receive a third frame from the first sensor;

determine the object is not present in the third frame; and

discard information associated with the object from the first tracking list in response to determining that the object is not present in the third frame.

17. The computer program of claim 12 , wherein:

the first sensor and the second sensor are members of a plurality of sensors configured as a sensor array; and

the sensor array is positioned parallel with the global plane.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 25, 2019
From: KRISHNAMURTHY, SAILESH BHARATHWAAJ; MIRZA, SHAHMEER ALI
To: 7-ELEVEN, INC.
Reel/Frame 050825/0497 →
Continuity (1)
Related Publication 20210124935A1 · Apr 29, 2021
Cited By (1)
US 12,190,589