IP Library Granted Patent US 7,319,479
Granted Patent B1
US 7,319,479 · App. 09/960,218 · Granted Jan 15, 2008

System and method for multi-camera linking and analysis

View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 7,319,479
App. No.
09/960,218
Granted
Jan 15, 2008
Kind
B1
Abstract

The present invention is directed to a system and method for tracking objects, such as customers in a retail environment. The invention is divided into two distinct software subsystems, the Customer Tracking system and the Track Analysis system. The Customer Tracking System tracks individuals through multiple cameras and reports various types of information regarding the tracking function, such as location by time and real world coordinates. The Track Analysis system converts individual tracks to reportable actions. Inputs to the Customer Tracking System are a set of analog or digital cameras that are positioned to view some physical environment. The output of the Customer Tracking System is a set of customer tracks that contain time and position information that describe the path that an individual took through the environment being monitored. The Track Analysis System reads the set of customer tracks produced by the tracking system and interprets the activities of individuals based on user defined regions of interest (ROI). Examples of a region of interest are: (1) a teller station in a bank branch, (2) a check-out lane in a retail store, (3) an aisle in a retail store, and the like.

Claims (51)

1. A computer-readable medium encoded with a computer program representing instructions to cause a computer to:

analyze information used to define a first path, the information used to define the first path including a first plurality of data elements, each data element from the first plurality of data elements associated with the first path including a spatial value and a time value, each spatial value associated with the first path indicating a position of a person associated with the first path at a time associated with the corresponding time value;

analyze information used to define a second path, the information used to define the second path including a second plurality of data elements, each data element from the second plurality of data elements associated with the second path including a spatial value and a time value, each spatial value associated with the second path indicating a position of a person associated with the second path at a time associated with the corresponding time value, the spatial values used to define the second path including spatial values not included in the information used to define the first path;

determine, at least partially based on the analysis of the information used to define the first path and the analysis of the information used to define the second path, if the person associated with the first path and the person associated with the second path are the same person; and

remove a data element from the first plurality of data elements when the data element is associated with an exclusion region, the data element associated with the exclusion region being removed based on a region of interest filter, the data element being removed to prevent linking of the first path with the second path based on the data element.

2. The computer-readable medium of claim 1 , wherein the computer program further representing instructions to cause the computer to:

output information used to define a third path, if it is determined that the person associated with the first path and the person associated with the second path are the same person, the information used to define the third path including the plurality of spatial values used to define the first path and at least a portion of the plurality of spatial values used to define the second path thereby creating a path representing a limiting of the first and second paths.

3. The computer-readable medium of claim 1 , wherein the information used to define the first path is information from a video image associated with the first path, each spatial value used to define the first path being a spatial value within the video image used to define the first person, the information associated with the second path being information from a video image associated with the second path, each spatial value used to define the second path being a spatial value within the video image.

4. The computer-readable medium of claim 3 , wherein the computer program further representing instructions to cause the computer to:

convert each spatial value used to define the first path to a spatial value within a universal coordinate system; and

convert each spatial value used to define the second path to a spatial value within the universal coordinate system.

5. The computer-readable medium of claim 1 , wherein the computer program further representing instructions to cause the computer to:

determine a confidence value that the person associated with the first path and the person associated with the second path are the same person, the computer program representing instructions to cause the computer to determine if the person associated with the first path and the person associated with the second path are the same person being configured to make a determination at least partially based on the confidence value.

6. The computer-readable medium of claim 1 , wherein the computer program representing instructions to cause the computer to determine if the person associated with the first path and the person associated with the second path are the same person includes instructions to determine at least whether an end spatial value of the first path is within a predetermined distance of a start spatial value of the second path, an end spatial value of the first path having a corresponding time value that is chronologically last of all time values uniquely associated with the plurality of spatial values used to define the first path, a start spatial value of the second path having a corresponding time value that is chronologically first of all time values uniquely associated with the plurality of spatial values used to define the second path.

7. The computer-readable medium of claim 1 , wherein the computer program representing instructions to cause the computer to determine if the person associated with the first path and the person associated with the second path are the same person includes instructions to determine at least whether a time value corresponding to an end spatial value of the first path is within a predetermined time of a time value corresponding to a start spatial value of the second path, the end spatial value of the first path having a corresponding time value that is chronologically last of all time values uniquely associated with the plurality of spatial values used to define the first path, the start spatial value of the second path having a corresponding time value that is chronologically first of all time values uniquely associated with the plurality of spatial values used to define the second path.

8. A computer-readable medium encoded with a computer program representing instructions to cause a computer to:

receive information associated with a plurality of paths, each path from the plurality of paths representing movement of an object defined over time;

iteratively determine, for each path from the plurality of paths, whether that path can be linked to another path from the plurality of paths at least partially based on a predetermined linking rule;

produce a first link between a first path from the plurality of paths and a second path from the plurality of paths based on the iteratively determining;

produce a second link between a third path from the plurality of paths and a fourth path from the plurality of paths; and

remove the first link between the first path and the second path and remove the second link between the third path and the fourth path when information associated with the first link and information associated with the second link satisfy a path-breaking rule,

the path-breaking rule includes a time-space bubble condition, the information associated with the first link includes a spatial coordinate and corresponding time value associated with the first link, the information associated with the second link includes a spatial coordinate and corresponding time value associated with the second link, the path-breaking rule is satisfied when the spatial coordinate and corresponding time value associated with the first link and the spatial coordinate and corresponding time value associated with the second link satisfy the time-space bubble condition.

9. The computer-readable medium of claim 8 , wherein the computer program representing instructions to cause the computer to resolve is configured to remove a path from a set of linked paths.

10. The computer-readable medium of claim 8 , wherein each path from the plurality of paths includes a plurality of data elements, each data element from the plurality of data elements associated with a path from the plurality of paths including a spatial value and a time value, the computer program representing instructions to cause the computer to iteratively determine being configured to extract and store at least one data element associated with each path from the plurality of paths, the at least one data element including at least one of a start spatial value, an end spatial value, a length between a start spatial value and an end spatial value.

11. The computer-readable medium of claim 10 , wherein the computer program further representing instructions to cause a computer to:

sort each path from the plurality of paths based on start time values associated with the plurality of paths;

sort each path from the plurality of paths based on end time values associated with the plurality of paths; and

iteratively compare each path from the plurality of paths sorted based on start time values with each path from the plurality of paths sorted based on end time values.

12. The computer-readable medium of claim 8 , wherein each path from the plurality of paths includes a plurality of data elements, each data element from the plurality of data elements associated with a path from the plurality of paths including a spatial value and a time value, the computer program representing instructions to cause the computer to resolve conflicts being configured to determine if at least one of

a distance between an end spatial value of a first path from the plurality of paths and a start spatial value of a second path from the plurality of paths is within a predetermined distance threshold or

a time between an end time value of the first path from the plurality of paths and a start time value of the second path from the plurality of paths is within a predetermined time threshold.

13. An apparatus, comprising:

a first image capture device configured to capture a plurality of images associated with a first physical area over a time period;

a second image capture device configured to capture a plurality of images associated with a second physical area over the time period, the second physical area being substantially different from the first physical area;

a processor in communication with the first image capture device and the second image capture device, the processor being configured to extract a plurality of sets of spatial values and corresponding time values associated with the first physical area from the images associated with the first physical area, the processor being configured to extract a plurality of sets of spatial values and corresponding time values associated with the second physical area from the images associated with the second physical area,

each set from the plurality of sets of spatial values and corresponding time values associated with the first physical area being used to define a path of an object within the first physical area, each set from the plurality of sets of spatial values and corresponding time values associated with the second physical area being used to define a path of an object within the second physical area,

the processor being configured to associate the path of the object within the first physical area and the path of the object within the second physical area if the processor determines that the plurality of sets of spatial values and corresponding time values associated with the object within the first physical area and the plurality of sets of spatial values and corresponding time values associated with the object within the second physical area satisfy a time-space bubble criteria,

the processor being configured to prevent merging of the path of the object within the first physical area and the path of the object within the second physical area into a combined path of the object if the processor determines that the combined path will intersect an exclusion region.

14. The computer-readable medium of claim 1 , wherein the computer program representing instructions to cause the computer to:

produce a link between the first path and the second path when the person associated with the first path and the person associated with the second path are the same person; and

resolve a conflict associated with the link based on a predetermined conflict-resolution rule.

15. A computer-readable medium encoded with a computer program representing instructions to cause a computer to:

analyze information used to define a first path, the information used to define the first path including a first plurality of data elements, each data element from the first plurality of data elements associated with the first path including a spatial value and a time value, each spatial value associated with the first path indicating a position of a person associated with the first path at a time associated with the corresponding time value;

analyze information used to define a second path, the information used to define the second path including a second plurality of data elements, each data element from the second plurality of data elements associated with the second path including a spatial value and a time value, each spatial value associated with the second path indicating a position of a person associated with the second path at a time associated with the corresponding time value, the spatial values used to define the second path including spatial values not included in the information used to define the first path;

determine, at least partially based on the analysis of the information used to define the first path and the analysis of the information used to define the second path, if the person associated with the first path and the person associated with the second path are the same person; and

prevent merging of the first path and the second path into a third path when the third path intersects an exclusion region.

16. The computer-readable medium of claim 1 , wherein the computer program representing instructions to cause the computer to:

produce a link between the first path and the second path when the person associated with the first path and the person associated with the second path are the same person; and

remove the link based on a predetermined conflict-resolution rule.

17. The computer-readable medium of claim 8 , wherein the predetermined conflict-resolution rule includes a path-breaking rule, the link being removed when the path-breaking rule is satisfied.

18. The apparatus of claim 13 , wherein the processor is configured to resolve a conflict associated with the link.

Assignments (13)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 29, 2017
From: POINT GREY RESEARCH INC.
To: FLIR INTEGRATED IMAGING SOLUTIONS, INC.
Reel/Frame 042866/0316 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 29, 2017
From: FLIR INTEGRATED IMAGING SOLUTIONS, INC.
To: FLIR COMMERCIAL SYSTEMS, INC.
Reel/Frame 042866/0713 →
RELEASE OF SECURITY INTEREST Recorded Oct 26, 2016
From: WESTERN ALLIANCE BANK, AS SUCCESSOR IN INTEREST TO BRIDGE BANK, NATIONAL ASSOCIATION
To: NOMI CORPORATION F/K/A BRICKSTREAM CORPORATION
Reel/Frame 040142/0953 →
RELEASE OF SECURITY INTEREST Recorded Oct 21, 2016
From: HORIZON TECHNOLOGY FINANCE CORPORATION
To: NOMI CORPORATION
Reel/Frame 040093/0912 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 25, 2016
From: NOMI (ASSIGNMENT FOR THE BENEFIT OF CREDITORS), LLC
To: POINT GREY RESEARCH INC.
Reel/Frame 039247/0904 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 25, 2016
From: NOMI CORPORATION
To: NOMI (ASSIGNMENT FOR THE BENEFIT OF CREDITORS), LLC
Reel/Frame 039247/0851 →
SECURITY INTEREST Recorded Jan 11, 2016
From: NOMI CORPORATION
To: HORIZON TECHNOLOGY FINANCE CORPORATION
Reel/Frame 037448/0920 →
CHANGE OF NAME Recorded Feb 20, 2015
From: BRICKSTREAM CORPORATION
To: NOMI CORPORATION
Reel/Frame 035058/0562 →
SECURITY AGREEMENT Recorded Apr 29, 2013
From: BRICKSTREAM CORPORATION
To: BRIDGE BANK, NATIONAL ASSOCIATION
Reel/Frame 030312/0582 →
RELEASE OF SECURITY INTEREST Recorded Apr 26, 2013
From: COMERICA BANK
To: BRICKSTREAM CORPORATION, A DELAWARE CORPORATION
Reel/Frame 030291/0942 →
SECURITY AGREEMENT Recorded Oct 3, 2008
From: BRICKSTREAM CORPORATION
To: COMERICA BANK
Reel/Frame 021630/0456 →
CHANGE OF NAME Recorded Aug 29, 2003
From: EMTERA CORPORATION
To: BRICKSTREAM CORPORATION
Reel/Frame 014440/0460 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 18, 2002
From: MOED, MICHAEL C.; CRABTREE, RALPH N.
To: EMTERA CORPORATION
Reel/Frame 012508/0364 →