Trajectory cluster model for learning trajectory patterns in video data
View Patent ↗Techniques are disclosed for analyzing and learning behavior in an acquired stream of video frames. In one embodiment, a trajectory analyzer clusters trajectories of objects depicted in video frames and builds a trajectory model including the trajectory clusters, a prior probability of assigning a trajectory to each cluster, and an intra-cluster probability distribution indicating the probability that a trajectory mapping to each cluster is least various distances away from the cluster. Given a new trajectory, a score indicating how unusual the trajectory is may be computed based on the product of the probability of the trajectory mapping to a particular cluster and the intra-cluster probability of the trajectory being a computed distance from the cluster. The distance used to match the trajectory to the cluster and determine intra-cluster probability is computed using a parallel Needleman-Wunsch algorithm, with cells in antidiagonals of a matrix and connected sub-matrices being computed in parallel.
1. A computer-implemented method for analyzing trajectories of objects depicted in video frames captured by a plurality of video cameras, comprising:
acquiring, via a video driver, a plurality of data streams to track an object, the plurality of data streams being processed video frames from a plurality of video cameras, each data stream including a tracked subject identifier that specifies the object being tracked, and a camera identifier associated with a corresponding video camera of the plurality of video cameras;
calculating, via at least one processor, a first distance between a received trajectory for the object in a first data stream of the plurality of data streams and a stored trajectory cluster from a plurality of stored trajectory clusters based on the tracked subject identifier;
determining, based on the first distance, whether the received trajectory maps to the stored trajectory cluster; and
upon determining that the received trajectory maps to the stored trajectory cluster:
determining a first score based on at least a joint probability computed as a product of a probability of the received trajectory mapping to the stored trajectory cluster and the probability of the received trajectory being the first distance from the stored trajectory cluster, and
upon determining that the first score exceeds a threshold value, issuing an alert to a user, the alert indicating an anomaly in the received trajectory of the object.
2. The method of claim 1 , further comprising generating a new trajectory including an (x,y) position of the object upon determining that the received trajectory does not map to the stored trajectory cluster, and storing the new trajectory in a trajectory buffer.
3. The method of claim 1 , wherein:
the probability of the received trajectory mapping to the stored trajectory cluster is determined based on at least a count of previous trajectories for the object which mapped to the stored trajectory cluster based on the tracked subject identifier.
4. The method of claim 1 , wherein the first score is determined as S=1.0−Pr×f i , Pr being the probability of the received trajectory mapping to the stored trajectory cluster, and f i being the probability of the received trajectory being the first distance from the stored trajectory cluster.
5. The method of claim 1 , wherein:
the stored trajectory cluster includes an ordered list of points in a two-dimensional (2D) image-pixel space; and
the first distance between the received trajectory and the stored trajectory cluster is a distance between a list of points of the received trajectory and the list of points of the stored trajectory cluster determined based on a dynamic programming technique.
6. The method of claim 5 , wherein:
the dynamic programming technique is a Needleman-Wunsch algorithm; and
cells in antidiagonals of a matrix used in the Needleman-Wunsch algorithm and all connected sub-matrices are computed in parallel.
7. The method of claim 1 , further comprising:
upon determining that the received trajectory does not map to a stored trajectory cluster:
determining a second score based on at least a cumulative probability distribution indicating a probability of the received trajectory being at least a second distance from a mature stored trajectory cluster that best matches the received trajectory, and
upon determining that the second score exceeds a second threshold value, issuing an alert to the user.
8. The method of claim 1 , wherein the received trajectory is assembled by appending tracked positions of the object to the received trajectory.
9. The method of claim 8 , wherein assembling the received trajectory includes:
clustering raw tracked object positions; and
appending the clustered tracked positions to the received trajectory.
10. A non-transitory computer-readable storage medium storing instructions, which when executed by a computer system, perform operations to:
acquire a plurality of data streams to track an object, the plurality of data streams being processed video frames from a plurality of video cameras, each data stream including a tracked subject identifier that specifies the object being tracked, and a camera identifier associated with a corresponding video camera of the plurality of video cameras;
calculate a first distance between a received trajectory for the object in a first data stream of the plurality of data streams and a stored trajectory cluster from a plurality of stored trajectory clusters based on the tracked subject identifier;
determine, based on the first distance, whether the received trajectory maps to the stored trajectory cluster; and
upon a determination that the received trajectory maps to the stored trajectory cluster:
determine a first score based on at least a joint probability computed as a product of a probability of the received trajectory mapping to the stored trajectory cluster and the probability of the received trajectory being the first distance from the stored trajectory cluster, and
upon a determination that the first score exceeds a threshold value, issue an alert to a user, the alert indicating an anomaly in the received trajectory of the object.
11. The non-transitory computer-readable storage medium of claim 10 , wherein:
the probability of the received trajectory mapping to the stored trajectory cluster is determined based on at least a count of previous trajectories for the object which mapped to the stored trajectory cluster based on the tracked subject identifier.
12. The non-transitory computer-readable storage medium of claim 10 , wherein the first score is determined as S=1.0−Pr×f i , Pr being the probability of the received trajectory mapping to the stored trajectory cluster, and f i being the probability of the received trajectory being the first distance from the stored trajectory cluster.
13. The non-transitory computer-readable storage medium of claim 10 , wherein:
the stored trajectory cluster includes an ordered list of points in a two-dimensional (2D) image-pixel space; and
the first distance between the received trajectory and the stored trajectory cluster is a distance between a list of points of the received trajectory and the list of points of the stored trajectory cluster determined based on a dynamic programming technique.
14. The non-transitory computer-readable storage medium of claim 13 , wherein:
the dynamic programming technique is a Needleman-Wunsch algorithm; and
cells in antidiagonals of a matrix used in the Needleman-Wunsch algorithm and all connected sub-matrices are computed in parallel.
15. The non-transitory computer-readable storage medium of claim 10 , the operations further comprising:
upon a determination that the received trajectory does not map to a stored trajectory cluster:
determine a second score based on at least a second cumulative probability distribution indicating probability of the received trajectory being at least a second distance from a mature stored trajectory cluster that best matches the received trajectory, and
upon a determination that the second score exceeds a second threshold value, issue an alert to the user.
16. The non-transitory computer-readable storage medium of claim 10 , wherein the received trajectory is assembled by appending tracked positions of the object to the received trajectory.
17. The computer-readable storage medium of claim 16 , wherein instructions to assemble the received trajectory includes operations to:
cluster raw tracked object positions; and
append the clustered tracked positions to the received trajectory.
18. A system, comprising:
a processor; and
a memory in communication with the processor, wherein the memory includes an application program configured to perform operations to:
acquire a plurality of data streams to track an object, the plurality of data streams being processed video frames from a plurality of video cameras, each data stream including a tracked subject identifier that specifies the object being tracked, and a camera identifier associated with a corresponding video camera of the plurality of video cameras;
calculate a first distance between a received trajectory for the object in a first data stream of the plurality of data streams and a stored trajectory cluster from a plurality of stored trajectory clusters based on the tracked subject identifier;
determine, based on the first distance, whether the received trajectory maps to the stored trajectory cluster; and
upon a determination that the received trajectory maps to the stored trajectory cluster:
determine a first score based on at least a joint probability computed as a product of a probability of the received trajectory mapping to the stored trajectory cluster and the probability of the received trajectory being the first distance from the stored trajectory cluster, and
upon a determination that the first score exceeds a threshold value, issue an alert to a user, the alert indicating an anomaly in the received trajectory of the object.
19. The system of claim 18 , wherein:
the first distance between the received trajectory and the stored trajectory cluster is a distance between a list of points of the received trajectory and the list of points of the stored trajectory cluster determined based on a Needleman-Wunsch algorithm; and
cells in antidiagonals of a matrix used in the Needleman-Wunsch algorithm and all connected sub-matrices are computed in parallel.
20. The system of claim 18 , the application program further configured to perform operations to generate a new trajectory including an (x,y) position of the object upon determining that the received trajectory does not map to the stored trajectory cluster, and storing the new trajectory in a trajectory buffer.