Cognitive model for a machine-learning engine in a video analysis system
View Patent ↗A machine-learning engine is disclosed that is configured to recognize and learn behaviors, as well as to identify and distinguish between normal and abnormal behavior within a scene, by analyzing movements and/or activities (or absence of such) over time. The machine-learning engine may be configured to evaluate a sequence of primitive events and associated kinematic data generated for an object depicted in a sequence of video frames and a related vector representation. The vector representation is generated from a primitive event symbol stream and a phase space symbol stream, and the streams describe actions of the objects depicted in the sequence of video frames.
1. A method for processing data generated from a sequence of video frames, the method comprising:
receiving, as a trajectory for a first object, a series of primitive events associated with a path of the first object in the sequence of video frames as the first object moves through the scene depicted in the sequence of video frames, wherein each primitive event includes at least an object type and a set of one or more kinematic variables associated with the first object;
after receiving the trajectory for the first object, receiving a first vector representation generated for the first object, wherein the first vector representation is generated from a primitive event symbol stream and a phase space symbol stream, and wherein the streams describe actions of at least the first object depicted in the sequence of video frames;
exciting one or more nodes of a perceptual associative memory using the trajectory and the first vector representation;
identifying, based on the one or more excited nodes, a percept;
copying the percept to a workspace;
in response to copying the percept to the workspace, selecting a codelet, wherein the codelet includes an executable sequence of instructions; and
invoking execution of the codelet.
2. The method of claim 1 , further comprising, receiving a series of primitive events associated with a path of a second object interacting with the first object in the sequence of video frames, wherein each primitive event includes at least an object type and a set of one or more kinematic variables associated with the second object.
3. The method of claim 1 , wherein the set of kinematic variables include at least a time variable, a position variable, and a velocity variable.
4. The method of claim 1 , further comprising:
prior to invoking execution of the codelet, identifying, based on the percept, one or more memories stored in a long-term memory data structure; and
copying the identified memories to the workspace.
5. The method of claim 4 , wherein the one or more long-term memories are identified based on a measure of similarity used to evaluate the memories stored in the long-term memory and the percept.
6. The method of claim 4 , wherein the codelet is configured to reinforce one or more aspects of the identified memories, based on the percept.
7. The method of claim 4 , wherein the long term memory data structure is implemented as a sparse distributed memory.
8. The method of claim 1 , wherein the codelet is configured to generate a long-term memory from the percept; and wherein the method further comprises, storing the long-term memory in a long-term memory data structure.
9. The method of claim 1 , wherein the codelet is configured to generate an episodic memory from the percept, and wherein the method further comprises, storing the episodic memory data structure in long-term memory.
10. The method of claim 1 , wherein the primitive event symbol stream includes a sequence of primitive event symbols, wherein each primitive event symbol represents a behavior engaged in by at least one of the objects depicted in the sequence of video.
11. The method of claim 1 , wherein the first vector representation is generated by applying a singular value decomposition (SVD) to a second first vector representation.
12. The method of claim 1 , wherein the perceptual associative memory includes a network of nodes, each node representing a behavior observed to have occurred in the scene and wherein links between nodes represents a relationship between two behaviors.
13. A non-transitory computer-readable storage medium containing a program, which, when executed on a processor is configured to perform an operation for processing data generated from a sequence of video frames, the operation comprising:
receiving, as a trajectory for a first object, a series of primitive events associated with a path of first object in a scene depicted by the sequence of video frames as the first object moves through the scene, wherein each primitive event includes at least an object type and a set of one or more kinematic variables associated with the first object;
after receiving the trajectory for the first object, receiving a first vector representation generated for the first object, wherein the first vector representation is generated from a primitive event symbol stream and a phase space symbol stream, and wherein the streams describe actions of at least the first object depicted in the sequence of video frames;
exciting one or more nodes of a perceptual associative memory using the trajectory and the first vector representation;
identifying, based on the one or more excited nodes, a percept;
copying the percept to a workspace;
in response to copying the percept to the workspace, selecting a codelet, wherein the codelet includes an executable sequence of instructions; and
invoking execution of the codelet.
14. The non-transitory computer-readable storage medium of claim 13 , wherein the operation further comprises:
prior to invoking execution of the codelet, identifying, based on the percept, one or more memories stored in a long-term memory data structure; and
copying the identified memories to the workspace.
15. The non-transitory computer-readable storage medium of claim 14 , wherein the one or more long-term memories are identified based on a measure of similarity used to evaluate the memories stored in the long-term memory and the percept.
16. The non-transitory computer-readable storage medium of claim 14 , wherein the codelet is configured to reinforce one or more aspects of the identified memories, based on the percept.
17. The non-transitory computer-readable storage medium of claim 14 , wherein the long term memory data structure is implemented as a sparse distributed memory.
18. The non-transitory computer-readable storage medium of claim 13 , wherein the codelet is configured to generate a long-term memory from the percept; and wherein the operation further comprises, storing the long-term memory in a long-term memory data structure.
19. The non-transitory computer-readable storage medium of claim 13 , wherein the codelet is configured to generate an episodic memory from the percept, and wherein the operation further comprises, storing the episodic memory data structure in long-term memory.
20. The non-transitory computer-readable storage medium of claim 13 , wherein the primitive event symbol stream includes a sequence of primitive event symbols, wherein each primitive event symbol represents a behavior engaged in by at least one of the objects depicted in the sequence of video.
21. The non-transitory computer-readable storage medium of claim 13 , wherein the first vector representation is generated by applying a singular value decomposition (SVD) to a second first vector representation.
22. The non-transitory computer-readable storage medium of claim 13 , wherein the perceptual associative memory includes a network of nodes, each node representing a behavior observed to have occurred in the scene and wherein links between nodes represents a relationship between two behaviors.
23. A system, comprising:
a video input source;
a processor; and
a memory storing a machine learning engine, wherein the machine learning engine is configured to:
receive, as a trajectory for a first object, a series of primitive events associated with a path of the first object in a scene depicted by the sequence of video frames as the first object moves through the scene, wherein each primitive event includes at least an object type and a set of one or more kinematic variables associated with the first object;
after receiving the trajectory for the first object, receive a first vector representation generated for the first object, wherein the first vector representation is generated from a primitive event symbol stream and a phase space symbol stream, and wherein the streams describe actions of at least the first object depicted in the sequence of video frames;
excite one or more nodes of a perceptual associative memory using the trajectory and the first vector representation;
identify, based on the one or more excited nodes, a percept;
copy the percept to a workspace;
in response to copying the percept to the workspace, select a codelet, wherein the codelet includes an executable sequence of instructions; and
invoke execution of the codelet.
24. The system of claim 23 , wherein the machine learning engine is further configured to:
prior to invoking execution of the codelet, identify, based on the percept, one or more memories stored in a long-term memory data structure; and
copy the identified memories to the workspace.
25. The system of claim 24 , wherein the one or more long-term memories are identified based on a measure of similarity used to evaluate the memories stored in the long-term memory and the percept.
26. The system of claim 24 , wherein the codelet is configured to reinforce one or more aspects of the identified memories, based on the percept.
27. The system of claim 23 , wherein the perceptual associative memory includes a network of nodes, each node representing a behavior observed to have occurred in the scene and wherein links between nodes represents a relationship between two behaviors.