Object interaction detection and inferences using semantic learning

1 . A method comprising:

converting, by a device, video data into a set of tracklets, each tracklet representing a different object depicted in the video data;

identifying, by the device and based on the set of tracklets, a particular object depicted in the video data as being an attractor or a repulsor with respect to one or more other objects depicted in the video data, the identifying including:

generating a first motion vector for a first tracklet associated with the particular object, the first motion vector indicative of a velocity and direction of motion of the particular object;

generating a second motion vector for a second tracklet associated with the one or more other objects, the second motion vector indicative of a velocity and direction of motion of the one or more other objects;

analyzing the first motion vector and the second motion vector to determine whether more than a threshold number of the one or more other objects are moving towards the particular object or away from the particular object;

determining that the particular object is an attractor when it is determined that the one or more objects are moving towards the particular object;

determining that the particular object is a repulsor when it is determined that the one or more objects are moving away from the particular object; and

dynamically applying and reassigning a focus of attention to the particular object over a plurality of frames of the video data based on a number of the one or more other objects moving toward or away from the particular object;

making, by the device and using a semantic reasoning engine that uses a knowledge base, an inference about the video data, based in part on the particular object being identified as an attractor or repulsor and an orientation of the particular object relative to the one or more other objects, the inference indicative of a dangerous or urgent situation, wherein the knowledge base represents concepts and relationships among the concepts that are domain-agnostic prior to runtime and instantiated with attractor-specific or repulsor-specific predicates during the inference, and has not been trained with the video data; and

providing, by the device, data for display based on the inference, wherein the data comprises both the identification of the attractor or the repulsor of the particular object and a predicted future convergence or divergence event between the particular object and the one or more other objects.

2 . The method as in claim 1 , wherein the video data is generated by a plurality of cameras deployed to a location.

3 . The method as in claim 2 , wherein identifying the particular object as being an attractor or repulsor comprises:

re-identifying the particular object across video data from different cameras in the plurality of cameras by correlating tracklets associated with the particular object.

4 . The method as in claim 1 , wherein identifying the particular object as being an attractor or repulsor comprises:

analyzing interactions between the first tracklet of the particular object and the second tracklet of the one or more other objects depicted in the video data.

5 . The method as in claim 1 , wherein making, by the device and using a semantic reasoning engine, the inference about the video data, based in part on the particular object being identified as an attractor or repulsor comprises:

using zero-shot learning to infer a condition of the particular object, without the device being trained to recognize the condition using sample data representative of that condition.

6 . The method as in claim 1 , wherein the semantic reasoning engine uses a knowledge graph comprising concepts and relationships, to make the inference about the video data.

7 . The method as in claim 6 , wherein the concepts of the knowledge graph represent at least one of: a medical condition, fighting, aggression, or a stampede.

8 . The method as in claim 1 , wherein the inference about the video data comprises a predicted future event.

9 . An apparatus, comprising:

a network interface to communicate with a computer network;

a processor coupled to the network interface; and

a memory configured to store one or more instructions, that when executed by the processor, configure the processor to:

convert video data into a set of tracklets, each tracklet representing a different object depicted in the video data;

identify a particular object depicted in the video data as being an attractor or a repulsor with respect to one or more other objects depicted in the video data by dynamically applying and reassigning a focus of attention to the particular object over a plurality of frames responsive to detecting relative motion patterns between the particular object and the one or more other objects, based on an analysis of their respective tracklets, wherein the particular object is identified as the attractor when the one or more other objects move towards it and is identified as the repulsor when the one or more other objects move away from it, wherein identification of the attractor or the repulsor is determined by analyzing motion vectors and spatial-temporal relationships between tracklets of the particular object and the one or more other objects without altering tracklet boundaries during tracking;

make, using a semantic reasoning engine that uses a knowledge base, an inference about the video data, based in part on the particular object being identified as an attractor or repulsor and an orientation of the particular object relative to the one or more other objects, the inference indicative of a dangerous or urgent situation, wherein the knowledge base represents concepts and relationships among the concepts that are domain-agnostic prior to runtime and instantiated with attractor-specific or repulsor-specific predicates during the inference, and has not been trained with the video data; and

provide data for display based on the inference, wherein the data comprises both the identification of the attractor or the repulsor of the particular object and a predicted future convergence or divergence event between the particular object and the one or more other objects.

10 . The apparatus as in claim 9 , wherein the video data is generated by a plurality of cameras deployed to a location.

11 . The apparatus as in claim 10 , wherein the apparatus identifies the particular object as being an attractor or repulsor by:

re-identifying the particular object across video data from different cameras in the plurality of cameras by correlating tracklets associated with the particular object.

12 . The apparatus as in claim 9 , wherein the apparatus identifies the particular object as being an attractor or repulsor by:

analyzing interactions between a tracklet of the particular object and those of the one or more other objects depicted in the video data.

13 . The apparatus as in claim 9 , wherein the apparatus makes, using a semantic reasoning engine, the inference about the video data, based in part on the particular object being identified as an attractor or repulsor comprises:

using zero-shot learning to infer a condition of the particular object, without the apparatus being trained to recognize the condition using sample data representative of that condition.

14 . The apparatus as in claim 9 , wherein the semantic reasoning engine uses a knowledge graph comprising concepts and relationships, to make the inference about the video data.

15 . The apparatus as in claim 14 , wherein the concepts of the knowledge graph represent at least one of: a medical condition, fighting, aggression, or a stampede.

16 . The apparatus as in claim 9 , wherein the inference about the video data comprises a predicted future event.

17 . A tangible, non-transitory, computer-readable medium storing program instructions that cause a device to execute a process comprising:

converting, by the device, video data into a set of tracklets, each tracklet representing a different object depicted in the video data;

identifying, by the device, a particular object depicted in the video data as being an attractor or a repulsor with respect to one or more other objects depicted in the video data by dynamically applying and reassigning a focus of attention to the particular object over a plurality of frames responsive to detecting relative motion patterns between the particular object and the one or more other objects, based on an analysis of their respective tracklets, wherein the particular object is identified as the attractor when the one or more other objects move towards it and is identified as the repulsor when the one or more other objects move away from it, wherein identification of the attractor or the repulsor is determined by analyzing motion vectors and spatial-temporal relationships between tracklets of the particular object and the one or more other objects without altering tracklet boundaries during tracking;

making, by the device and using a semantic reasoning engine that uses a knowledge base, an inference about the video data, based in part on the particular object being identified as an attractor or repulsor and an orientation of the particular object relative to the one or more other objects, the inference indicative of a dangerous or urgent situation, wherein the knowledge base represents concepts and relationships among the concepts that are domain-agnostic prior to runtime and instantiated with attractor-specific or repulsor-specific predicates during the inference, and has not been trained with the video data; and

providing, by the device, data for display based on the inference, wherein the data comprises both the identification of the attractor or the repulsor of the particular object and a predicted future convergence or divergence event between the particular object and the one or more other objects.

18 . The method as in claim 1 , wherein the identifying the particular object as being an attractor or a repulsor includes:

analyzing motion vectors for a set of objects that are within a predetermined distance to the particular object.

19 . The apparatus as in claim 9 , wherein the identifying the particular object as being an attractor or a repulsor includes:

identifying a set of other objects within a radius of the particular object based on their respective tracklets;

generating, for each of the set of other objects, a relative velocity vector between the particular object and each of the set of other objects;

generating, for each of the set of other objects, an inter-tracklet vector between the particular object and each of the set of other objects; and

determining, based on a product of the relative velocity vector and the inter-tracklet vector for each of the set of other objects, whether each of the set of other objects is moving toward or away from the particular object.

20 . The tangible, non-transitory, computer-readable medium as in claim 17 , wherein the identifying the particular object as being an attractor or a repulsor includes:

identifying a set of other objects within a radius of the particular object based on their respective tracklets;

generating, for each of the set of other objects, a relative velocity vector between the particular object and each of the set of other objects;

generating, for each of the set of other objects, an inter-tracklet vector between the particular object and each of the set of other objects; and

determining, based on a product of the relative velocity vector and the inter-tracklet vector for each of the set of other objects, whether each of the set of other objects is moving toward or away from the particular object.