System and method for automated detection of situational awareness

1. A method implemented in a computer comprising a processor, memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor, the method comprising:

receiving, at the computer system, data capturing an event;

generating, at the computer system, a narrativization of the data characterizing the event captured in the data, wherein generating the narrativization comprises captioning, at the computer system, image data, captioning, at the computer system, video data, recognizing, at the computer system, speech included in audio data, generating summary data, at the computer system, characterizing text data, and generating summary data, at the computer system, characterizing sensor data;

detecting, at the computer system, at least one entity involved in the event captured in the data, wherein the at least one entity comprises at least one of an object, activity, situation, and person detected from the image data, video data, speech included in audio data, text data, and sensor data;

obtaining, at the computer system, ontology information based on the generated narrativization and the detected at least one entity, wherein the ontology information includes concepts and categories relating to users and data showing properties and relations between the users and user's data, real world data, concepts and categories relating to a context of data including data showing properties and relations between the contexts and the other data, concepts and categories relating to intents of monitored entities including data showing properties and the relations between the monitored entities, actions of the monitored entities, and characteristics of the monitored entities;

determining, at the computer system, an intent of the at least one detected entity involved in the event captured in the data, wherein the intent is determined using the generated captioning of the image data, the generated captioning of the video data, the recognized speech included in the audio data, the generating summary data characterizing the text data, and the generated summary data characterizing the sensor data, the at least one entity detected from the image data, video data, speech included in audio data, text data, and sensor data, and the ontology information obtained based on the generated narrativization and the detected at least one entity;

classifying the determined intent based on duration according to Strategic Intent: that which the user wants to achieve over the long-term in a specific domain, and Tactical Intent: that which the user wants to achieve over the short-term, and based on form of expression according to Explicit Intent: the intent is explicitly presented to the system and can be directly detected, and Implicit Intent: the intent is to be derived from one or a combination of data sources that do not express the intent directly; and

performing, at the computer system, an action responsive to the determined intent, wherein the action responsive to the determined intent is determined using an intent extractor and actuator to infer an appropriate action for a given situation and an overall goal of a use case.

2. The method of claim 1 , wherein the data capturing an event comprises at least one of image data, video data, text data, audio data, and sensor data.

3. The method of claim 2 , wherein the data capturing an event comprises at least one of real-time data relating to events occurring contemporaneously and stored data relating events that occurred in the past.

4. The method of claim 1 , wherein detecting at least one entity comprises:

detecting, at the computer system, the at least one entity comprising at least one of an object, activity, situation, and person from image data using at least one of image object recognition models, image movement recognition models, image facial recognition models, and image situation recognition models;

detecting, at the computer system, the at least one entity comprising at least one of an object, activity, situation, and person from video data using at least one of video object recognition models, video movement recognition models, video facial recognition models, and video situation recognition models;

detecting, at the computer system, the at least one entity comprising at least one of an object, activity, situation, and person from audio data using at least one of audio object recognition models, audio movement recognition models, audio speaker recognition models, and audio situation recognition models;

detecting, at the computer system, the at least one entity comprising at least one of an object, activity, situation, and person from text data using at least one of text object recognition models, text activity recognition models, text situation recognition models, and text person recognition models; and

detecting, at the computer system, the at least one entity comprising at least one of an object, activity, situation, and person from text data using at least one of sensor object recognition models, sensor activity recognition models, sensor situation recognition models, and sensor person recognition models.

5. A computer system comprising a processor, memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor to perform:

receiving data capturing an event;

generating a narrativization of the data characterizing the event captured in the data, wherein generating the narrativization comprises captioning, at the computer system, image data, captioning, at the computer system, video data, recognizing, at the computer system, speech included in audio data, generating summary data, at the computer system, characterizing text data, and generating summary data, at the computer system, characterizing sensor data;

detecting at least one entity involved in the event captured in the data, wherein the at least one entity comprises at least one of an object, activity, situation, and person detected from the image data, video data, speech included in audio data, text data, and sensor data;

obtaining ontology information based on the generated narrativization and the detected at least one entity, wherein the ontology information includes concepts and categories relating to users and data showing properties and relations between the users and user's data, real world data, concepts and categories relating to a context of data including data showing properties and relations between the contexts and the other data, concepts and categories relating to intents of monitored entities including data showing properties and the relations between the monitored entities, actions of the monitored entities, and characteristics of the monitored entities;

determining an intent of the at least one detected entity involved in the event captured in the data, wherein the intent is determined using the generated captioning of the image data, the generated captioning of the video data, the recognized speech included in the audio data, the generating summary data characterizing the text data, and the generated summary data characterizing the sensor data, the at least one entity detected from the image data, video data, speech included in audio data, text data, and sensor data, and the ontology information obtained based on the generated narrativization and the detected at least one entity;

classifying the determined intent based on duration according to Strategic Intent: that which the user wants to achieve over the long-term in a specific domain, and Tactical Intent: that which the user wants to achieve over the short-term, and based on form of expression according to Explicit Intent: the intent is explicitly presented to the system and can be directly detected, and Implicit Intent: the intent is to be derived from one or a combination of data sources that do not express the intent directly; and

performing an action responsive to the determined intent, wherein the action responsive to the determined intent is determined using an intent extractor and actuator to infer an appropriate action for a given situation and an overall goal of a use case.

6. The system of claim 5 , wherein the data capturing an event comprises at least one of image data, video data, text data, audio data, and sensor data.

7. The system of claim 6 , wherein the data capturing an event comprises at least one of real-time data relating to events occurring contemporaneously and stored data relating events that occurred in the past.

8. The system of claim 5 , wherein detecting at least one entity comprises:

detecting the at least one entity comprising at least one of an object, activity, situation, and person from image data using at least one of image object recognition models, image movement recognition models, image facial recognition models, and image situation recognition models;

detecting the at least one entity comprising at least one of an object, activity, situation, and person from video data using at least one of video object recognition models, video movement recognition models, video facial recognition models, and video situation recognition models;

detecting the at least one entity comprising at least one of an object, activity, situation, and person from audio data using at least one of audio object recognition models, audio movement recognition models, audio speaker recognition models, and audio situation recognition models;

detecting the at least one entity comprising at least one of an object, activity, situation, and person from text data using at least one of text object recognition models, text activity recognition models, text situation recognition models, and text person recognition models; and

detecting the at least one entity comprising at least one of an object, activity, situation, and person from text data using at least one of sensor object recognition models, sensor activity recognition models, sensor situation recognition models, and sensor person recognition models.

9. A computer program product comprising a non-transitory computer readable storage having program instructions embodied therewith, the program instructions executable by a computer, to cause the computer to perform a method comprising:

receiving, at the computer system, data capturing an event;

generating, at the computer system, a narrativization of the data characterizing the event captured in the data, wherein generating the narrativization comprises captioning, at the computer system, image data, captioning, at the computer system, video data, recognizing, at the computer system, speech included in audio data, generating summary data, at the computer system, characterizing text data, and generating summary data, at the computer system, characterizing sensor data;

detecting, at the computer system, at least one entity involved in the event captured in the data, wherein the at least one entity comprises at least one of an object, activity, situation, and person detected from the image data, video data, speech included in audio data, text data, and sensor data;

obtaining, at the computer system, ontology information based on the generated narrativization and the detected at least one entity, wherein the ontology information includes concepts and categories relating to users and data showing properties and relations between the users and user's data, real world data, concepts and categories relating to a context of data including data showing properties and relations between the contexts and the other data, concepts and categories relating to intents of monitored entities including data showing properties and the relations between the monitored entities, actions of the monitored entities, and characteristics of the monitored entities;

determining, at the computer system, an intent of the at least one detected entity involved in the event captured in the data, wherein the intent is determined using the generated captioning of the image data, the generated captioning of the video data, the recognized speech included in the audio data, the generating summary data characterizing the text data, and the generated summary data characterizing the sensor data, the at least one entity detected from the image data, video data, speech included in audio data, text data, and sensor data, and the ontology information obtained based on the generated narrativization and the detected at least one entity;

classifying the determined intent based on duration according to Strategic Intent: that which the user wants to achieve over the long-term in a specific domain, and Tactical Intent: that which the user wants to achieve over the short-term, and based on form of expression according to Explicit Intent: the intent is explicitly presented to the system and can be directly detected, and Implicit Intent: the intent is to be derived from one or a combination of data sources that do not express the intent directly; and

performing, at the computer system, an action responsive to the determined intent, wherein the action responsive to the determined intent is determined using an intent extractor and actuator to infer an appropriate action for a given situation and an overall goal of a use case.

10. The computer program product of claim 9 , wherein the data capturing an event comprises at least one of image data, video data, text data, audio data, and sensor data.

11. The computer program product of claim 10 , wherein the data capturing an event comprises at least one of real-time data relating to events occurring contemporaneously and stored data relating events that occurred in the past.

12. The computer program product of claim 9 , wherein detecting at least one entity comprises:

detecting, at the computer system, the at least one entity comprising at least one of an object, activity, situation, and person from image data using at least one of image object recognition models, image movement recognition models, image facial recognition models, and image situation recognition models;

detecting, at the computer system, the at least one entity comprising at least one of an object, activity, situation, and person from video data using at least one of video object recognition models, video movement recognition models, video facial recognition models, and video situation recognition models;

detecting, at the computer system, the at least one entity comprising at least one of an object, activity, situation, and person from audio data using at least one of audio object recognition models, audio movement recognition models, audio speaker recognition models, and audio situation recognition models;

detecting, at the computer system, the at least one entity comprising at least one of an object, activity, situation, and person from text data using at least one of text object recognition models, text activity recognition models, text situation recognition models, and text person recognition models; and

detecting, at the computer system, the at least one entity comprising at least one of an object, activity, situation, and person from text data using at least one of sensor object recognition models, sensor activity recognition models, sensor situation recognition models, and sensor person recognition models.