SYSTEM AND METHOD FOR AUTOMATED DETECTION OF SITUATIONAL AWARENESS

1 . A method implemented in a computer comprising a processor, memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor, the method comprising:

receiving, at the computer system, data capturing an event;

generating, at the computer system, a narrativization of the data characterizing the event captured in the data using at least one model for each type of the data, the models stored in a models database comprising at least some of text models, video models, audio models, combination models, enrichment models, encoder models, and decoder models;

detecting, at the computer system, at least one entity involved in the event captured in the data;

obtaining, at the computer system, ontology information based on the generated narrativization and the detected at least one entity;

determining, at the computer system, an intent of the at least one detected entity involved in the event captured in the data; and

performing, at the computer system, an action responsive to the determined intent.

2 . The method of claim 1 , wherein the data capturing an event comprises at least one of image data, video data, text data, audio data, and sensor data.

3 . The method of claim 2 , wherein the data capturing an event comprises at least one of real-time data relating to events occurring contemporaneously and stored data relating events that occurred in the past.

4 . The method of claim 2 , wherein generating the narrativization comprises at least one of:

captioning, at the computer system, image data,

captioning, at the computer system, image data video data,

recognizing, at the computer system, speech included in audio data,

generating summary data, at the computer system, characterizing text data, and

generating summary data, at the computer system, characterizing sensor data.

5 . The method of claim 1 , wherein detecting at least one entity comprises at least one of:

detecting, at the computer system, the at least one entity comprising at least one of an object, activity, situation, and person from image data using at least one of image object recognition models, image movement recognition models, image facial recognition models, and image situation recognition models;

detecting, at the computer system, the at least one entity comprising at least one of an object, activity, situation, and person from video data using at least one of video object recognition models, video movement recognition models, video facial recognition models, and video situation recognition models;

detecting, at the computer system, the at least one entity comprising at least one of an object, activity, situation, and person from audio data using at least one of audio object recognition models, audio movement recognition models, audio speaker recognition models, and audio situation recognition models;

detecting, at the computer system, the at least one entity comprising at least one of an object, activity, situation, and person from text data using at least one of text object recognition models, text activity recognition models, text situation recognition models, and text person recognition models; and

detecting, at the computer system, the at least one entity comprising at least one of an object, activity, situation, and person from text data using at least one of sensor object recognition models, sensor activity recognition models, sensor situation recognition models, and sensor person recognition models.

6 . A system comprising a processor, memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor to perform:

receiving data capturing an event;

generating a narrativization of the data characterizing the event captured in the data using at least one model for each type of the data, the models stored in a models database comprising at least some of text models, video models, audio models, combination models, enrichment models, encoder models, and decoder models;

detecting at least one entity involved in the event captured in the data;

obtaining ontology information based on the generated narrativization and the detected at least one entity;

determining an intent of the at least one detected entity involved in the event captured in the data; and

performing an action responsive to the determined intent.

7 . The system of claim 6 , wherein the data capturing an event comprises at least one of image data, video data, text data, audio data, and sensor data.

8 . The system of claim 7 , wherein the data capturing an event comprises at least one of real-time data relating to events occurring contemporaneously and stored data relating events that occurred in the past.

9 . The system of claim 7 , wherein generating the narrativization comprises at least one of:

captioning image data,

captioning image data video data,

recognizing speech included in audio data,

generating summary data characterizing text data, and

generating summary data characterizing sensor data.

10 . The system of claim 6 , wherein detecting at least one entity comprises at least one of:

detecting the at least one entity comprising at least one of an object, activity, situation, and person from image data using at least one of image object recognition models, image movement recognition models, image facial recognition models, and image situation recognition models;

detecting the at least one entity comprising at least one of an object, activity, situation, and person from video data using at least one of video object recognition models, video movement recognition models, video facial recognition models, and video situation recognition models;

detecting the at least one entity comprising at least one of an object, activity, situation, and person from audio data using at least one of audio object recognition models, audio movement recognition models, audio speaker recognition models, and audio situation recognition models;

detecting the at least one entity comprising at least one of an object, activity, situation, and person from text data using at least one of text object recognition models, text activity recognition models, text situation recognition models, and text person recognition models; and

detecting the at least one entity comprising at least one of an object, activity, situation, and person from text data using at least one of sensor object recognition models, sensor activity recognition models, sensor situation recognition models, and sensor person recognition models.

11 . A computer program product comprising a non-transitory computer readable storage having program instructions embodied therewith, the program instructions executable by a computer, to cause the computer to perform a method comprising:

receiving, at the computer system, data capturing an event;

generating, at the computer system, a narrativization of the data characterizing the event captured in the data using at least one model for each type of the data, the models stored in a models database comprising at least some of text models, video models, audio models, combination models, enrichment models, encoder models, and decoder models;

detecting, at the computer system, at least one entity involved in the event captured in the data;

obtaining, at the computer system, ontology information based on the generated narrativization and the detected at least one entity;

determining, at the computer system, an intent of the at least one detected entity involved in the event captured in the data; and

performing, at the computer system, an action responsive to the determined intent.

12 . The computer program product of claim 11 , wherein the data capturing an event comprises at least one of image data, video data, text data, audio data, and sensor data.

13 . The computer program product of claim 12 , wherein the data capturing an event comprises at least one of real-time data relating to events occurring contemporaneously and stored data relating events that occurred in the past.

14 . The computer program product of claim 12 , wherein generating the narrativization comprises at least one of:

captioning, at the computer system, image data,

captioning, at the computer system, image data video data,

recognizing, at the computer system, speech included in audio data,

generating summary data, at the computer system, characterizing text data, and

generating summary data, at the computer system, characterizing sensor data.

15 . The computer program product of claim 11 , wherein detecting at least one entity comprises at least one of:

detecting, at the computer system, the at least one entity comprising at least one of an object, activity, situation, and person from image data using at least one of image object recognition models, image movement recognition models, image facial recognition models, and image situation recognition models;

detecting, at the computer system, the at least one entity comprising at least one of an object, activity, situation, and person from video data using at least one of video object recognition models, video movement recognition models, video facial recognition models, and video situation recognition models;

detecting, at the computer system, the at least one entity comprising at least one of an object, activity, situation, and person from audio data using at least one of audio object recognition models, audio movement recognition models, audio speaker recognition models, and audio situation recognition models;

detecting, at the computer system, the at least one entity comprising at least one of an object, activity, situation, and person from text data using at least one of text object recognition models, text activity recognition models, text situation recognition models, and text person recognition models; and

detecting, at the computer system, the at least one entity comprising at least one of an object, activity, situation, and person from text data using at least one of sensor object recognition models, sensor activity recognition models, sensor situation recognition models, and sensor person recognition models.