Graphical rendition of multi-modal data
Changes and anomalies in multi-modal data are detected, collected and abstracted into understandable and actionable formats utilizing, for example, color, intensity, icons and texture creating a rendition of current and developing situations and events. Changes and anomalies in multi-modal sensor data are detected, aggregated, abstracted and filtered using case-based reasoning providing a tractable data dimensionality. From this collection of data situations are recognized and presented in a means so as to assist a user in accessing an environment and formulate the basis a recommended course of action.
1. A system for rendition of multi-modal data, comprising:
a plurality of sources of multi-modal data configured to collect data from an environment wherein each of the plurality of sources is associated with a distinct sensor modality having distinct data modalities;
a processor communicatively coupled to each of the plurality of sources of multi-modal data; and
a storage medium communicatively coupled to the processor and tangibly embodying a plurality of executable modules of instructions wherein each module of instructions is connected to the processor via a bus, the plurality of executable modules including
a data collection module, configured to asynchronously collect event data across the plurality of data modalities from each of the plurality of sources of multi-modal data and store the collected event data in memory,
an anomaly detection module, configured to independently detect from the collected event data one or more event anomalies for each of a plurality of data modalities against background/historical data and to associate each event anomaly with a data modality temporal identifier and a data modality spatial identifier,
an event anomaly recognition module configured to identify a spatial and a temporal relationship between each of the one or more event anomalies among the plurality of data modalities,
an anomaly correlation module configured to combine identified event anomalies to form a coherent common event representation that is cross-correlated based on a predetermined modality temporal window and a predetermined spatial window, and
a rendition module configured to render cross-correlated common representations of detected event anomalies and the environment from a plurality of perspectives.
2. The system for rendition of multi-modal data according to claim 1 , wherein collected event data from each data modality is processed asynchronously and in parallel with collected event data from other data modalities.
3. The system for rendition of multi-modal data according to claim 1 , wherein the plurality of sources of multi-modal data includes a ultra-wide band transceiver.
4. The system for rendition of multi-modal data according to claim 1 , wherein the collected event data collected from the plurality of sources includes a spatial and temporal reference.
5. The system for rendition of multi-modal data according to claim 1 , wherein the anomaly correlation module combines two or more related event anomalies within a data modality.
6. The system for rendition of multi-modal data according to claim 1 , wherein the anomaly detection module prioritizes event anomalies based on one or more predetermined criteria.
7. The system for rendition of multi-modal data according to claim 1 , wherein the anomaly detection module filters detected anomalies based on background/historical data.
8. The system for rendition of multi-modal data according to claim 1 , wherein the rendition module represents clusters of anomalies in visual formats.
9. A computer implemented method for rendering anomaly events based on multi-modal data, comprising executing on a processor the steps of:
collecting from a plurality of sources of multi-modal data within an environment static and/or dynamic multimodal data wherein each of the plurality of sources is associated with a distinct sensor modality having distinct data modalities;
detecting independently from the collected static and/for dynamic multimodal data and based on background and historical data, one or more anomalies;
associating each of the one or more detected anomalies with a data modality temporal identifier and a data modality spatial identifier;
identifying from the one or more detected anomalies, one or more anomaly events based on a correlation between the data modality temporal identifiers and the data modality spatial identifiers;
combining identified anomaly events to form a coherent common event representation that is cross-correlated based on a predetermined modality temporal window and a predetermined spatial window; and
rendering on a graphical display the coherent common event representation of identified anomaly events and the environment in which they occur from a plurality of perspectives.
10. The method for rendering anomaly events based on multi-modal data according to claim 9 , further comprising abstracting collected static and/or dynamic data.
11. The method for rendering anomaly events based on multi-modal data according to claim 10 , further comprising comparing abstracted collected data to pre-identified event temporal and spatial constraints.
12. The method for rendering anomaly events based on multi-modal data according to claim 9 , wherein the plurality of sources of multi-modal data includes a ultra-wide band transceiver.
13. The method for rendering anomaly events based on multi-modal data according to claim 12 , wherein change analysis includes motion tracking and occupancy change analysis.
14. The method for rendering anomaly events based on multi-modal data according to claim 12 , wherein change analysis includes radio frequency analysis.
15. The method for rendering anomaly events based on multi-modal data according to claim 12 , wherein change analysis includes acoustical change analysis.
16. The method for rendering anomaly events based on multi-modal data according to claim 12 , wherein change analysis includes visual and thermal imagery analysis.
17. The method for rendering anomaly events based on multi-modal data according to claim 12 , wherein change analysis includes chemical change analysis.
18. The method for rendering anomaly events based on multi-modal data according to claim 9 , wherein detecting includes recognizing data fluctuations within each modality.
19. The method for rendering anomaly events based on multi-modal data according to claim 9 , wherein identifying includes determining whether two or more detected anomalies occur within a pre-identified temporal period of time and within a pre-identified spatial proximity.
20. The method for rendering anomaly events based on multi-modal data according to claim 19 , wherein the pre-identified spatial proximity is based on environmental features.
21. The method for rendering anomaly events based on multi-modal data according to claim 9 , wherein identifying includes recognizing physical and temporal relationships between and among the detected one or more anomalies.
22. The method for rendering anomaly events based on multi-modal data according to claim 9 , further comprising clustering the one or more anomalies based on common spatially salient features.
23. A non-transitory computer-readable storage medium comprising instructions stored thereon for rendering anomaly events based on multi-modal data that, when executed on a processor, perform the steps of:
collecting from a plurality of sources of multi-modal data within an environment static and/or dynamic multimodal data wherein each of the plurality of sources is associated with a distinct sensor modality having distinct data modalities;
detecting independently from the collected static and/or dynamic multimodal data and based on background and historical data, one or more anomalies;
associating each of the one or more detected anomalies with a data modality temporal identifier and a data modality spatial identifier;
identifying from the one or more detected anomalies, one or more anomaly events based on a correlation between the data modality temporal identifiers and the data modality spatial identifiers;
combining identified anomaly events to form a coherent common event representation that is cross-correlated based on a predetermined modality temporal window and a predetermined spatial window; and
rendering on a graphical display the coherent common event representation of identified anomaly events and the environment in which they occur from a plurality of perspectives.
24. The non-transitory computer-readable storage medium of claim 23 , further comprising instructions for abstracting collected static and/or dynamic multimodal data.
25. The non-transitory computer-readable storage medium of claim 24 , further comprising instructions for comparing abstracted collected data to pre-identified event characteristics.
26. The non-transitory computer-readable storage medium of claim 23 , further comprising instructions for clustering the detected one or more anomalies based on common spatially salient features.
27. The non-transitory computer-readable storage medium of claim 23 , wherein the instructions for identifying includes identifying physical and temporal relationships between and among the detected one or more anomalies.