Recognition system using multimodality dataset

1. A computer-implemented method for generating a multimodality dataset, comprising:

extracting motion features from a video recording of a real-world event, captured from a video recording device, to generate a motion capture dataset, the motion capture dataset including human models of human motion;

generating a synthetic dataset based on derivatives of the motion capture dataset;

converting the synthetic dataset to a heatmap of vectors and velocities of human actions and inanimate object movements;

generating an augmented motion capture dataset based on modified conditions of the motion capture dataset;

generating an augmented synthetic dataset based on modified conditions of the synthetic dataset; and

generating a multimodality dataset based on a combination of the motion capture dataset, the synthetic dataset, the augmented motion capture dataset, and the augmented synthetic dataset.

2. The computer-implemented method of claim 1 , wherein the derivatives include at least one of shape, height and sex of the human models.

3. The computer-implemented method of claim 1 , wherein the modified conditions include at least one of orientation, location, scale and brightness.

4. The computer-implemented method of claim 1 , further comprising: filtering and rendering at least one of the motion capture dataset, the synthetic dataset, the augmented motion capture dataset, or the augmented synthetic dataset to form a colourized heatmap representing vectors and velocities.

5. The computer-implemented method of claim 4 , wherein the colourized heatmap is a hue saturation value representation.

6. The computer-implemented method of claim 4 , wherein the vectors and velocities represent vectors and velocities of human actions and inanimate object movements.

7. The computer-implemented method of claim 1 , further comprising converting at least one of the motion capture dataset, the synthetic dataset, the augmented motion capture dataset, or the augmented synthetic dataset into a music or sound notation to form an audio saturation value representation.

8. The computer-implemented method of claim 1 , wherein the real-world event is simulated.

9. The computer-implemented method of claim 1 , wherein the multimodality dataset is for use in training a machine learning algorithm.

10. The computer-implemented method of claim 9 , wherein the machine learning algorithm is a motion recognition deep learning algorithm for predicting actions based on a video input.

11. A computer system comprising:

a processor;

a memory in communication with the processor, the memory storing instructions that, when executed by the processor cause the processor to:

extract motion features from a video recording of a real-world event, captured from a video recording device, to generate a motion capture dataset, the motion capture dataset including human models of human motion;

generate a synthetic dataset based on derivatives of the motion capture dataset;

convert the synthetic dataset to a heatmap of vectors and velocities of human actions and inanimate object movements;

generate an augmented motion capture dataset based on modified conditions of the motion capture dataset;

generate an augmented synthetic dataset based on modified conditions of the synthetic dataset; and

generate a multimodality dataset based on a combination of the motion capture dataset, the synthetic dataset, the augmented motion capture dataset, and the augmented synthetic dataset.

12. The computer system as claimed in claim 11 , wherein the processor is configured to:

filter and render at least one of the motion capture dataset, the synthetic dataset, the augmented motion capture dataset, or the augmented synthetic dataset to form a colourized heatmap representing vectors and velocities.

13. The computer system as claimed in claim 12 , wherein at least one of:

the colourized heatmap is a hue saturation value representation; or

the vectors and velocities represent vectors and velocities of human actions and inanimate object movements.

14. The computer system as claimed in claim 11 , wherein the processor is configured to:

convert at least one of the motion capture dataset, the synthetic dataset, the augmented motion capture dataset, or the augmented synthetic dataset into a dynamic music or sound notation to form an audio saturation value representation that is determined from a deep learning trained algorithm.

15. The computer system as claimed in claim 11 , wherein the multimodality dataset is for use in training a motion recognition deep learning algorithm for predicting actions based on a video input or other passive or active proximity detection recording device.

16. An electronic device for machine learning motion recognition, comprising a processor and memory storing code to configure the processor to:

at a training phase,

extract motion features from event data to generate a motion capture dataset;

generate a synthetic dataset based on the motion capture dataset;

convert the synthetic dataset to a heatmap of vectors and velocities of human actions and inanimate object movements;

generate an augmented fully redacted multimodality dataset based on a combination of the motion capture dataset and the synthetic dataset;

using deep learning, build a recognition model using the generated datasets;

at a prediction phase,

receive features of additional event data;

process the features using the recognition models to generate predicted motion data;

generate alert data based on the generated predicted motion data; and

transmit the alert data to a reporting device.

17. The electronic device of claim 16 , wherein the processor is further configured to:

convert the motion capture dataset to a heatmap of vectors and velocities of human actions and inanimate object movements.

18. The electronic device as claimed in claim 16 , wherein the processor is configured to:

at least one of:

filter and render at least one of the motion capture dataset, the synthetic dataset, the augmented motion capture dataset, or the augmented synthetic dataset to form a colourized heatmap representing vectors and velocities; or

convert at least one of the motion capture dataset, the synthetic dataset, the augmented motion capture dataset, or the augmented synthetic dataset into a dynamic music or sound notation to form an audio saturation value representation that is determined from a deep learning trained algorithm.

19. The electronic device as claimed in claim 18 , wherein at least one of:

the colourized heatmap is a hue saturation value representation; or

the vectors and velocities represent vectors and velocities of human actions and inanimate object movements.

20. The electronic device as claimed in claim 16 , wherein the multimodality dataset is for use in training a motion recognition deep learning algorithm for predicting actions based on a video input or other passive or active proximity detection recording device.