Multimodal machine learning for emotion metrics
Techniques are described for machine-trained analysis for multimodal machine learning. A computing device captures a plurality of information channels, wherein the plurality of information channels includes contemporaneous audio information and video information from an individual. A multilayered convolutional computing system learns trained weights using the audio information and the video information from the plurality of information channels, wherein the trained weights cover both the audio information and the video information and are trained simultaneously, and wherein the learning facilitates emotional analysis of the audio information and the video information. A second computing device captures further information and analyzes the further information using trained weights to provide an emotion metric based on the further information. Additional information is collected with the plurality of information channels from a second individual and learning the trained weights factors in the additional information. The further information can include only video data or audio data.
1. A computer-implemented method for machine-trained analysis comprising:
capturing, into a computing device, a plurality of information channels, wherein the plurality of information channels includes contemporaneous audio information and video information from an individual;
learning, on a multilayered convolutional network, trained weights using the audio information and the video information from the plurality of information channels, wherein the trained weights are trained from both the audio information and the video information and are trained simultaneously, and wherein the learning facilitates emotional analysis of the audio information and the video information;
learning image descriptors, as part of the multilayered convolutional network, for emotional content; and
capturing, into a second computing device, further information and analyzing the further information using the trained weights to provide an emotion metric based on the further information.
2. The method of claim 1 further comprising collecting additional information with the plurality of information channels from a second individual and learning the trained weights factoring in the additional information.
3. The method of claim 1 wherein the further information includes only video data.
4. The method of claim 1 wherein the further information includes only audio data.
5. The method of claim 1 wherein the audio information and the video information comprise multimodal emotional information.
6. The method of claim 1 wherein the learning comprises early fusion combination of video data and audio data for the analyzing of the further information.
7. The method of claim 1 wherein the computing device and the second computing device are a same computing device.
8. The method of claim 1 wherein the learning comprises hybrid learning using early fusion combination of portions of the audio information and the video information along with late fusion combination of portions of the audio information and the video information.
9. The method of claim 1 wherein the capturing further information comprises intermittent information.
10. The method of claim 9 wherein during capturing further information, one channel from the plurality of information channels drops out, and the analyzing the further information continues without the one channel that dropped out.
11. The method of claim 1 further comprising tuning the multilayered convolutional network.
12. The method of claim 1 wherein the analyzing of the further information further comprises counting occurrences of a specific emotion type.
13. The method of claim 12 further comprising building a summary emotional intensity metric based on a number of the occurrences.
14. The method of claim 1 wherein the multilayered convolutional network includes multiple layers that include one or more convolutional layers and one or more hidden layers.
15. The method of claim 14 wherein a last layer within the multiple layers provides output indicative of emotional state.
16. The method of claim 15 further comprising tuning the last layer within the multiple layers for a particular emotional state.
17. The method of claim 14 wherein the learning comprises assigning weights to inputs on one or more layers within the multilayered convolutional network.
18. The method of claim 17 wherein the assigning weights is accomplished during a feed-forward pass through the multilayered convolutional network.
19. The method of claim 17 wherein the weights are updated during a backpropagation process through the multilayered convolutional network.
20. The method of claim 1 further comprising using iterative semi-supervised updates of model weights in the learning.
21. The method of claim 1 further comprising performing unsupervised learning as part of training the multilayered convolutional network.
22. The method of claim 1 further comprising training an emotion classifier, as part of the multilayered convolutional network, for emotional content.
23. A computer program product embodied in a non-transitory computer readable medium for machine-trained analysis, the computer program product comprising code which causes one or more processors to perform operations of:
capturing, into a computing device, a plurality of information channels, wherein the plurality of information channels includes contemporaneous audio information and video information from an individual;
learning, on a multilayered convolutional network, trained weights using the audio information and the video information from the plurality of information channels, wherein the trained weights are trained from both the audio information and the video information and are trained simultaneously, and wherein the learning facilitates emotional analysis of the audio information and the video information;
learning image descriptors, as part of the multilayered convolutional network, for emotional content; and
capturing, into a second computing device, further information and analyzing the further information using the trained weights to provide an emotion metric based on the further information.
24. A computer system for machine trained analysis comprising:
a memory which stores instructions;
one or more processors attached to the memory wherein the one or more processors, when executing the instructions which are stored, are configured to:
capture, into a computing device, a plurality of information channels, wherein the plurality of information channels includes contemporaneous audio information and video information from an individual;
learn, on a multilayered convolutional network, trained weights using the audio information and the video information from the plurality of information channels, wherein the trained weights are trained from both the audio information and the video information and are trained simultaneously, and wherein the learning facilitates emotional analysis of the audio information and the video information;
learn image descriptors, as part of the multilayered convolutional network, for emotional content; and
capture, into a second computing device, further information and analyze the further information using the trained weights to provide an emotion metric based on the further information.