Recurrent multimodal attention system based on expert gated networks

1. A system comprising:

a plurality of feature extraction modules each configured to receive data associated with a respective input modality of a plurality of input modalities and extract features associated with the respective input modality;

a plurality of recurrent neural network (RNN) modules each configured to receive the features extracted by a respective one of the feature extraction modules and predict a class associated with the features extracted by the respective feature extraction module;

a gate expert configured to receive the features extracted by each of the plurality of feature extraction modules and output a set of weights for the plurality of input modalities based on a first neural network, the first neural network being trained to produce the set of weights based on the features extracted by each of the plurality of feature extraction modules; and

a fusion module configured to generate a weighted prediction based on the predicted classes and the set of weights.

2. The system of claim 1 , wherein the plurality of input modalities includes images, video and audio.

3. The system of claim 1 , wherein the feature extraction modules comprise an image expert, a video expert and an audio expert.

4. The system of claim 1 , wherein each feature extraction module comprises a respective neural network different than the first neural network.

5. The system of claim 1 , wherein at least one of the RNN modules comprises a long short-term memory network.

6. The system of claim 1 , wherein the gate expert comprises a long short-term memory network.

7. The system of claim 1 , further comprising a co-learning framework.

8. A method comprising:

receiving a plurality of data streams associated with a plurality of input modalities, respectively;

extracting, from each data stream of the plurality of data streams, features associated with a respective input modality of the plurality of input modalities;

predicting a plurality of classes based on a plurality of recurrent neural networks (RNNs), respectively, each RNN of the plurality of RNNs being configured to predict the respective class based on the features associated a respective input modality of the plurality of input modalities;

generating a set of weights for the plurality of input modalities based on a first neural network, the first neural network being trained to produce the set of weights based on the features associated with each input modality of the plurality of input modalities; and

generating a weighted prediction based on the predicted classes and the set of weights.

9. The method of claim 8 , wherein receiving the plurality of data streams comprises sensing, using a plurality of sensor types, one or more conditions in an environment, and wherein each of the plurality of sensor types has a corresponding input modality.

10. The method of claim 9 , wherein each of the plurality of data streams contributes to the weighted prediction with a different degree of confidence depending on sensed conditions.

11. The method of claim 9 , wherein the set of weights is generated dynamically to combine information from the plurality of sensor types in accordance with sensed conditions.

12. The method of claim 8 , wherein the first neural network comprises a gating modular neural network trained to dynamically generate a set of weights for outputs of a plurality of sensor networks, at least in part, by balancing a utility of each data stream.

13. The method of claim 8 , wherein the generating of the weighted prediction comprises:

detecting voice activity in the plurality of data streams; and

providing corresponding input frames to one or more applications for further processing.

14. The method of claim 8 , wherein the plurality of input modalities includes images, video and audio.

15. A system comprising:

a memory storing instructions;

a processor coupled to the memory and configured to execute the instructions to cause the system to perform operations comprising:

receiving a plurality of data streams associated with a plurality of input modalities, respectively;

extracting, from each data stream of the plurality of data streams, features associated with a respective input modality of the plurality of input modalities;

predicting a plurality of classes based on a plurality of recurrent neural networks (RNNs), respectively, each RNN of the plurality of RNNs being configured to predict the respective class based on the features associated with a respective input modality of the plurality of input modalities;

generating a set of weights for the plurality of input modalities based on a first neural network, the first neural network being trained to produce the set of weights based on the features associated with each input modality of the plurality of input modalities; and

generating a weighted prediction based on the plurality of classes and the set of weights.

16. The system of claim 15 , wherein the plurality of input modalities includes images, video and audio.

17. The system of claim 15 , wherein extracting features associated with the respective input modality comprises inputting the data stream to a trained neural network.

18. The system of claim 15 , wherein each of the plurality of RNNs comprises a long short-term memory network.

19. The system of claim 15 , wherein the first neural network comprises a long short-term memory network.