Deep compositional frameworks for human-like language acquisition in virtual environments

1. A computer-implemented method of human-like language acquisition for a compositional framework, the method comprising:

receiving, at a perception module, a vision input;

receiving, at a language module, a natural language input comprising one or more words related to the vision input;

transforming the vision input, at the perception module, into a feature map comprising a visual feature map and a spatial feature map;

obtaining, at the language module, an attention distribution over words in the natural language input;

obtaining an averaged word embedding based on the attention distribution;

using a convolution of a masked portion of the averaged word embedding with at least portion of the feature map, at the language module, to generate an attention map; and

responsive to the natural language input being a question, using a recognition module to output an answer by performing steps comprising:

obtaining a question intention from the question using an embedding mask;

obtaining extracted features using at least a portion of the feature map weighted by the attention map;

generating an embedding mask from the question intention;

masking the extracted image features with the embedding mask to obtain one or more masked features; and

classifying the one or more masked features to output the answer in response to the question.

2. The computer-implemented method of claim 1 wherein the natural language input comprises at least a word not pre-parsed or pre-linked to the visual input.

3. The computer-implemented method of claim 1 wherein the visual input is an environmental image.

4. The computer-implemented method of claim 1 wherein the visual feature map is transformed from the visual input through a Convolutional Neural Network (CNN) in the perception module.

5. The computer-implemented method of claim 1 wherein attention of a word in the natural language input for the attention distribution is determined by at least a global sentential context and a local context at each word.

6. The computer-implemented method of claim 1 wherein both the global sentential context and local contexts are computed based on a syntax embedding that is a projection of word embeddings converted from the natural language input.

7. The computer-implemented method of claim 1 further comprising:

in response to the natural language input as a navigation command or request:

convolving the visual feature map with a filter to get an environment map;

stacking the environment map and the attention map, at an action module, to project a state vector using an action Convolutional Neural Network (CNN); and

projecting the state vector to a distribution over a plurality of actions.

8. The computer-implemented method of claim 1 wherein the embedding mask in the step of obtaining a question intention from the question using an embedding mask is obtained by performing steps comprising:

converting the question into a sequence of word embeddings;

projecting the sequence of word embeddings into a sequence of functionality embeddings;

obtaining an averaged functionality embedding based on the sequence of functionality embeddings; and

obtaining the embedding mask using mask computation on the averaged functionality embedding.

9. The computer-implemented method of claim 8 wherein the language module further comprising a bidirectional RNN, which outputs a booting vector to initialize a programmer RNN within the language module for obtaining attention distribution over words.

10. A computer-implemented method for end-to-end training a compositional framework for human-like language acquisition, the method comprising:

converting a natural language input comprising one or more words related to an image into word embeddings by using a word embedding table;

generating a feature map comprising a visual feature map and a spatial feature map from the image using a convolution neural network;

projecting the word embeddings using a first set of one or more layers to form syntax embeddings;

projecting the word embeddings using a second set of one or more layers to form functionality embeddings;

using the syntax embeddings as an input to a neural network to obtain a distribution of attentions over the plurality of words in the natural language input;

obtaining an averaged word embedding based on the distribution of attentions;

using the functionality embeddings and one or more neural networks to obtain a mask;

using the mask to obtain a masked averaged word embedding;

convolving the masked average word embedding with at least a portion of the feature map to obtain a convolved output;

using the convolved output to generate a spatial attention map of the image; and

generating a response to the natural language input based at least on the spatial attention map.

11. The computer-implemented method of claim 10 wherein:

in response to the natural language input is a navigation command, the response is one or more actions.

12. The computer-implemented method of claim 10 wherein:

in response to the natural language input is a recognition question, the response is a most possible word or words corresponding to an extracted feature extracted from the feature map weighted by the spatial attention map.

13. The computer-implemented method of claim 10 wherein the step of using the convolved output to generate a spatial attention map of the image comprises:

inputting the convolved output to a softmax layer to obtain a sum-to-one attention map; and

given a cached attention map from a previous step, generating the spatial attention map by convolving a translation of the sum-to-one attention map with the cached attention map.

14. The computer-implemented method of claim 11 wherein an action from the one or more actions is obtain by performing the steps comprising:

convolving the visual feature map with a filter to get an environment map;

obtaining a state vector by inputting the environment map and the spatial attention map into a Convolutional Neural Network (CNN) and projecting its output;

projecting the state vector to a distribution over a plurality of actions; and

selecting the action using the distribution.

15. A computer-implemented method for training a compositional framework for human-like language acquisition, the method comprising:

receiving a natural language input comprising one or more words related to an image;

converting the natural language input into a sequence of word embeddings;

projecting the sequence of word embeddings into syntax embeddings and functionality embeddings;

feeding the syntax embeddings into a recurrent neural network (RNN) to obtain word context vectors;

computing attention for each word from its word context vector;

obtaining a weighted average attention based on the attention for each word;

using a convolved output obtained by convolving the weighted average attention with a feature map extracted from the image to get a generated attention map, the feature map is formed using a convolutional neural network and comprises a visual feature map and a spatial feature map;

combining the generated attention map with a cached attention map in a previous step, if available, to obtain a current attention map; and

generating a response to the natural language input based at least on the current attention map.

16. The computer-implemented method of claim 15 further comprising caching the current attention map via a forget gate using a weighted combination of the cached attention map and the current attention map.

17. The computer-implemented method of claim 15 wherein the cached attention map in the previous step is convolved with a 180-degree rotation of the generated attention map.

18. The computer-implemented method of claim 15 further comprising:

in response to the natural language input being a navigation command:

inputting the current attention map and an environment map into an action Convolutional Neural Network (CNN) and projecting its output to obtain a state vector, the environment map is obtained by convolving the visual feature map with a filter; and

projecting the state vector to a distribution over a plurality of actions in one or more time steps.

19. The computer-implemented method of claim 18 further comprising:

assigning a positive reward number if the navigation command is accomplished correctly before a time threshold; and

assigning a negative reward number if the navigation command is not implemented correctly or accomplished above the time threshold.