Generating natural language representations of mental content from functional brain images
By way of introduction, the present embodiments described below include apparatuses and methods for generating natural language representations of mental content from functional brain images. Given functional imaging data acquired while a subject reads a text passage, a reconstruction of the text passage is produced. Linguistic semantic vector representations are assigned ( 1301 ) to words, phrases or sentences to be used as training stimuli. Basis learning is performed ( 1305 ), using brain imaging data acquired ( 1303 ) when a subject is exposed to the training stimuli and the corresponding semantic vectors for training stimuli, to learn an image basis directly. Semantic vector decoding ( 1309 ) is performed with functional brain imaging data for test stimuli and using the image basis to generate a semantic vector representing the test imaging stimuli. Text generation ( 1311 ) is then performed using the decoded semantic vector representing the test imaging stimuli.
1. A method for generating natural language representations of mental content from functional brain images, the method comprising:
capturing a first set of functional brain imaging data while a subject is exposed to a plurality of stimuli comprising training text, wherein the training text is assigned semantic vectors;
decomposing the first set of functional brain imaging data into a weighted combination of basis images, each basis image comprising a spatial pattern of activation;
mapping, for each stimuli, the weighted combination of basis images to corresponding values in each dimension of a semantic vector representing the respective stimuli;
capturing a second set of functional brain imaging data while a subject is exposed to a test stimuli;
identifying a linear combination of basis images in the second set of functional brain imaging data;
decoding a semantic vector representation for the test stimuli by estimating a weight of each basis image in the linear combination of basis images; and
generating a text output based on the decoded semantic vector.
2. The method of claim 1 wherein semantic vector representations of the plurality of stimuli are based on context in which the stimuli appears.
3. The method of claim 1 wherein the generated text is a natural language representation of the second set of functional brain imaging data.
4. The method of claim 1 wherein the semantic vectors are multidimensional.
5. A system for generating natural language representations of mental content from functional brain images, the system comprising:
a linguistic semantic vector representor configured to assign semantic vectors to training text;
a basis learner configured to decompose a first set of functional brain imaging data acquired in the presence of the training text into a weighted combination of basis images, each basis image comprising a spatial pattern of activation and map, for the training text, the weighted combination of basis images to corresponding values in each dimension of a semantic vector representing the respective text of the training text;
a semantic vector decoder configured to identify a linear combination of basis images in a second set of functional brain imaging data and decode a semantic vector representation by estimating a weight of each basis image in the linear combination of basis images; and
a text generator configured to generate a text output based on the decoded semantic vector.
6. The system of claim 5 wherein the training text comprises words, phrases or sentences.
7. The system of claim 5 wherein assigning a semantic vector to the training text comprises averaging vectors representing each word in a sentence.
8. The system of claim 5 wherein the basis learner is further configured to identify common characteristics in the functional brain imaging data for overlapping semantic vectors.
9. The system of claim 5 wherein the basis learner and the semantic vector decoder are based on a ridge regression.
10. The system of claim 5 wherein the text generator is based on a recurrent neural network.
11. The system of claim 5 wherein the text output is a sequence of words forming a sentence predicted from the decoded semantic vector.
12. A method for generating natural language representations of mental content from functional brain images, the method comprising:
assigning multidimensional linguistic semantic vector representations to a plurality of training text;
decomposing a first set of functional brain imaging data acquired in the presence of the training text into a weighted combination of basis images, each basis image comprising a spatial pattern of activation;
mapping the weighted combination of basis images to corresponding values in each dimension of a semantic vector representing the respective text of the training text;
identifying a linear combination of basis images in a second set of functional brain imaging data;
decoding a semantic vector representation by estimating a weight of each basis image in the linear combination of basis images; and
generating text based on the semantic vector representation.
13. The method of claim 12 , wherein the training text comprise words, phrases or sentences.
14. The method of claim 12 , wherein the generated text comprises a word, phrase or sentence.