Long-term memory networks for knowledge extraction from text and publications

1. A long-term memory network method for text comprehension, said method comprising:

providing a recurrent neural network that includes an external memory and a long-short term memory unit, wherein said recurrent neural network encodes raw text information into vector representations, forms memories, finds relevant sentences to answer questions, and generates multi-word answers to said questions utilizing said long-short term memory unit; and

controlling the long-short term memory unit with a set of three gates comprising an input gate, an output gate and a forget gate, wherein the long-short term memory unit accumulates knowledge from input data at each time step among a plurality of time steps and based on values of the input gate, the output gate, and the forget gate, stores the knowledge in internal states,

wherein said raw text information comprises input sentences and questions and wherein the long-term memory network combines an End-to-End Memory Network (MemN2N) with a Long-Short Term Memory network, wherein the Long-Short Term Memory network includes the long-short term memory unit and wherein said recurrent neural network comprises a weakly supervised architecture, and

wherein the long-short term memory unit includes a memory module comprising a plurality of sub-units including a plurality of sentence representation units, a question representation unit, a matching probability vector unit, and a unit representing the output of the MemN2N, wherein output from the matching probability vector unit is provided as input to the unit representing the output of the MemN2N.

2. The method of claim 1 wherein said input sentences comprise at least one sentence containing a word belonging to a dictionary and ending with an end-of-sentence token.

3. The method of claim 1 wherein said recurrent neural network is trained in an end-to-end manner.

4. The method of claim 1 wherein said recurrent neural network is trained in an end-to-end manner with a loss defined by a cross-entropy between a true answer and a predicted output, wherein the predicted output is represented using one-hot encoding.

5. The method of claim 1 wherein said recurrent neural network further comprises a memory module, an answer module, a question module, and an input module, wherein the memory module calculates matching probabilities the input sentences and the questions by computing an inner product followed by a softmax function.

6. The method of claim 5 wherein said answer module includes said long-short term memory unit.

7. The method of claim 1 wherein said vector representations include a matching probability vector.

8. A long-term memory system for text comprehension, said system comprising:

a recurrent neural network that comprises an external memory and a long-short term memory unit, wherein said recurrent neural network encodes raw text information into vector representations, forms memories, finds relevant sentences to answer questions, and generates multi-word answers to said questions utilizing said long-short term memory unit; and

wherein the long-short term memory unit is controlled by a set of three gates comprising an input gate, an output gate, and a forget gate and wherein the long-short term memory unit accumulates knowledge from input data at each time step among a plurality of time steps and based on values of the input gate, the output gate, and the forget gate, stores the knowledge in internal states,

wherein said raw text information comprises input sentences and questions and wherein the long-term memory network combines an End-to-End Memory Network (MemN2N) with a Long-Short Term Memory network, wherein the Long-Short Term Memory network includes the long-short term memory unit, and,

wherein said recurrent neural network comprises a weakly supervised architecture and wherein the long-short term memory unit includes a memory module comprising a plurality of sub-units including a plurality of sentence representation units, a question representation unit, a matching probability vector unit, and a unit representing the output of the MemN2N, wherein output from the matching probability vector unit is provided as input to the unit representing the output of the MemN2N.

9. The system of claim 8 wherein said input sentences comprise at least one sentence containing a word belonging to a dictionary and ending with an end-of-sentence token.

10. The system of claim 9 wherein said recurrent neural network is trained in an end-to-end manner.

11. The system of claim 8 wherein said recurrent neural network is trained in an end-to-end manner with a loss defined by a cross-entropy between a true answer and a predicted output, wherein the predicted output is represented using one-hot encoding.

12. The system of claim 8 wherein said recurrent neural network further comprises a memory, an answer unit, a question unit, and an input unit, wherein the memory calculates matching probabilities the input sentences and the questions by computing an inner product follows by a softmax function.

13. The system of claim 12 wherein said answer unit includes said long-short term memory unit.

14. The system of claim 8 wherein said vector representations include a matching probability vector.

15. A long-term memory network system for text comprehension, said system comprising:

at least one processor; and

a computer-usable medium embodying computer program code, said computer-usable medium capable of communicating with said at least one processor, said computer program code comprising instructions executable by said at least one processor and configured for:

configuring a recurrent neural network with an external memory and long-short term memory unit; and instructing said recurrent neural network to encode raw text information into vector representations, form memories, find relevant sentences to answer questions, and generate multi-word answers to said questions utilizing said long short term memory unit; and

controlling the long-short term memory unit with a set of three gates comprising an input gate, an output gate and a forget gate, wherein the long-short term memory unit accumulates knowledge from input data at each time step among a plurality of time steps and based on values of the input gate, the output gate, and the forget gate, stores the knowledge in internal states,

wherein said raw text information comprises input sentences and questions, and wherein the long-term memory network combines an End-to-End Memory Network (MemN2N) with a Long-Short Term Memory network, wherein the Long-Short Term Memory network includes the long-short term memory unit, and

wherein said recurrent neural network comprises a weakly supervised architecture and wherein the long-short term memory unit includes a memory module comprising a plurality of sub-units including a plurality of sentence representation units, a question representation unit, a matching probability vector unit, and a unit representing the output of the MemN2N, wherein output from the matching probability vector unit is provided as input to the unit representing the output of the MemN2N.

16. The system of claim 15 wherein input sentences comprise at least one sentence containing a word belonging to a dictionary and ending with an end-of-sentence token.

17. The system of claim 16 wherein said recurrent neural network is trained in an end-to-end manner.

18. The system of claim 15 wherein said recurrent neural network is trained in an end-to-end manner with a loss defined by a cross-entropy between a true answer and a predicted output, wherein the predicted output is represented using one-hot encoding.

19. The system of claim 15 wherein said recurrent neural network further comprises a memory, an answer unit, a question unit, and an input and wherein said answer unit includes said long-short term memory unit, wherein the memory calculates matching probabilities the input sentences and the questions by computing an inner product followed by a softmax function.

20. The system of claim 19 wherein said vector representations include a matching probability vector.