Reading comprehension neural networks

1. A method comprising:

receiving a document comprising a plurality of document tokens;

receiving a question associated with the document, the question comprising a plurality of question tokens;

processing the document tokens and the question tokens using a reader neural network to generate a joint numeric representation of the document and the question, including

processing the document tokens using a document neural network to generate a respective numeric representation of the document tokens,

processing the question tokens using a question neural network to generate a respective numeric representation of the question tokens, and

generating the joint numeric representation of the document and the question by combining the representation of the document tokens and the representation of the question tokens; and

selecting, from the plurality of document tokens, an answer to the question using the joint numeric representation of the document and the question.

2. The method of claim 1 , wherein the respective numeric representation of the document tokens, the respective numeric representation of the question tokens, and the joint numeric representation of the document and the question are a vectors of numeric values.

3. The method of claim 1 , wherein selecting the answer to the question using the joint numeric representation of the document and the question comprises:

processing the joint numeric representation using a selection neural network to generate a respective token score for each of a plurality of token types; and

selecting a token that is of a highest-scoring token type as the answer to the question.

4. The method of claim 1 , wherein the document neural network comprises a document forward long short-term memory (LSTM) neural network and a document backward LSTM neural network, and wherein generating a respective numeric representation of the document tokens further comprises:

processing each document token in a forward order using the document forward LSTM neural network to generate a respective forward representation for each document token;

processing each document token in a backward order using the document backward LSTM neural network to generate a respective backward representation of the document; and

for each document token, combining the forward representation of the document token and the backward representation of the document token to generate a composite representation of the document token.

5. The method of claim 4 , wherein the question neural network comprises a question forward LSTM neural network and a question backward LSTM neural network, and wherein generating a respective numeric representation of the question tokens further comprises:

processing each question token in a forward order using the question forward LSTM neural network to generate a respective forward representation for each question token;

processing each question token in a backward order using the question backward LSTM neural network to generate a respective backward representation of each question token;

for each question token, combining the forward representation of the question token and the backward representation of the question token to generate a composite representation of the question token; and

combining the forward representation of the last question token and the backward representation of the first question token to generate a composite question representation of the question.

6. The method of claim 5 , wherein generating the joint numeric representation of the document and the question comprises:

identifying an initial document representation for the document;

for each question token:

processing the composite representation of the question token, the composite representations of the document tokens, and a current document representation for the document through one or more neural network layers to update a respective weight for each document token, and

combining the composite representations of the document tokens in accordance with the updated weights for the document tokens to generate an updated document representation; and

generating the joint representation by combining the document representation after the last question token and the composite question representation.

7. The method of claim 1 , wherein the question neural network comprises a question forward long short-term memory (LSTM) neural network and a question backward LSTM neural network, and wherein generating a respective numeric representation of the question tokens comprises:

processing each question token in a forward order using the question forward LSTM neural network to generate a forward representation of the question;

processing each question token in a backward order using the question backward LSTM neural network to generate a backward representation of the question; and

combining the forward representation of the question and the backward representation of the question to generate a composite question representation.

8. The method of claim 7 , wherein the document neural network comprises a document forward LSTM neural network and a document backward LSTM neural network, and wherein generating a respective numeric representation of the document tokens comprises:

processing each document token in a forward order using the document forward LSTM neural network to generate a respective forward representation for each document token;

processing each document token in a backward order using the document backward LSTM neural network to generate a respective backward representation of the document; and

for each document token, combining the forward representation of the document and the backward representation of the document to generate a composite representation of the document token.

9. The method of claim 8 , wherein generating the joint numeric representation of the document and the question comprises:

determining a respective weight for each document token;

determining, for each document token, a weighted composite representation by computing the product of the weight for the document token and the composite representation of the document token;

determining a composite document representation by combining the weighted composite representations of the document tokens; and

generating the joint representation by combining the composite document representation and the composite question representation.

10. The method of claim 9 , wherein determining the respective weight for each document token comprises:

processing the composite representation of the document token and the composite query representation through one or more neural network layers to generate the weight for the document token.

11. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

receiving a document comprising a plurality of document tokens;

receiving a question associated with the document, the question comprising a plurality of question tokens;

processing the document tokens and the question tokens using a reader neural network to generate a joint numeric representation of the document and the question, including

using a document neural network to generate a respective numeric representation of the document tokens,

using a question neural network to generate a respective numeric representation of the question tokens, and

generating the joint numeric representation of the document and the question by combining the representation of the document tokens and the representation of the question tokens; and

selecting, from the plurality of document tokens, an answer to the question using the joint numeric representation of the document and the question.

12. The system of claim 11 , wherein selecting the answer to the question using the joint numeric representation of the document and the question comprises:

processing the joint numeric representation using a selection neural network to generate a respective token score for each of a plurality of token types; and

selecting a token that is of a highest-scoring token type as the answer to the question.

13. The system of claim 11 , wherein the document neural network comprises a document forward long short-term memory (LSTM) neural network and a document backward LSTM neural network, and wherein generating a respective numeric representation of the document tokens further comprises:

processing each document token in a forward order using the document forward LSTM neural network to generate a respective forward representation for each document token;

processing each document token in a backward order using the document backward LSTM neural network to generate a respective backward representation of the document; and

for each document token, combining the forward representation of the document token and the backward representation of the document token to generate a composite representation of the document token.

14. The system of claim 13 , wherein the question neural network comprises a question forward LSTM neural network and a question backward LSTM neural network, and wherein generating a respective numeric representation of the question tokens comprises:

processing each question token in a forward order using the question forward LSTM neural network to generate a respective forward representation for each question token;

processing each question token in a backward order using the question backward LSTM neural network to generate a respective backward representation of each question token;

for each question token, combining the forward representation of the question token and the backward representation of the question token to generate a composite representation of the question token; and

combining the forward representation of the last question token and the backward representation of the first question token to generate a composite question representation of the question.

15. The system of claim 14 , wherein generating the joint numeric representation of the document and the question comprises:

identifying an initial document representation for the document;

for each question token:

processing the composite representation of the question token, the composite representations of the document tokens, and a current document representation for the document through one or more neural network layers to update a respective weight for each document token, and

combining the composite representations of the document tokens in accordance with the updated weights for the document tokens to generate an updated document representation; and

generating the joint representation by combining the document representation after the last question token and the composite question representation.

16. The system of the question neural network comprises a question forward long short-term memory (LSTM) neural network and a question backward LSTM neural network, and wherein generating a respective numeric representation of the question tokens comprises:

processing each question token in a forward order using the question forward LSTM neural network to generate a forward representation of the question;

processing each question token in a backward order using the question backward LSTM neural network to generate a backward representation of the question; and

combining the forward representation of the question and the backward representation of the question to generate a composite question representation.

17. The system of claim 16 , wherein the document neural network comprises a document forward LSTM neural network and a document backward LSTM neural network, and wherein generating a respective numeric representation of the document tokens comprises:

processing each document token in a forward order using the document forward LSTM neural network to generate a respective forward representation for each document token;

processing each document token in a backward order using the document backward LSTM neural network to generate a respective backward representation of the document; and

for each document token, combining the forward representation of the document and the backward representation of the document to generate a composite representation of the document token.

18. The system of claim 17 , wherein generating the joint numeric representation of the document and the question comprises:

determining a respective weight for each document token;

determining, for each document token, a weighted composite representation by computing the product of the weight for the document token and the composite representation of the document token;

determining a composite document representation by combining the weighted composite representations of the document tokens; and

generating the joint representation by combining the composite document representation and the composite question representation.

19. The system of claim 18 , wherein determining the respective weight for each document token comprises:

processing the composite representation of the document token and the composite query representation through one or more neural network layers to generate the weight for the document token.

20. One or more non-transitory computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

receiving a document comprising a plurality of document tokens;

receiving a question associated with the document, the question comprising a plurality of question tokens;

processing the document tokens and the question tokens using a reader neural network to generate a joint numeric representation of the document and the question, including

using a document neural network to generate a respective numeric representation of the document tokens,

using a question neural network to generate a respective numeric representation of the question tokens, and

generating the joint numeric representation of the document and the question by combining the representation of the document tokens and the representation of the question tokens; and

selecting, from the plurality of document tokens, an answer to the question using the joint numeric representation of the document and the question.

21. The non-transitory computer storage media of claim 20 , wherein selecting the answer to the question using the joint numeric representation of the document and the question comprises:

processing the joint numeric representation using a selection neural network to generate a respective token score for each of a plurality of token types; and

selecting a token that is of a highest-scoring token type as the answer to the question.

22. The non-transitory computer storage media of claim 20 , wherein the question neural network comprises a question forward long short-term memory (LSTM) neural network and a question backward LSTM neural network, and wherein generating a respective numeric representation of the question tokens comprises:

processing each question token in a forward order using the question forward LSTM neural network to generate a forward representation of the question;

processing each question token in a backward order using the question backward LSTM neural network to generate a backward representation of the question; and

combining the forward representation of the question and the backward representation of the question to generate a composite question representation.

23. The non-transitory computer storage media of claim 22 , wherein the document neural network comprises a document forward LSTM neural network and a document backward LSTM neural network, and wherein generating a respective numeric representation of the document tokens comprises:

processing each document token in a forward order using the document forward LSTM neural network to generate a respective forward representation for each document token;

processing each document token in a backward order using the document backward LSTM neural network to generate a respective backward representation of the document; and

for each document token, combining the forward representation of the document and the backward representation of the document to generate a composite representation of the document token.

24. The non-transitory computer storage media of claim 23 , wherein generating the joint numeric representation of the document and the question comprises:

determining a respective weight for each document token;

determining, for each document token, a weighted composite representation by computing the product of the weight for the document token and the composite representation of the document token;

determining a composite document representation by combining the weighted composite representations of the document tokens; and

generating the joint representation by combining the composite document representation and the composite question representation.

25. The non-transitory computer storage media of claim 24 , wherein determining the respective weight for each document token comprises:

processing the composite representation of the document token and the composite query representation through one or more neural network layers to generate the weight for the document token.