Training a document classification neural network

1. A method for training a first neural network to perform a document processing task, wherein the first neural network comprises one or more initial neural network layers and a first output layer, wherein the one or more initial neural network layers have parameters, and wherein the method comprises:

training a language model neural network to predict missing text inputs in text sequences that each include a respective plurality of text inputs, wherein the language model neural network comprises the one or more initial neural network layers and a language model output layer, and wherein training the language model neural network comprises determining pre-trained values of the parameters of the one or more initial neural network layers from initial values of the parameters of the one or more initial neural network layers; and

training the first neural network on a plurality of training documents to determine trained values of the parameters of the one or more initial neural network layers from the pre-trained values of the parameters of the one or more initial neural network layers.

2. The method of claim 1 , wherein training the first neural network on a plurality of training documents to determine trained values of the parameters of the one or more initial neural network layers from the pre-trained values of the parameters of the one or more initial neural network layers comprises:

initializing, for the training of the first neural network, the values of the parameters of the one or more initial neural network layers to be the pre-trained values of the parameters of the one or more initial neural network layers.

3. The method of claim 1 , wherein the language model neural network further comprises:

an embedding input layer configured to, for each text input in an input sequence of text inputs, receive the text input and map the text input to a numeric embedding of the text input.

4. The method of claim 3 , wherein training the language model neural network comprises, for a particular input document:

obtaining a sequence of input text inputs that comprises text inputs from the particular input document arranged in the order in which the text inputs appear in the particular input document;

obtaining a held-out text input that appears in a corresponding position in the particular input document relative to the text inputs in the sequence of input text inputs;

generating an initial output by processing each input text input in the sequence of input text inputs using the language model neural network in accordance with current values of the parameters of the initial neural network layers; and

processing the initial output using the language model output layer to generate a set of text input scores, wherein the set of text input scores comprises a respective score for each of a plurality of vocabulary text inputs that represents a likelihood that the vocabulary text input is the text input that appears in the corresponding position in the particular input document relative to the text inputs in the sequence of input text inputs.

5. The method of claim 4 , wherein training the language model neural network further comprises, for the particular input document:

adjusting the current values of the parameters of the initial neural network layers using the set of text input scores.

6. The method of claim 5 , wherein adjusting the current values of the parameters of the initial neural network layers comprises:

determining an error based on a text input score for the held-out text input; and

updating values of the parameters of the initial neural network layers based on the error through backpropagation.

7. The method of claim 1 , wherein training the language model neural network comprises training the language model neural network on sequences from unlabeled training documents, and wherein training the first neural network comprises training the first neural network on labeled training documents.

8. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for training a first neural network to perform a document processing task, wherein the first neural network comprises one or more initial neural network layers and a first output layer, wherein the one or more initial neural network layers have parameters, and wherein the operations comprise:

training a language model neural network to predict missing text inputs in text sequences that each include a respective plurality of text inputs, wherein the language model neural network comprises the one or more initial neural network layers and a language model output layer, and wherein training the language model neural network comprises determining pre-trained values of the parameters of the one or more initial neural network layers from initial values of the parameters of the one or more initial neural network layers; and

training the first neural network on a plurality of training documents to determine trained values of the parameters of the one or more initial neural network layers from the pre-trained values of the parameters of the one or more initial neural network layers.

9. The system of claim 8 , wherein training the first neural network on a plurality of training documents to determine trained values of the parameters of the one or more initial neural network layers from the pre-trained values of the parameters of the one or more initial neural network layers comprises:

initializing, for the training of the first neural network, the values of the parameters of the one or more initial neural network layers to be the pre-trained values of the parameters of the one or more initial neural network layers.

10. The system of claim 8 , wherein the language model neural network further comprises:

an embedding input layer configured to, for each text input in an input sequence of text inputs, receive the text input and map the text input to a numeric embedding of the text input.

11. The system of claim 10 , wherein training the language model neural network comprises, for a particular input document:

obtaining a sequence of input text inputs that comprises text inputs from the particular input document arranged in the order in which the text inputs appear in the particular input document;

obtaining a held-out text input that appears in a corresponding position in the particular input document relative to the text inputs in the sequence of input text inputs;

generating an initial output by processing each input text input in the sequence of input text inputs using the language model neural network in accordance with current values of the parameters of the initial neural network layers; and

processing the initial output using the language model output layer to generate a set of text input scores, wherein the set of text input scores comprises a respective score for each of a plurality of vocabulary text inputs that represents a likelihood that the vocabulary text input is the text input that appears in the corresponding position in the particular input document relative to the text inputs in the sequence of input text inputs.

12. The system of claim 11 , wherein training the language model neural network further comprises, for the particular input document:

adjusting the current values of the parameters of the initial neural network layers using the set of text input scores.

13. The system of claim 12 , wherein adjusting the current values of the parameters of the initial neural network layers comprises:

determining an error based on a text input score for the held-out text input; and

updating values of the parameters of the initial neural network layers based on the error through backpropagation.

14. The system of claim 8 , wherein training the language model neural network comprises training the language model neural network on sequences from unlabeled training documents, and wherein training the first neural network comprises training the first neural network on labeled training documents.

15. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a first neural network to perform a document processing task, wherein the first neural network comprises one or more initial neural network layers and a first output layer, wherein the one or more initial neural network layers have parameters, and wherein the operations comprise:

training a language model neural network to predict missing text inputs in text sequences that each include a respective plurality of text inputs, wherein the language model neural network comprises the one or more initial neural network layers and a language model output layer, and wherein training the language model neural network comprises determining pre-trained values of the parameters of the one or more initial neural network layers from initial values of the parameters of the one or more initial neural network layers; and

training the first neural network on a plurality of training documents to determine trained values of the parameters of the one or more initial neural network layers from the pre-trained values of the parameters of the one or more initial neural network layers.

16. The computer-readable storage media of claim 15 , wherein training the first neural network on a plurality of training documents to determine trained values of the parameters of the one or more initial neural network layers from the pre-trained values of the parameters of the one or more initial neural network layers comprises:

initializing, for the training of the first neural network, the values of the parameters of the one or more initial neural network layers to be the pre-trained values of the parameters of the one or more initial neural network layers.

17. The computer-readable storage media of claim 15 , wherein the language model neural network further comprises:

an embedding input layer configured to, for each text input in an input sequence of text inputs, receive the text input and map the text input to a numeric embedding of the text input.

18. The computer-readable storage media of claim 17 , wherein training the language model neural network comprises, for a particular input document:

obtaining a sequence of input text inputs that comprises text inputs from the particular input document arranged in the order in which the text inputs appear in the particular input document;

obtaining a held-out text input that appears in a corresponding position in the particular input document relative to the text inputs in the sequence of input text inputs;

generating an initial output by processing each input text input in the sequence of input text inputs using the language model neural network in accordance with current values of the parameters of the initial neural network layers; and

processing the initial output using the language model output layer to generate a set of text input scores, wherein the set of text input scores comprises a respective score for each of a plurality of vocabulary text inputs that represents a likelihood that the vocabulary text input is the text input that appears in the corresponding position in the particular input document relative to the text inputs in the sequence of input text inputs.

19. The computer-readable storage media of claim 18 , wherein training the language model neural network further comprises, for the particular input document:

adjusting the current values of the parameters of the initial neural network layers using the set of text input scores.

20. The computer-readable storage media of claim 15 , wherein training the language model neural network comprises training the language model neural network on sequences from unlabeled training documents, and wherein training the first neural network comprises training the first neural network on labeled training documents.