Granted Patent B1

US 12,307,367 · App. 18/407,329 · Granted May 20, 2025

Inventors: Andrew M. Dai (San Francisco, CA); Quoc V. Le (Sunnyvale, CA)

Assignee: Google LLC

G06N3/08G06N3/044G06N3/045

Abstract

Claims (47)

1. A method for training a first neural network to perform a task, wherein the first neural network comprises one or more initial neural network layers and a first output layer, wherein the one or more initial neural network layers have parameters, and wherein the method comprises:

training a language model neural network to predict missing text inputs in sequences that each include a respective plurality of text inputs, wherein the language model neural network comprises the one or more initial neural network layers, and wherein training the language model neural network comprises determining pre-trained values of the parameters of the one or more initial neural network layers from initial values of the parameters of the one or more initial neural network layers; and

training the first neural network on a plurality of training documents to determine trained values of the parameters of the one or more initial neural network layers from the pre-trained values of the parameters of the one or more initial neural network layers.

2. The method of claim 1 , wherein training the first neural network on a plurality of training documents to determine trained values of the parameters of the one or more initial neural network layers from the pre-trained values of the parameters of the one or more initial neural network layers comprises:

initializing, for the training of the first neural network, the values of the parameters of the one or more initial neural network layers to be the pre-trained values of the parameters of the one or more initial neural network layers.

3. The method of claim 1 , wherein the language model neural network further comprises:

an embedding input layer configured to, for each text input in an input sequence of text inputs, receive the text input and map the text input to a numeric embedding of the text input.

4. The method of claim 1 , wherein training the language model neural network comprises, for a particular input document:

obtaining a sequence that comprises text inputs from the particular input document arranged in the order in which the text inputs appear in the particular input document;

obtaining a held-out text input that appears in a corresponding position in the particular input document relative to the text inputs in the sequence of input text inputs;

processing each text input in the sequence that comprises text inputs from the particular input document using the language model neural network in accordance with current values of the parameters of the initial neural network layers to generate a set of text input scores, wherein the set of text input scores comprises a respective score for each of a plurality of vocabulary text inputs that represents a likelihood that the vocabulary text input is the text input that appears in the corresponding position in the particular input document relative to the text inputs in the sequence that comprises text inputs from the particular input document.

5. The method of claim 4 , wherein training the language model neural network further comprises, for the particular input document:

adjusting the current values of the parameters of the initial neural network layers using the set of text input scores.

6. The method of claim 4 , wherein the corresponding position is after a last text input in the sequence that comprises text inputs from the particular input document.

7. The method of claim 5 , wherein adjusting the current values of the parameters of the initial neural network layers comprises:

determining an error based on a text input score for the held-out text input; and

updating values of the parameters of the initial neural network layers based on the error through backpropagation.

8. The method of claim 1 , wherein training the language model neural network comprises training the language model neural network on sequences from unlabeled training documents, and wherein training the first neural network comprises training the first neural network on labeled training documents.

9. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for training a first neural network to perform a task, wherein the first neural network comprises one or more initial neural network layers and a first output layer, wherein the one or more initial neural network layers have parameters, and wherein the operations comprise:

training a language model neural network to predict missing text inputs in sequences that each include a respective plurality of text inputs, wherein the language model neural network comprises the one or more initial neural network layers, and wherein training the language model neural network comprises determining pre-trained values of the parameters of the one or more initial neural network layers from initial values of the parameters of the one or more initial neural network layers; and

training the first neural network on a plurality of training documents to determine trained values of the parameters of the one or more initial neural network layers from the pre-trained values of the parameters of the one or more initial neural network layers.

10. The system of claim 9 , wherein training the first neural network on a plurality of training documents to determine trained values of the parameters of the one or more initial neural network layers from the pre-trained values of the parameters of the one or more initial neural network layers comprises:

initializing, for the training of the first neural network, the values of the parameters of the one or more initial neural network layers to be the pre-trained values of the parameters of the one or more initial neural network layers.

11. The system of claim 9 , wherein the language model neural network further comprises:

an embedding input layer configured to, for each text input in an input sequence of text inputs, receive the text input and map the text input to a numeric embedding of the text input.

12. The system of claim 9 , wherein training the language model neural network comprises, for a particular input document:

obtaining a sequence that comprises text inputs from the particular input document arranged in the order in which the text inputs appear in the particular input document;

obtaining a held-out text input that appears in a corresponding position in the particular input document relative to the text inputs in the sequence that comprises text inputs from the particular input document;

processing each input text input in the sequence that comprises text inputs from the particular input document using the language model neural network in accordance with current values of the parameters of the initial neural network layers to generate a set of text input scores, wherein the set of text input scores comprises a respective score for each of a plurality of vocabulary text inputs that represents a likelihood that the vocabulary text input is the text input that appears in the corresponding position in the particular input document relative to the text inputs in the sequence of input text inputs.

13. The system of claim 12 , wherein training the language model neural network further comprises, for the particular input document:

adjusting the current values of the parameters of the initial neural network layers using the set of text input scores.

14. The system of claim 13 , wherein adjusting the current values of the parameters of the initial neural network layers comprises:

determining an error based on a text input score for the held-out text input; and

updating values of the parameters of the initial neural network layers based on the error through backpropagation.

15. The system of claim 12 , wherein the corresponding position is after a last text input in the sequence that comprises text inputs from the particular input document.

16. The system of claim 9 , wherein training the language model neural network comprises training the language model neural network on sequences from unlabeled training documents, and wherein training the first neural network comprises training the first neural network on labeled training documents.

17. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a first neural network to perform a task, wherein the first neural network comprises one or more initial neural network layers and a first output layer, wherein the one or more initial neural network layers have parameters, and wherein the operations comprise:

training a language model neural network to predict missing text inputs in sequences that each include a respective plurality of text inputs, wherein the language model neural network comprises the one or more initial neural network layers, and wherein training the language model neural network comprises determining pre-trained values of the parameters of the one or more initial neural network layers from initial values of the parameters of the one or more initial neural network layers; and

training the first neural network on a plurality of training documents to determine trained values of the parameters of the one or more initial neural network layers from the pre-trained values of the parameters of the one or more initial neural network layers.

18. The computer-readable storage media of claim 17 , wherein training the first neural network on a plurality of training documents to determine trained values of the parameters of the one or more initial neural network layers from the pre-trained values of the parameters of the one or more initial neural network layers comprises:

initializing, for the training of the first neural network, the values of the parameters of the one or more initial neural network layers to be the pre-trained values of the parameters of the one or more initial neural network layers.

19. The computer-readable storage media of claim 17 , wherein the language model neural network further comprises:

an embedding input layer configured to, for each text input in an input sequence of text inputs, receive the text input and map the text input to a numeric embedding of the text input.

20. The computer-readable storage media of claim 17 , wherein training the language model neural network comprises, for a particular input document:

obtaining a sequence that comprises text inputs from the particular input document arranged in the order in which the text inputs appear in the particular input document;

obtaining a held-out text input that appears in a corresponding position in the particular input document relative to the text inputs in the sequence that comprises text inputs from the particular input document;

processing each input text input in the sequence that comprises text inputs from the particular input document using the language model neural network in accordance with current values of the parameters of the initial neural network layers to generate a set of text input scores, wherein the set of text input scores comprises a respective score for each of a plurality of vocabulary text inputs that represents a likelihood that the vocabulary text input is the text input that appears in the corresponding position in the particular input document relative to the text inputs in the sequence of input text inputs.

Assignments (2)

Continuity (4)

References Cited (11)