IP Library › Granted Patent US 12670356
Granted Patent B2
US 12670356 · App. 17/643,736 · Granted Jun 30, 2026

Generating representations of input sequences using neural networks

Inventors: Oriol Vinyals (London, GB); Quoc V. Le (Sunnyvale, CA); Ilya Sutskever (San Francisco, CA)
Assignee: Google LLC
G06N3/02G06F40/40G06N3/0442G06N3/0455
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12670356
App. No.
17/643,736
Granted
Jun 30, 2026
Kind
B2
Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating representations of input sequences. One of the methods includes obtaining an input sequence, the input sequence comprising a plurality of inputs arranged according to an input order; processing the input sequence using a first long short term memory (LSTM) neural network to convert the input sequence into an alternative representation for the input sequence; and processing the alternative representation for the input sequence using a second LSTM neural network to generate a target sequence for the input sequence, the target sequence comprising a plurality of outputs arranged according to an output order.

Claims (36)

1 . A method performed by one or more computers, the method comprising:

obtaining an input sequence, the input sequence comprising a plurality of inputs arranged according to an input order;

processing the input sequence with an encoder network to convert the input sequence into an alternative representation of the input sequence;

processing the alternative representation of the input sequence with a decoder network in accordance with an initial version of a hidden state of the decoder network to predict a first output of a target sequence for the input sequence, wherein the target sequence comprises a plurality of outputs arranged according to an output order, wherein the hidden state of the decoder network is updated as a result of processing the alternative representation for the input sequence;

for each output of the target sequence after the first output, processing a preceding output that was predicted for a preceding position of the target sequence with the decoder network in accordance with a current version of the hidden state of the decoder network to predict a next output of the target sequence, wherein the hidden state of the decoder network is updated as a result of processing each preceding output to predict each next output of the target sequence, wherein the alternative representation of the input sequence is processed with the decoder network just once to predict only the first output of the target sequence.

2 . The method of claim 1 , wherein the input sequence is a variable length input sequence.

3 . The method of claim 2 , wherein the alternative representation is a vector of fixed dimensionality.

4 . The method of claim 1 , comprising:

adding an end-of-sentence token to the end of the input sequence to generate a modified input sequence; and

processing the modified input sequence with the encoder network to generate the alternative representation.

5 . The method of claim 1 , wherein processing the alternative representation of the input sequence with the decoder network comprises initializing the hidden state of the decoder network to the alternative representation of the input sequence.

6 . The method of claim 1 , wherein processing the alternative representation of the input sequence with the decoder network comprises using a left to right beam search decoding technique.

7 . The method of claim 1 , comprising training the encoder network and the decoder network using Stochastic Gradient Descent.

8 . The method of claim 1 , wherein the input sequence is a sequence of words in a first language and the target sequence is a translation of the sequence of words into a second language.

9 . The method of claim 1 , wherein the input sequence is a sequence of words and the target sequence is an autoencoding of the input sequence.

10 . The method of claim 1 , wherein the input sequence is a sequence of graphemes and the target sequence is a phoneme representation of the sequence of graphemes.

11 . The method of claim 1 , wherein the encoder network comprises a first long short-term memory (LSTM) neural network and the decoder network comprises a second LSTM neural network.

12 . A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

obtaining an input sequence, the input sequence comprising a plurality of inputs arranged according to an input order;

processing the input sequence with an encoder network to convert the input sequence into an alternative representation of the input sequence;

processing the alternative representation of the input sequence with a decoder network in accordance with an initial version of a hidden state of the decoder network to predict a first output of a target sequence for the input sequence, wherein the target sequence comprises a plurality of outputs arranged according to an output order, wherein the hidden state of the decoder network is updated as a result of processing the alternative representation for the input sequence;

for each output of the target sequence after the first output, processing a preceding output that was predicted for a preceding position of the target sequence with the decoder network in accordance with a current version of the hidden state of the decoder network to predict a next output of the target sequence, wherein the hidden state of the decoder network is updated as a result of processing each preceding output to predict each next output of the target sequence, wherein the alternative representation of the input sequence is processed with the decoder network just once to predict only the first output of the target sequence.

13 . The system of claim 12 , wherein the input sequence is a variable length input sequence.

14 . The system of claim 13 , wherein the alternative representation is a vector of fixed dimensionality.

15 . The system of claim 12 , wherein the operations comprise:

adding an end-of-sentence token to the end of the input sequence to generate a modified input sequence; and

processing the modified input sequence with the encoder network to generate the alternative representation.

16 . The system of claim 12 , wherein processing the alternative representation of the input sequence with the decoder network comprises initializing the hidden state of the decoder network to the alternative representation of the input sequence.

17 . The system of claim 12 , wherein the input sequence is a sequence of words in a first language and the target sequence is a translation of the sequence of words into a second language.

18 . The system of claim 12 , wherein the input sequence is a sequence of words and the target sequence is an autoencoding of the input sequence.

19 . The system of claim 12 , wherein the input sequence is a sequence of graphemes and the target sequence is a phoneme representation of the sequence of graphemes.

20 . A computer program product encoded on one or more non-transitory storage media, the computer program product comprising instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

obtaining an input sequence, the input sequence comprising a plurality of inputs arranged according to an input order;

processing the input sequence with an encoder network to convert the input sequence into an alternative representation of the input sequence;

processing the alternative representation of the input sequence with a decoder network in accordance with an initial version of a hidden state of the decoder network to predict a first output of a target sequence for the input sequence, wherein the target sequence comprises a plurality of outputs arranged according to an output order, wherein the hidden state of the decoder network is updated as a result of processing the alternative representation for the input sequence;

for each output of the target sequence after the first output, processing a preceding output that was predicted for a preceding position of the target sequence with the decoder network in accordance with a current version of the hidden state of the decoder network to predict a next output of the target sequence, wherein the hidden state of the decoder network is updated as a result of processing each preceding output to predict each next output of the target sequence, wherein the alternative representation of the input sequence is processed with the decoder network just once to predict only the first output of the target sequence.