IP Library › Granted Patent US 12,217,173
Granted Patent B2
US 12,217,173 · App. 17/467,096 · Granted Feb 4, 2025

Attention-based sequence transduction neural networks

Inventors: Noam M. Shazeer (Palo Alto, CA); Aidan Nicholas Gomez (Toronto, CA); Lukasz Mieczyslaw Kaiser (Mountain View, CA); Jakob D. Uszkoreit (Berlin, DE); Llion Owen Jones (San Francisco, CA); Niki J. Parmar (Sunnyvale, CA); Illia Polosukhin (Mountain View, CA); Ashish Teku Vaswani (San Francisco, CA)
Assignee: Google LLC
G06N3/08G06N3/04G06N3/045G06N20/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,217,173
App. No.
17/467,096
Granted
Feb 4, 2025
Kind
B2
Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. In one aspect, one of the systems includes an encoder neural network configured to receive the input sequence and generate encoded representations of the network inputs, the encoder neural network comprising a sequence of one or more encoder subnetworks, each encoder subnetwork configured to receive a respective encoder subnetwork input for each of the input positions and to generate a respective subnetwork output for each of the input positions, and each encoder subnetwork comprising: an encoder self-attention sub-layer that is configured to receive the subnetwork input for each of the input positions and, for each particular input position in the input order: apply an attention mechanism over the encoder subnetwork inputs using one or more queries derived from the encoder subnetwork input at the particular input position.

Claims (75)

1. A method of autoregressively generating an output sequence having a respective network output at each of a plurality of output positions in an output order, the method comprising, at each of a plurality of time steps corresponding to respective output positions:

receiving one or more network outputs generated at respective preceding time steps of the plurality of time steps and corresponding to respective preceding outputs positions in the output order; and

processing the one or more received network outputs using a neural network to generate the network output for the output position corresponding to the time step,

wherein the neural network comprises a sequence of one or more subnetworks, each subnetwork being configured to (i) receive a respective subnetwork input for each of the one or more received network outputs corresponding to the respective preceding output positions and (ii) generate a respective subnetwork output for each of the preceding output positions,

wherein each subnetwork comprises:

a self-attention sub-layer that is configured to, at each time step, receive the respective subnetwork input for each of the preceding output positions and, for each particular preceding output position of the preceding output positions:

apply a self-attention mechanism over the subnetwork inputs at the preceding output positions to generate a respective self-attention output for the particular preceding output position, wherein applying a self-attention mechanism comprises: determining a query according to the subnetwork input at the particular preceding output position, determining keys according to the subnetwork inputs at the preceding output positions, determining values according to the subnetwork inputs at the preceding output positions, and using the determined query, keys, and values to generate the respective self-attention output for the particular preceding output position.

2. The method of claim 1 , wherein the neural network further comprises:

an embedding layer configured to, at each time step:

for each of the one or more received network outputs at the respective preceding output positions:

map the network output to an embedded representation of the network output, and

combine the embedded representation of the network output with a positional embedding of the corresponding preceding output position to generate a combined embedded representation of the network output; and

provide the combined embedded representations of the network output as input to a first subnetwork in the sequence of subnetworks.

3. The method of claim 1 , wherein at least one of the subnetworks comprises:

a position-wise feed-forward layer that is configured to, at each time step:

for each particular preceding output position of the preceding output positions:

receive a feed-forward input at the particular preceding output position, and

apply a sequence of transformations to the feed-forward input at the particular preceding output position to generate a feed-forward output for the particular preceding output position.

4. The method of claim 3 , wherein the sequence of transformations comprises a plurality of learned linear transformations separated by at least one activation function.

5. The method of claim 3 , wherein the at least one subnetwork further comprises:

a residual connection layer that combines the feed-forward outputs of the position-wise feed-forward layer with the feed-forward inputs to the position-wise feed-forward layer to generate a residual output, and

a layer normalization layer that applies layer normalization to the residual output.

6. The method of claim 1 , wherein each self-attention sub-layer is configured to, at each time step:

apply a learned query linear transformation to the subnetwork input at each particular preceding output position to generate a respective query for each particular preceding output position,

apply a learned key linear transformation to the subnetwork input at each particular preceding output position to generate a respective key for each particular preceding output position,

apply a learned value linear transformation to the subnetwork input at each particular preceding output position to generate a respective key for each particular preceding output position, and

for each particular preceding output position of the preceding output positions,

determine a respective output-position specific weight corresponding to each preceding output position by applying a comparison function between the query for the particular preceding output position and the keys, and

determine the self-attention output for the particular preceding output position by determining a weighted sum of the values weighted by the corresponding output-position specific weights.

7. The method of claim 6 , wherein for each particular preceding output position, determining the respective output-position specific weights corresponding to the preceding output positions comprises determining a non-zero output-position specific weight only for output positions that precede, in the output order, the particular preceding output position.

8. The method of claim 1 , wherein each self-attention sub-layer comprises a plurality of self-attention layers, wherein:

each self-attention layer is configured to, at each time step, generate a respective initial self-attention output for each preceding output position, and

the self-attention sub-layer is configured to, at each time step, combine the initial self-attention outputs generated by the self-attention layers to generate the self-attention output for the self-attention sub-layer.

9. The method of claim 8 , wherein the self-attention layers operate in parallel.

10. The method of claim 1 , wherein each subnetwork further comprises:

a residual connection layer that combines the self-attention outputs of the self-attention sub-layer with the inputs to the self-attention sub-layer to generate a residual output, and

a layer normalization layer that applies layer normalization to the residual output.

11. The method of claim 1 , wherein each network output in the output sequence represents a respective text token of a text represented by the output sequence.

12. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations for autoregressively generating an output sequence having a respective network output at each of a plurality of output positions in an output order, the operations comprising, at each of a plurality of time steps corresponding to respective output positions:

receiving one or more network outputs generated at respective preceding time steps of the plurality of time steps and corresponding to respective preceding outputs positions in the output order; and

processing the one or more received network outputs using a neural network to generate the network output for the output position corresponding to the time step,

wherein the neural network comprises a sequence of one or more subnetworks, each subnetwork being configured to (i) receive a respective subnetwork input for each of the one or more received network outputs corresponding to the respective preceding output positions and (ii) generate a respective subnetwork output for each of the preceding output positions,

wherein each subnetwork comprises:

a self-attention sub-layer that is configured to, at each time step, receive the respective subnetwork input for each of the preceding output positions and, for each particular preceding output position of the preceding output positions:

apply a self-attention mechanism over the subnetwork inputs at the preceding output positions to generate a respective self-attention output for the particular preceding output position, wherein applying a self-attention mechanism comprises: determining a query according to the subnetwork input at the particular preceding output position, determining keys according to the subnetwork inputs at the preceding output positions, determining values according to the subnetwork inputs at the preceding output positions, and using the determined query, keys, and values to generate the respective self-attention output for the particular preceding output position.

13. The system of claim 12 , wherein each self-attention sub-layer is configured to, at each time step:

apply a learned query linear transformation to the subnetwork input at each particular preceding output position to generate a respective query for each particular preceding output position,

apply a learned key linear transformation to the subnetwork input at each particular preceding output position to generate a respective key for each particular preceding output position,

apply a learned value linear transformation to the subnetwork input at each particular preceding output position to generate a respective key for each particular preceding output position, and

for each particular preceding output position of the preceding output positions,

determine a respective output-position specific weight corresponding to each preceding output position by applying a comparison function between the query for the particular preceding output position and the keys, and

determine the self-attention output for the particular preceding output position by determining a weighted sum of the values weighted by the corresponding output-position specific weights.

14. The system of claim 13 , wherein for each particular preceding output position, determining the respective output-position specific weights corresponding to the preceding output positions comprises determining a non-zero output-position specific weight only for output positions that precede, in the output order, the particular preceding output position.

15. The system of claim 12 , wherein each self-attention sub-layer comprises a plurality of self-attention layers, wherein:

each self-attention layer is configured to, at each time step, generate a respective initial self-attention output for each preceding output position, and

the self-attention sub-layer is configured to, at each time step, combine the initial self-attention outputs generated by the self-attention layers to generate the self-attention output for the self-attention sub-layer.

16. The system of claim 15 , wherein the self-attention layers operate in parallel.

17. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for autoregressively generating an output sequence having a respective network output at each of a plurality of output positions in an output order, the operations comprising, at each of a plurality of time steps corresponding to respective output positions:

receiving one or more network outputs generated at respective preceding time steps of the plurality of time steps and corresponding to respective preceding outputs positions in the output order; and

processing the one or more received network outputs using a neural network to generate the network output for the output position corresponding to the time step,

wherein the neural network comprises a sequence of one or more subnetworks, each subnetwork being configured to (i) receive a respective subnetwork input for each of the one or more received network outputs corresponding to the respective preceding output positions and (ii) generate a respective subnetwork output for each of the preceding output positions,

wherein each subnetwork comprises:

a self-attention sub-layer that is configured to, at each time step, receive the respective subnetwork input for each of the preceding output positions and, for each particular preceding output position of the preceding output positions:

apply a self-attention mechanism over the subnetwork inputs at the preceding output positions to generate a respective self-attention output for the particular preceding output position, wherein applying a self-attention mechanism comprises: determining a query according to the subnetwork input at the particular preceding output position, determining keys according to the subnetwork inputs at the preceding output positions, determining values according to the subnetwork inputs at the preceding output positions, and using the determined query, keys, and values to generate the respective self-attention output for the particular preceding output position.

18. The non-transitory computer storage media of claim 17 , wherein each self-attention sub-layer is configured to, at each time step:

apply a learned query linear transformation to the subnetwork input at each particular preceding output position to generate a respective query for each particular preceding output position,

apply a learned key linear transformation to the subnetwork input at each particular preceding output position to generate a respective key for each particular preceding output position,

apply a learned value linear transformation to the subnetwork input at each particular preceding output position to generate a respective key for each particular preceding output position, and

for each particular preceding output position of the preceding output positions,

determine a respective output-position specific weight corresponding to each preceding output position by applying a comparison function between the query for the particular preceding output position and the keys, and

determine the self-attention output for the particular preceding output position by determining a weighted sum of the values weighted by the corresponding output-position specific weights.

19. The non-transitory computer storage media of claim 18 , wherein for each particular preceding output position, determining the respective output-position specific weights corresponding to the preceding output positions comprises determining a non-zero output-position specific weight only for output positions that precede, in the output order, the particular preceding output position.

20. The non-transitory computer storage media of claim 17 , wherein each self-attention sub-layer comprises a plurality of self-attention layers, wherein:

each self-attention layer is configured to, at each time step, generate a respective initial self-attention output for each preceding output position, and

the self-attention sub-layer is configured to, at each time step, combine the initial self-attention outputs generated by the self-attention layers to generate the self-attention output for the self-attention sub-layer.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 9, 2021
From: SHAZEER, NOAM M.; GOMEZ, AIDAN NICHOLAS; KAISER, LUKASZ MIECZYSLAW; USZKOREIT, JAKOB D.; JONES, LLION OWEN; PARMAR, NIKI; POLOSUKHIN, ILLIA; VASWANI, ASHISH TEKU
To: GOOGLE INC.
Reel/Frame 057430/0251 →
CHANGE OF NAME Recorded Sep 9, 2021
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 057453/0807 →
Continuity (7)
Continuation 16932422 · Jul 17, 2020
Continuation 16559392 · Sep 3, 2019
Continuation 16021971 · Jun 28, 2018
Continuation PCTUS2018034224 · May 23, 2018
Provisional Application 62541594 · Aug 4, 2017
Provisional Application 62510256 · May 23, 2017
Related Publication 20220051099A1 · Feb 17, 2022
References Cited (65)
US 20090319267A1 · Kurki-Suonio · 2009 [cited by applicant]
US 20160232440A1 · Gregor et al. · 2016 [cited by applicant]
US 20170039174A1 · Strope et al. · 2017 [cited by applicant]
US 20170124433A1 · Chandraker et al. · 2017 [cited by applicant]
US 20170140271A1 · Reed et al. · 2017 [cited by applicant]
US 20180181592A1 · Chen et al. · 2018 [cited by applicant]
US 20180341860A1 · Shazeer · 2018 [cited by applicant]
US 20200327359A1 · Blundell · 2020 [cited by applicant]
CN 105159890A · 2015 [cited by applicant]
CN 105868829A · 2016 [cited by applicant]
CN 106372577A · 2017 [cited by applicant]
CN 106462802A · 2017 [cited by applicant]
KR 1020150037986 · 2015 [cited by applicant]
TW 501074B · 2002 [cited by applicant]
Extended European Search Report in European Appln. No. 22204636.9, dated Mar. 1, 2023, 11 pages. [cited by applicant]
Extended European Search Report in European Appln. No. 22204664.1, dated Mar. 1, 2023, 10 pages. [cited by applicant]
Extended European Search Report in European Appln. No. 22204659.1, dated Mar. 1, 2023, 10 pages. [cited by applicant]
Extended European Search Report in European Appln. No. 22204647.6, dated Mar. 1, 2023, 9 pages. [cited by applicant]
Extended European Search Report in European Appln. No. 22204640.1, dated Mar. 1, 2023, 11 pages. [cited by applicant]
Extended European Search Report in European Appln. No. 22204654.2, dated Mar. 1, 2023, 11 pages. [cited by applicant]
Office Action in Chinese Appln. No. 201880007309.X, dated Mar. 15, 2023, 10 pages (with English Translation). [cited by applicant]
Office Action in Chinese Appln. No. 201880007309.X, dated Dec. 2, 2022, 16 pages (with English Translation). [cited by applicant]
Office Action in Korean Appln. No. 10-2022-7033030, dated Dec. 26, 2022, 6 pages (with English Translation). [cited by applicant]
Xu, “Research on Noise Resistant Speech Recognition Based on Feature Extraction,” Doctoral Dissertation, South China University of Technology, Nov. 15, 2006, 123 pages (with English Abstract). [cited by applicant]
Zheng et al., “Anomaly detection in smart grid based on encoder-decoder framework with recurrent neural network,” The Journal of China Universities of Posts and Telecommunications, Dec. 30, 2017, 7 pages. [cited by applicant]
AU Examination Report in Australian Appln. No. 2018271931, dated Feb. 28, 2020, 3 pages. [cited by applicant]
AU Office Action in Australian Application No. 2018271931, dated Apr. 28, 2020, 3 pages. [cited by applicant]
Ba et al. “Layer Normalization,” arXiv, Jul. 21, 2016, 14 pages. [cited by applicant]
Bahdanau et al. “Neural Machine Translation by Jointly Learning to Align and Translate,” arXiv, mailed on May 19, 2016, 15 pages. [cited by applicant]
Britz et al.“Massive exploration of neural machine translation architectures,” arXiv, Mar. 21, 2017, 9 pages. [cited by applicant]
Cheng et al. “Long short-term memory-networks for machine reading,” arXiv, Sep. 20, 2016, 11 pages. [cited by applicant]
Cho et al. “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” arXiv, Sep. 3, 2014, 15 pages. [cited by applicant]
Chollet. “Xception: Deep Learning with depthwise separable convolution,” arXiv, Apr. 4, 2017, 8 pages. [cited by applicant]
Chung et al. “Empirical evaluation of gated recurrent neural networks on sequence modeling,” arXiv, Dec. 11, 2014, 9 pages. [cited by applicant]
Daniluk et al., “Frustratingly Short Attention Spans in Neural Language Modeling,” arXiv, Feb. 2017, 10 pages. [cited by applicant]
Gehring et al. “Convolutional sequence to sequence learning,” arXiv, May 12, 2017, 15 pages. [cited by applicant]
Hochreiter et al. “Gradient flow in recurrent nets: the difficulty of learning long-term dependencies,” A field Guide to Dynamical Recurrent Neural Networks, 2001, 15 pages. [cited by applicant]
Hochreiter et al. “Long short term memory,” Neural Computation 9(8), Nov. 1997, 46 pages. [cited by applicant]
IN Office Action in Indian Application No. 201927025719, dated May 5, 2021, 7 pages (with English translation). [cited by applicant]
International Preliminary Report on Patentability issued in International Application No. PCT/US2018/034224, mailed on Nov. 26, 2019, 8 pages. [cited by applicant]
International Search Report and Written Opinion issued in International Application No. PCT/US2018/034224, mailed on Sep. 24, 2018, 14 pages. [cited by applicant]
Jozefowiz et al. “Exploring the limits of language modeling,” arXiv, Feb. 7, 2016, 11 pages. [cited by applicant]
JP Office Action in Japanese Application No. 2019-538514, dated Jan. 4, 2021, 4 pages (with English translation). [cited by applicant]
JP Office Action in Japanese Application No. 2019-538514, dated Jul. 20, 2020, 4 pages (with English translation). [cited by applicant]
Kaiser et al. “Can active memory replace attention?” Advances in Neural Information Processing Systems, Dec. 2016, 9 pages. [cited by applicant]
Kaiser et al. “Neural GPUs learn algorithms,” International Conference on Learning Representations, arXiv, Mar. 15, 2016, 9 pages. [cited by applicant]
Kalchbrenner et al. “Neural machine translation in linear time,” arXiv, Mar. 15, 2017, 9 pages. [cited by applicant]
Kim et al. “Structured attention networks,” arXiv, Feb. 16, 2017, 21 pages. [cited by applicant]
Kingma et al. “Adam: A method for stochastic optimization,” arXiv, Jul. 23, 2015, 15 pages. [cited by applicant]
KR Notice of Allowance in Korean Application No. 10-2019-7019186, dated Sep. 3, 2020, 4 pages (with English translation). [cited by applicant]
KR Office Action in Korean Appln. No. 10-2020-7032482, dated Jun. 29, 2022, 5 pages (with English Translation). [cited by applicant]
Kuchaiev et al. “Factorization tricks for lstm networks,” arXiv, Feb. 24, 2018, 6 pages. [cited by applicant]
Lin et al. “A structured self-attentive sentence embedding,” arXiv, Mar. 9, 2017, 15 pages. [cited by applicant]
Luong et al. “Effective approaches to attention based neural machine translation,” arXiv, Sep. 20, 2015, 11 pages. [cited by applicant]
Parikh et al. “A decomposable attention model for natural language inference,” Proceedings of the Empirical Methods in Natural Language Processing conference, Nov. 2016, 7 pages. [cited by applicant]
Paulus et al. “A deep reinforced model for abstractive summarization,” arXiv, Nov. 13, 2017, 12 pages. [cited by applicant]
RU Office Action in Russian Application No. 2019122632/28, mailed Sep. 1, 2020, 14 pages (with English translation). [cited by applicant]
Sennrich et al. “Neural Machine Translation of rare words with subword units,” arXiv, Jun. 10, 2016, 11 pages. [cited by applicant]
Shazeer et al. “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” arXiv, Jan. 23, 2017, 19 pages. [cited by applicant]
Srivastava et al. “Dropout: a simple way to prevent neural network from overfitting,” Journal of Machine Learning Research, 15(1), Jan. 2014, 30 pages. [cited by applicant]
Sukhbaatar et al., “End-to-End Memory Networks,” arXiv, Nov. 24, 2015, 11 pages. [cited by applicant]
Sutskever et al. “Sequence to sequence learning with neural networks,” Advances in Neural Information Processing Systems, Dec. 2014, 9 pages. [cited by applicant]
Szegedy et al. “Rethinking the inception architecture for computer vision,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2016, 9 pages. [cited by applicant]
Vaswan et al. “Attention is All You Need,” 31st Conference on Neural Information Processing Systems, Jun. 12, 2017, 15 pages. [cited by applicant]
Wu et al. “Google's neural machine translation system: Bridging the gap between human and machine translation,” arXiv, Oct. 8, 2016, 23 pages. [cited by applicant]
Cited By (2)
US 12,646,130 US 12,651,286