IP Library › Granted Patent US 12,254,411
Granted Patent B2
US 12,254,411 · App. 17/175,567 · Granted Mar 18, 2025

Attention neural networks with linear units

Inventor: Noam M. Shazeer (Palo Alto, CA)
Assignee: Google LLC
G06N3/082G06N3/048
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,254,411
App. No.
17/175,567
Granted
Mar 18, 2025
Kind
B2
Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a machine learning task on a network input to generate a network output. In one aspect, one of the systems includes an attention neural network configured to perform the machine learning task, the attention neural network including one or more attention layers, each attention layer comprising an attention sub-layer and a feed-forward sub-layer that applies an element-wise multiplication between two vectors generated as a result of two different linear transformations performed on the same attended layer input.

Claims (69)

1. A system for performing a machine learning task on a network input to generate a network output, the system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement:

an attention neural network configured to perform the machine learning task, the attention neural network comprising a plurality of attention layers, each attention layer comprising an attention sub-layer and a feed-forward sub-layer, the attention sub-layer configured to:

receive an input sequence for the attention layer comprising a respective layer input at each of one or more positions; and

generate an attended input sequence at least in part by applying an attention mechanism to the input sequence for the attention layer, the attended input sequence comprising a respective attended layer input at each of the one or more positions, and the feed-forward sub-layer configured to:

receive the attended input sequence generated by the attention sub-layer of the attention layer; and

generate an output sequence for the attention layer from the attended input sequence, the output sequence comprising a respective layer output at each of the one or more positions, and the generating comprising, for each of the positions:

generating a first transformed input from the attended layer input at the position in the attended input sequence generated by the attention sub-layer of the attention layer, comprising applying a first linear transformation to the attended layer input at the position;

generating a second transformed input from the attended layer input at the position in the attended input sequence generated by the attention sub-layer of the attention layer, comprising applying a second, different linear transformation to the attended layer input at the position, wherein:

the first and second linear transformations have been learned during the training of the attention neural network to perform the machine learning task,

the training of the attention neural network comprises training the attention neural network on an unsupervised data set through unsupervised learning;

the same first linear transformation is applied to the attended layer inputs at each of the positions in the input sequence, and

the same second linear transformation is applied to the attended layer inputs at each of the positions in the input sequence;

generating a third transformed input by performing an element-wise multiplication between (i) the first transformed input generated from the attended layer input at the position in the attended input sequence generated by the attention sub-layer of the attention layer and (ii) the second transformed input generated from the attended layer input at the position in the attended input sequence generated by the attention sub-layer of the attention layer; and

generating the layer output at the position from the third transformed input;

wherein the attention neural network further comprises one or more output layers that are configured to process at least one of the layer outputs in an output sequence generated by one of the attention layers to generate at least a portion of the network output for performing the machine learning task.

2. The system of claim 1 , wherein generating the layer output at the position from the third transformed input comprises:

applying a third linear transformation to the third transformed input to generate a fourth transformed input; and

generating the layer output at the position from the fourth transformed input.

3. The system of claim 2 , wherein generating the layer output from the third transformed input comprises:

applying a residual connection and normalization to the third transformed inputs at the positions to generate the output sequence.

4. The system of claim 1 , wherein the first transformed input is the output of the first linear transformation.

5. The system of claim 1 , wherein generating the first transformed input further comprises applying a non-linear activation function to the output of the first linear transformation.

6. The system of claim 5 , wherein the non-linear activation function is a ReLu activation function.

7. The system of claim 5 , wherein the non-linear activation function is a sigmoid activation function.

8. The system of claim 5 , wherein the non-linear activation function is a Gaussian Error Linear Units activation function.

9. The system of claim 5 , wherein the non-linear activation function is a Swish activation function.

10. One or more non-transitory computer-readable storage media storing instructions that when executed by the one or more computers cause the one or more computers to implement:

an attention neural network configured to perform a machine learning task on a network input to generate a network output, the attention neural network comprising a plurality of attention layers, each attention layer comprising an attention sub-layer and a feed-forward sub-layer, the attention sub-layer configured to:

receive an input sequence for the attention layer comprising a respective layer input at each of one or more positions; and

generate an attended input sequence at least in part by applying an attention mechanism to the input sequence for the attention layer, the attended input sequence comprising a respective attended layer input at each of the one or more positions, and the feed-forward sub-layer configured to:

receive the attended input sequence generated by the attention sub-layer of the attention layer; and

generate an output sequence for the attention layer from the attended input sequence, the output sequence comprising a respective layer output at each of the one or more positions, and the generating comprising, for each of the positions:

generating a first transformed input from the attended layer input at the position in the attended input sequence generated by the attention sub-layer of the attention layer, comprising applying a first linear transformation to the attended layer input at the position;

generating a second transformed input from the attended layer input at the position in the attended input sequence generated by the attention sub-layer of the attention layer, comprising applying a second, different linear transformation to the attended layer input at the position, wherein:

the first and second linear transformations have been learned during the training of the attention neural network to perform the machine learning task,

the training of the attention neural network comprises training the attention neural network on an unsupervised data set through unsupervised learning;

the same first linear transformation is applied to the attended layer inputs at each of the positions in the input sequence, and

the same second linear transformation is applied to the attended layer inputs at each of the positions in the input sequence;

generating a third transformed input by performing an element-wise multiplication between (i) the first transformed input generated from the attended layer input at the position in the attended input sequence generated by the attention sub-layer of the attention layer and (ii) the second transformed input generated from the attended layer input at the position in the attended input sequence generated by the attention sub-layer of the attention layer; and

generating the layer output at the position from the third transformed input;

wherein the attention neural network further comprises one or more output layers that are configured to process at least one of the layer outputs in an output sequence generated by one of the attention layers to generate at least a portion of the network output for performing the machine learning task.

11. The non-transitory computer-readable storage media of claim 10 , wherein generating the layer output at the position from the third transformed input comprises:

applying a third linear transformation to the third transformed input to generate a fourth transformed input; and

generating the layer output at the position from the fourth transformed input.

12. The non-transitory computer-readable storage media of claim 11 , wherein generating the layer output from the third transformed input comprises:

applying a residual connection and normalization to the third transformed inputs at the positions to generate the output sequence.

13. The non-transitory computer-readable storage media of claim 10 , wherein the first transformed input is the output of the first linear transformation.

14. The non-transitory computer-readable storage media of claim 10 , wherein generating the first transformed input further comprises applying a non-linear activation function to the output of the first linear transformation.

15. The non-transitory computer-readable storage media of claim 14 , wherein the non-linear activation function is a ReLu activation function.

16. The non-transitory computer-readable storage media of claim 14 , wherein the non-linear activation function is a sigmoid activation function.

17. The non-transitory computer-readable storage media of claim 14 , wherein the non-linear activation function is a Gaussian Error Linear Units activation function.

18. The non-transitory computer-readable storage media of claim 14 , wherein the non-linear activation function is a Swish activation function.

19. A method performed by one or more computers, the method comprising:

receiving a network input; and

processing the network input using an attention neural network configured to perform a machine learning task on the network input to generate a network output, the attention neural network comprising a plurality of attention layers, each attention layer comprising an attention sub-layer and a feed-forward sub-layer, the attention sub-layer configured to:

receive an input sequence for the attention layer comprising a respective layer input at each of one or more positions; and

generate an attended input sequence at least in part by applying an attention mechanism to the input sequence for the attention layer, the attended input sequence comprising a respective attended layer input at each of the one or more positions, and the feed-forward sub-layer configured to:

receive an input sequence for the attention layer comprising a respective layer input at each of one or more positions; and

generate an attended input sequence at least in part by applying an attention mechanism to the input sequence for the attention layer, the attended input sequence comprising a respective attended layer input at each of the one or more positions, and the feed-forward sub-layer configured to:

receive the attended input sequence generated by the attention sub-layer of the attention layer; and

generate an output sequence for the attention layer from the attended input sequence, the output sequence comprising a respective layer output at each of the one or more positions, and the generating comprising, for each of the positions:

generating a first transformed input from the attended layer input at the position in the attended input sequence generated by the attention sub-layer of the attention layer, comprising applying a first linear transformation to the attended layer input at the position;

generating a second transformed input from the attended layer input at the position in the attended input sequence generated by the attention sub-layer of the attention layer, comprising applying a second, different linear transformation to the attended layer input at the position, wherein:

the first and second linear transformations have been learned during the training of the attention neural network to perform the machine learning task,

the training of the attention neural network comprises training the attention neural network on an unsupervised data set through unsupervised learning;

the same first linear transformation is applied to the attended layer inputs at each of the positions in the input sequence, and

the same second linear transformation is applied to the attended layer inputs at each of the positions in the input sequence;

generating a third transformed input by performing an element-wise multiplication between (i) the first transformed input generated from the attended layer input at the position in the attended input sequence generated by the attention sub-layer of the attention layer and (ii) the second transformed input generated from the attended layer input at the position in the attended input sequence generated by the attention sub-layer of the attention layer; and

generating the layer output at the position from the third transformed input, wherein the attention neural network further comprises one or more output lavers that are configured to process at least one of the layer outputs in an output sequence generated by one of the attention layers to generate at least a portion of the network output for performing the machine learning task.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 18, 2021
From: SHAZEER, NOAM M.
To: GOOGLE LLC
Reel/Frame 055312/0783 →
Continuity (2)
Provisional Application 62975707 · Feb 12, 2020
Related Publication 20210248473A1 · Aug 12, 2021
References Cited (20)
US 20180261214A1 · Gehring · 2018 [cited by examiner]
US 20180341860A1 · Shazeer · 2018 [cited by examiner]
US 20200258496A1 · Yang · 2020 [cited by examiner]
Medina, “Parallel Attention Mechanisms in Neural Machine Translation”, 2018 17th IEEE International Conference on Machine Learning and Applications. (Year: 2018). [cited by examiner]
Sukhbaatar, “Augmenting Self-attention with Persistent Memory”, 2019. (Year: 2019). [cited by examiner]
Gehring, “Convolutional Sequence to Sequence Learning”, Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, PMLR 70, 2017. (Year: 2017). [cited by examiner]
Dai et al., “Transformer-xl: Attentive language models beyond a fixed-length context,” arXiv preprint arXiv:1901.02860, Jan. 2019, 20 pages. [cited by applicant]
Dauphin et al., “Language modeling with gated convolutional networks,” International conference on machine learning, Jul. 2017, pp. 933-941. [cited by applicant]
Devlin et al., “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, Oct. 2018, 16 pages. [cited by applicant]
Glorot et al., “Deep sparse rectifier neural networks,” Proceedings of the fourteenth international conference on artificial intelligence and statistics, Jun. 2011, pp. 315-323. [cited by applicant]
Hendrycks et al., “Gaussian error linear units (gelus),” arXiv preprint arXiv:1606.08415, Jun. 2016, 9 pages. [cited by applicant]
Kitaev et al., “Reformer: The efficient transformer,” arXiv preprint arXiv:2001.04451, Jan. 2020, 12 pages. [cited by applicant]
Mnih et al., “Three new graphical models for statistical language modelling,” Proceedings of the 24th international conference on Machine learning, Jun. 2007, pp. 641-648. [cited by applicant]
Raffel et al., “Exploring the limits of transfer learning with a unified text-to-text transformer,” arXiv preprint arXiv:1910.10683, Oct. 2019, 67 pages. [cited by applicant]
Rajpurkar et al., “Squad: 100,000+ questions for machine comprehension of text,” arXiv preprint arXiv:1606.05250, Jun. 2016, 10 pages. [cited by applicant]
Ramachandran et al., “Searching for activation functions,” arXiv preprint arXiv:1710.05941, Oct. 2017, 13 pages. [cited by applicant]
Shazeer et al., “Adafactor: Adaptive learning rates with sublinear memory cost,” International Conference on Machine Learning, Jul. 2018, pp. 4596-4604. [cited by applicant]
Vaswani et al., “Attention is all you need,” arXiv preprint arXiv:1706.03762, Jun. 2017, 15 pages. [cited by applicant]
Wang et al., “GLUE: A multi-task benchmark and analysis platform for natural language understanding,” arXiv preprint arXiv:1804.07461, Apr. 2018, 20 pages. [cited by applicant]
Wang et al., “Superglue: A stickier benchmark for general-purpose language understanding systems,” arXiv preprint arXiv:1905.00537, May 2019, 29 pages. [cited by applicant]