IP Library › Granted Patent US 12,393,840
Granted Patent B2
US 12,393,840 · App. 18/823,611 · Granted Aug 19, 2025

Granular neural network architecture search over low-level primitives

Inventors: David Richard So (Brooklyn, NY); Quoc V. Le (Sunnyvale, CA); Hanxiao Liu (Santa Clara, CA); Wojciech Andrzej Manke (Zürich, CH); Zihang Dai (Pittsburgh, PA); Noam M. Shazeer (Palo Alto, CA)
Assignee: Google LLC
G06N3/08
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,393,840
App. No.
18/823,611
Granted
Aug 19, 2025
Kind
B2
Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for performing a machine learning task on a network input to generate a network output. One of the systems includes an attention neural network configured to perform the machine learning task. The attention neural network includes one or more attentions layers that each include a squared ReLU activation layer, a depth-wise convolution layer, or both.

Claims (83)

1. A system for performing a machine learning task on a network input to generate a network output, the system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement:

an attention neural network configured to perform the machine learning task, the attention neural network comprising one or more layers, each layer comprising an attention sub-layer and a feed-forward sub-layer, the attention sub-layer configured to:

receive an input sequence for the layer comprising a respective layer input at each of a plurality of positions; and

generate an attended input sequence at least in part by applying an attention mechanism that uses a softmax activation function over the input sequence for the attention sub-layer, the attended input sequence comprising a respective attended layer input at each of the plurality of positions, and the feed-forward sub-layer configured to:

receive the attended input sequence from the attention sub-layer; and

generate an output sequence for the feed-forward sub-layer from the attended input sequence, the output sequence comprising a respective layer output at each of the plurality of positions, and the generating comprising, for each of the plurality of positions:

applying a first linear transformation to the attended layer input at the position to generate a first transformed input at the position, comprising;

applying a rectified polynomial activation function with a degree higher than one to the first transformed input to generate a rectified polynomial activated input at the position;

applying a second linear transformation to the rectified polynomial activated input at the position to generate a second transformed input at the position; and

generating the respective layer output at the position from the second transformed input at the position.

2. The system of claim 1 , wherein the rectified polynomial activation function with a degree higher than one comprises a squared ReLU activation function, and wherein applying the rectified polynomial activation function with a degree higher than one to the first transformed input to generate the rectified polynomial activated input at the position comprises:

applying a ReLU activation function to the first transformed input to generate a ReLU activated input at the position; and

applying a square function to the ReLU activated input to generate a squared ReLU activated input at the position.

3. The system of claim 1 , wherein generating the respective layer output at the position from the second transformed input at the position comprises:

applying a residual connection to the second transformed input to generate the respective layer output at the position.

4. The system of claim 1 , wherein generating the respective layer output at the position from the second transformed input at the position comprises:

applying a layer normalization to the second transformed input to generate the respective layer output at the position.

5. The system of claim 1 , wherein the attention neural network is configured to provide the output sequence for the feed-forward sub-layer to another layer in the attention neural network for further processing.

6. The system of claim 1 , wherein the attention mechanism comprises a multi-head depth-wise convolution attention mechanism.

7. The system of claim 1 , wherein the machine learning task comprises:

an audio processing task, wherein the network input is a sequence representing a spoken utterance, and the network output is a piece of text that is a transcript of the utterance, an indication whether a particular word or phrase was spoken in the utterance, or an identification of a natural language in which the utterance was spoken;

a text to speech task, wherein the network input is text in a natural language or features of text in a natural language, and the network output is a spectrogram, a waveform, or other data defining audio of the text being spoken in the natural language;

a computer vision task, wherein the network input is an image or a point cloud, and the network output is a computer vision output for the image or point cloud, and optionally wherein the computer vision output is a classification output including a score for each of a plurality of categories, each score representing the likelihood that the image or point cloud includes an object belonging to the category;

an image generation task, wherein the network input is a conditioning input, and the network output is a sequence of intensity value inputs for the pixels of an image;

a neural machine translation task, wherein the network input is a sequence of text in one language, and the network output is a translation of the sequence of text into another language;

an agent control task, wherein the network input is a sequence of observations or other data characterizing states of an environment, and the network output defines an action to be performed by the agent in response to a most recent data in the sequence;

a health prediction task, wherein the network input is a sequence derived from electronic health record data for a patient, and the network output is a predicted diagnosis for the patient; or

a genomics task, wherein the network input is a sequence representing a fragment of a DNA sequence or other molecule sequence, and the network output is either an embedding of the fragment for use in a downstream task, or an output for the downstream task.

8. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to implement

an attention neural network configured to perform a machine learning task on a network input to generate a network output, the attention neural network comprising one or more layers, each layer comprising an attention sub-layer and a feed-forward sub-layer, the attention sub-layer configured to:

receive an input sequence for the layer comprising a respective layer input at each of a plurality of positions;

and generate an attended input sequence at least in part by applying an attention mechanism that uses a softmax activation function over the input sequence for the attention sub-layer, the attended input sequence comprising a respective attended layer input at each of the plurality of positions, and the feed-forward sub-layer configured to:

receive the attended input sequence from the attention sub-layer;

and generate an output sequence for the feed-forward sub-layer from the attended input sequence, the output sequence comprising a respective layer output at each of the plurality of positions, and the generating comprising, for each of the plurality of positions:

applying a first linear transformation to the attended layer input at the position to generate a first transformed input at the position, comprising;

applying a rectified polynomial activation function with a degree higher than one to the first transformed input to generate a rectified polynomial activated input at the position;

applying a second linear transformation to the rectified polynomial activated input at the position to generate a second transformed input at the position; and

generating the respective layer output at the position from the second transformed input at the position.

9. The non-transitory computer-readable storage media of claim 8 , wherein the rectified polynomial activation function with a degree higher than one comprises a squared ReLU activation function, and wherein applying the rectified polynomial activation function with a degree higher than one to the first transformed input to generate the rectified polynomial activated input at the position comprises:

applying a ReLU activation function to the first transformed input to generate a ReLU activated input at the position; and

applying a square function to the ReLU activated input to generate a squared ReLU activated input at the position.

10. The non-transitory computer-readable storage media of claim 8 , wherein generating the respective layer output at the position from the second transformed input at the position comprises:

applying a residual connection to the second transformed input to generate the respective layer output at the position.

11. The non-transitory computer-readable storage media of claim 8 , wherein generating the respective layer output at the position from the second transformed input at the position comprises:

applying a layer normalization to the second transformed input to generate the respective layer output at the position.

12. The non-transitory computer-readable storage media of claim 8 , wherein the attention neural network is configured to provide the output sequence for the feed-forward sub-layer to another layer in the attention neural network for further processing.

13. The non-transitory computer-readable storage media of claim 8 , wherein the attention mechanism comprises a multi-head depth-wise convolution attention mechanism.

14. The non-transitory computer-readable storage media of claim 8 , wherein the machine learning task comprises:

an audio processing task, wherein the network input is a sequence representing a spoken utterance, and the network output is a piece of text that is a transcript of the utterance, an indication whether a particular word or phrase was spoken in the utterance, or an identification of a natural language in which the utterance was spoken;

a text to speech task, wherein the network input is text in a natural language or features of text in a natural language, and the network output is a spectrogram, a waveform, or other data defining audio of the text being spoken in the natural language;

a computer vision task, wherein the network input is an image or a point cloud, and the network output is a computer vision output for the image or point cloud, and optionally wherein the computer vision output is a classification output including a score for each of a plurality of categories, each score representing the likelihood that the image or point cloud includes an object belonging to the category;

an image generation task, wherein the network input is a conditioning input, and the network output is a sequence of intensity value inputs for the pixels of an image;

a neural machine translation task, wherein the network input is a sequence of text in one language, and the network output is a translation of the sequence of text into another language;

an agent control task, wherein the network input is a sequence of observations or other data characterizing states of an environment, and the network output defines an action to be performed by the agent in response to a most recent data in the sequence;

a health prediction task, wherein the network input is a sequence derived from electronic health record data for a patient, and the network output is a predicted diagnosis for the patient;

or a genomics task, wherein the network input is a sequence representing a fragment of a DNA sequence or other molecule sequence, and the network output is either an embedding of the fragment for use in a downstream task, or an output for the downstream task.

15. A computer-implemented method for performing a machine learning task on a network input to generate a network output, wherein the method comprises:

receiving, at an attention sub-layer of an attention neural network, an input sequence for the attention sub-layer comprising a respective layer input at each of a plurality of positions; and

generating an attended input sequence at least in part by applying an attention mechanism that uses a softmax activation function over the input sequence for the attention sub-layer, the attended input sequence comprising a respective attended layer input at each of the plurality of positions;

receiving, at a feed-forward sub-layer of the attention neural network, the attended input sequence from the attention sub-layer; and

generating an output sequence for the feed-forward sub-layer from the attended input sequence, the output sequence comprising a respective layer output at each of the plurality of positions, and the generating comprising, for each of the plurality of positions:

applying a first linear transformation to the attended layer input at the position to generate a first transformed input at the position, comprising;

applying a rectified polynomial activation function with a degree higher than one to the first transformed input to generate a rectified polynomial activated input at the position;

applying a second linear transformation to the rectified polynomial activated input at the position to generate a second transformed input at the position; and

generating the respective layer output at the position from the second transformed input at the position.

16. The method of claim 15 , wherein the rectified polynomial activation function with a degree higher than one comprises a squared ReLU activation function, and wherein applying the rectified polynomial activation function with a degree higher than one to the first transformed input to generate the rectified polynomial activated input at the position comprises:

applying a ReLU activation function to the first transformed input to generate a ReLU activated input at the position; and

applying a square function to the ReLU activated input to generate a squared ReLU activated input at the position.

17. The method of claim 15 , wherein generating the respective layer output at the position from the second transformed input at the position comprises:

applying a residual connection to the second transformed input to generate the respective layer output at the position.

18. The method of claim 15 , wherein generating the respective layer output at the position from the second transformed input at the position comprises:

applying a layer normalization to the second transformed input to generate the respective layer output at the position.

19. The method of claim 15 , wherein the attention neural network is configured to provide the output sequence for the feed-forward sub-layer to another layer in the attention neural network for further processing.

20. The method of claim 15 , wherein the attention mechanism comprises a multi-head depth-wise convolution attention mechanism.

21. The method of claim 15 , wherein the machine learning task comprises:

an audio processing task, wherein the network input is a sequence representing a spoken utterance, and the network output is a piece of text that is a transcript of the utterance, an indication whether a particular word or phrase was spoken in the utterance, or an identification of a natural language in which the utterance was spoken;

a text to speech task, wherein the network input is text in a natural language or features of text in a natural language, and the network output is a spectrogram, a waveform, or other data defining audio of the text being spoken in the natural language;

a computer vision task, wherein the network input is an image or a point cloud, and the network output is a computer vision output for the image or point cloud, and optionally wherein the computer vision output is a classification output including a score for each of a plurality of categories, each score representing the likelihood that the image or point cloud includes an object belonging to the category;

an image generation task, wherein the network input is a conditioning input, and the network output is a sequence of intensity value inputs for the pixels of an image;

a neural machine translation task, wherein the network input is a sequence of text in one language, and the network output is a translation of the sequence of text into another language;

an agent control task, wherein the network input is a sequence of observations or other data characterizing states of an environment, and the network output defines an action to be performed by the agent in response to a most recent data in the sequence;

a health prediction task, wherein the network input is a sequence derived from electronic health record data for a patient, and the network output is a predicted diagnosis for the patient; or

a genomics task, wherein the network input is a sequence representing a fragment of a DNA sequence or other molecule sequence, and the network output is either an embedding of the fragment for use in a downstream task, or an output for the downstream task.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 12, 2024
From: SO, DAVID RICHARD; LE, QUOC V.; LIU, HANXIAO; MANKE, WOJCIECH ANDRZEJ; DAI, ZIHANG; SHAZEER, NOAM M.
To: GOOGLE LLC
Reel/Frame 068575/0736 →
Continuity (3)
Continuation 17827362 · May 27, 2022
Provisional Application 63194889 · May 28, 2021
Related Publication 20240428071A1 · Dec 26, 2024
References Cited (67)
US 10452978B2 · Shazeer et al. · 2019 [cited by applicant]
US 10997503B2 · Dohan et al. · 2021 [cited by applicant]
US 11144831B2 · Huang et al. · 2021 [cited by applicant]
US 20200320399A1 · Huang et al. · 2020 [cited by applicant]
US 20210150370A1 · McDonnell et al. · 2021 [cited by applicant]
Vaswani et al., Attention is All You Need, 2017 (Year: 2017). [cited by examiner]
Yang et al., GLoMo: Unsupervisedly Learned Relational Graphs as Transferable Representations, 2018 (Year: 2018). [cited by examiner]
Wu et al., Pay Less Attention With Lightweight and Dynamic Convolutions, 2019 (Year: 2019). [cited by examiner]
Hansson et al., Feedforward neural networks with ReLU activation functions are linear splines, 2017 (Year: 2017). [cited by examiner]
Abadi et al., “TensorFlow: A System for Large-Scale Machine Learning,” Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation, Nov. 2016, 21 pages. [cited by applicant]
Adiwardana et al., “Towards a Human-like Open-Domain Chatbot,” arXiv, Feb. 27, 2020, 38 pages. [cited by applicant]
Ba et al., “Layer normalization,” arXiv, Jul. 21, 2016, 14 pages. [cited by applicant]
Baevski et al., “Adaptive Input Representations for Neural Language Modeling,” arXiv, Feb. 22, 2019, 13 pages. [cited by applicant]
Bender et al., “Can Weight Sharing Outperform Random Architecture Search? An Investigation With TuNAS,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2020, pp. 14323-1433… [cited by applicant]
Brown et al., “Language Models are Few-Shot Learners,” Advances in Neural Information Processing Systems, 2020, 25 pages. [cited by applicant]
Cai et al., “ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware,” arXiv, Feb. 23, 2019, 13 pages. [cited by applicant]
Chelba et al., “One billion word benchmark for measuring progress in statistical language modeling,” Interspeech, Sep. 2014, pp. 2635-2639. [cited by applicant]
Dauphin et al., “Language Modeling with Gated Convolutional Networks,” Proceedings of the 34th International Conference on Machine Learning, 2017, 9 pages. [cited by applicant]
Devlin et al., “BERT: Pre-training of deep bidirectional transformers for language understanding,” arXiv, May 24, 2019, 16 pages. [cited by applicant]
Elfadel, “On the stability of analog ReLU Networks,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, Dec. 3, 2020, 5 pages. [cited by applicant]
Elsken et al., “Efficient Multi-objective Neural Architecture Search via Lamarckian Evolution,” arXiv, Feb. 26, 2019, 23 pages. [cited by applicant]
Elsken et al., “Neural architecture search: A survey,” Journal of Machine Learning Research, Mar. 2019, 20(55):1-21. [cited by applicant]
Fedus et al., “Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity,” arXiv, Jan. 11, 2021, 31 pages. [cited by applicant]
Gao et al., “Making Pre-trained Language Models Better Few-shot Learners,” arXiv, Dec. 31, 2020, 15 pages. [cited by applicant]
Gulati et al., “Conformer: Convolution-augmented Transformer for Speech Recognition,” Interspeech, Oct. 2020, 5 pages. [cited by applicant]
Hendrycks et al., “Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear Units,” arXiv, Nov. 4, 2016, 10 pages. [cited by applicant]
International Search Report and Written Opinion in International Appln. No. PCT/US2022/031468, mailed on Oct. 14, 2022, 24 pages. [cited by applicant]
Jiang et al., “An ConvBert: Improving BERT with Span-based Dynamic Convolution,” Advances in Neural Information Processing Systems, Dec. 2020, 12 pages. [cited by applicant]
Kaplan et al., “Scaling laws for neural language models,” arXiv, Jan. 23, 2020, 30 pages. [cited by applicant]
Karnin et al., “Almost Optimal Exploration in Multi-Armed Bandits,” Proceedings of the 30th International Conference on Machine Learning, 2013, 9 pages. [cited by applicant]
Krotov et al., “Dense Associative Memory for Pattern Recognition,” Advances in Neural Information Processing Systems, 2016, 9 pages. [cited by applicant]
Li et al., “Hyperband: A novel bandit-based approach to hyperparameter optimization,” Journal of Machine Learning Research, Apr. 2018, 18(185):1-52. [cited by applicant]
Li et al., “Random Search and Reproducibility for Neural Architecture Search,” Proceedings of the 35th Uncertainty in Artificial Intelligence Conference, 2020, 11 pages. [cited by applicant]
Liu et al., “DARTS: Differentiable architecture search,” arXiv, Apr. 23, 2019, 13 pages. [cited by applicant]
Liu et al., “RoBERTa: A Robustly Optimized BERT Pretraining Approach,” arXiv, Jul. 26, 2019, 13 pages. [cited by applicant]
Narang et al., “Do Transformer Modifications Transfer Across Implementations and Applications?,” arXiv, Sep. 10, 2021, 16 pages. [cited by applicant]
Patterson et al., “Carbon Emissions and Large Neural Network Training,” arXiv, Apr. 23, 2021, 22 pages. [cited by applicant]
Pham et al., “Efficient neural architecture search via parameter sharing,” arXiv, Feb. 12, 2018, 11 pages. [cited by applicant]
Radford et al., “Language models are unsupervised multitask learners,” Technical Report, OpenAI, 2019, 24 pages. [cited by applicant]
Raffel et al., “Exploring the limits of transfer learning with a unified text-to-text transformer,” arXiv, Jul. 28, 2020, 67 pages. [cited by applicant]
Ramachandran et al., “Searching for Activation Functions,” arXiv, Oct. 27, 2017, 13 pages. [cited by applicant]
Real et al., “AutoML-Zero: Evolving Machine Learning Algorithms From Scratch,” Proceedings of the 37th International Conference on Machine Learning, 2020, 13 pages. [cited by applicant]
Real et al., “Large-Scale Evolution of Image Classifiers,” Proceedings of the 34th International Conference on Machine Learning, 2017, 10 pages. [cited by applicant]
Real et al., “Regularized Evolution for Image Classifier Architecture Search,” Proceedings of the AAAI Conference on Artificial Intelligence, Jul. 17, 2019, pp. 4780-4789. [cited by applicant]
Schick et al., “It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners,” arXiv, Apr. 12, 2021, 14 pages. [cited by applicant]
Schmidhuber, “Evolutionary Principles in Self-Referential Learning. On Learning now to Learn: The Meta-Meta-Meta . . . —Hook,” Diploma thesis, Technische Universitat Munchen, Germany, May 14, 1987, 63 pages. [cited by applicant]
Shazeer et al., “Mesh-TensorFlow: Deep Learning for Supercomputers,” Advances in Neural Information Processing Systems, 2018, 10 pages. [cited by applicant]
Shazeer, “Glu variants improve transformer,” arXiv, Feb. 12, 2020, 5 pages. [cited by applicant]
So et al., “The Evolved Transformer,” Proceedings of the 36th International Conference on Machine Learning, 2019, 10 pages. [cited by applicant]
Stanley et al., “Designing neural networks through neuroevolution,” Nature Machine Intelligence, Jan. 7, 2019, 1:24-35. [cited by applicant]
Sutskever et al., “Sequence to Sequence Learning with Neural Networks,” Advances in Neural Information Processing Systems, 2014, 9 pages. [cited by applicant]
Tan et al., “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks,” Proceedings of the 36th International Conference on Machine Learning, 2019, 10 pages. [cited by applicant]
Tan et al., “MnasNet: Platform-Aware Neural Architecture Search for Mobile,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2019, pp. 2820-2828. [cited by applicant]
Tay et al., “Synthesizer: Rethinking Self-Attention in Transformer Models,” arXiv, Oct. 6, 2020, 13 pages. [cited by applicant]
Vaswani et al., “Attention is all you need,” arXiv, Dec. 6, 2017, 15 pages. [cited by applicant]
Vaswani et al., “Tensor2tensor for neural machine translation,” arXiv, Mar. 16, 2018, 9 pages. [cited by applicant]
Wang et al., “Entailment as Few-Shot Learner,” arXiv, Apr. 29, 2021, 15 pages. [cited by applicant]
Wu et al., “CvT: Introducing Convolutions to Vision Transformers,” arXiv, Mar. 29, 2021, 10 pages. [cited by applicant]
Xiong et al., “On layer normalization in the transformer architecture,” arXiv, Jun. 29, 2020, 17 pages. [cited by applicant]
Yang et al., “XLNet: Generalized Autoregressive Pretraining for Language Understanding,” Advances in Neural Information Processing Systems, 2019, 11 pages. [cited by applicant]
Yang et al., GLoMo: Unsupervisedly Learned Relational Graphs as Transferable Representations, 2018. [cited by applicant]
Yao, “Evolving artificial neural networks,” Proceedings of the IEEE, Sep. 1999, 87(9):1423-1447. [cited by applicant]
Yu et al., “Evaluating the Search Phase of Neural Architecture Search,” ICLR, 2020, 16 pages. [cited by applicant]
Yu et al., “QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension,” ICLR, Feb. 15, 2018, 16 pages. [cited by applicant]
Zhang et al., “Root Mean Square Layer Normalization,” Advances in Neural Information Processing Systems, 2019, 12 pages. [cited by applicant]
Zhu et al., “An Empirical Study of Spatial Attention Mechanisms in Deep Networks,” Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Oct. 2019, pp. 6688-6697. [cited by applicant]
Xu et al., “Efficient Transformer for Remote Sensing Image Segmentation,” Remote Sensing, Sep. 2021, 13(18):3585, 24 pages. [cited by applicant]
Cited By (1)
US 12,633,105