Resource-efficient attention in a neural network

1 . A method, performed by a computing system, for generating attention information in a neural network having machine-trained weighting values, comprising, as part of an expedited processing path having plural subnetworks:

obtaining original query information;

obtaining shared key and value (KV) information and storing the shared KV information in a cache memory of a hardware processor of the computing system;

in a first subnetwork of the neural network, generating a plurality of instances of head-specific query information based on the original query information by linearly projecting the original query information using head-specific machine-trained query weighting matrices, the plurality of instances of head-specific query information being associated with a plurality of respective representational subspaces;

in a second subnetwork of the neural network, generating a plurality of instances of probability information based on a product of the plurality of respective instances of head-specific query information and the shared KV information; and in a third subnetwork of the neural network, generating the attention information based on the plurality of instances of probability information and the shared KV information, the attention information expressing an amount of attention to be paid to different parts of the shared KV information in interpreting the original query information,

wherein said generating attention information uses the neural network to generate a plurality of head-specific products, each head-specific product corresponding to a representational subspace associated with a particular head, and being computed as a product of a head-specific instance of the plurality of instances of probability information generated by the second subnetwork, the shared KV information, a head-specific value weighting matrix, and a head-specific linear transformation matrix, and

wherein said generating the plurality of instances of probability information and said generating the attention information being performed using the hardware processor of the computing system, and being performed by the plural subnetworks of the neural network without storing a plurality of instances of head-specific key information and a plurality of instances of head-specific value information in the cache memory of the hardware processor, and without generating the plurality of instances of the head-specific key information and the plurality of instances of head-specific value information by linearly projecting original key information and original value information using head-specific key and value machine-trained weighting matrices.

2 . The method of claim 1 , wherein the method further includes obtaining input text, and using a text generation system to generate synthesized text by decoding the input text based on the attention information.

3 . The method of claim 2 , wherein the input text is query text, and the synthesized text is keyword information that is based on the query text, and wherein the method further includes storing the keyword information, and serving information items based on the keyword information.

4 . The method of claim 2 , wherein the input text is text contained in an identified web page, and the synthesized text is a reformulation of the text contained in the web page.

5 . The method of claim 1 , wherein the method is performed by at least one decoder block of a decoder.

6 . The method of claim 5 , wherein the decoder incrementally generates output tokens, each output token that is generated being added to decoder input information that is fed to the decoder for processing in a subsequent decoding operation.

7 . The method of claim 5 , wherein the decoder is a part of an encoder-decoder system, and wherein the shared KV information fed to each decoder block originates from an output of an encoder of the encoder-decoder system.

8 . The method of claim 5 , wherein the decoder is part of a standalone decoder system that does not include an encoder, and wherein the shared KV information fed to each decoder block originates directly or indirectly from output information produced by the decoder.

9 . The method of claim 5 ,

wherein decoder input information that is fed to the decoder includes input text that includes one or more input tokens, combined with one or more output tokens predicted by the decoder, and wherein the method applies the expedited processing path to said one or more input tokens of the input text, and applies another processing path to said one or more output tokens,

wherein the other processing path takes into consideration head-specific instances of key information and head-specific instances of value information.

10 . The method of claim 5 , wherein, for a given beam setting, the decoder produces a plurality of candidate output tokens at a particular instance of time, and wherein the method subsequently processes the plurality of candidate output tokens using a same instance of shared KV information.

11 . The method of claim 1 ,

wherein said generating a plurality of instances of head-specific query information uses the neural network to generate a product, for each representational subspace associated with a particular head, of the original query information, a head-specific query weighting matrix, and a head-specific key weighting matrix.

12 . The method of claim 1 , wherein said generating attention information sums together the plurality of head-specific products.

13 . A computing system for performing attention, comprising:

a neural network having machine-trained weighting values that implements at least one attention operation using plural subnetworks, said at least one attention operation having an expedited processing path that is configured to operate by:

obtaining original query information;

obtaining shared key and value (KV) information and storing the shared KV information in a cache memory of a hardware processor of the computing system;

in a first subnetwork of the neural network, generating a plurality of instances of head-specific query information based on the original query information by linearly projecting the original query information using head-specific machine-trained query weighting matrices, the plurality of instances of head-specific query information being associated with a plurality of respective representational subspaces;

in a second subnetwork of the neural network, generating a plurality of instances of probability information based on a product of the plurality of respective instances of head-specific query information and the shared KV information; and

in a third subnetwork of the neural network, generating attention information based on the plurality of instances of probability information and the shared KV information, the attention information expressing an amount of attention to be paid to different parts of the shared KV information in interpreting the original query information,

wherein said generating attention information uses the neural network to generate a plurality of head-specific products, each head-specific product corresponding to a representational subspace associated with a particular head, and being computed as a product of a head-specific instance of the plurality of instances of probability information generated by the second subnetwork, the shared KV information, a head-specific value weighting matrix, and a head-specific linear transformation matrix, and

wherein said generating the plurality of instances of probability information and said generating the attention information being performed by the plural subnetworks of the neural network without storing a plurality of instances of head-specific key information and a plurality of instances of head-specific value information in the cache memory of the hardware processor, and without generating the plurality of instances of the head-specific key information and the plurality of instances of head-specific value information by linearly projecting original key information and original value information using head-specific key and value machine-trained weighting matrices.

14 . The computing system of claim 13 , wherein the computing system includes a decoder, and wherein said at least one attention operation is a part of at least one decoder block of the decoder.

15 . The computing system of claim 14 , wherein the decoder is configured to incrementally generate output tokens, each output token that is generated being added to decoder input information that is fed to the decoder for processing in a subsequent decoding operation.

16 . The computing system of claim 14 , wherein the computing system includes an encoder-decoder system, wherein the decoder is a part of the encoder-decoder system, and wherein the shared KV information fed to each decoder block originates from an output of an encoder of the encoder-decoder system.

17 . The computing system of claim 14 , wherein the computing system includes a standalone decoder system that does not include an encoder, wherein the decoder is part of the standalone decoder system, and wherein the shared KV information fed to each decoder block in the decoder originates directly or indirectly from output information produced by the decoder.

18 . The computer-readable storage medium for storing computer-readable instructions, the computer-readable instructions, when executed by one or more hardware processors, performing a method, using a neural network having plural subnetworks, that comprises:

obtaining input text; and

generating synthesized text by decoding the input text, said generating synthesized text involving, in an expedited processing path, at least one attention operation comprising:

obtaining original query information;

obtaining shared key and value (KV) information and storing the shared KV information in a cache memory of a hardware processor of a computing system;

in a first subnetwork of the neural network, generating a plurality of instances of head-specific query information based on the original query information by linearly projecting the original query information using head-specific machine-trained query weighting matrices, the plurality of instances of head-specific query information being associated with a plurality of respective representational subspaces;

in a second subnetwork of the neural network, generating a plurality of instances of probability information based on a product of the plurality of respective instances of head-specific query information and the shared KV information; and

in a third subnetwork of the neural network, generating the attention information based on the plurality of instances of probability information and the shared KV information, the attention information expressing an amount of attention to be paid to different parts of the shared KV information in interpreting the original query information,

wherein said generating attention information uses the neural network to generate a plurality of head-specific products, each head-specific product corresponding to a representational subspace associated with a particular head, and being computed as a product of a head-specific instance of the plurality of instances of probability information generated by the second subnetwork, the shared KV information, ahead-specific value weighting matrix, and a head-specific linear transformation matrix, and

wherein said generating the plurality of instances of probability information and said generating the attention information being performed by the plural subnetworks of the neural network without storing a plurality of instances of head-specific key information and a plurality of instances of head-specific value information in the cache memory of the hardware processor, and without generating the plurality of instances of the head-specific key information and the plurality of instances of head-specific value information by linearly projecting original key information and original value information using head-specific key and value machine-trained weighting matrices.

19 . The method of claim 1 , wherein the hardware processor is a graphics processing unit.