Language processing using a neural network

1. A computer-implemented method for language processing using a neural network, the method comprising:

performing, at an embedding layer of a neural network, embedding processing on a current input to the embedding layer to obtain a plurality of feature vectors, wherein:

the neural network comprises the embedding layer and N transformer layers, and

the N transformer layers comprise a first transformer layer comprising an attention layer and a pooling layer, wherein the first transformer layer is an nth transformer layer in the N transformer layers;

obtaining, at the attention layer of the first transformer layer, a quantity P of input vectors from an upper layer of the first transformer layer; and

determining P intermediate vectors based on the P input vectors, including, for each input vector of the P input vectors:

obtaining, using the input vector as a center, a respective intermediate vector based on correlation values calculated between the input vector and each other input vector in a predetermined attention window range,

determining a quantity Q based on a quantity of feature vectors in the plurality of feature vectors, a value of a coverage radius of an attention window, N, and n, wherein the quantity Q is negatively correlated with n, and wherein determining the quantity Q comprises:

determining a first difference between the quantity of feature vectors in the plurality of feature vectors and the value of the coverage radius;

determining a first proportion of n to N;

determining a value of a first product of the first difference and the first proportion; and

determining the quantity Q based on the quantity of feature vectors in the plurality of feature vectors and the value of the first product;

combining the P intermediate vectors at the pooling layer to form Q output vectors, wherein Q<P; and

generating, as a feature representation of the current input, a plurality of output vectors obtained at a last transformer layer in the N transformer layers.

2. The computer-implemented method of claim 1 , wherein:

the first transformer layer is a lower layer of the embedding layer; and

obtaining the P input vectors from the upper layer of the first transformer layer comprises:

obtaining the plurality of feature vectors from the embedding layer as the P input vectors.

3. The computer-implemented method of claim 1 , wherein:

the N transformer layers further comprise a second transformer layer, and the second transformer layer is an upper layer of the first transformer layer; and

obtaining the P input vectors from an upper layer of the first transformer layer comprises obtaining output vectors of the second transformer layer from the second transformer layer as the P input vectors.

4. The computer-implemented method of claim 1 , wherein a difference between the quantity Q and the quantity P is a predetermined fixed value.

5. The computer-implemented method of claim 1 , wherein each intermediate vector is a vector with a quantity H of dimensions; and

the combining the P intermediate vectors to form Q output vectors comprises:

fixing a sequence of the P intermediate vectors;

selecting Q values from the quantity P of values of the P intermediate vectors in any of the H dimensions;

forming a value sequence in each dimension of the H dimensions based on the sequence of the P intermediate vectors; and

forming the Q output vectors based on an H×Q matrix formed by sequentially arranging value sequences in all of the H dimensions.

6. The computer-implemented method of claim 1 , wherein:

the embedding layer comprises an input embedding layer and a positional encoding layer; and

performing embedding processing on the current input to the embedding layer to obtain the plurality of feature vectors comprises:

performing word embedding processing on a plurality of words in the current input at the input embedding layer to obtain a plurality of word embedding vectors;

generating, at the positional encoding layer, a position vector of each of the plurality of words based on a position of the word in the current input; and

combining the position vector of each word with a word embedding vector corresponding to the word to obtain the plurality of feature vectors.

7. A non-transitory, computer-readable medium storing one or more instructions executable by a computer system to perform operations comprising:

performing, at an embedding layer of a neural network, embedding processing on a current input to the embedding layer to obtain a plurality of feature vectors, wherein:

the neural network comprises the embedding layer and N transformer layers, and

the N transformer layers comprise a first transformer layer comprising an attention layer and a pooling layer, wherein the first transformer layer is an nth transformer layer in the N transformer layers;

obtaining, at the attention layer of the first transformer layer, a quantity P of input vectors from an upper layer of the first transformer layer; and

determining P intermediate vectors based on the P input vectors, including, for each input vector of the P input vectors:

obtaining, using the input vector as a center, a respective intermediate vector based on correlation values calculated between the input vector and each other input vector in a predetermined attention window range,

determining a quantity Q based on a quantity of feature vectors in the plurality of feature vectors, a value of a coverage radius of an attention window, N, and n, wherein the quantity Q is negatively correlated with n, and wherein determining the quantity Q comprises:

determining a first difference between the quantity of feature vectors in the plurality of feature vectors and the value of the coverage radius;

determining a first proportion of n to N;

determining a value of a first product of the first difference and the first proportion; and

determining the quantity Q based on the quantity of feature vectors in the plurality of feature vectors and the value of the first product;

combining the P intermediate vectors at the pooling layer to form Q output vectors, wherein Q<P; and

generating, as a feature representation of the current input, a plurality of output vectors obtained at a last transformer layer in the N transformer layers.

8. The non-transitory, computer-readable medium of claim 7 , wherein:

the first transformer layer is a lower layer of the embedding layer; and

obtaining the P input vectors from the upper layer of the first transformer layer comprises:

obtaining the plurality of feature vectors from the embedding layer as the P input vectors.

9. A computer-implemented system, comprising:

one or more computers; and

one or more computer memory devices interoperably coupled with the one or more computers and having tangible, non-transitory, machine-readable media storing one or more instructions that, when executed by the one or more computers, perform one or more operations comprising:

performing, at an embedding layer of a neural network, embedding processing on a current input to the embedding layer to obtain a plurality of feature vectors, wherein:

the neural network comprises the embedding layer and N transformer layers, and

the N transformer layers comprise a first transformer layer comprising an attention layer and a pooling layer, wherein the first transformer layer is an nth transformer layer in the N transformer layers;

obtaining, at the attention layer of the first transformer layer, a quantity P of input vectors from an upper layer of the first transformer layer; and

determining P intermediate vectors based on the P input vectors, including, for each input vector of the P input vectors:

obtaining, using the input vector as a center, a respective intermediate vector based on correlation values calculated between the input vector and each other input vector in a predetermined attention window range,

determining a quantity Q based on a quantity of feature vectors in the plurality of feature vectors, a value of a coverage radius of an attention window, N, and n, wherein the quantity Q is negatively correlated with n, and wherein determining the quantity Q comprises:

determining a first difference between the quantity of feature vectors in the plurality of feature vectors and the value of the coverage radius;

determining a first proportion of n to N;

determining a value of a first product of the first difference and the first proportion; and

determining the quantity Q based on the quantity of feature vectors in the plurality of feature vectors and the value of the first product;

combining the P intermediate vectors at the pooling layer to form Q output vectors, wherein Q<P; and

generating, as a feature representation of the current input, a plurality of output vectors obtained at a last transformer layer in the N transformer layers.

10. The computer-implemented system of claim 9 , wherein:

the first transformer layer is a lower layer of the embedding layer; and

obtaining the P input vectors from the upper layer of the first transformer layer comprises:

obtaining the plurality of feature vectors from the embedding layer as the P input vectors.

11. The computer-implemented system of claim 9 , wherein:

the N transformer layers further comprise a second transformer layer, and the second transformer layer is an upper layer of the first transformer layer; and

obtaining the P input vectors from an upper layer of the first transformer layer comprises obtaining output vectors of the second transformer layer from the second transformer layer as the P input vectors.

12. The computer-implemented system of claim 9 , wherein a difference between the quantity Q and the quantity P is a predetermined fixed value.

13. The computer-implemented system of claim 9 , wherein each intermediate vector is a vector with a quantity H of dimensions; and

the combining the P intermediate vectors to form Q output vectors comprises:

fixing a sequence of the P intermediate vectors;

selecting Q values from the quantity P of values of the P intermediate vectors in any of the H dimensions;

forming a value sequence in each dimension of the H dimensions based on the sequence of the P intermediate vectors; and

forming the Q output vectors based on an H×Q matrix formed by sequentially arranging value sequences in all of the H dimensions.

14. The computer-implemented system of claim 9 , wherein:

the embedding layer comprises an input embedding layer and a positional encoding layer; and

performing embedding processing on the current input to the embedding layer to obtain the plurality of feature vectors comprises:

performing word embedding processing on a plurality of words in the current input at the input embedding layer to obtain a plurality of word embedding vectors;

generating, at the positional encoding layer, a position vector of each of the plurality of words based on a position of the word in the current input; and

combining the position vector of each word with a word embedding vector corresponding to the word to obtain the plurality of feature vectors.

15. The non-transitory, computer-readable medium of claim 7 , wherein:

the N transformer layers further comprises a second transformer layer, and the second transformer layer is an upper layer of the first transformer layer; and

obtaining the P input vectors from an upper layer of the first transformer layer comprises obtaining output vectors of the second transformer layer from the second transformer layer as the P input vectors.

16. The non-transitory, computer-readable medium of claim 7 , wherein a difference between the quantity Q and the quantity P is a predetermined fixed value.

17. The non-transitory, computer-readable medium of claim 7 , wherein each intermediate vector is a vector with a quantity H of dimensions; and

the combining the P intermediate vectors to form Q output vectors comprises:

fixing a sequence of the P intermediate vectors;

selecting Q values from the quantity P of values of the P intermediate vectors in any of the H dimensions;

forming a value sequence in each dimension of the H dimensions based on the sequence of the P intermediate vectors; and

forming the Q output vectors based on an H×Q matrix formed by sequentially arranging value sequences in all of the H dimensions.

18. The non-transitory, computer-readable medium of claim 7 , wherein:

the embedding layer comprises an input embedding layer and a positional encoding layer; and

performing embedding processing on the current input to the embedding layer to obtain the plurality of feature vectors comprises:

performing word embedding processing on a plurality of words in the current input at the input embedding layer to obtain a plurality of word embedding vectors;

generating, at the positional encoding layer, a position vector of each of the plurality of words based on a position of the word in the current input; and

combining the position vector of each word with a word embedding vector corresponding to the word to obtain the plurality of feature vectors.