Method and apparatus for sequence processing

1 . A method for sequence processing, comprising:

receiving an input sequence, wherein the input sequence comprises a plurality of elements in order;

determining M windows based on a position of a first element in the input sequence, to balance a self-attention calculation amount and a self-attention dependency range;

performing, by at least one processor executing a neural network comprising one or more self-attention layers, self-attention calculation on the first element in the input sequence by using a respective element from each of the M windows, to obtain a representation of the first element, wherein each of the M windows comprises one element or a plurality of consecutive elements in the input sequence, there is an interval of at least one element between different windows of the M windows, at least one of the M windows does not include the first element, and M is an integer greater than 1; and

obtaining, based on the representation of the first element, an output sequence corresponding to the input sequence.

2 . The method according to claim 1 , wherein

the M windows comprise a first window, the first window comprises an element that is in the input sequence and whose dependency length from the first element is greater than or equal to a and is less than b, a is an integer greater than 1, b is an integer greater than a, and the dependency length indicates a distance between the first element and an element in the M windows.

3 . The method according to claim 2 , wherein the method is applied to a plurality of self-attention layers comprising at least a current self-attention layer and a preceding self-attention layer that precedes the current self-attention layer, and the input sequence is a sequence that is output by the preceding self-attention layer and is input to the current self-attention layer, wherein

values of b and a are set so that there is no repeated calculation between self-attention calculation performed for the current self-attention layer on the first element and self-attention calculation performed for the preceding self-attention layer on the first element.

4 . The method according to claim 3 , wherein the preceding self-attention layer performs self-attention calculation on the first element based on an element comprised in a second window, wherein the second window comprises an element that is in the input sequence and whose dependency length from the first element is greater than or equal to a1 and is less than b1, wherein b1 is a positive integer, and a1 is a non-negative integer less than b1; and

a value of a is greater than a value of b1.

5 . The method according to claim 1 , wherein a value of M is preset.

6 . The method according to claim 5 , wherein the M windows comprise:

a second window, wherein the second window comprises an element that is located before the first element in the input sequence; and/or

a third window, wherein the third window comprises an element that is located after the first element in the input sequence.

7 . The method according to claim 5 , wherein the M windows comprise a fourth window, and the fourth window comprises the first element and an adjacent element of the first element.

8 . The method according to claim 1 , wherein the input sequence is a speech sequence or a text sequence.

9 . A data processing apparatus, comprising:

a memory, configured to store executable instructions; and

a processor, configured to invoke and run the executable instructions stored in the memory, to perform operations comprising:

receiving an input sequence, wherein the input sequence comprises a plurality of elements in order;

determining M windows based on a position of a first element in the input sequence, to balance a self-attention calculation amount and a self-attention dependency range;

performing, by executing a neural network comprising one or more self-attention layers, self-attention calculation on the first element in the input sequence by using a respective element from each of the M windows, to obtain a representation of the first element, wherein each of the M windows comprises one element or a plurality of consecutive elements in the input sequence, there is an interval of at least one element between different windows of the M windows, at least one of the M windows does not include the first element, and M is an integer greater than 1; and

obtaining, based on the representation of the first element, an output sequence corresponding to the input sequence.

10 . The apparatus according to claim 9 , wherein

the M windows based on a position of the first element in the input sequence, wherein the M windows comprise a first window, the first window comprises an element that is in the input sequence and whose dependency length from the first element is greater than or equal to a and is less than b, a is an integer greater than 1, b is an integer greater than a, and the dependency length indicates a distance between the first element and an element in the M windows.

11 . The apparatus according to claim 10 , wherein the apparatus is applied to a plurality of self-attention layers comprising at least a current self-attention layer and a preceding self-attention layer that precedes the current self-attention layer, and the input sequence is a sequence that is output by the preceding self-attention layer and is input to the current self-attention layer, wherein

values of b and a are set so that there is no repeated calculation between self-attention calculation performed by the current self-attention layer on the first element and self-attention calculation performed by the preceding self-attention layer on the first element.

12 . The apparatus according to claim 11 , wherein the preceding self-attention layer performs self-attention calculation on the first element based on an element comprised in a second window, wherein the second window comprises an element that is in the input sequence and whose dependency length from the first element is greater than or equal to a1 and is less than b1, wherein b1 is a positive integer, and a1 is a non-negative integer less than b1; and

a value of a is greater than a value of b1.

13 . The apparatus according to claim 9 , wherein a value of M is preset.

14 . The apparatus according to claim 11 , wherein the M windows comprise:

a second window, wherein the second window comprises an element that is located before the first element in the input sequence; and/or

a third window, wherein the third window comprises an element that is located after the first element in the input sequence.

15 . A non-transitory computer-readable storage medium storing program instructions that when executed by a processor, causes the processor to perform operations comprising:

receiving an input sequence, wherein the input sequence comprises a plurality of elements in order;

determining M windows based on a position of a first element in the input sequence, to balance a self-attention calculation amount and a self-attention dependency range;

performing, by the processor executing a neural network comprising one or more self-attention layers, self-attention calculation on the first element in the input sequence by using a respective element from each of the M windows, to obtain a representation of the first element, wherein each of the M windows comprises one element or a plurality of consecutive elements in the input sequence, there is an interval of at least one element between different windows of the M window, at least one of the M windows does not include the first element, and M is an integer greater than 1; and

obtaining, based on the representation of the first element, an output sequence corresponding to the input sequence.

16 . The computer-readable storage medium according to claim 15 , wherein

the M windows comprise a first window, the first window comprises an element that is in the input sequence and whose dependency length from the first element is greater than or equal to a and is less than b, a is an integer greater than 1, b is an integer greater than a, and the dependency length indicates a distance between the first element and an element in the M windows.

17 . The computer-readable storage medium according to claim 16 , wherein the computer-readable storage medium is applied to a plurality of self-attention layers comprising at least a current self-attention layer and a preceding self-attention layer that precedes the current self-attention layer, and the input sequence is a sequence that is output by the preceding self-attention layer and is input to the current self-attention layer, wherein

values of b and a are set so that there is no repeated calculation between self-attention calculation performed for the current self-attention layer on the first element and self-attention calculation performed for the preceding self-attention layer on the first element.

18 . The computer-readable storage medium according to claim 17 , wherein the preceding self-attention layer performs self-attention calculation on the first element based on an element comprised in a second window, wherein the second window comprises an element that is in the sequence and whose dependency length from the first element is greater than or equal to a1 and is less than b1, b1 is a positive integer, and a1 is a non-negative integer less than b1; and

a value of a is greater than a value of b1.

19 . The computer-readable storage medium according to claim 15 , wherein a value of M is preset.

20 . The computer-readable storage medium according to claim 19 , wherein the M windows comprise:

a second window, wherein the second window comprises an element that is located before the first element in the input sequence; and/or

a third window, wherein the third window comprises an element that is located after the first element in the input sequence.