Neural processing unit for performing RMS norm operation and control method thereof

1 . A neural processing unit for performing inference operations of a large-scale language model based on an artificial neural network, the neural processing unit comprising:

a processing element core configured to perform an attention mechanism-based operation based on input data in vector format to output an operation result;

a special function unit comprising a plurality of arithmetic circuits including

at least one vector-dedicated arithmetic circuit that exclusively performs vector operations, the at least one vector-dedicated arithmetic circuit comprising a first vector-dedicated arithmetic circuits, and

at least one mixed arithmetic circuit capable of performing both vector and scalar operations, the at least one mixed arithmetic circuit comprising first and second mixed arithmetic circuits and being configured to perform a special function operation on the operation result; and

a controller configured to, upon receiving an RMS normalization operation execution command, activate at least one of the plurality of arithmetic circuits to control the special function unit to perform an operation of converting at least one of the operation result or the input data into a normalized vector whose magnitude is adjusted based on a root mean square (RMS),

wherein the operation result is an attention score for the input data, and

wherein the controller is further configured to control the special function unit to perform an RMS normalization operation based on one operation code so that

the first mixed arithmetic circuit squares each element of the input data,

the first vector-dedicated arithmetic circuit sums each of the squared elements to output a summed value, and

the second mixed arithmetic circuit calculates an average value for the summed value and outputs a value obtained by adding a correction constant to the average value.

2 . The neural processing unit of claim 1 , wherein the at least one mixed arithmetic circuit is further configured to calculate a normalization coefficient based on the sum or average of the squared values, or to normalize the input data based on the normalization coefficient.

3 . The neural processing unit of claim 1 , wherein the plurality of arithmetic circuits are configured to be connected in series.

4 . The neural processing unit of claim 1 ,

wherein the controller is further configured to, by at least two operation codes, selectively activate some of the plurality of arithmetic circuits,

wherein an intermediate result value is obtained by performing an operation of a first operation step by selectively activating at least one of the arithmetic circuits by a first operation code, and

wherein an RMS normalization operation is performed by performing an operation of a second operation step based on the intermediate result value by selectively activating at least one of the arithmetic circuits by a second operation code, thereby controlling the special function unit.

5 . The neural processing unit of claim 4 ,

wherein the at least one mixed arithmetic circuit of the special function unit further comprises a third mixed arithmetic circuit, and

wherein the controller is further configured to, in the first operation step, based on the first operation code, control the special function unit so that the third mixed arithmetic circuit performs a square root operation on the value obtained by adding the correction constant to the average value and takes the reciprocal to store intermediate operation values, which are normalization coefficients, in a memory of the neural processing unit.

6 . The neural processing unit of claim 5 ,

wherein the at least one mixed arithmetic circuit of the special function unit further comprises a fourth mixed arithmetic circuit, and

wherein the controller is further configured to, in the second operation step, based on the second operation code, control the special function unit so that the fourth mixed arithmetic circuit calculates a normalized vector by performing scaling for each element by multiplying each element of the input data and the intermediate operation values, based on the intermediate operation values.

7 . The neural processing unit of claim 1 ,

wherein the at least one mixed arithmetic circuit of the special function unit further comprises third and fourth mixed arithmetic circuits, and

wherein the controller is further configured to, based on the one operation code, control the special function unit so that

the third mixed arithmetic circuit performs a square root operation on the value obtained by adding the correction constant to the average value and takes the reciprocal to calculate a normalization coefficient, and

the fourth mixed arithmetic circuit calculates a normalized vector by performing scaling for each element by multiplying each element of the input data and the normalization coefficient.

8 . A special function unit included in a neural processing unit that performs a special function operation and performs an inference operation of a large-scale language model based on an artificial neural network, the special function unit comprising:

a data input unit for reading input data necessary for an operation from an external memory;

a plurality of arithmetic circuits including

at least one vector-dedicated arithmetic circuit that exclusively performs vector operations and is configured to sum the squared values of the input data in vector format or calculate an average and

at least one mixed arithmetic circuit capable of performing both vector and scalar operations,

wherein the plurality of arithmetic circuits are configured to generate an operation result,

wherein the special function unit is configured to, upon receiving an RMS normalization operation execution command, perform an operation of converting at least one of the operation result or the input data into a normalized vector whose magnitude is adjusted based on a root mean square (RMS), and

wherein the at least one mixed arithmetic circuit comprises a plurality of lanes for performing an operation and is configured to

identify an operation type of an operation to be currently processed, the identified operation type being a vector operation or a scalar operation,

activate only one of the plurality of lanes if the identified operation type is a scalar operation, and

activate two or more of the plurality of lanes if the identified operation type is a vector operation.

9 . The special function unit of claim 8 , wherein the at least one mixed arithmetic circuit is further configured to calculate a normalization coefficient based on the sum or average of the squared values, or to normalize the input data based on the normalization coefficient.

10 . The special function unit of claim 8 , wherein the plurality of arithmetic circuits are further configured to be connected in series.

11 . The special function unit of claim 8 ,

wherein the special function unit is further configured to, by at least two operation codes, selectively activate some of the plurality of arithmetic circuits,

wherein an intermediate result value is obtained by performing an operation of a first operation step by selectively activating at least one of the arithmetic circuits by a first operation code, and

wherein an RMS normalization operation is performed by performing an operation of a second operation step based on the intermediate result value by selectively activating at least one of the arithmetic circuits by a second operation code.

12 . The special function unit of claim 11 , wherein the special function unit is further configured to, in the first operation step, based on the first operation code, have

a first mixed arithmetic circuit square each element of the input data,

a first vector-dedicated arithmetic circuit sum each of the squared elements,

a second mixed arithmetic circuit calculate an average value for the summed value and then sum a correction constant, and

a third mixed arithmetic circuit perform a square root operation on the value obtained by summing the correction constant to the average value and take the reciprocal to store intermediate operation values, which are normalization coefficients, in a memory of the neural processing unit.

13 . The special function unit of claim 12 , wherein the special function unit is further configured to, in the second operation step, based on the second operation code, have a fourth mixed arithmetic circuit calculate a normalized vector by performing scaling for each element by multiplying each element of the input data and the intermediate operation values, based on the intermediate operation values.

14 . The special function unit of claim 8 , wherein the special function unit is further configured to perform an RMS normalization operation based on one operation code.

15 . The special function unit of claim 14 , wherein the special function unit is further configured to, based on the one operation code, have

a first mixed arithmetic circuit square each element of the input data,

a first vector-dedicated arithmetic circuit sum each of the squared elements,

a second mixed arithmetic circuit calculate an average value for the summed value and then sum a correction constant,

a third mixed arithmetic circuit perform a square root operation on the value obtained by summing the correction constant to the average value and take the reciprocal to calculate a normalization coefficient, and

a fourth mixed arithmetic circuit calculate a normalized vector by performing scaling for each element by multiplying each element of the input data and the normalization coefficient.

16 . The special function unit of claim 8 ,

wherein the special function unit comprises a selector circuit configured to output a lane activation signal corresponding to the identified operation type, at least one of the plurality of lanes being activated by the lane activation signal, and

wherein the at least one mixed arithmetic circuit performs the vector or scalar operation using the at least one of the plurality of lanes activated according to the lane activation signal.

17 . An arithmetic device comprising:

a plurality of arithmetic circuits including

at least one vector-dedicated arithmetic circuit that exclusively performs vector operations, the at least one vector-dedicated arithmetic circuit comprising first and second mixed arithmetic circuits, and

at least one mixed arithmetic circuit capable of performing both vector and scalar operations, the at least one mixed arithmetic circuit comprising a first vector-dedicated arithmetic circuit,

wherein the arithmetic device is configured to:

selectively activate the plurality of arithmetic circuits to perform a plurality of sets of operations by a combination of the activated arithmetic circuits, the plurality of sets of operations comprising an RMS normalization operation among attention operations, which are inference operations based on an artificial neural network, and

perform an RMS normalization operation based on one operation code so that

the first mixed arithmetic circuit squares each element of the input data,

the first vector-dedicated arithmetic circuit sums each of the squared elements to output a summed value, and

the second mixed arithmetic circuit calculates an average value for the summed value and outputs a value obtained by adding a correction constant to the average value.