Neural processing unit and method of operation thereof
An operating method for a neural processing unit is provided. The method includes determining, by a controller, that an operation performed in a first convolution layer is a transpose convolution operation, dividing, by the controller, a kernel used for the transpose convolution operation into a plurality of sub-kernels, and performing, by at least one processing element, a convolution operation between an input feature map and each of the plurality of sub-kernels in the first convolution layer.
1. A method of operating a neural processing unit having a systolic array structure, the method comprising:
determining, by a controller, that an operation performed in a first convolution layer is a transpose convolution operation;
dividing, by the controller, a kernel used for the transpose convolution operation into a plurality of sub-kernels; and
performing a convolution operation between an input feature map and each of the plurality of sub-kernels in the first convolution layer, the convolution operation performed by each of a plurality of processing elements,
wherein each of the plurality of processing elements is configured to perform a process of reusing at least one of an output feature map, each of the plurality of sub-kernels, and the input feature map, which are values stored in a local memory of each of the plurality of processing elements, and
wherein the reusing process is performed by a first processing element of the plurality of processing elements transferring the stored values of the at least one of the output feature map, each of the plurality of sub-kernels, and the output feature map to a second processing element of the plurality of processing elements,
wherein the systolic array structure includes a plurality of structures arranged, in parallel, in correspondence to the values stored in the local memory, the stored values being used in successive convolution operations,
wherein a multiply-and-accumulate (MAC) operation mode of the plurality of processing elements corresponds to one of the plurality of structures and is switched based on a calculated MAC operation time, and
wherein the MAC operation mode includes an output stationary mode where the output feature map is reused, a weight stationary mode where each of the plurality of sub-kernels is reused, and an input stationary mode where the input feature map is reused.
2. The method of claim 1 ,
wherein an input of the first convolution layer is the input feature map, and
wherein the input feature map is a feature map obtained by padding input data with at least one zero.
3. The method of claim 2 , further comprising:
identifying a size of the input data and a size of the input feature map;
determining a size of a stride for the plurality of sub-kernels; and
determining a size of output data based on at least one of the size of the input feature map and the size of the stride.
4. The method of claim 1 ,
wherein the kernel includes at least one weight value, and
wherein at least one weight value is included in one of the plurality of sub-kernels.
5. The method of claim 3 ,
wherein the plurality of sub-kernels includes a first sub-kernel and a second sub-kernel,
wherein the first sub-kernel is determined according to a location of the input data included in a first window when the kernel corresponds to the first window of the input feature map, and
wherein the second sub-kernel is determined according to a location of the input data included in a second window when the kernel corresponds to the second window of the input feature map.
6. The method of claim 5 , wherein the performing the convolution operation includes:
performing a convolution operation between the input feature map and the first sub-kernel in the first convolution layer; and
performing a convolution operation between the input feature map and the second sub-kernel in the first convolution layer after performing the convolution operation between the input feature map and the first sub-kernel.
7. The method of claim 6 , wherein the convolution operation between the input feature map and the first sub-kernel is a convolution operation between the input data included in the input feature map and the first sub-kernel.
8. The method of claim 7 ,
wherein an output of the first convolution layer is the output feature map, and
wherein the output feature map has a larger size than the input data.
9. The method of claim 8 , further comprising:
determining, by the controller, the plurality of sub-kernels based on at least one of the size of the input feature map, a size of the output feature map, at least one zero included in the input feature map, the location of the input data, the size of the stride, and a size of the kernel.
10. The method of claim 8 , further comprising:
mapping, by the controller, the output data included in the output feature map to an address of an internal memory; and
storing, by the internal memory, the output data included in the output feature map at the mapped address.
11. A neural processing unit having a systolic array structure, the neural processing unit comprising:
an internal memory configured to store an artificial neural network model;
a plurality of processing elements each of which is configured to access the internal memory and to process a convolution operation of the artificial neural network model; and
a controller operably connected with the internal memory and the plurality of processing elements,
wherein the artificial neural network model is an artificial neural network-based model configured to perform image enhancement using an image as an input,
wherein the controller is configured to
determine that an operation performed in a first convolution layer is a transpose convolution operation,
divide a kernel used for the transpose convolution operation into a plurality of sub-kernels, and
control each of the plurality of processing elements to perform a convolution operation between an input feature map and each of the plurality of sub-kernels in the first convolution layer,
wherein each of the plurality of processing elements is further configured to perform a process of reusing at least one of an output feature map, each of the plurality of sub-kernels, and the input feature map, which are values stored in a local memory of each of the plurality of processing elements, and
wherein the reusing process is performed by a first processing element of the plurality of processing elements transferring the stored values of the at least one of the output feature map, each of the plurality of sub-kernels, and the output feature map to a second processing element of the plurality of processing elements,
wherein the systolic array structure includes a plurality of structures arranged, in parallel, in correspondence to the values stored in the local memory, the stored values being used in successive convolution operations,
wherein a multiply-and-accumulate (MAC) operation mode of the plurality of processing elements corresponds to one of the plurality of structures and is switched based on a calculated MAC operation time, and
wherein the MAC operation mode includes an output stationary mode where the output feature map is reused, a weight stationary mode where each of the plurality of sub-kernels is reused, and an input stationary mode where the input feature map is reused.
12. The neural processing unit of claim 11 ,
wherein an input of the first convolution layer is the input feature map, and
wherein the input feature map is a feature map obtained by padding input data with at least one zero.
13. The neural processing unit of claim 12 , wherein the controller is further configured to
identify a size of the input data and a size of the input feature map;
determine a size of a stride for the plurality of sub-kernels; and
determine a size of output data based on at least one of the size of the input feature map and the size of the stride.
14. The neural processing unit of claim 13 ,
wherein the plurality of sub-kernels include a first sub-kernel and a second sub-kernel,
wherein the first sub-kernel is determined according to a location of the input data included in a first window when the kernel corresponds to the first window of the input feature map, and
wherein the second sub-kernel is determined according to a location of the input data included in a second window when the kernel corresponds to the second window of the input feature map.
15. The neural processing unit of claim 14 , wherein the controller is further configured to
perform a convolution operation between the input feature map and the first sub-kernel in the first convolution layer, and
perform a convolution operation between the input feature map and the second sub-kernel after performing the convolution operation between the input feature map and the first sub-kernel in the first convolution layer.
16. The neural processing unit of claim 15 , wherein the convolution operation between the input feature map and the first sub-kernel is a convolution operation between the input data included in the input feature map and the first sub-kernel.
17. The neural processing unit of claim 16 ,
wherein an output of the first convolution layer is the output feature map, and
wherein the output feature map has a larger size than the input data.
18. The neural processing unit of claim 17 , wherein the controller is further configured to
determine the plurality of sub-kernels based on at least one of the size of the input feature map, a size of the output feature map, at least one zero included in the input feature map, the location of the input data, the size of the stride, and a size of the kernel.
19. The neural processing unit of claim 17 , wherein the controller is further configured to
map the output data included in the output feature map to an address of an internal memory, and
control the internal memory to store the output data included in the output feature map at the mapped address.
20. A neural processing unit having a systolic array structure, the neural processing unit comprising:
a controller operatively connected to an internal memory and a plurality of processing elements,
wherein the internal memory is configured to store one or more instructions executed by the controller,
wherein the controller is configured to execute the one or more instructions to
determine that an operation performed in a first convolution layer is a transpose convolution operation,
divide a kernel used for the transpose convolution operation into a plurality of sub-kernels, and
control a processing element to perform a convolution operation between an input feature map and each of the plurality of sub-kernels in the first convolution layer,
wherein each of the plurality of processing elements is configured to perform a process of reusing at least one of an output feature map, each of the plurality of sub-kernels, and the input feature map, which are values stored in a local memory of each of the plurality of processing elements, and
wherein the reusing process is performed by a first processing element of the plurality of processing elements transferring the stored values of the at least one of the output feature map, each of the plurality of sub-kernels, and the output feature map to a second processing element of the plurality of processing elements,
wherein the systolic array structure includes a plurality of structures arranged, in parallel, in correspondence to the values stored in the local memory, the stored values being used in successive convolution operations,
wherein a multiply-and-accumulate (MAC) operation mode of the plurality of processing elements corresponds to one of the plurality of structures and is switched based on a calculated MAC operation time, and
wherein the MAC operation mode includes an output stationary mode where the output feature map is reused, a weight stationary mode where each of the plurality of sub-kernels is reused, and an input stationary mode where the input feature map is reused.