Method and apparatus with neural network operation processing
A processor-implemented neural network method includes: obtaining a first weight kernel of a weight model and pruning information of the first weight kernel; determining, based on the pruning information, a processing range of an input feature map for each weight element vector of the first weight kernel; performing a convolution operation between the input feature map and the first weight kernel based on the determined processing range; and generating an output feature map of a neural network layer based on an operation result of the convolution operation.
1. A processor-implemented neural network method, the method comprising:
obtaining a first weight kernel of a weight model and pruning information of the first weight kernel, the pruning information comprises information of a number of valid vectors corresponding to non-zero vectors included in the first weight kernel, information of positions of the valid vectors in the first weight kernel, or combination thereof;
determining, based on the pruning information, a processing range of an input feature map for each weight element vector of the first weight kernel;
performing a convolution operation between the input feature map and the first weight kernel based on the determined processing range; and
generating an output feature map of a neural network layer based on an operation result of the convolution operation.
2. The method of claim 1 , wherein
the pruning information comprises information of a number of weight element vectors included in the first weight kernel and information of positions of the weight element vectors in the first weight kernel, and
each of the weight element vectors in the first weight kernel corresponds to a valid vector distinguished from an invalid vector removed from the first weight kernel in a pruning process.
3. The method of claim 1 , wherein
the determining of the processing range for each weight element vector comprises determining a first processing range corresponding to a first weight element vector of the first weight kernel in the input feature map based on the pruning information, and
the performing of the convolution operation comprises generating a first intermediate output plane by performing a convolution operation between the first weight element vector and the first processing range.
4. The method of claim 3 , wherein the convolution operation between the first weight element vector and the first processing range comprises a 1×1 convolution operation.
5. The method of claim 4 , wherein the convolution operation between the first weight element vector and the first processing range comprises 1×1 convolution operations between each input element vector of the first processing range and the first weight element vector.
6. The method of claim 3 , wherein the generating of the output feature map comprises generating a first output plane of the output feature map by accumulating intermediate output planes comprising the first intermediate output plane.
7. The method of claim 1 , wherein the generating of the output feature map comprises generating a first output plane of the output feature map by accumulating intermediate output planes corresponding to the operation result of the convolution operation.
8. The method of claim 7 , wherein the first weight kernel and the first output plane correspond to a first output channel.
9. The method of claim 1 , wherein
the weight model comprises a plurality of weight kernels including the first weight kernel, and
weight element vectors of the weight kernels corresponds to non-zero vectors.
10. The method of claim 1 , wherein sparsity of each of weight kernels of the weight model is increased through a training process using a respective base weight kernel and scale matrix.
11. The method of claim 10 , wherein the training process comprises, for each of the weight kernels:
inducing a sparsity to the scale matrix based on a loss function; and
generating the weight kernel based on a pointwise multiplication between the base weight kernel and the sparsity-induced scale matrix.
12. The method of claim 1 , further comprising:
generating an input image using a camera; and
determining the input feature map, the input feature map corresponding to the input image.
13. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the method of claim 1 .
14. A neural network apparatus, the apparatus comprising:
a processor configured to:
obtain a first weight kernel of a weight model and pruning information of the first weight kernel, the pruning information comprises information of a number of valid vectors corresponding to non-zero vectors included in the first weight kernel, information of positions of the valid vectors in the first weight kernel, or combination thereof;
determine, based on the pruning information, a processing range of an input feature map for each weight element vector of the first weight kernel;
perform a convolution operation between the input feature map and the first weight kernel based on the determined processing range; and
generate an output feature map of a neural network layer based on an operation result of the convolution operation.
15. The apparatus of claim 14 , wherein
the pruning information comprises information of a number of weight element vectors included in the first weight kernel and information of positions of the weight element vectors in the first weight kernel, and
each of the weight element vectors in the first weight kernel corresponds to a valid vector distinguished from an invalid vector removed from the first weight kernel in a pruning process.
16. The apparatus of claim 14 , wherein the processor is configured to:
for the determining of the processing range for each weight element vector, determine a first processing range corresponding to a first weight element vector of the first weight kernel in the input feature map based on the pruning information; and
for the performing of the convolution operation, generate a first intermediate output plane by performing a convolution operation between the first weight element vector and the first processing range.
17. The apparatus of claim 16 , wherein, for the generating of the output feature map, the processor is configured to generate a first output plane of the output feature map by accumulating intermediate output planes comprising the first intermediate output plane.
18. The apparatus of claim 14 , wherein, for the generating of the output feature map, the processor is configured to generate a first output plane of the output feature map by accumulating intermediate output planes corresponding to the operation result of the convolution operation.
19. The apparatus of claim 14 , wherein
the weight model comprises a plurality of weight kernels including the first weight kernel, and
weight element vectors of the weight kernels correspond to non-zero vectors.
20. The apparatus of claim 14 , wherein sparsity of each of weight kernels of the weight model is increased through a training process using a respective base weight kernel and scale matrix.
21. The data processing apparatus of claim 14 , wherein
the apparatus is an electronic apparatus comprising a camera configured to generate an input image, and
the processor is configured to determine the input feature map, the input feature map corresponding to the input image.
22. The apparatus of claim 14 , further comprising a memory storing instructions that, when executed by the processor, configure the processor to perform the obtaining of the first weight kernel, the determining of the processing range, the performing of the convolution, and the generating of the output feature map.
23. An electronic apparatus comprising:
a camera configured to generate an input image; and
a processor configured to:
determine an input feature map corresponding to the input image;
obtain a first weight kernel of a weight model and pruning information of the first weight kernel, the pruning information comprises information of a number of valid vectors corresponding to non-zero vectors included in the first weight kernel, information of positions of the valid vectors in the first weight kernel, or combination thereof;
determine, based on the pruning information, a processing range of the input feature map for each weight element vector of the first weight kernel;
perform a convolution operation between the input feature map and the first weight kernel based on the determined processing range; and
generate an output feature map of a neural network layer based on an operation result of the convolution operation.
24. The electronic apparatus of claim 23 , wherein the processor is configured to:
for the determining of the processing range for each weight element vector, determine a first processing range corresponding to a first weight element vector of the first weight kernel in the input feature map based on the pruning information; and
for the performing of the convolution operation, generate a first intermediate output plane by performing a convolution operation between the first weight element vector and the first processing range.
25. The electronic apparatus of claim 23 , wherein the processor is configured to generate a recognition result for the input image based on the output feature map.
26. A processor-implemented neural network method, the method comprising:
determining, for each valid vector of valid vectors of a pruned weight kernel, a processing range of an input feature map based on a position of the valid vector within the pruned weight kernel;
performing, for each valid vector, a convolution operation between the valid vector and the corresponding processing range of the input feature map; and
generating a channel of an output feature map of a neural network layer by accumulating results of the convolution operation performed for each valid vector.
27. The method of claim 26 , wherein a position of each processing range within the input feature map corresponds to the position of the corresponding valid vector within the pruned weight kernel.
28. The method of claim 26 , wherein the valid vectors correspond to non-zero vectors of the pruned weight kernel.
29. A processor-implemented neural network method, the method comprising:
obtaining valid vectors of pruned weight kernels and pruning information of the valid vectors, the pruning information comprises information of a number of valid vectors corresponding to non-zero vectors included in the first weight kernel, information of positions of the valid vectors in the first weight kernel, or combination thereof;
determining, for each valid vector, a processing range of an input feature map based on the pruning information;
generating, for each valid vector, an intermediate output plane by performing a convolution operation between the valid vector and the corresponding processing range; and
generating an output feature map by generating, for each weight kernel, an output plane of an output feature map by accumulating the intermediate output planes corresponding to the weight kernel.