Z-first reference neural processing unit for mapping winograd convolution and a method thereof
A z-first reference neural processing unit (NPU) for mapping Winograd Convolution is disclosed where the NPU includes memory banks configured to store input feature maps (IFMs) in a z-first data storage layout, each of the memory banks being configured to store the IFMs in one of a direct convolution (DConv) mode or a Winograd convolution (WgConv) mode, a reconfigurable IFM distributor configured to receive the IFMs from the memory banks, a parallel reconfigurable Winograd forward transform module configured to receive the IFMs from the reconfigurable IFM distributor and to transform the IFMs in a Winograd domain to transformed IFMs in the WgConv mode, multiply and accumulate (MAC) units configured to perform dot product operations on one of IFMs in the DConv mode and the transformed IFMs in the WgConv mode to obtain intermediate output feature maps (OFMs), and a reconfigurable OFM adder and Winograd inverse transform module configured to generate one of an OFM from the intermediate OFMs in the DConv mode and OFMs from the intermediate OFMs in the WgConv.
1. A z-first reference neural processing circuitry (NPU) for mapping Winograd Convolution supporting both direct convolution (DConv) mode and Winograd convolution (WgConv) mode, the NPU comprising:
memory banks configured to store input feature maps (IFMs) in a z-first data storage layout, each of the memory banks being configured to store the IFMs in one of the DConv mode or the WgConv mode;
a reconfigurable IFM distributor circuitry configured to receive the IFMs from the memory banks;
a parallel reconfigurable Winograd forward transform circuitry configured to receive the IFMs from the reconfigurable IFM distributor and to transform the IFMs in a Winograd domain to transformed IFMs in the WgConv mode;
multiply and accumulate (MAC) circuitries configured to perform dot product operations on one of IFMs in the DConv mode and the transformed IFMs in the WgConv mode to obtain intermediate output feature maps (OFMs); and
a reconfigurable OFM adder and Winograd inverse transform circuitry configured to generate one of an OFM from the intermediate OFMs in the DConv mode and OFMs from the intermediate OFMs in the WgConv mode,
wherein each of the memory banks are further configured to store a batch of IFMs from the IFMs in the DConv mode and store channels of each of coordinates of each of the IFMs in the WgConv mode as IFM blocks.
2. The NPU of claim 1 , wherein a size of each of the IFM block is 4×4.
3. The NPU of claim 1 , wherein the parallel reconfigurable Winograd forward transform circuitry is further configured to:
select a transform matrix and a transposed transform matrix based on a size of kernels and a position of IFM window; and
transform the IFM blocks based on the size of the kernels, the transform matrix, and the transposed transform matrix to obtain the transformed IFMs.
4. The NPU of claim 3 , wherein:
the size of the kernels is 3×3; and
a size of the transformed IFMs is 4×4.
5. The NPU of claim 1 , wherein the parallel reconfigurable Winograd forward transform circuitry is further configured to:
select a transform matrix and a transposed transform matrix based on a size of kernels; and
transform the IFM blocks based on the size of the kernels and one of the transform matrix and the transposed transform matrix to obtain the transformed IFMs.
6. The NPU of claim 5 , wherein:
the size of the kernels is one of 3×1 and 1×3; and
a size of the transformed IFMs is one of 4×1 and 1×4.
7. The NPU of claim 1 , wherein the reconfigurable OFM adder and Winograd inverse transform circuitry is built using an OFM adder tree.
8. The NPU of claim 7 , wherein the reconfigurable OFM adder and Winograd inverse transform circuitry is further configured to:
add the intermediate OFMs to obtain the OFM in the DConv mode.
9. The NPU of claim 7 , wherein the reconfigurable OFM adder and Winograd inverse transform circuitry is further configured to:
select an inverse transformation matrix and a transposed inverse transformation matrix based on a size of kernels; and
transform the intermediate OFMs based on the size of the kernels, the inverse transformation matrix, and the transposed inverse transformation matrix to obtain the OFMs in the WgConv mode.
10. The NPU of claim 9 , wherein the size of the kernels is 3×3.
11. The NPU of claim 7 , wherein the reconfigurable OFM adder and Winograd inverse transform circuitry is further configured to:
select an inverse transformation matrix and a transposed inverse transformation matrix based on a size of kernels; and
transform the intermediate OFMs based on the size of the kernels and one of the inverse transformation matrix and the transposed inverse transformation matrix to obtain the OFMs in the WgConv mode.
12. The NPU of claim 11 , wherein the size of the kernels is one of 3×1 and 1×3.
13. The NPU of claim 1 , further comprising zero-skip circuitries to skip zero-value elements from one of the IFMs received from IFM buffers in the DConv mode and the transformed IFMs received from the parallel reconfigurable Winograd forward transform circuitry in the WgConv mode.
14. A method for mapping Winograd Convolution on a z-first reference neural processing circuitry (NPU) supporting both direct convolution (DConv) mode and Winograd convolution (WgConv) mode comprising:
receiving, by a reconfigurable IFM distributor circuitry, input feature maps (IFMs) from memory banks, wherein the IFMs are stored in a z-first data storage layout in the memory banks, each of the memory banks being configured to store the IFMs in one of the DConv mode or the WgConv mode;
receiving, by a parallel reconfigurable Winograd forward transform circuitry, the IFMs from the reconfigurable IFM distributor circuitry;
transforming, by the parallel reconfigurable Winograd forward transform circuitry, the IFMs in a Winograd domain to obtain transformed IFMs in the WgConv mode;
performing, by multiply and accumulate (MAC) circuitries, dot product operations on one of the IFMs in the DConv mode and the transformed IFMs in the WgConv mode to obtain intermediate output feature maps (OFMs); and
generating, by a reconfigurable OFM adder and Winograd inverse transform circuitry, one of an OFM from the intermediate OFMs in the DConv mode and OFMs from the intermediate OFMs in the WgConv mode,
wherein each of the memory banks are further configured to store a batch of IFMs from the IFMs in the DConv mode and store channels of each of coordinates of each of the IFMs in the WgConv mode as IFM blocks.
15. The method of claim 14 , wherein the transforming of the IFMs comprises:
selecting, by the parallel reconfigurable Winograd forward transform circuitry, a transform matrix and a transposed transform matrix based on a size of kernels and a position of IFM window; and
transforming, by the parallel reconfigurable Winograd forward transform circuitry, the IFMs based on the size of the kernels, the transform matrix, and the transposed transform matrix to obtain the transformed IFMs.
16. The method of claim 15 , wherein the size of the kernels is 3×3.
17. The method of claim 14 , further comprising:
selecting, by the parallel reconfigurable Winograd forward transform circuitry, a transform matrix and a transposed transform matrix based on a size of kernels; and
transforming, by the parallel reconfigurable Winograd forward transform circuitry, the IFMs based on the size of the kernels and one of the transform matrix and the transposed transform matrix to obtain intermediate transformed IFMs.
18. The method of claim 17 , wherein:
the size of the kernels is one of 3×1 and 1×3;
a size of the intermediate transformed IFMs is one of 4×1 and 1×4; and
a size of the transformed IFMs is 4×4.