Efficient data layouts for convolutional neural networks
View Patent ↗Systems and methods for efficient implementation of a convolutional layer of a convolutional neural network are disclosed. In one aspect, weight values of kernels in a kernel stack of a convolutional layer can be reordered into a tile layout with tiles of runnels. Pixel values of input activation maps of the convolutional layer can be reordered into an interleaved layout comprising a plurality of clusters of input activation map pixels. The output activation maps can be determined using the clusters of the input activation map pixels and kernels tile by tile.
1. A method implemented by a system of one more processors, the method comprising:
receiving a convolutional layer of a convolutional neural network, wherein the convolutional layer comprises kernels in a kernel stack;
receiving input activation maps of the convolutional layer; and
determining output activation maps of the convolutional layer based at least partly on the kernels and the input activation maps, wherein the output activation maps are in an interleaved output activation map layout.
2. The method of claim 1 , wherein the input activation maps are in an interleaved input activation map layout.
3. The method of claim 1 , wherein the input activation maps are in a basic input activation map layout.
4. The method of claim 3 , wherein the method further comprises:
reordering pixel values of the input activation maps from the basic input activation map layout into the interleaved input activation map layout comprising a plurality of clusters of input activation map pixels.
5. The method of claim 1 , wherein the kernels of the kernel stack are in a basic kernel layout.
6. The method of claim 5 , wherein weight values of the kernels of the kernel stack are reordered from the basic kernel layout into a tile kernel layout comprising a plurality of kernel tiles.
7. The method of claim 6 , wherein a kernel tile comprises a plurality of kernel runnels, and wherein a kernel runnel comprises a number of the weight values of the kernels of the kernel stack.
8. The method of claim 7 , wherein the output activation maps are determined based on the plurality of kernel tiles and the input activation maps.
9. The method of claim 8 , wherein the input activation maps are in an interleaved input activation map layout comprising a plurality of clusters of input activation map pixels, and wherein determining the output activation maps comprises:
performing fused-multiply-add operations tile by tile on the plurality of kernel tiles and the plurality of clusters of input activation map pixels.
10. The method of claim 7 , wherein a first kernel runnel of the kernel tile corresponds a first kernel stack width boundary, and wherein a last kernel runnel of the kernel tile corresponds to a second kernel stack width boundary subsequent of the first kernel stack width boundary.
11. The method of claim 7 , wherein a size of a first kernel runnel in bits and a size of a first input activation map runnel in bits are the same.
12. The method of claim 11 , wherein the size of the first kernel runnel in bits and a size of a first output activation map runnel in bits are the same.
13. The method of claim 11 , wherein the size of the first kernel runnel in bits and a size of a first register of at least one of the processors in bits are the same.
14. The method of claim 1 , wherein the interleaved output activation map layout comprises a plurality of clusters of output activation map pixels.
15. The method of claim 1 , further comprising reordering pixel values of the output activation maps from the interleaved output activation map layout into a basic output activation map layout.
16. The method of claim 1 , wherein at least one of the one or more processors comprises a single instruction, multiple data processor.
17. A system comprising non-transitory memory storing instructions that when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving a convolutional layer of a convolutional neural network, wherein the convolutional layer comprises kernels in a kernel stack;
receiving input activation maps of the convolutional layer; and
determining output activation maps of the convolutional layer based at least partly on the kernels and the input activation maps, wherein the output activation maps are in an interleaved output activation map layout.
18. The system of claim 17 , wherein weight values of the kernels of the kernel stack are reordered into a tile kernel layout comprising a plurality of kernel tiles, wherein a kernel tile comprises a plurality of kernel runnels, and wherein a kernel runnel comprises a number of the weight values of the kernels of the kernel stack.
19. The system of claim 18 , wherein a size of a first kernel runnel in bits and a size of a first input activation map runnel in bits are the same.
20. The system of claim 18 , wherein a size of a first kernel runnel in bits and a size of a first register of at least one of the processors in bits are the same.