Accelerated convolution of neural networks
Accelerated convolution of neural networks can be performed by executing N computing engines (CEs) of a neural network processor in parallel. An input dataset can be divided spatially into N chunks such that a respective last portion of each chunk overlaps with a respective first portion of a subsequent chunk. Portions of each chunk can be processed by a respective CE to generate a respective portion of an output dataset. The overlapping intermediate states computed by each CE from processing the overlapping portion can be stored locally for sharing with a subsequent CE using an on-chip bus.
1. A computer-implemented method to accelerate a dilated convolutional neural network using N computing engines (CEs), the method comprising:
receiving a neural network model for the dilated convolutional neural network to generate an output dataset for an inference on an input dataset using convolution operations, wherein the N CEs comprise at least two CEs and are part of a neural network processor, wherein the input dataset is to be transferred to the neural network processor from an external memory and comprises N chunks, each chunk to be processed by a separate CE to generate a respective portion of the output dataset, and wherein a last portion of each chunk overlaps with a first portion of a subsequent chunk; and
generating, by a compiler of a host system external to the neural network processor and based on the neural network model, instructions to be executed by each CE in parallel for:
processing the last portion of a respective chunk to compute a respective set of intermediate states across a stack of dilated convolutional layers having different dilation rates, wherein:
a subset of the respective set of intermediate states forms shared states to be used by a respective subsequent CE and which correspond to outputs of different convolutional layers in the stack,
each convolutional layer in the stack involves a convolution operation in which a corresponding convolution kernel is applied to input data of the convolutional layer,
the stack comprises a first convolutional layer for which the input data is the input data set and at least one additional convolutional layer,
the input data for each convolutional layer after the first convolutional layer corresponds to convolution operation results from an immediately preceding convolutional layer, and
the processing of the last portion is performed in parallel with corresponding operations performed by the respective subsequent CE and a respective preceding CE;
storing the respective set of intermediate states in a respective local memory during the processing of the last portion;
processing a second portion of the respective chunk across the stack of dilated convolutional layers to generate a first sub-portion of the respective portion of the output dataset as an output of a last convolutional layer in the stack, wherein the first sub-portion has no dependency on shared states computed by the respective preceding CE, while transmitting the shared states computed through processing the last portion of the respective chunk to the respective subsequent CE via an internal bus connected to local memories of the N CEs, and wherein the processing of the second portion and the transmitting are performed in parallel with corresponding operations performed by the respective subsequent CE and the respective preceding CE;
receiving the shared states computed by the respective preceding CE via the internal bus, thereby avoiding re-computation of the shared states computed by the respective preceding CE as part of processing the respective chunk, wherein the receiving occurs during the processing of the second portion; and
generating a second sub-portion of the respective portion of the output dataset as another output of the last convolutional layer in the stack, using the shared states computed by the respective preceding CE, wherein the generating of the second sub-portion is performed in parallel with corresponding operations performed by the respective subsequent CE and the respective preceding CE.
2. The computer-implemented method of claim 1 , the N chunks comprising a first chunk to be processed by a first CE, a second chunk to be processed by a second CE, and a third chunk to be processed by a third CE, wherein a last portion of the first chunk overlaps with a first portion of the second chunk, and a last portion of the second chunk overlaps with a first portion of the third chunk.
3. The computer-implemented method of claim 2 , wherein the first CE processes the last portion of the first chunk to compute a first set of states comprising first shared states using the convolution operations to be used by the second CE, and the second CE processes the last portion of the second chunk to compute a second set of states comprising second shared states using the convolution operations to be used by the third CE.
4. The computer-implemented method of claim 2 , wherein shared states received by the first CE from a preceding CE are pre-computed states.
5. A computer-implemented method comprising:
receiving a neural network model for a dilated convolutional neural network configured to generate output data from an input dataset using convolution operations, wherein the convolution operations are to be performed by at least a first computing engine (CE) and a second CE in a neural network processor, the output data comprises a first output dataset and a second output dataset, the input dataset is to be transferred to the neural network processor from an external memory and comprises a first chunk and a second chunk, and wherein a last portion of the first chunk overlaps with a first portion of the second chunk; and
generating, by a compiler of a host system external to the neural network processor and based on the neural network model, instructions for:
processing, by the first CE, the last portion of the first chunk to compute a set of intermediate states across a stack of dilated convolutional layers having different dilation rates, wherein:
a subset of the set of intermediate states forms shared states to be used by the second CE and which correspond to outputs of different convolutional layers in the stack,
each convolutional layer in the stack involves a convolution operation in which a corresponding convolution kernel is applied to input data of the convolutional layer,
the stack comprises a first convolutional layer for which the input data is the input data set and at least one additional convolutional layer,
the input data for each convolutional layer after the first convolutional layer corresponds to convolution operation results from an immediately preceding convolutional layer, and
the processing of the last portion of the first chunk is performed in parallel with corresponding operations performed by the second CE with respect to a last portion of the second chunk;
storing, by the first CE, the set of intermediate states in a local memory of the first CE;
processing, by the second CE, a second portion of the second chunk across the stack of dilated convolutional layers to generate a first portion of the second output dataset as an output of a last convolutional layer in the stack, wherein the first portion of the second output dataset has no dependency on the shared states computed by the first CE, and wherein the processing of the second portion of the second chunk is performed in parallel with corresponding operations performed by first CE with respect to a second portion of the first chunk;
transmitting, by the first CE while the second CE is processing the second portion of the second chunk and the first CE is processing the second portion of the first chunk, the shared states to the second CE via an internal bus connecting the local memory of the first CE to a local memory of the second CE; and
generating, by the second CE as part of processing the first portion of the second chunk, a second portion of the second output dataset as another output of the last convolutional layer in the stack, using the shared states transmitted by the first CE.
6. The computer-implemented method of claim 5 , further comprising generating an additional instruction for:
generating, by the first CE while the second CE is generating the first portion of the second output dataset, a first portion of the first output dataset using a second portion of the first chunk and the set of intermediate states.
7. The computer-implemented method of claim 5 , further comprising generating an additional instruction for:
processing, by the second CE while the first CE is processing the last portion of the first chunk, a last portion of the second chunk to compute another set of intermediate states using the convolution operations that are used to generate the first portion of the second output dataset.
8. The computer-implemented method of claim 7 , further comprising generating an additional instruction for:
storing, by the second CE, the other set of intermediate states in an on-chip memory in the second CE.
9. The computer-implemented method of claim 6 , further comprising generating an additional instruction for:
processing, by the first CE while the second CE is generating the second portion of the second output dataset, a first portion of the first chunk to generate a second portion of the first output dataset.
10. The computer-implemented method of claim 6 , further comprising generating an additional instruction for:
generating, by the first CE while the second CE is generating the second portion of the second output dataset, a second portion of the first output dataset using pre-computed states.
11. The computer-implemented method of claim 10 , wherein the pre-computed states have static values or values that were computed prior to applying the input dataset to the dilated convolutional neural network.
12. The computer-implemented method of claim 5 , wherein the dilated convolutional neural network comprises a stack of dilated convolutional layers comprising, from bottom to top, an input layer followed by one or more hidden layers and an output layer.
13. The computer-implemented method of claim 12 , wherein the set of intermediate states are computed using convolution operations performed by the input layer and the one or more hidden layers.
14. The computer-implemented method of claim 12 , wherein the convolution operations are performed by each layer at a respective dilation rate that increases from the bottom to the top for each of the one or more hidden layers and the output layer.
15. The computer-implemented method of claim 5 , wherein the local memory is a first on-chip memory, and wherein the first CE comprises the first on-chip memory, and a first systolic array including first processing elements arranged in rows and columns to perform the convolution operations, and the second CE comprises a second on-chip memory, and a second systolic array including second processing elements arranged in rows and columns to perform the convolution operations.
16. The computer-implemented method of claim 5 , wherein the first CE and the second CE are a first processor core and a second processor core of the neural network processor.
17. A non-transitory computer readable medium having stored therein instructions that, when executed by one or more processors, cause the one or more processors of a host system to execute a compiler, the compiler performing operations including:
receiving a neural network model for a dilated convolutional neural network configured to generate output data from an input dataset using convolution operations, wherein the convolution operations are to be performed by at least a first computing engine (CE) and a second CE in a neural network processor external to the host system, the output data comprises a first output dataset and a second output dataset, the input dataset is to be transferred to the neural network processor from an external memory and comprises a first chunk and a second chunk, and wherein a last portion of the first chunk overlaps with a first portion of the second chunk; and
generating instructions for:
processing, by the first CE, the last portion of the first chunk to compute a set of intermediate states across a stack of dilated convolutional layers having different dilation rates, wherein:
a subset of the set of intermediate states forms shared states to be used by the second CE and which correspond to outputs of different convolutional layers in the stack,
each convolutional layer in the stack involves a convolution operation in which a corresponding convolution kernel is applied to input data of the convolutional layer,
the stack comprises a first convolutional layer for which the input data is the input data set and at least one additional convolutional layer,
the input data for each convolutional layer after the first convolutional layer corresponds to convolution operation results from an immediately preceding convolutional layer, and
the processing of the last portion of the first chunk is performed in parallel with corresponding operations performed by the second CE with respect to a last portion of the second chunk;
storing, by the first CE, the set of intermediate states in a local memory of the first CE;
processing, by the second CE, a second portion of the second chunk across the stack of dilated convolutional layers to generate a first portion of the second output dataset as an output of a last convolutional layer in the stack, wherein the first portion of the second output dataset has no dependency on the shared states computed by the first CE, and wherein the processing of the second portion of the second chunk is performed in parallel with corresponding operations performed by first CE with respect to a second portion of the first chunk;
transmitting, by the first CE while the second CE is processing the second portion of the second chunk and the first CE is processing the second portion of the first chunk, the shared states to the second CE via an internal bus connecting the local memory of the first CE to a local memory of the second CE; and
generating, by the second CE as part of processing the first portion of the second chunk, a second portion of the second output dataset as another output of the last convolutional layer in the stack, using the shared states transmitted by the first CE.
18. The non-transitory computer readable medium of claim 17 , wherein the compiler is executed on a first system, and wherein the first CE and the second CE are part of an acceleration engine in a second system.
19. The non-transitory computer readable medium of claim 17 , wherein the first CE and the second CE are integrated circuits, and the local memory is an on-chip memory in the first CE.
20. The non-transitory computer readable medium of claim 17 , wherein the internal bus is configured to permit the first CE to communicate with the second CE using a shared bus protocol.
21. A method comprising:
receiving, by a first computing engine (CE) and a second CE in a neural network processor, an input dataset to generate output data, wherein the input dataset is transferred to the neural network processor from an external memory and is spatially divided into chunks comprising a first chunk and a second chunk, and wherein a last portion of the first chunk overlaps with a first portion of the second chunk; and
executing, by the neural network processor, instructions generated by a compiler of a host system external to the neural network processor, wherein the instructions cause the neural network processor to performing operations comprising:
performing, by the first CE, convolution operations on the last portion of the first chunk to compute a set of intermediate states across a stack of dilated convolutional layers having different dilation rates, wherein:
a subset of the set of intermediate states forms shared states to be used by the second CE and which correspond to outputs of different convolutional layers in the stack,
each convolutional layer in the stack involves a convolution operation in which a corresponding convolution kernel is applied to input data of the convolutional layer,
the stack comprises a first convolutional layer for which the input data is the input data set and at least one additional convolutional layer,
the input data for each convolutional layer after the first convolutional layer corresponds to convolution operation results from an immediately preceding convolutional layer, and
the convolution operations on the last portion of the first chunk are performed in parallel with corresponding convolution operations performed by the second CE with respect to a last portion of the second chunk;
storing, by the first CE, the set of intermediate states in a local memory of the first CE;
processing, by the second CE, a second portion of the second chunk across the stack of dilated convolutional layers to generate a first portion of an output dataset as an output of a last convolutional layer in the stack, wherein the first portion of the output dataset has no dependency on the shared states computed by the first CE, and wherein the processing of the second portion of the second chunk is performed in parallel with corresponding operations performed by first CE with respect to a second portion of the first chunk;
transmitting, by the first CE while the second CE is processing the second portion of the second chunk and the first CE is processing the second portion of the first chunk, the shared states to the second CE via an internal bus connecting the local memory of the first CE to a local memory of the second CE; and
generating, by the second CE as part of processing the first portion of the second chunk, a second portion of the output dataset as another output of the last convolutional layer in the stack, using the shared states transmitted by the first CE.