Acceleration of neural networks with stacks of convolutional layers
A neural network processor is configured to execute instructions in parallel on different computing engines (CEs) to perform convolution operations on an input dataset. The input dataset is divided into overlapping chunks, including a first chunk and a second chunk. Each CE processes a last portion of a respective chunk to compute respective shared states and receives additional shared states for processing. The first chunk is the respective chunk for a first CE. The second chunk is the respective chunk for a second CE. The additional shared states received by the second CE are the respective shared states computed by the first CE. The second CE receives the respective shared states computed by the first CE as a substitute for intermediate states that would otherwise have been computed by the second CE using a first portion of the second chunk.
1 . A computer-implemented method comprising:
receiving a neural network model providing for an output dataset based on convolution operations; and
generating instructions for performing the convolution operations on a neural network processor and using an input dataset from an external memory, wherein the input dataset comprises a first chunk and a second chunk, and wherein the instructions are to be executed in parallel by a first computing engine (CE) and a second CE of the neural network processor to:
process a last portion of a respective chunk to compute respective shared states;
process a middle portion of the respective chunk to generate a first sub-portion of a respective portion of the output dataset;
receive, while processing the middle portion of the respective chunk, additional shared states for processing, wherein:
the first chunk is the respective chunk for the first CE,
the second chunk is the respective chunk for the second CE,
the last portion of the first chunk overlaps with a first portion of the second chunk,
the additional shared states received by the second CE are the respective shared states computed by the first CE and are obtained from a local memory of the first CE, and
the second CE receives the respective shared states computed by the first CE as a substitute for intermediate states that would otherwise have been computed by the second CE using the first portion of the second chunk; and
generate a second sub-portion of the respective portion of the output dataset using the received additional shared states.
2 . The computer-implemented method of claim 1 , wherein the instructions for performing the convolution operations comprise a same set of instructions for execution by the first CE and the second CE.
3 . The computer-implemented method of claim 1 , wherein the instructions for performing the convolution operations comprise an instruction for:
storing the respective shared states in a respective local memory while processing the last portion of the respective chunk.
4 . The computer-implemented method of claim 1 , wherein the instructions for performing the convolution operations comprise an instruction to provide the first CE with pre-computed states as the additional shared states.
5 . The computer-implemented method of claim 4 , wherein the pre-computed states have known or static values.
6 . The computer-implemented method of claim 4 , wherein:
the input dataset is a first input dataset in an input feature map, and
the respective shared states computed by the second CE are used as the pre-computed states for the first CE during processing of a second input dataset in the input feature map.
7 . The computer-implemented method of claim 1 , wherein the neural network model is a model of a dilated convolutional neural network (CNN), the dilated CNN comprising a first stack of dilated convolutional layers.
8 . The computer-implemented method of claim 7 , wherein the first chunk and the second chunk are inputs to a first layer in the first stack, and wherein the output dataset corresponds to outputs of a last layer in the first stack.
9 . The computer-implemented method of claim 8 , wherein the respective shared states correspond to outputs of the first layer and outputs of intervening layers between the first layer and the last layer.
10 . The computer-implemented method of claim 8 , wherein the dilated CNN further comprises a second stack of dilated convolutional layers, and wherein the outputs of the last layer in the first stack correspond to inputs of a first layer in the second stack.
11 . A neural network processor comprising:
a first computing engine (CE);
a second CE; and
an internal bus connecting the first CE and the second CE, wherein the neural network processor is configured to receive instructions for performing convolution operations using an input dataset from an external memory, wherein the input dataset comprises a first chunk and a second chunk, and wherein the instructions are to be executed in parallel by the first CE and the second CE to:
process a last portion of a respective chunk to compute respective shared states;
process a middle portion of the respective chunk to generate a first sub-portion of a respective portion of an output dataset;
receive, while processing the middle portion of the respective chunk, additional shared states for processing, wherein:
the first chunk is the respective chunk for the first CE,
the second chunk is the respective chunk for the second CE,
the last portion of the first chunk overlaps with a first portion of the second chunk,
the additional shared states received by the second CE are the respective shared states computed by the first CE and are obtained from a local memory of the first CE, and
the second CE receives the respective shared states computed by the first CE as a substitute for intermediate states that would otherwise have been computed by the second CE using the first portion of the second chunk; and
generate a second sub-portion of the respective portion of the output dataset using the received additional shared states.
12 . The neural network processor of claim 11 , wherein the instructions for performing the convolution operations comprise a same set of instructions for execution by the first CE and the second CE.
13 . The neural network processor of claim 11 , wherein the instructions for performing the convolution operations comprise an instruction for:
storing the respective shared states in a respective local memory while processing the last portion of the respective chunk.
14 . The neural network processor of claim 11 , wherein the instructions for performing the convolution operations comprise an instruction to provide the first CE with pre-computed states as the additional shared states.
15 . The neural network processor of claim 14 , wherein:
the input dataset is a first input dataset in an input feature map, and
the respective shared states computed by the second CE are used as the pre-computed states for the first CE during processing of a second input dataset in the input feature map.
16 . The neural network processor of claim 11 , wherein the instructions for performing the convolution operations implement a model of a dilated convolutional neural network (CNN), the dilated CNN comprising a first stack of dilated convolutional layers.
17 . The neural network processor of claim 16 , wherein the first chunk and the second chunk are inputs to a first layer in the first stack, and wherein the output dataset corresponds to outputs of a last layer in the first stack.
18 . The neural network processor of claim 17 , wherein the respective shared states correspond to outputs of the first layer and outputs of intervening layers between the first layer and the last layer.
19 . The neural network processor of claim 17 , wherein the dilated CNN further comprises a second stack of dilated convolutional layers, and wherein the outputs of the last layer in the first stack correspond to inputs of a first layer in the second stack.
20 . A non-transitory computer readable medium having stored therein instructions that, when executed by one or more processors, cause the one or more processors to execute a compiler, the compiler performing operations including:
receiving a neural network model providing for an output dataset based on convolution operations; and
generating instructions for performing the convolution operations on a neural network processor and using an input dataset from an external memory, wherein the input dataset comprises a first chunk and a second chunk, and wherein the instructions are to be executed in parallel by a first computing engine (CE) and a second CE of the neural network processor to:
process a last portion of a respective chunk to compute respective shared states;
process a middle portion of the respective chunk to generate a first sub-portion of a respective portion of the output dataset;
receive, while processing the middle portion of the respective chunk, additional shared states for processing, wherein:
the first chunk is the respective chunk for the first CE,
the second chunk is the respective chunk for the second CE,
the last portion of the first chunk overlaps with a first portion of the second chunk,
the additional shared states received by the second CE are the respective shared states computed by the first CE and are obtained from a local memory of the first CE, and
the second CE receives the respective shared states computed by the first CE as a substitute for intermediate states that would otherwise have been computed by the second CE using the first portion of the second chunk; and
generate a second sub-portion of the respective portion of the output dataset using the received additional shared states.