Vector operation acceleration with convolution computation unit
This application describes hybrid hardware accelerators, systems, and apparatus for performing various computations in neural network applications using the same set of hardware resources. An example accelerator may include weight selectors, activation input interfaces, and a plurality of Multiplier-Accumulation (MAC) circuits organized as a plurality of MAC lanes Each of the plurality of MAC lanes may be configured to: receive a control signal indicating whether to perform convolution or vector operations; receive one or more weights according to the control signal; receive one or more activations according to the control signal; and generate output data based on the one or more weights and the one or more input activations according to the control signal and feed the output data into an output buffer. Each of the plurality of MAC lanes includes a plurality of multiplier circuits and a plurality of adder-subtractor circuits.
1. A neural network accelerator, comprising:
an instruction decoder configured to decode a neural network computation instruction from a processor into a weight load control signal, an activation load control signal, and a compute control signal;
a plurality of weight selectors configured to obtain weights according to the weight load control signal, wherein the weight load control signal indicates whether to obtain the weights from a weight cache or from a weight generator;
a plurality of activation selectors configured to obtain activations or vectors from a memory according to the activation load control signal, wherein the activation load control signal indicates whether to obtain the activations or the vectors; and
a plurality of lanes of circuits, each lane of circuits being configured to:
receive the weights obtained by the plurality of weight selectors and the activations or the vectors obtained by the plurality of activation selectors,
determine whether to perform convolution operations or vector operations according to the compute control signal, and
perform the convolution operations or vector operations based on the weights and the activations or the vectors to generate output data; and
wherein the instruction decoder is further configured to, in response to the weights have a pattern, instruct the plurality of weight selectors to obtain the weights from the weight generator rather than obtaining the weights from the weight cache to reduce memory access.
2. The neural network accelerator of claim 1 , wherein the weight cache comprises weights of filters corresponding to a convolution layer in a neural network for convolution operations, and the weight generator generates weights to perform vector operations.
3. The neural network accelerator of claim 1 , wherein the weight load control signal indicates whether a current operation is a convolution operation or a vector operation, and
in response to the current operation being the convolution operation, the plurality of weight selectors obtain the weights from the weight cache; and
in response to the current operation being the vector operation, the plurality of weight selectors obtain the weights from the weight generator.
4. The neural network accelerator of claim 1 , wherein:
the compute control signal instructs the plurality of lanes of circuits to perform summations for convolution operations or subtractions for vector operations.
5. The neural network accelerator of claim 1 , wherein:
according to the compute control signal, a subset of the plurality of lanes of circuits is configured to perform convolution operations, and at a same time, remaining lanes of circuits are configured to perform vector operations.
6. The neural network accelerator of claim 1 , wherein each lane of the circuits are organized as a tree,
the circuits at a leaf level of the tree are configured to perform multiplications based on one or more of the weights and one or more of the activations or the vectors, and
the circuits at non-leaf levels of the tree are configured according to the compute control signal indicating whether to perform addition or subtraction based on data received from circuits of a prior level of the tree.
7. The neural network accelerator of claim 1 , wherein the vector operations comprise one or more of reduce mean, reduce minimum, reduce maximum, reduce average, reduce add, or pooling.
8. The neural network accelerator of claim 1 , wherein each of the plurality of weight selectors comprises a multiplexer coupled with the weight generator and the weight cache.
9. A hybrid convolution-vector operation processing system, comprising:
an instruction decoder configured to decode a neural network computation instruction from a processor into a weight load control signal and a compute control signal;
a plurality of weight selectors configured to obtain weights according to the weight load control signal, wherein the weight load control signal indicates whether to obtain the weights from a weight cache or from a weight generator;
a plurality of activation selectors configured to obtain activations or vectors from a memory; and
a plurality of lanes of circuits, each lane of circuits being configured to:
receive the weights obtained by the plurality of weight selectors and the activations or the vectors obtained by the plurality of activation selectors,
determine whether to perform convolution operations or vector operations according to the compute control signal, and
perform the convolution operations or vector operations based on the weights and the activations or the vectors to generate output data; and
wherein the instruction decoder is further configured to, in response to the weights have a pattern, instruct the plurality of weight selectors to obtain the weights from the weight generator rather than obtaining the weights from the weight cache to reduce memory access.
10. The hybrid convolution-vector operation processing system of claim 9 , wherein the weight cache comprises weights of filters corresponding to a convolution layer in a neural network for convolution operations, and the weight generator generates weights to perform vector operations.
11. The hybrid convolution-vector operation processing system of claim 9 , wherein the weight load control signal indicates whether a current operation is a convolution operation or a vector operation, and
in response to the current operation being the convolution operation, the plurality of weight selectors obtain the weights from the weight cache; and
in response to the current operation being the vector operation, the plurality of weight selectors obtain the weights from the weight generator.
12. The hybrid convolution-vector operation processing system of claim 9 , wherein the compute control signal instructs the plurality of lanes of circuits to perform summations for convolution operations or subtractions for vector operations.
13. The hybrid convolution-vector operation processing system of claim 9 , wherein, according to the compute control signal, a subset of the plurality of lanes of circuits is configured to perform convolution operations, and at a same time, remaining lanes of circuits are configured to perform vector operations.
14. The hybrid convolution-vector operation processing system of claim 9 , wherein each lane of the circuits are organized as a tree,
the circuits at a leaf level of the tree are configured to perform multiplications based on one or more of the weights and one or more of the activations or the vectors, and
the circuits at non-leaf levels of the tree are configured according to the compute control signal indicating whether to perform addition or subtraction based on data received from circuits of a prior level of the tree.
15. The hybrid convolution-vector operation processing system of claim 9 , wherein the vector operations comprise one or more of reduce mean, reduce minimum, reduce maximum, reduce average, reduce add, or pooling.
16. The hybrid convolution-vector operation processing system of claim 9 , wherein each of the plurality of weight selectors comprises a multiplexer coupled with the weight generator and the weight cache.
17. A computer-implemented method, comprising:
receiving a neural network computation instruction from a processor;
decoding, by an instruction decoder, the neural network computation instruction into a weight load control signal, an activation load control signal, and a compute control signal;
sending the weight load control signal to a plurality of weight selectors for obtaining weights, wherein the weight load control signal indicates whether the plurality of weight selectors obtain the weights from a weight cache or from a weight generator;
sending the activation load control signal to a plurality of activation selectors to obtain activations or vectors from a memory, wherein the activation load control signal indicates whether to obtain the activations or the vectors;
sending the compute control signal to a plurality of lanes of circuits, wherein the plurality of lanes of circuits are configured according to the compute control signal to:
receive the weights obtained by the plurality of weight selectors and the activations or the vectors obtained by the plurality of activation selectors,
determine whether to perform convolution operations or vector operations according to the compute control signal, and
perform the convolution operations or vector operations based on the weights and the activations or the vectors to generate output data; and
wherein the instruction decoder is further configured to, in response to the weights have a pattern, instruct the plurality of weight selectors to obtain the weights from the weight generator rather than obtaining the weights from the weight cache to reduce memory access.
18. The computer-implemented method of claim 17 , wherein the weight load control signal indicates whether a current operation is a convolution operation or a vector operation, and
in response to the current operation being the convolution operation, the obtaining weights comprises obtaining the weights from the weight cache; and
in response to the current operation being the vector operation, the obtaining weights comprises obtaining the weights from the weight generator.