Near memory sparse matrix computation in deep neural network
A DNN accelerator includes a multiplication controller controlling whether to perform matrix computation based on weight values. The multiplication controller reads a weight matrix from a WRAM in the DNN accelerator and determines a row value for a row in the weight matrix. In an embodiment where the row value is one, a first switch sends a read request to the WRAM to read weights in the row and a second switch forms a data transmission path from an IRAM in the DNN accelerator to a PE in the DNN accelerator. The PE receives the weights and input data stored in the IRAM and performs MAC operations. In an embodiment where the row value is zero, the first and second switches are not triggered. No read request is sent to the WRAM and the data transmission path is not formed. The PE will not perform any MAC operations.
1 . An apparatus for accelerating a deep neural network (DNN) layer, the apparatus comprising:
a multiplication controller to:
access a weight matrix, the weight matrix comprising a plurality of rows, and
determine a row value for each respective row of the plurality of rows based on values of weights in the respective row;
a buffer to store the row values determined by the multiplication controller;
a processing element configured to perform multiplication operations and accumulation operations in the DNN layer;
a weight memory to store the weight matrix;
an input memory to store an activation matrix of the DNN layer;
a first switch, the first switch to control weight transmission from the weight memory to the processing element in accordance with the row value; and
a second switch, the second switch to control activation transmission from the input memory to the processing element in accordance with the row value.
2 . The apparatus of claim 1 , wherein the multiplication controller is configured to determine the row value for each respective row of the plurality of rows based on the values of the weights in the respective row by:
determining whether the values of the weights in the respective row include a non-zero value; and
in response to determining that the values of the weights in the respective row include a non-zero value, determining that the row value equals to one.
3 . The apparatus of claim 2 , wherein the multiplication controller is configured to determine the row value for each respective row of the plurality of rows based on the values of the weights in the respective row further by:
in response to determining that the values of the weights in the respective row do not include a non-zero value, determining that the row value equals to zero.
4 . The apparatus of claim 1 , wherein the first switch is to control the weight transmission from the weight memory to the processing element by:
accessing a row value from the buffer;
determining whether the row value equals to one; and
in response to determining that the row value equals to one, sending a read request to the weight memory to read the weights in the row corresponding to the row value.
5 . The apparatus of claim 1 , further comprising:
a bus configured to connect the input memory to the processing element, the bus controlled by the second switch.
6 . The apparatus of claim 5 , wherein the second switch is to control activation transmission from the input memory to the processing element by:
accessing a row value from the buffer;
determining whether the row value equals to one; and
in response to determining that the row value equals to one, controlling the bus to connect the input memory to the processing element for transmitting one or more activations from the input memory to the processing element.
7 . The apparatus of claim 1 , wherein the processing element comprises:
multipliers configured to perform the multiplication operations; and
accumulators configured to perform the accumulator operations.
8 . The apparatus of claim 7 , wherein the processing element further comprises:
a demultiplexer coupled to the multipliers and accumulators, the demultiplexer configured to perform demultiplexing operations on outputs of the multipliers, a demultiplexing operation comprising adding elements having values of zero into an output of a multiplier.
9 . The apparatus of claim 8 , wherein each respective multiplier of the multipliers corresponds to a different accumulator of the accumulators, and the demultiplexer is further configured to send a result of a demultiplexing operation on an output of the respective multiplier to the corresponding accumulator.
10 . The apparatus of claim 8 , further comprising:
a compression module that generates the activation matrix by compressing an input feature map matrix of the DNN layer.
11 . A method for controlling sparse matrix computation in deep learning, the method comprising:
storing an activation matrix of a deep neural network (DNN) layer in an input memory;
storing a weight matrix of the DNN layer in a weight memory, the weight matrix comprising a plurality of rows;
identifying a row of the weight matrix, the row comprising a plurality of weights;
determining whether a weight in the plurality of weights has a non-zero value;
in response to determining that the weight has the non-zero value, determining that a row value for the row is one;
storing the row value in a buffer;
controlling, by a first switch in accordance with the row value, weight transmission from the weight memory to a processing element;
controlling, by a second switch in accordance with the row value, activation transmission from the input memory to the processing element; and
perform, by the processing element using one or more activations from the input memory and one or more weights from the weight memory, multiplication operations and accumulation operations in the DNN layer.
12 . The method of claim 11 , further comprising:
identifying an additional row of the weight matrix, the additional row comprising additional weights;
determining whether any of the additional weights have a non-zero value;
in response to determine that none of the additional weights have a non-zero value, determining that an additional row value for the additional row is zero; and
storing the additional row value in the buffer, the additional row value used to prevent the additional weights from being read from the weight memory.
13 . The method of claim 11 , wherein determining the row value comprises:
determining whether the values of the weights in the respective row include a non-zero value; and
in response to determining that the values of the weights in the respective row include a non-zero value, determining that the row value equals to one.
14 . The method of claim 13 , wherein determining the row value further comprises:
in response to determining that the values of the weights in the respective row do not include a non-zero value, determining that the row value equals to zero.
15 . The method of claim 11 , further comprising:
generating the activation matrix by compressing an input feature map matrix of the DNN layer.
16 . One or more non-transitory computer-readable media storing instructions executable to perform operations for accelerating a deep neural network (DNN) layer, the operations comprising:
storing an activation matrix of a deep neural network (DNN) layer in an input memory;
storing a weight matrix of the DNN layer in a weight memory, the weight matrix comprising a plurality of rows;
determining a row value for each respective row of the plurality of rows based on values of weights in the respective row;
storing the row value in a buffer;
controlling, by a first switch in accordance with the row value, weight transmission from the weight memory to a processing element;
controlling, by a second switch in accordance with the row value, activation transmission from the input memory to the processing element; and
perform, by the processing element using one or more activations from the input memory and one or more weights from the weight memory, multiplication operations and accumulation operations in the DNN layer.
17 . The one or more non-transitory computer-readable media of claim 16 , wherein the operations further comprise:
identifying an additional row of the weight matrix, the additional row comprising additional weights;
determining whether any of the additional weights have a non-zero value;
in response to determine that none of the additional weights have a non-zero value, determining that an additional row value for the additional row is zero; and
storing the additional row value in the buffer, the additional row value used to prevent read of the additional weights from the weight memory.
18 . The one or more non-transitory computer-readable media of claim 16 , wherein determining the row value comprises:
determining whether the values of the weights in the respective row include a non-zero value; and
in response to determining that the values of the weights in the respective row include a non-zero value, determining that the row value equals to one.
19 . The one or more non-transitory computer-readable media of claim 18 , wherein determining the row value further comprises:
in response to determining that the values of the weights in the respective row do not include a non-zero value, determining that the row value equals to zero.
20 . The one or more non-transitory computer-readable media of claim 16 , wherein the operations further comprise:
generating the activation matrix by compressing an input feature map matrix of the DNN layer.