Adjusting enhancement coefficients for neural network engine
View Patent ↗Some embodiments include a special-purpose hardware accelerator that can perform specialized machine learning tasks during both training and inference stages. For example, this hardware accelerator uses a systolic array having a number of data processing units (“DPUs”) that are each connected to a small number of other DPUs in a local region. Data from the many nodes of a neural network is pulsed through these DPUs with associated tags that identify where such data was originated or processed, such that each DPU has knowledge of where incoming data originated and thus is able to compute the data as specified by the architecture of the neural network. These tags enable the systolic neural network engine to perform computations during backpropagation, such that the systolic neural network engine is able to support training.
1. A device for training a convolutional neural network comprising a plurality of layers, the device comprising:
an array comprising a plurality of processing units including processing circuitry and memory, wherein the array is configured to transmit data systolically between particular processing units;
a computer-readable memory storing instructions for using the array to perform computations of the neural network during the training; and
a controller configured by the instructions to:
provide input data representing an image into the array, the image including an array of pixels;
perform a forward pass of the input data through the plurality of layers;
for a particular location in the array of pixels, generate a pixel vector representing values output by the plurality of layers for that particular location, wherein the pixel vector includes a first value generated by a first layer of the plurality of layers and a second value generated by a second layer of the plurality of layers, wherein the second layer is deeper along the plurality of layers of the convolutional neural network than the first layer; and
adjust an enhancement coefficient of the first value of the first layer based on the second value of the second layer.
2. The device of claim 1 , wherein to adjust the enhancement coefficient includes increasing a weighting for the particular location based on finding a correspondence.
3. The device of claim 2 , wherein the correspondence includes an identification of similar values occurring at similar pixel locations.
4. The device of claim 1 , wherein to adjust the enhancement coefficient includes decreasing a weighting for the particular location based on not finding a correspondence.
5. The device of claim 1 , wherein the controller is further configured by the instructions to adjust the first value based on the enhancement coefficient to generate an adjusted output value; and provide the adjusted output value to the second layer.
6. The device of claim 1 , wherein to adjust the enhancement coefficient is based on image-correlated disturbance mode training by correlating disturbances to the image without a learned parameter.
7. The device of claim 1 , wherein to adjust the enhancement coefficient is based on set-correlated enhancement mode training using one or more learned parameters.
8. The device of claim 1 , wherein to adjust the enhancement coefficient includes computing the enhancement coefficient via an enhancement matrix by summing over corresponding positions across maps using masks of different volumes, the mask volumes producing coefficients to be placed in an enhancement matrix.
9. The device of claim 8 , wherein the different volumes of masks include at least one of: a 1×1, a 3×3, a 5×5, or a 7×7 mask.
10. The device of claim 1 , wherein the controller is further configured by the instructions to randomly turn off one or more neurons of a layer in the neural network during the training.
11. A method for performing computations of a neural network comprising a plurality of layers including at least a first layer and a second layer, the method comprising:
accessing data representing an image including an array of pixels;
performing a forward pass of the data through the plurality of layers;
for a particular location in the array of pixels, generating a pixel vector representing values output by the plurality of layers for that particular location, wherein the pixel vector includes a first value generated by a first layer of the plurality of layers and a second value generated by a second layer of the plurality of layers, wherein the second layer is deeper along the plurality of layers of the neural network than the first layer; and
adjusting an enhancement coefficient of the first value of the first layer based on the second value of the second layer.
12. The method of claim 11 , wherein adjusting the enhancement coefficient includes increasing a weighting for the particular location based on finding a correspondence.
13. The method of claim 12 , wherein the correspondence includes an identification of similar values occurring at similar pixel locations.
14. The method of claim 11 , wherein adjusting the enhancement coefficient includes decreasing a weighting for the particular location based on not finding a correspondence.
15. The method of claim 11 , wherein the method further includes adjusting the first value based on the enhancement coefficient to generate an adjusted output value; and providing the adjusted output value to the second layer.
16. The method of claim 11 , wherein adjusting the enhancement coefficient is based on image-correlated disturbance mode training by correlating disturbances to the image without a learned parameter.
17. The method of claim 11 , wherein adjusting the enhancement coefficient is based on set-correlated enhancement mode training using one or more learned parameters.
18. A controller comprising one or more processors configured to:
perform a forward pass of input data of an array of pixels through a plurality of layers of a neural network;
for a particular location in the array of pixels, generate a pixel vector representing values output by the plurality of layers for that particular location, wherein the pixel vector includes a first value generated by a first layer of the plurality of layers and a second value generated by a second layer of the plurality of layers; and
adjust an enhancement coefficient of the first value of the first layer based on the second value of the second layer.
19. The controller of claim 18 , wherein adjustment of the enhancement coefficient includes increase of a weighting for the particular location based on finding a correspondence.
20. The controller of claim 19 , wherein the correspondence includes an identification of similar values occurring at similar pixel locations, wherein the controller performs the operations by computer executable instructions stored in a non-transitory computer-readable medium.