Apparatus and methods for training in fully connected layers of convolutional networks
Aspects for backpropagation in a fully connect layer of a convolutional neural network are described herein. The aspects may include a direct memory access unit configured to receive input data and one or more first data gradients from a storage device. The aspects may further include a master computation module configured to transmit the input data and the one or more first data gradients to one or more slave computation modules. The slave computation modules are respectively configured to multiply one of the one or more first data gradients with the input data to generate a default weight gradient vector.
1. An integrated circuit (IC) chip for backpropagation in a fully connected layer of a neural network, comprising:
a controller circuit configured to receive an instruction; and
one or more computation circuits that include:
a master computation circuit,
one or more slave computation circuits, and
an interconnection circuit communicatively connected to the master computation circuit and the one or more slave computation circuits,
wherein the master computation circuit configured to
receive input data and one or more first data gradients in response to the instruction, and
transmit the input data and the one or more first data gradients to the one or more slave computation circuits, and
wherein the one or more slave computation circuits are respectively configured to multiply one of the one or more first data gradients with the input data to generate a default weight gradient vector,
wherein the master computation circuit is further configured to update one or more weight values based on the default weight gradient vector,
wherein the master computation circuit is further configured to apply a derivative of an activation function to the one or more first data gradients to generate one or more input gradients,
wherein the one or more slave computation circuits are respectively configured to multiply one of the one or more input gradients with one or more weight vectors in a weight matrix to generate one or more multiplication results, and
wherein the interconnection circuit is configured to combine the one or more multiplication results of a lower dimension into an output gradient vector of a higher dimension.
2. The IC chip of claim 1 , wherein the master computation circuit is further configured to
calculate a scaled weight gradient vector based on the default weight gradient vector and a predetermined threshold value; and
update one or more weight values based on the scaled weight gradient vector.
3. The IC chip of claim 1 , wherein the interconnection circuit is configured to channel data between the master computation circuit and the one or more slave computation circuits.
4. The IC chip of claim 1 , wherein each of the one or more slave computation circuits includes a slave neuron caching circuit, wherein the slave neuron caching circuit is configured to store the one or more first data gradients with the input data.
5. The IC chip of claim 1 , wherein each of the one or more slave computation circuits includes a weight value caching circuit, wherein the weight value caching circuit is configured to store the weight matrix that includes the one or more weight vectors.
6. A method for backpropagation in a fully connected layer of a neural network, comprising:
receiving, by a controller circuit, an instruction;
receiving, by a master computation circuit, input data and one or more first data gradients in response to the instruction;
transmitting, by the master computation circuit, the input data and the one or more first data gradients to one or more slave computation circuits, wherein the master computation circuit is communicatively connected to the one or more slave computation circuits via an interconnection circuit;
respectively multiplying, by the one or more slave computation circuits, one of the one or more first data gradients with the input data to generate a default weight gradient vector;
updating, by the master computation circuit, one or more weight values based on the default weight gradient vector;
applying, by the master computation circuit, a derivative of an activation function to the one or more first data gradients to generate one or more input gradients;
respectively multiplying, by the one or more slave computation circuits, one of the one or more input gradients with one or more weight vectors in a weight matrix to generate one or more multiplication results; and
combining, by the interconnection circuit, the one or more multiplication results of a lower dimension into an output gradient vector of a higher dimension.
7. The method of claim 6 , further comprising
calculating, by the master computation circuit, a scaled weight gradient vector based on the default weight gradient vector and a predetermined threshold value; and
updating, by the master computation circuit, one or more weight values based on the scaled weight gradient vector.
8. The method of claim 6 , further comprising channeling, by the interconnection circuit, data between the master computation circuit and the one or more slave computation circuits.
9. The method of claim 6 , further comprising storing, by a slave neuron caching circuit of each of the one or more slave computation circuits, the one or more first data gradients with the input data.
10. The method of claim 6 , further comprising storing, by a weight value caching circuit of each of the one or more slave computation circuits, the weight matrix that includes the one or more weight vectors.