System and method for implementing neural networks in integrated circuits
A neural network system includes an input layer, one or more hidden layers, and an output layer. A first layer circuit implements a first layer of the one or more hidden layers. The first layer includes a first weight space including one or more subgroups. A forward path circuit of the first layer circuit includes a multiply and accumulate circuit to receive an input from a layer preceding the first layer; and provide a first subgroup weighted sum using the input and a first plurality weights associated with a first subgroup. A scaling coefficient circuit provides a first scaling coefficient associated with the first subgroup, and applies the first scaling coefficient to the first subgroup weighted sum to generate a first subgroup scaled weighted sum. An activation circuit generates an activation based on the first subgroup scaled weighted sum and provide the activation to a layer following the first layer.
1. A neural network system, comprising:
one or more hardware circuits configured to perform a method comprising:
providing a first layer of one or more hidden layers, wherein the first layer includes a first weight space including one or more subgroups receiving an input from a layer preceding the first layer;
generating a first subgroup weighted sum using the input and a first plurality of weights associated with a first subgroup of the one or more subgroups;
performing a first training step using first-step scaling coefficients for the first subgroup;
generating a single, second-step scaling coefficient for the first subgroup based on the first-step scaling coefficients;
performing a second training step using the single, second-step scaling coefficient for the first subgroup to generate a first subgroup scaled weighted sum;
generating an activation based on the first subgroup scaled weighted sum; and
providing the activation to a layer following the first layer.
2. The neural network system of claim 1 , wherein the first layer includes a kernel filter having a size of K1×K2, and an input feature map having a size of I,
wherein K1, K2, and I are positive integers,
wherein the first weight space includes:
first and second dimensions corresponding to the kernel filter and have sizes of K1 and K2 respectively; and
a third dimension corresponding to the input feature map and having a size of I; and
wherein the first plurality of weights have a same location in at most two of the first, second, and third dimensions.
3. The neural network system of claim 2 , wherein the first layer is a convolutional layer, and
wherein the first weight space includes a fourth dimension corresponding to an output feature map of the convolutional layer and having a size of N, N being an integer.
4. The neural network system of claim 3 , wherein the first subgroup is a pixel-based subgroup,
wherein the first plurality of weights have a same first location in the first dimension and a same second location in the second dimension, and
wherein the first plurality of weights have different locations in the third dimension.
5. The neural network system of claim 3 , wherein the first subgroup is a row-based subgroup,
wherein the first plurality of weights have a same location in the first dimension, and
wherein the first plurality of weights have variable locations in each of the second dimension and the third dimension.
6. The neural network system of claim 2 ,
wherein a second layer of the one or more hidden layers includes a second weight space having a single layer-based subgroup, and
wherein a single scaling coefficient is applied to weights of the second layer.
7. The neural network system of claim 1 , wherein the method further comprises:
performing multiplications of the input and each of the first plurality of weights associated with the first subgroup in parallel to generate a first subgroup weighted sum.
8. The neural network system of claim 1 , wherein the method further includes:
multiplying the single, second-step scaling coefficient with a first subgroup weighted sum to generate the first subgroup scaled weighted sum.
9. The neural network system of claim 1 , wherein the method further includes:
accumulating a plurality of subgroup scaled weighted sums for the one or more subgroups to generate an accumulated scaled weighted sum; and
generating the activation based on the accumulated scaled weighted sum.
10. The neural network system of claim 1 , wherein the first plurality of weights have binary or ternary values.
11. A method, comprising:
providing a first layer of one or more hidden layers, wherein the first layer includes a first weight space including one or more subgroups;
receiving an input from a layer preceding the first layer;
generating a first subgroup weighted sum using the input and a first plurality of weights associated with a first subgroup of the one or more subgroups;
performing a first training step using first-step scaling coefficients for the first subgroup;
generating a single, second-step scaling coefficient for the first subgroup based on the first-step scaling coefficients;
performing a second training step using the single, second-step scaling coefficient for the first subgroup to generate a first subgroup scaled weighted sum;
generating an activation based on the first subgroup scaled weighted sum; and
providing the activation to a layer following the first layer.
12. The method of claim 11 , wherein the first layer includes a kernel filter having a size of K1×K2, and an input feature map having a size of I,
wherein K1, K2, and I are positive integers,
wherein the first weight space includes:
first and second dimensions corresponding to the kernel filter and have sizes of K1 and K2 respectively; and
a third dimension corresponding to the input feature map and having a size of I; and
wherein the first plurality of weights have a same location in at most two of the first, second, and third dimensions.
13. The method of claim 12 , wherein the first layer is a convolutional layer, and
wherein the first weight space includes a fourth dimension corresponding to an output feature map of the convolutional layer and having a size of N, N being a positive integer.
14. The method of claim 13 , wherein the first subgroup is a pixel-based subgroup,
wherein the first plurality of weights have a same first location in the first dimension and a same second location in the second dimension, and
wherein the first plurality of weights have different locations in the third dimension.
15. The method of claim 13 , wherein the first subgroup is a row-based subgroup,
wherein the first plurality of weights have a same location in the first dimension, and
wherein the first plurality of weights have different locations in each of the second dimension and the third dimension.
16. The method of claim 12 , wherein a second layer of the one or more hidden layers includes a second weight space having a single layer-based subgroup, and
wherein a single scaling coefficient is applied to weights of the second layer.
17. The method of claim 16 , wherein the second layer is a fully connected layer.
18. The method of claim 11 , further comprising:
receiving, by a multiplexer, one or more scaling coefficients corresponding to the one or more subgroups respectively; and
receiving, by the multiplexer from a counter, a select signal to select the first-step scaling coefficients corresponding to the first subgroup.
19. The method of claim 11 , further comprising:
accumulating a plurality of subgroup scaled weighted sums for the one or more subgroups to generate an accumulated scaled weighted sum;
wherein the activation is generated based on the accumulated scaled weighted sum.
20. The method of claim 11 , wherein the first plurality of weights have binary or ternary values.