System, circuit, device and/or processes for accumulating neural network signals
Example methods, devices and/or circuits to be implemented in a processing device to perform neural network-based computing operations. According to an embodiment, an accumulation of weighted activation input values may be computed on accumulation cycles at least in part by multiplying and/or scaling accumulated activation input values by an associated neural network weight.
1 . A method comprising:
combining, at a first circuit formed in an electronic processing device, input signal values, the input signal values having a first fixed-word and/or byte format and being associated with a same weight cluster of a plurality of weight clusters, to generate an intermediate accumulated input signal value on accumulation cycles, each weight cluster comprising weight values associated with nodes and/or edges connecting nodes in a neural network mapped to a particular unique centroid weight value, the particular unique centroid weight value to be applied at the associated nodes and/or edges connecting nodes in the neural network, at least one of the input signal values selected from among a plurality of input signals based, at least in part, on an index value uniquely associated with the same weight cluster; and
combining, at a second circuit formed in the electronic processing device, the intermediate accumulated input signal value and a weight value to generate an output signal value having a second fixed-word and/or byte format on the accumulation cycles.
2 . The method of claim 1 , wherein:
the weight value comprises an 8-bit expression;
the intermediate accumulated input signal value comprises a 16-bit expression; and
the output signal value comprises a 32-bit expression.
3 . The method of claim 1 , wherein the weight value comprises a centroid weight value to be applied for weight values that are members of the same weight cluster.
4 . The method of claim 1 , and further comprising retrieving the weight value from a non-transitory storage medium based, at least in part, on an index value uniquely associated with a weight value.
5 . The method of claim 4 , and further comprising:
selecting the intermediate accumulated input signal value from among signals at a plurality of input terminals of a multiplexer circuit based, at least in part, on the index value.
6 . The method of claim 1 , and further comprising:
receiving the input signal values from an upstream layer in a neural network; and
providing the output signal value to a downstream layer in the neural network.
7 . The method of claim 1 , wherein the output signal value comprises a single output feature map pixel in a convolutional layer of a convolutional neural network.
8 . The method of claim 1 , and further comprising:
storing neural network weight values in a non-transitory memory device;
updating one or more of the stored neural network weight values based, at least in part, on the input signal values; and
mapping the updated one or more of the stored neural network weight values to an associated neural network weight cluster of the plurality of weight clusters.
9 . The method of claim 1 , and further comprising:
storing neural network weight values in a non-transitory memory device in association with weight indices, and wherein accumulating activation input signal values associated with the same weight cluster to generate an intermediate accumulated input signal value on accumulation cycles further comprises:
adding the at least one of the input signal values selected from among the plurality of input signals with the intermediate accumulated input signal value to generate an updated intermediate accumulated input signal value.
10 . An electronic processing device comprising:
a first circuit to combine input signal values associated with a same weight cluster of a plurality of weight clusters to generate an intermediate accumulated input signal value on accumulation cycles, each weight cluster comprising weight values associated with nodes and/or edges connecting nodes in a neural network mapped to a particular unique centroid weight value, the particular unique centroid weight value to be applied at the associated nodes and/or edges connecting nodes in the neural network, the first circuit to combine the input signal values associated with the same weight cluster to select an input signal value from among a plurality of input signals based, at least in part, on an index value uniquely associated with the same weight cluster; and
a second circuit to combine a weight value and the intermediate accumulated input signal value to generate an output signal value on the accumulation cycles.
11 . The electronic processing device of claim 10 , wherein:
the weight value to comprise an 8-bit expression;
the intermediate accumulated input signal value to comprise a 16-bit expression; and
the output signal value to comprise a 32-bit expression.
12 . The electronic processing device of claim 10 , wherein the weight value to comprise a centroid weight value to be applied for neural network weights that are members of the same weight cluster.
13 . The electronic processing device of claim 10 , and further comprising a third circuit to retrieve the weight value from a non-transitory storage medium based, at least in part, on an index value uniquely associated with a neural network weight value.
14 . The electronic processing device of claim 13 , and further comprising:
a fourth circuit to select the intermediate accumulated input signal value from among signals at a plurality of input terminals of a multiplexer circuit based, at least in part, on the index value.
15 . The electronic processing device of claim 11 , and further comprising:
a circuit to receive the input signal values from an upstream layer in a neural network; and
a circuit to provide the output signal value to a downstream layer in the neural network.
16 . The electronic processing device of claim 11 , wherein the output signal value comprises a single output feature map pixel in a convolutional layer of a convolutional neural network.
17 . The electronic processing device of claim 10 , and further comprising:
a non-transitory memory device to store neural network weight values;
a circuit to update one or more of the stored neural network weight values based, at least in part, on the input signal values; and
a circuit to map the updated one or more of the stored neural network weight values to an associated neural network weight cluster of the plurality of weight clusters.
18 . The electronic processing device of claim 10 , and further comprising:
a non-transitory memory device to store neural network weight values in association with weight indices, and wherein accumulation of input signal values associated with a weight cluster of the plurality of weight clusters to generate the intermediate accumulated input signal value on accumulation cycles to further comprise:
a sum of the selected input signal value with the intermediate accumulated input signal value to generate an updated intermediate accumulated input signal value.
19 . An article comprising:
a non-transitory storage medium comprising computer-readable instructions stored thereon that are executable by one or more processors of a computing device to:
express a first circuit, to be formed in electronic processing device, to combine input values associated with a same weight cluster of a plurality of weight clusters to generate an intermediate accumulated input signal value on accumulation cycles, each weight cluster comprising weight values associated with nodes and/or edges connecting nodes in a neural network mapped to a particular unique centroid weight value, the particular unique centroid weight value to be applied at the associated nodes and/or edges connecting nodes in the neural network, the first circuit to combine the input values associated with the same weight cluster to select an input value from among a plurality of input signals based, at least in part, on an index value uniquely associated with the same weight cluster; and
express a second circuit, to be formed in the electronic processing device, to combine a weight and the intermediate accumulated input signal value to generate an output signal value on the accumulation cycles.
20 . The article of claim 19 , the computer-readable instructions are formatted according to a register description language.