DIGITAL NEURAL NETWORK
Various embodiments provide for digital neural network (DNN) that form part of a machine learning (ML) processor and that perform a compute intensive function, such as convolutions or matrix multiplies, which can facilitate operation of a ML model. According to some embodiments, the DNN comprises a combinatorial tree that uses multiple-accumulate (MAC) units, and a sequencer that reads values from memory devices and feeds the combinatorial tree.
1 . A hardware circuit comprising:
a combinatorial tree having a plurality of rows and a plurality of columns, the combinatorial tree using a plurality of hardware units to generate one or more dot product values based on input values received at a first column the combinatorial tree;
a first memory device for storing the input values;
a sequencer configured to obtain the input values from the first memory device and reorganize the input values in a switch fabric;
a clock configured to control a time for when a portion of the reorganized input values is provided to the first column of the combinatorial tree from the switch fabric; and
a plurality of hardware units where each hardware unit comprises an accumulator configured to accumulate the one or more dot product values, wherein during one or more subsequent time periods defined by the clock, additional portions of the reorganized input values are provided to accumulators of hardware units associated with the first column, and wherein during one or more additional time periods controlled by the clock, a value associated with each of the accumulators of the hardware units in the first column is written to the first memory device.
2 . The hardware circuit of claim 1 , wherein the hardware units comprise at least one fused multiple-accumulate (FMA) unit.
3 . The hardware circuit of claim 1 , wherein the combinatorial tree comprises an array of dot product units configured to receive the input values, and to generate the one or more dot product values based on the input values as received by the first column, a single dot product unit of the array comprising two or more of the hardware units.
4 . The hardware circuit of claim 3 , further comprising a control circuit, the array of dot product units comprising data paths, the control circuit being configured to use the data paths to cause at least two weight values to swap between two neighboring columns of the array or to propagate a weight value from a first row of the array to a second row of the array.
5 . The hardware circuit of claim 1 , wherein the sequencer includes the switch fabric that operatively couples the combinatorial tree to the first memory device.
6 . The hardware circuit of claim 1 , wherein the sequencer determines a sequence of input vectors provided to the first column based on a set of parameters, the set of parameters comprising at least one of a stride value, a dilation value, or a kernel size value.
7 . The hardware circuit of claim 1 , wherein the sequencer is configured to generate an individual input vector of the sequence of input vectors by reading a plurality of weight values and a plurality of operand values read from the first memory device during one or more clock cycles, the individual input vector comprising the plurality of the weight values and the plurality of the operand values.
8 . The hardware circuit of claim 7 , wherein the first memory device comprises memory banks, and wherein the plurality of weight values and the plurality of operand values read by the sequencer during the one or more clock cycles are selected for the reading such that:
the plurality of weight values and the plurality of operand values are permuted by the sequencer in the switch fabric after receipt from the memory bank, and each of the weight values and the operand valued are read by the sequencer in a single pass.
9 . The hardware circuit of claim 7 , wherein the first memory devices comprise memory banks, and wherein the plurality of weight values and the plurality of operand values read by the sequencer during one or more clock cycles are selected for the reading such that:
the plurality of weight values and the plurality of operand values are permuted by the sequencer in the switch fabric after receipt from the memory bank, and each of the weight values and the operand values are read by the sequencer in a single pass; and
data is read from each of the memory banks, wherein a number of rows of the memory banks is substantially equivalent to a number of rows in the first column of the combinatorial tree.
10 . The hardware circuit of claim 7 , wherein each successive read of the first memory device, by the sequencer, results in reading of new weight values and new operand values from the first memory device not read by the sequencer during a previous clock cycle.
11 . The hardware circuit of claim 1 , wherein a single hardware unit is operatively coupled to at least two adjacent hardware units such that an output value generated by the single hardware unit is shared as an input value to each of the at least two adjacent hardware units.
12 . The hardware circuit of claim 1 , wherein the accumulator comprises a plurality of accumulators configured to accumulate partial dot product values generated by a single hardware unit as values flow through the single hardware unit.
13 . The hardware circuit of claim 1 , wherein the combinatorial tree comprises registers associated with the hardware units configured to maintain clock alignment of values flowing through the hardware units.
14 . The hardware circuit of claim 1 , comprising logic that is configured to generate a set of final dot product values for the input values after all of a sequence of input vectors have been inputted to the combinatorial tree, the set of final dot product values being constructed from values stored in accumulators of the hardware units of the combinatorial tree.
15 . The hardware circuit of claim 14 , wherein the logic is configured to store the set of final dot product values to the first memory device.
16 . The hardware circuit of claim 1 , wherein the input values comprise at least one weight value and at least one operand value.
17 . The hardware circuit of claim 1 , wherein the hardware circuit is part of a photonic integrated circuit (PIC).
18 . A method comprising:
reading, by a sequencer, a first plurality of weight values and a first plurality of operand values from a set of memory devices operatively coupled to the sequencer;
receiving, at a first column of a combinatorial tree, a plurality of first input vectors from the sequencer, the first plurality of input vectors comprising the first plurality of weight values and the first plurality of operand values;
processing the first plurality of input vectors by math units at the first column of the combinatorial tree, the math units being associated with each of the input vectors, each of the math units having at least one accumulator, the combinatorial tree being configured to generate a dot product value in an accumulator of each of the math unit based on values received by the math units at the first column of the combinatorial tree;
reading, by the sequencer, a second plurality of weight values and a second plurality of operand values from the set of memory devices;
receiving, at the first column of the combinatorial tree, a second plurality of input vectors from the sequencer, the second plurality of input vectors comprising the second plurality of weight values and the second plurality of operand values;
processing the second plurality of input vectors by the math units at the first column of the combinatorial tree, the first plurality of input vectors and the second plurality of inputs vector being part of a sequence of input vectors to be received by the math units in the first column of the combinatorial tree; and
after all of the sequence of input vectors has been received by the first column of the combinatorial tree and processed by associated math units, providing a set of final dot product values by retrieving a value from each accumulator in each of the math units residing in every row of the first column of the combinatorial tree.
19 . The method of claim 18 , wherein the at least one accumulator comprises a plurality of accumulators in each of the math units, the plurality of accumulators being configured to accumulate partial dot product values generated by individual math units as values flow through the individual math units.
20 . A hardware circuit comprising:
an array of dot product units organized as a combinatorial tree that generates dot product values based on weight values and activation values received as input by the array of dot product units, the array of dot product units comprises 64 rows by 32 columns of dot product units, each dot product unit being configured to receive as input two 8-bit values and having an accumulator; and
a switch fabric operatively coupling the array to a set of memory devices, the set of memory devices storing a plurality of weight values and a plurality of activation values, the switch fabric being configured to provide 64 input vectors as input to a first column of the array, during a first clock sequence comprising one or more clock cycles,
wherein one or more additional clock sequences cause the switch fabric to provide a plurality of additional sets of 64 input vectors to the first column of the array, and wherein when each of the accumulators in the first column of the array has 32 bits, writing each value from each accumulator back into the set of memory devices.
21 . The hardware circuit of claim 20 , wherein the dot product units comprise a plurality of FMA units.
22 . The hardware circuit of claim 20 , wherein the dot product units are configured to receive a 32-byte input.
23 . The hardware circuit of claim 20 , wherein the set of memory devices comprises 64 banks of 32-byte wide memory devices.