Neural processing device
A neural processing device is provided. The neural processing device includes a plurality of processing engine groups, a first memory shared by the plurality of engine groups, a first interconnection configured to transmit data between the first memory and the plurality of processing engine groups. The neural processing device is configured to provide hardware resource to the plurality of processing engine groups. At least one of the plurality of processing engine groups includes a plurality of processing engines, each of the plurality of processing engines includes an array of a plurality of processing elements interconnected by a mesh style network, the processing elements being reconfigurable. The neural processing device includes a second memory shared by the plurality of processing engines, and a second interconnection configured to transmit data between the second memory and the plurality of processing engines.
1. A processing device comprising processing circuitry comprising:
a plurality of processing engine groups;
a first memory shared by the plurality of engine groups; and
a first interconnection configured to transmit data between the first memory and the plurality of processing engine groups, wherein the processing circuitry is configured to provide hardware resource to the plurality of processing engine groups, at least one of the plurality of processing engine groups comprises:
a plurality of processing engines, each of the plurality of processing engines comprising an array of a plurality of processing elements;
a second memory shared by the plurality of processing engines;
a second interconnection configured to transmit data between the second memory and the plurality of processing engines; and
a compiler stack, wherein the compiler stack comprises:
a compute library configured to store a preset calculation code;
an adaptation layer configured to quantize a deep learning graph to generate a quantization model;
a frontend compiler configured to receive the quantization model and convert the quantization model into intermediate representation (IR);
a backend compiler configured to convert the IR into a binary code by referring to the calculation code; and
wherein the compiler stack is further configured to perform, on a circuit, optimization scheduling of the plurality of processing engines, wherein the performing the optimization scheduling comprises:
generating a control-flow graph (CFG) according to the deep learning graph;
unrolling a loop of the CFG to generate an unrolling CFG;
generating a hyperblock of the unrolling CFG to generate a hyperblocking CFG; and
generating a calculation code at a processing engine level by scheduling the hyperblocking CFG.
2. The processing device of claim 1 , wherein the processing circuitry is configured to perform monitoring at least one of the plurality of processing engines, the first interconnection, or the first memory, and individually provides the hardware resource according to a monitoring.
3. The processing device of claim 1 , wherein latency sensitivity of the second interconnection is higher than latency sensitivity of the first interconnection.
4. The processing device of claim 1 , wherein a bandwidth of the first interconnection is greater than a bandwidth of the second interconnection.
5. The processing device of claim 1 , further comprising a first processing engine cluster, and
wherein the first processing engine cluster comprises:
a first set of processing engine groups among the plurality of processing engine groups; and
a local interconnection between the first set of processing engine groups.
6. The processing device of claim 5 , further comprising a second processing engine cluster, comprising a second set of processing engine groups among the plurality of processing engine groups, and
wherein a processing engine group of the second set of processing engine groups is different from a processing engine group of the first set of processing engine groups, and
wherein the first processing engine cluster and the second processing engine cluster are managed by separate modules.
7. The processing device of claim 6 , wherein the plurality of processing engine groups are managed by separate modules.
8. The processing device of claim 1 , further comprising a first processing engine cluster, comprising at least one processing engine group belonging to a first region and at least one processing engine group belonging to a second region,
wherein the at least one processing engine group belonging to the first region and the at least one processing engine group belonging to the second region are managed by separate modules.
9. The processing device of claim 1 , wherein the each of the plurality of processing engines further comprises:
at least one third memory storing input data input to the processing elements and output data output from the processing elements; and
at least one fourth memory providing an instruction for an operation of the processing elements.
10. The processing device of claim 1 , wherein the compiler stack further comprises:
a first compiler configured to compile operations of the plurality of processing engines; and
a second compiler configured to compile operations of the first memory, the first interconnection, and at least one of the plurality of processing engine groups.
11. The processing device of claim 10 , wherein
the first compiler is further configured to determine a dimension of the plurality of processing engines, and
perform, on a circuit, the optimization scheduling of the plurality of processing engines.
12. A processing device comprising processing circuitry comprising:
a plurality of processing engines, each of the plurality of processing engine including a processing element (PE) array of a plurality of processing elements, at least one first memory configured to store data for the PE array, at least one second memory configured to provide instructions for operating the plurality of processing elements, and at least one load/store unit (LSU) configured to perform load and store for the data;
a third memory shared by the plurality of processing engines;
an interconnection configured to transmit data between the third memory and the plurality of processing engines; and
a compiler stack,
wherein the compiler stack comprises:
a compute library configured to store a preset calculation code;
an adaptation layer configured to quantize a deep learning graph to generate a quantization model;
a frontend compiler configured to receive the quantization model and convert the quantization model into intermediate representation (IR); and
a backend compiler configured to convert the IR into a binary code by referring to the calculation code,
wherein the compiler stack is further configured to perform, on a circuit, optimization scheduling of the plurality of processing engines, wherein the performing the optimization scheduling comprises:
generating a control-flow graph (CFG) according to the deep learning graph;
unrolling a loop of the CFG to generate an unrolling CFG;
generating a hyperblock of the unrolling CFG to generate a hyperblocking CFG; and
generating a calculation code at a processing engine level by scheduling the hyperblocking CFG.
13. The processing device of claim 12 , wherein
the processing circuitry is configured to provide a hardware resource to the plurality of processing engines according to importance of operations performed by the plurality of processing engines.
14. The processing device of claim 13 , wherein
a first set of processing engines are included in a first processing engine group, and
a second set of processing engines are included in a second processing engine group.
15. The processing device of claim 14 , wherein
the first processing group is managed by an upper module;
a first subset of processing engines in the first processing engine group is managed by a first lower module associated with the upper module; and
a second subset of processing engine in the first processing engine group is managed by a second lower module associated with the upper module.
16. The processing device of claim 12 , wherein each of the plurality of processing elements comprises:
an instruction queue configured to receive and divide an instruction including precision; and
an input formatter and an output formatter configured to perform precision conversion through the precision.
17. A processing device comprising processing circuitry comprising:
at least one processing engine group comprising a plurality of processing engines; and
a compiler stack comprising a compute library configured to store a preset calculation code, an adaptation layer configured to quantize a deep learning graph to generate a quantization model, a frontend compiler configured to receive the quantization model and convert the quantization model into intermediate representation (IR), and a backend compiler configured to convert the IR into a binary code by referring to the calculation code,
wherein at least one of the plurality of processing engines comprises a plurality of processing elements,
the processing circuitry is configured to provide the plurality of processing engines with hardware resources,
wherein at least one of the plurality of processing elements comprises:
an instruction queue configured to receive an instruction including precision, a source, an opcode, and a destination;
a first register configured to receive the source and the precision from the instruction queue;
an input formatter configured to determine an operand through the first register and configured to perform precision conversion;
a second register configured to receive the opcode from the instruction queue and configured to determine an operator; and
a third register configured to receive the destination and the precision from the instruction queue,
wherein the compiler stack is further configured to perform, on a circuit, optimization scheduling of the plurality of processing engines, wherein the performing the optimization scheduling comprises:
generating a control-flow graph (CFG) according to the deep learning graph;
unrolling a loop of the CFG to generate an unrolling CFG;
generating a hyperblock of the unrolling CFG to generate a hyperblocking CFG; and
generating a calculation code at a processing engine level by scheduling the hyperblocking CFG.
18. The processing device of claim 17 , further comprising an output formatter configured to perform the precision conversion of an output according to the operator of the operand through the third register.
19. The processing device of claim 18 , wherein the input formatter receives the output in bypass by the output formatter.