Data structure descriptors for deep learning acceleration
Techniques in advanced deep learning provide improvements in one or more of accuracy, performance, and energy efficiency. An array of processing elements performs flow-based computations on wavelets of data. Each processing element has a respective compute element and a respective routing element. Instructions executed by the compute element include operand specifiers, some specifying a data structure register storing a data structure descriptor describing an operand as a fabric vector or a memory vector. The data structure descriptor further describes the memory vector as one of a one-dimensional vector, a four-dimensional vector, or a circular buffer vector. Optionally, the data structure descriptor specifies an extended data structure register storing an extended data structure descriptor. The extended data structure descriptor specifies parameters relating to a four-dimensional vector or a circular buffer vector.
1. A system comprising:
a plurality of processing elements, each processing element respectively comprising a coupled fabric router and compute element, the processing elements being interconnected at least in part via the respective fabric routers;
each compute element comprising
a memory,
an instruction decoder enabled to decode each of a plurality of vector instructions comprising at least one respective operand identifier of at least one respective operand comprising an operand type of a mutually exclusive one of a memory operand type accessible via the memory and a fabric operand type accessible via the coupled fabric router,
data structure registers enabled to store data structure descriptors, each stored data structure descriptor specifying the operand type and other attributes of a vector instruction operand, and
a data sequencer enabled for each of the plurality of vector instructions to determine locations of one or more data elements of the at least one respective operand, the determination being based at least in part on at least one of the data structure descriptors read from the data structure registers, the reading being based at least in part on the at least one respective operand identifier; and
wherein the compute element is enabled for each of the plurality of vector instructions to access the one or more data elements of the at least one respective operand in accordance with the at least one of the data structure descriptors.
2. The system of claim 1 , wherein each compute element is further enabled, responsive to the operand type being the fabric operand type and the at least one respective operand being a source, to access the operand via reading data elements from an input queue of the compute element, the input queue comprising at least in part the coupling with the fabric router.
3. The system of claim 1 , wherein each compute element is further enabled, responsive to the operand type being the fabric operand type and the at least one respective operand being a destination, to access the operand via writing data elements to an output queue of the compute element, the output queue comprising at least in part the coupling with the fabric router.
4. The system of claim 1 , wherein the at least one of the data structure descriptors is enabled to identify one of a plurality of extended operand descriptors.
5. The system of claim 4 , wherein the extended operand descriptors are enabled to specify one or more of stride information and dimension information of a four-dimensional memory vector.
6. The system of claim 4 , wherein the extended operand descriptors are enabled to specify one or more of a start address and an end address of a circular memory buffer.
7. The system of claim 4 , wherein the extended operand descriptors are enabled to specify FIFO or non-FIFO operation of a circular memory buffer.
8. The system of claim 1 , wherein execution of one or more of the plurality of vector instructions implements at least a portion of any one or more of: computing an activation of a neural network, computing a partial sum of activations of a neural network, computing an error of a neural network, computing a gradient estimate of a neural network, and updating a weight of a neural network.
9. The system of claim 1 , wherein the at least one respective operand comprises at least a portion of any one or more of: a weight of a neural network, an activation of a neural network, a partial sum of activations of a neural network, an error of a neural network, a gradient estimate of a neural network, and a weight update of a neural network.
10. The system of claim 1 , wherein a substantially whole wafer comprises the processing elements.
11. The system of claim 1 , wherein the compute element is enabled to perform an iteration one or more of the plurality of vector instructions via accessing, in accordance with an access pattern described by the at least one of the data structure descriptors, sufficient data elements of a vector for the iteration.
12. The system of claim 11 , wherein the access pattern is one of a fabric vector, a one-dimensional memory vector, a four-dimensional memory vector, and a circular memory buffer.
13. The system of claim 1 , wherein the compute element is enabled to read from the memory when the operand type is the memory operand type and the operand is a source.
14. The system of claim 1 , wherein the compute element is enabled to write to the memory when the operand type is the memory operand type and the operand is a destination.
15. The system of claim 1 , wherein the other attributes comprises information describing a length of the vector instruction operand.
16. The system of claim 1 , wherein the compute element is enabled to execute a first of the plurality of vector instructions, and the at least one of the data structure descriptors comprises microthreading information describing how the compute element is to operate when there is a stall accessing the one or more data elements of the at least one respective operand.
17. The system of claim 16 , wherein responsive to the stall and the microthreading information indicating microthreading not enabled, the compute element stalling.
18. The system of claim 16 , wherein responsive to the stall and the microthreading information indicating microthreading is enabled, the compute element suspending processing of the first of the plurality of vector instructions and selecting a first of one or more other instructions for processing.
19. The system of claim 18 , wherein the first of the plurality of vector instructions is associated with a first task and the first of one or more other instructions is associated with a second task.
20. The system of claim 1 , wherein the at least one respective operand comprises at least a portion of one or more of: the vector, a matrix, and a tensor.
21. The system of claim 1 , wherein when the operand type is the fabric operand type, the at least one of the data structure descriptors is associated with a fabric virtual channel of the interconnected processing elements.
22. The system of claim 1 , wherein the at least one of the data structure descriptors indicates how many of the data elements of the at least one respective operand to process in parallel.
23. The system of claim 1 , wherein the at least one of the data structure descriptors comprises an indicator of whether to terminate processing responsive at least in part to a control fabric packet, conveying one of the data elements of the at least one respective operand, being received via the fabric.
24. The system of claim 1 , wherein the at least one of the data structure descriptors comprises an indicator of a virtual channel to selectively activate responsive to completion of a first of the plurality of vector instructions.
25. The system of claim 1 , wherein the interconnected processing elements are enabled to perform dataflow-based and instruction-based processing.
26. The system of claim 1 , wherein the interconnection of the processing elements is via a fabric, wherein the fabric is at least in part a collection of couplings between the processing elements, the couplings comprising one or more of logical couplings and physical couplings.
27. The system of claim 1 , wherein each processing element is enabled to selectively communicate fabric packets with others of the processing elements at least in part via the fabric router of the respective processing element.
28. A method comprising:
storing a plurality of data structure descriptors in data structure registers of a compute element, each stored data structure descriptor specifying an operand type and other attributes of a vector instruction operand, the compute element further comprising a memory, an instruction decoder, and a data sequencer;
wherein the compute element and a coupled fabric router are comprised in a processing element, the processing element being one of a plurality of processing elements each comprising a respective compute element and a respective fabric router, the processing elements being interconnected at least in part via the respective fabric routers;
in the compute element, for each of a plurality of vector instructions,
decoding by the instruction decoder at least one respective operand identifier of at least one respective operand,
accessing by the data sequencer at least one of the plurality of data structure descriptors from the data structure registers based at least in part on the at least one respective operand identifier,
determining locations by the data sequencer of one or more data elements of the at least one respective operand based at least in part on the at least one of the data structure descriptors, and
accessing the one or more data elements of the at least one respective operand; and
wherein the operand type comprises a mutually exclusive one of a memory operand type accessible via the memory and a fabric operand type accessible via the coupled fabric router.
29. The method of claim 28 , further comprising:
wherein responsive to the operand type being the fabric operand type and the at least one respective operand being a source, the accessing the one or more data elements of the at least one respective operand is performed via reading the one or more data elements from an input queue of the compute element, the input queue comprising at least in part the coupling with the fabric router; and
wherein responsive to the operand type being the fabric operand type and the at least one respective operand being a destination, the accessing the one or more data elements of the at least one respective operand is performed via writing data elements to an output queue of the compute element, the output queue comprising at least in part the coupling with the fabric router.
30. The method of claim 28 , wherein the at least one of the data structure descriptors is enabled to identify one of a plurality of extended operand descriptors enabled to specify one or more of stride information and dimension information of a four-dimensional memory vector.
31. The method of claim 28 , wherein the at least one of the data structure descriptors is enabled to identify one of a plurality of extended operand descriptors enabled to specify one or more of a start address and an end address of a circular memory buffer and further enabled to specify FIFO or non-FIFO operation of the circular memory buffer.
32. The method of claim 28 , wherein the compute element is enabled to perform an iteration one or more of the plurality of vector instructions via accessing, in accordance with an access pattern described by the at least one of the data structure descriptors, sufficient data elements of a vector for the iteration.
33. The method of claim 32 , wherein the access pattern is one of a fabric vector, a one-dimensional memory vector, a four-dimensional memory vector, and a circular memory buffer.
34. The method of claim 28 , wherein when the operand type is the fabric operand type, the at least one of the data structure descriptors is associated with a fabric virtual channel of the interconnected processing elements.
35. The method of claim 28 , wherein the at least one of the data structure descriptors indicates how many of the data elements of the at least one respective operand to process in parallel.
36. The method of claim 28 , wherein the at least one of the data structure descriptors comprises an indicator of whether to terminate processing responsive at least in part to a control fabric packet, conveying one of the data elements of the at least one respective operand, being received via the fabric.
37. The method of claim 28 , wherein the at least one of the data structure descriptors comprises an indicator of a virtual channel to selectively activate responsive to completion of a first of the plurality of vector instructions.
38. A system comprising:
means for storing a plurality of data structure descriptors in a compute element, each stored data structure descriptor specifying an operand type and other attributes of a vector instruction operand;
wherein the compute element and a coupled fabric router are comprised in a processing element, the processing element being one of a plurality of processing elements each comprising a respective compute element and a respective fabric router, the processing elements being interconnected at least in part via the respective fabric routers;
means for decoding in the compute element, for each of a plurality of vector instructions, at least one respective operand identifier of at least one respective operand;
means for accessing in the compute element, for each of the plurality of vector instructions, at least one of the plurality of data structure descriptors from the data structure registers based at least in part on the at least one respective operand identifier;
means for determining in the compute element, for each of the plurality of vector instructions, locations of one or more data elements of the at least one respective operand based at least in part on the at least one of the data structure descriptors; and
means for accessing in the compute element, for each of the plurality of vector instructions, the one or more data elements of the at least one respective operand; and
wherein the operand type comprises a mutually exclusive one of a memory operand type accessible via the memory and a fabric operand type accessible via the coupled fabric router.
39. The system of claim 38 , further comprising:
wherein each compute element is further enabled, responsive to the operand type being the fabric operand type and the at least one respective operand being a source, to access the operand via reading data elements from an input queue of the compute element, the input queue comprising at least in part the coupling with the fabric router; and
wherein each compute element is further enabled, responsive to the operand type being the fabric operand type and the at least one respective operand being a destination, to access the operand via writing data elements to an output queue of the compute element, the output queue comprising at least in part the coupling with the fabric router.
40. The system of claim 38 , wherein the at least one of the data structure descriptors is enabled to identify one of a plurality of extended operand descriptors enabled to specify one or more of stride information and dimension information of a four-dimensional memory vector.
41. The system of claim 38 , wherein the at least one of the data structure descriptors is enabled to identify one of a plurality of extended operand descriptors enabled to specify one or more of a start address and an end address of a circular memory buffer and further enabled to specify FIFO or non-FIFO operation of the circular memory buffer.
42. The system of claim 38 , wherein the compute element is enabled to perform an iteration one or more of the plurality of vector instructions via accessing, in accordance with an access pattern described by the at least one of the data structure descriptors, sufficient data elements of a vector for the iteration.
43. The system of claim 42 , wherein the access pattern is one of a fabric vector, a one-dimensional memory vector, a four-dimensional memory vector, and a circular memory buffer.
44. The system of claim 38 , wherein when the operand type is the fabric operand type, the at least one of the data structure descriptors is associated with a fabric virtual channel of the interconnected processing elements.
45. The system of claim 38 , wherein the at least one of the data structure descriptors indicates how many of the data elements of the at least one respective operand to process in parallel.
46. The system of claim 38 , wherein the at least one of the data structure descriptors comprises an indicator of whether to terminate processing responsive at least in part to a control fabric packet, conveying one of the data elements of the at least one respective operand, being received via the fabric.
47. The system of claim 38 , wherein the at least one of the data structure descriptors comprises an indicator of a virtual channel to selectively activate responsive to completion of a first of the plurality of vector instructions.