Wavelet representation for accelerated deep learning
Techniques in advanced deep learning provide improvements in one or more of accuracy, performance, and energy efficiency. An array of processing elements performs flow-based computations on wavelets of data. Each processing element has a compute element with dedicated storage and a routing element. Each router enables communication with nearest neighbors in a 2D mesh. The communication is via wavelets in accordance with a representation comprising an index specifier, a virtual channel specifier, a task specifier, a data element specifier, and an optional control/data specifier. The virtual channel specifier and the task specifier are associated with one or more instructions. The index specifier and the data element are optionally associated with operands of the one or more instructions.
1. A method comprising:
sending a wavelet from a sending processing element to a receiving processing element via a fabric coupling the sending processing element and the receiving processing element, the wavelet comprising a color field, a control field, an optional index field, and a data field, wherein the sending is in accordance with a virtual channel specified by the color field, and the control field is enabled to indicate a mutually exclusive one of a control state and a data state;
in the receiving processing element and responsive to the wavelet being chosen for processing, conditionally executing one or more instructions beginning at an instruction address;
in the receiving processing element and responsive to the control field indicating the control state, conditionally determining the instruction address based at least in part on the optional index field;
in the receiving processing element and responsive to the control field indicating the data state, conditionally determining the instruction address based at least in part on the color field; and
in the receiving processing element and responsive to the wavelet being chosen for processing, conditionally using information from the data field as an input operand to an operation performed by the receiving processing element.
2. The method of claim 1 , wherein the conditionally executing is dependent on a wavelet-indicator enabled to indicate a mutually exclusive one of a task-initiation state and a data-processing state, and further comprising, in the receiving processing element, setting the wavelet-indicator to the task-initiation state responsive to executing a terminate instruction and setting the wavelet-indicator to the data-processing state responsive to initiating a task.
3. The method of claim 2 , wherein the wavelet is a first wavelet, the instruction address is a first instruction address, and the initiating the task comprises determining a second instruction address based at least in part on a second wavelet received by the receiving processing element.
4. The method of claim 2 , wherein the wavelet is a first wavelet, the control field is a first control field; further comprising, in the receiving processing element, processing an operand descriptor comprising a field; and wherein the field indicates that receiving, associated with the operand descriptor, a second wavelet comprising a second color field indicating the control state, is to result in the wavelet-indicator being set to the task-initiation state.
5. The method of claim 1 , wherein the information is in accordance with one of a floating-point value and an integer value.
6. The method of claim 1 , wherein the wavelet is in accordance with one of a plurality of mutually exclusive formats comprising a sparse format and a dense format, the sparse format is in accordance with the wavelet comprising the optional index field and the data field comprising a single data element, the dense format is in accordance with the data field comprising a plurality of data elements;
and further comprising, in the receiving processing element, accessing a data-indicator to determine which of the formats the wavelet is in accordance with.
7. The method of claim 6 , wherein the data-indicator is enabled to indicate a mutually exclusive one of a single-instruction-single-data state and a single-instruction-multiple-data state.
8. The method of claim 7 , wherein the data-indicator is set in accordance with information from an operand descriptor of a particular instruction.
9. The method of claim 6 , further comprising, when the wavelet is in accordance with the sparse format, the conditionally using information from the data field comprises using the single data element as the input operand.
10. The method of claim 6 , further comprising, in the receiving processing element and when the wavelet is in accordance with the sparse format, using information from the optional index field at least in part to determine one or more of: an identifier of a neuron of a neural network, an identifier of a portion of a vector, and an identifier of a portion of a matrix.
11. The method of claim 10 , wherein the portion of the vector comprises an element of the vector.
12. The method of claim 10 , wherein the portion of the matrix comprises an element of the matrix.
13. The method of claim 6 , further comprising, when the wavelet is in accordance with the dense format, the conditionally using information from the data field comprises using the plurality of data elements as the input operand.
14. The method of claim 13 , wherein the operation is in accordance with SIMD processing.
15. The method of claim 1 , wherein the conditionally determining the instruction address based at least in part on the index field and the conditionally determining the instruction address based at least in part on the color field are based at least in part on a same base register.
16. The method of claim 1 , wherein the conditionally determining the instruction address based at least in part on the color field comprises interpreting contents of the color field as an integer and computing an integer multiple of the integer.
17. The method of claim 1 , wherein the wavelet comprises a data field and at least one of the one or more instructions uses at least a portion of the data field as an operand.
18. The method of claim 1 , further comprising, in the receiving processing element, receiving within a same clock cycle the color field, the control field, and the optional index field.
19. The method of claim 1 , wherein the sending processing element and the receiving processing element are fabricated via wafer-scale integration.
20. The method of claim 1 , wherein the sending processing element and the receiving processing element implement at least portions of a neuron of a neural network.
21. The method of claim 1 , wherein the virtual channel implements at least a portion of connectivity between neurons of a neural network.
22. A system comprising:
means for sending a wavelet from a sending processing element to a receiving processing element via a fabric coupling the sending processing element and the receiving processing element, the wavelet comprising a color field, a control field, an optional index field, and a data field, wherein the means for sending is operable in accordance with a virtual channel specified by the color field, and the control field is enabled to indicate a mutually exclusive one of a control state and a data state;
in the receiving processing element and responsive to the wavelet being chosen for processing, means for conditionally executing one or more instructions beginning at an instruction address;
in the receiving processing element and responsive to the control field indicating the control state, means for conditionally determining the instruction address based at least in part on the optional index field;
in the receiving processing element and responsive to the control field indicating the data state, means for conditionally determining the instruction address based at least in part on the color field; and
in the receiving processing element and responsive to the wavelet being chosen for processing, means for conditionally using information from the data field as an input operand to an operation performed by the receiving processing element.
23. The system of claim 22 , wherein the means for conditionally executing is dependent on a wavelet-indicator enabled to indicate a mutually exclusive one of a task-initiation state and a data-processing state, and further comprising, in the receiving processing element, means for setting the wavelet-indicator to the task-initiation state responsive to executing a terminate instruction and setting the wavelet-indicator to the data-processing state responsive to initiating a task.
24. The system of claim 23 , wherein the wavelet is a first wavelet, the instruction address is a first instruction address, and the means for initiating the task comprises means for determining a second instruction address based at least in part on a second wavelet received by the receiving processing element.
25. The system of claim 23 , wherein the wavelet is a first wavelet, the control field is a first control field; further comprising, in the receiving processing element, means for processing an operand descriptor comprising a field; and wherein the field indicates that receiving, associated with the operand descriptor, a second wavelet comprising a second color field indicating the control state, is to result in the wavelet-indicator being set to the task-initiation state.
26. The system of claim 22 , wherein the information is in accordance with one of a floating-point value and an integer value.
27. The system of claim 22 , wherein the wavelet is in accordance with one of a plurality of mutually exclusive formats comprising a sparse format and a dense format, the sparse format is in accordance with the wavelet comprising the optional index field and the data field comprising a single data element, the dense format is in accordance with the data field comprising a plurality of data elements; and further comprising, in the receiving processing element, means for accessing a data-indicator to determine which of the formats the wavelet is in accordance with.
28. The system of claim 27 , wherein the data-indicator is enabled to indicate a mutually exclusive one of a single-instruction-single-data state and a single-instruction-multiple-data state.
29. The system of claim 28 , wherein the data-indicator is set in accordance with information from an operand descriptor of a particular instruction.
30. The system of claim 27 , wherein when the wavelet is in accordance with the sparse format, the means for conditionally using information from the data field uses the single data element as the input operand.
31. The system of claim 27 , further comprising, in the receiving processing element and when the wavelet is in accordance with the sparse format, means for using information from the optional index field at least in part to determine one or more of: an identifier of a neuron of a neural network, an identifier of a portion of a vector, and an identifier of a portion of a matrix.
32. The system of claim 31 , wherein the portion of the vector comprises an element of the vector.
33. The system of claim 31 , wherein the portion of the matrix comprises an element of the matrix.
34. The system of claim 27 , wherein when the wavelet is in accordance with the dense format, the means for conditionally using information from the data field uses the plurality of data elements as the input operand.
35. The system of claim 34 , wherein the operation is in accordance with SIMD processing.
36. The system of claim 22 , wherein the means for conditionally determining the instruction address based at least in part on the index field and the means for conditionally determining the instruction address based at least in part on the color field are based at least in part on a same base register.
37. The system of claim 22 , wherein the means for conditionally determining the instruction address based at least in part on the color field comprises means for interpreting contents of the color field as an integer and means for computing an integer multiple of the integer.
38. The system of claim 22 , wherein the wavelet comprises a data field and at least one of the one or more instructions uses at least a portion of the data field as an operand.
39. The system of claim 22 , further comprising, in the receiving processing element, means for receiving within a same clock cycle the color field, the control field, and the optional index field.
40. The system of claim 22 , wherein the sending processing element and the receiving processing element are fabricated via wafer-scale integration.
41. The system of claim 22 , wherein the sending processing element and the receiving processing element implement at least portions of a neuron of a neural network.
42. The system of claim 22 , wherein the virtual channel implements at least a portion of connectivity between neurons of a neural network.