Task synchronization for accelerated deep learning
Techniques in advanced deep learning provide improvements in one or more of accuracy, performance, and energy efficiency. An array of processing elements performs flow-based computations on wavelets of data. Each processing element has a compute element and a routing element. Each compute element has memory. Each router enables communication via wavelets with at least nearest neighbors in a 2D mesh. Routing is controlled by respective virtual channel specifiers in each wavelet and routing configuration information in each router. A compute element conditionally selects for task initiation a previously received wavelet specifying a particular one of the virtual channels. The conditional selecting excludes the previously received wavelet for selection until at least block/unblock state maintained for the particular virtual channel is in an unblock state. The compute element executes block/unblock instructions to modify the block/unblock state.
1. A method comprising:
sending a first fabric packet by a sending processing element, the first fabric packet comprising a virtual channel specifier specifying one of a plurality of virtual channels;
routing the first fabric packet from the sending processing element to a receiving processing element via one or more of routing elements in accordance with the virtual channel specifier;
in the receiving processing element, picking for processing the first fabric packet, the picking in accordance with a respective block/unblock state comprised in the receiving processing element and maintained for each of the plurality of virtual channels, the picking excluding the first fabric packet from being picked until at least the respective block/unblock state maintained for the specified virtual channel is in the unblock state;
wherein the sending processing element implements at least a portion of a first node of a plurality of nodes of a dataflow graph and the receiving processing element implements at least a portion of a second node of the dataflow graph, and the sending processing element and the receiving processing element are respective instances of a plurality of processing elements interconnected as a fabric; and
wherein the first fabric packet is an instance of a plurality of fabric packets and the routing of the first fabric packet is an instance of routing between the processing elements of the fabric, and wherein the routing between the processing elements of the fabric comprises each processing element transmitting each of one or more of the plurality of fabric packets over a selected group of preconfigured groups of one or more physical couplings between connected neighbors of the processing element, and the virtual channel specifier of each transmitted fabric packet is used to select the group.
2. The method of claim 1 , further comprising:
receiving in the receiving processing element a second fabric packet, the second fabric packet enabling execution of a selected one of a block instruction and an unblock instruction, the selected instruction having been stored in the receiving processing element prior to the receiving and comprising an immediate source operand specifying one or more of the virtual channels;
decoding the selected instruction in the receiving processing element; and
in the receiving processing element, setting the respective block/unblock state maintained for the specified one or more of the virtual channels in accordance with the decoding.
3. The method of claim 1 , further comprising:
receiving in the receiving processing element a second fabric packet of the fabric packets, the second fabric packet enabling execution of a selected one of a block instruction and an unblock instruction, the selected instruction having been stored in the receiving processing element prior to the receiving and comprising other than an immediate source operand;
decoding the selected instruction in the receiving processing element; and
in the receiving processing element, setting the respective block/unblock state maintained for all the virtual channels in accordance with the decoding.
4. The method of claim 1 , further comprising, in the receiving processing element, setting the respective block/unblock state maintained for a particular virtual channel of the virtual channels to a blocked state in response to a block instruction specifying the particular virtual channel of the virtual channels.
5. The method of claim 1 , further comprising, in the receiving processing element, setting the respective block/unblock state maintained for a particular virtual channel of the virtual channels to an unblocked state in response to an unblock instruction specifying the particular virtual channel of the virtual channels.
6. The method of claim 1 , wherein the sending processing element and the receiving processing element are fabricated via wafer-scale integration on separate die of a single wafer.
7. The method of claim 1 , wherein the sending processing element implements at least a portion of a first neuron of a neural network and the receiving processing element implements at least a portion of a second neuron of the neural network.
8. The method of claim 1 , wherein the sending processing element implements at least a portion of a first layer of a neural network and the receiving processing element implements at least a portion of a second layer of the neural network.
9. The method of claim 1 , wherein the sending processing element and the receiving processing element implement respective portions of at least a partitioned neuron of a neural network.
10. The method of claim 1 , further comprising, with respect to the receiving processing element, managing fabric packet input queues to have generally equal average rates of production and consumption by stalling/resuming task activities via manipulation of the respective block/unblock state.
11. The method of claim 1 , further comprising, with respect to the receiving processing element, managing one or more priorities within and between tasks by stalling/resuming task activities via manipulation of the respective block/unblock state.
12. The method of claim 1 , further comprising, with respect to the receiving processing element, managing one or more dependencies within and between tasks by stalling/resuming task activities via manipulation of the respective block/unblock state.
13. The method of claim 1 , further comprising, with respect to the receiving processing element, synchronizing one or more of computations and communications of one or more tasks, via manipulation of the respective block/unblock state.
14. The method of claim 1 , further comprising, with respect to the receiving processing element, implementing task software interlocks via manipulation of the respective block/unblock state.
15. The method of claim 1 , further comprising synchronizing data sourced via unequal delay paths by manipulation of the respective block/unblock state of one or more of the processing elements along the delay paths.
16. The method of claim 1 , further comprising shaping at least some dataflow in at least part of the fabric by manipulation of the respective block/unblock state of one or more of the processing elements.
17. The method of claim 1 , further comprising:
wherein the one of the plurality of virtual channels is used for communicating at least one of control and data associated with one or more of: computing an activation of a neural network, computing a partial sum of activations of the neural network, computing an error of the neural network, computing a gradient estimate of the neural network, and updating a weight of the neural network; and
wherein the first fabric packet comprises the at least one of control and data associated with the one or more of: computing the activation of the neural network, computing the partial sum of activations of the neural network, computing the error of the neural network, computing the gradient estimate of the neural network, and updating the weight of the neural network.
18. The method of claim 1 , wherein the sending processing element, the one or more of the routing elements, and the receiving processing element are fabricated via wafer-scale integration.
19. The method of claim 1 , further comprising:
initializing the fabric with all parameters and task software required for concurrent execution of communications and computations respectively corresponding to the dataflow graph; and
concurrently executing all layers of the dataflow graph using all-digital techniques for one or more of inference and training.
20. The method of claim 19 , wherein except for defects, the fabric is homogeneous, the plurality of processing elements numbers in the hundreds of thousands, each processing element comprises in the tens of kB of private local storage for instructions and data, and the parameters and task software comprises in the tens of GB of instruction and data storage for the one or more of inference and training.
21. The method of claim 19 , wherein the concurrently executing does not require any access to storage external to the fabric for any intermediate state or additional parameters of the dataflow graph.
22. The method of claim 19 , wherein the dataflow graph is a neural network, the plurality of nodes corresponds to neurons, and at least some of parameters of the dataflow graph correspond to a plurality of weights of the neural network.
23. The method of claim 1 , further comprising:
wherein the processing comprises selective dataflow-based processing and selective instruction-based processing;
wherein the selective dataflow-based processing comprises the routing and the picking, each in accordance with the virtual channel specifier; and
wherein the selective instruction-based processing is in accordance with a starting address for fetching instructions executable by the processing elements, the starting address identified at least in part by a selected one of the virtual channel specifier and a task specifier of the first fabric packet.
24. An apparatus comprising:
means for sending a first fabric packet by a sending processing element, the first fabric packet comprising a virtual channel specifier specifying one of a plurality of virtual channels;
means for routing the first fabric packet from the sending processing element to a receiving processing element via one or more of routing elements in accordance with the virtual channel specifier;
in the receiving processing element, means for picking for processing the first fabric packet, the picking in accordance with a respective block/unblock state comprised in the receiving processing element and maintained for each of the plurality of virtual channels, the picking excluding the first fabric packet from being picked until at least the respective block/unblock state maintained for the specified virtual channel is in the unblock state;
wherein the sending processing element implements at least a portion of a first node of a plurality of nodes of a dataflow graph and the receiving processing element implements at least a portion of a second node of the dataflow graph, and the sending processing element and the receiving processing element are respective instances of a plurality of processing elements interconnected as a fabric; and
wherein the first fabric packet is an instance of a plurality of fabric packets and the means for routing of the first fabric packet is a part of means for routing between the processing elements of the fabric, and wherein the means for routing between the processing elements of the fabric comprises means for each processing element transmitting each of one or more of the plurality of fabric packets over a selected group of preconfigured groups of one or more physical couplings between connected neighbors of the processing element, and the virtual channel specifier of each transmitted fabric packet is used to select the group.
25. The apparatus of claim 24 , further comprising:
means for receiving in the receiving processing element a second fabric packet, the second fabric packet enabling execution of a selected one of a block instruction and an unblock instruction, the selected instruction having been stored in the receiving processing element prior to the receiving and comprising an immediate source operand specifying one or more of the virtual channels;
means for decoding the selected instruction in the receiving processing element; and
in the receiving processing element, means for setting the respective block/unblock state maintained for the specified one or more of the virtual channels in accordance with the decoding.
26. The apparatus of claim 24 , further comprising:
means for receiving in the receiving processing element a second fabric packet of the fabric packets, the second fabric packet enabling execution of a selected one of a block instruction and an unblock instruction, the selected instruction having been stored in the receiving processing element prior to the receiving and comprising other than an immediate source operand;
means for decoding the selected instruction in the receiving processing element; and
in the receiving processing element, means for setting the respective block/unblock state maintained for all the virtual channels in accordance with the decoding.
27. The apparatus of claim 24 , further comprising, in the receiving processing element, means for setting the respective block/unblock state maintained for a particular virtual channel of the virtual channels to a blocked state in response to a block instruction specifying the particular virtual channel of the virtual channels.
28. The apparatus of claim 24 , further comprising, in the receiving processing element, means for setting the respective block/unblock state maintained for a particular virtual channel to an unblocked state in response to an unblock instruction specifying the particular virtual channel of the virtual channels.
29. The apparatus of claim 24 , wherein the sending processing element and the receiving processing element are fabricated via wafer-scale integration on separate die of a single wafer.
30. The apparatus of claim 24 , wherein the sending processing element implements at least a portion of a first neuron of a neural network and the receiving processing element implements at least a portion of a second neuron of a neural network.
31. The apparatus of claim 24 , wherein the sending processing element implements at least a portion of a first layer of a neural network and the receiving processing element implements at least a portion of a second layer of a neural network.
32. The apparatus of claim 24 , wherein the sending processing element and the receiving processing element implement respective portions of at least a partitioned neuron of a neural network.
33. A method comprising:
specifying communications and computations respectively corresponding to a plurality of nodes of a dataflow graph, wherein a sending processing element implements at least a portion of a first node of the dataflow graph and a receiving processing element implements at least a portion of a second node of the dataflow graph, and the sending processing element and the receiving processing element are respective instances of a plurality of processing elements interconnected as a fabric;
initializing the fabric with all parameters and task software required for a concurrent execution of the communications and computations respectively corresponding to the dataflow graph;
sending a first fabric packet by the sending processing element, the first fabric packet comprising a virtual channel specifier specifying one of a plurality of virtual channels;
routing the first fabric packet from the sending processing element to the receiving processing element via one or more of routing elements in accordance with the virtual channel specifier;
in the receiving processing element, the virtual channel specifier of the first fabric packet identifying a respective one of a plurality of queues of the receiving processing element, the respective queue holding at least a task specifier of the first fabric packet; and
in the receiving processing element, picking for processing the first fabric packet, the picking in accordance with a respective block/unblock state maintained for each of the plurality of queues of the receiving processing element, the picking excluding the first fabric packet from being picked until at least the respective block/unblock state maintained for the respective queue is in the unblock state.
34. The method of claim 33 , further comprising, in the receiving processing element, setting the respective block/unblock state maintained for a particular virtual channel of the virtual channels to a blocked state in response to a block instruction specifying the particular virtual channel of the virtual channels.
35. The method of claim 34 , wherein the block instruction is an instance of the task software stored in the receiving processing element by the initializing.
36. The method of claim 34 , wherein the block instruction is associated with a previously stored fabric packet received via the fabric.
37. The method of claim 33 , further comprising, in the receiving processing element, setting the respective block/unblock state maintained for a particular virtual channel to an unblocked state in response to an unblock instruction specifying the particular virtual channel of the virtual channels.
38. The method of claim 37 , wherein the unblock instruction is an instance of the task software stored in the receiving processing element by the initializing.
39. The method of claim 37 , wherein the unblock instruction is associated with a previously stored fabric packet received via the fabric.
40. The method of claim 33 , further comprising, with respect to the receiving processing element, managing one or more dependencies within and between tasks by stalling/resuming task activities via manipulation of the respective block/unblock state.
41. The method of claim 33 , further comprising, with respect to the receiving processing element, synchronizing one or more of computations and communications of one or more tasks, via manipulation of the respective block/unblock state.
42. The method of claim 33 , further comprising, with respect to the receiving processing element, implementing task software interlocks via manipulation of the respective block/unblock state.
43. The method of claim 33 , further comprising synchronizing data sourced via unequal delay paths by manipulation of the respective block/unblock state of one or more of the processing elements along the delay paths.
44. The method of claim 33 , further comprising shaping at least some dataflow in at least part of the fabric by manipulation of the respective block/unblock state of one or more of the processing elements.
45. The method of claim 33 , further comprising:
wherein the dataflow graph is a neural network for one or more of inference and training, the plurality of nodes corresponds to neurons, and at least some of parameters of the dataflow graph correspond to a plurality of weights of the neural network; and
concurrently executing all layers of the dataflow graph using all-digital techniques, the concurrently executing comprising the sending, routing, identifying, and picking.
46. The method of claim 33 , wherein the sending processing element implements at least a portion of a first neuron of a neural network and the receiving processing element implements at least a portion of a second neuron of the neural network.
47. The method of claim 33 , wherein the sending processing element implements at least a portion of a first layer of a neural network and the receiving processing element implements at least a portion of a second layer of the neural network.
48. The method of claim 33 , wherein the sending processing element and the receiving processing element implement respective portions of at least a partitioned neuron of a neural network.