Software-defined tensor streaming multiprocessor for large-scale machine learning
A system contains a network of processors arranged in a plurality of nodes. Each node comprises a respective plurality of processors connected via local links, and different nodes are connected via global links. The processors of the network communicate with each other to establish a global counter for the network, enabling deterministic communication between the processors of the network. A compiler is configured to explicitly schedule communication traffic across the global and local links of the network of processors based upon the deterministic links between the processors, which enable software-scheduled networking with explicit send or receive instructions executed by functional units of the processors at specific times, to establish a specific ordering of operations performed by the network of processors. In some embodiments, the processors of the network of processors are tensor streaming processors (TSPs).
1. A system comprising:
a network of processors, comprising:
a plurality of processors arranged in a plurality of nodes, comprising at least a first node having a first plurality of deterministic processors connected by local links;
wherein the plurality of nodes are connected by global links; and wherein the plurality of processors communicate with each other to establish a global counter for the network, enabling deterministic communication between the plurality of processors of the network; and
a compiler to explicitly schedule communication traffic across the global and local links with explicit send or receive instructions executed at specific times to establish a specific ordering of operations performed by the network of processors.
2. The system of claim 1 , wherein each processor of the network of processors is a deterministic processor.
3. The system of claim 2 , wherein each deterministic processor comprises one or more deterministic data paths connecting rows of deterministic functional units, each functional unit configured to process data received from, or output processed data onto, at least one data path, in accordance with a plurality of instructions which are executed at a known clock cycle.
4. The system of claim 1 , wherein each of the plurality of processors comprises n local links and m global links, and wherein the first plurality of processors comprises n+1 processors, wherein each of the n+1 processors is connected via a respective local link to each other processor of the first node, and wherein each local link comprises a synchronous link.
5. The system of claim 4 , wherein the plurality of nodes comprises up to ((n+1)*m)+1 nodes, and wherein each of the up to ((n+1)*m)+1 nodes is connected to each remaining node of the up to ((n+1)*m)+1 nodes via a respective global link and wherein each global link comprises a synchronous link.
6. The system of claim 1 , wherein the plurality of nodes are grouped into racks each having a set of global links, each rack having a set of nodes connected to each other via a first portion of the set of global links, and is connected to each remaining rack of the plurality of racks via a second portion of the set of global links.
7. The system of claim 6 , wherein the set of nodes within each rack are doubly connected to each other via the first portion of the set of global links, and wherein the racks are singly connected via the second portion of the set of global links.
8. The system of claim 6 , wherein the set of nodes of each rack includes a hot spare node, and wherein, responsive to detection of a critical error, the system is configured to perform a runtime reply of an inference during which the critical error was detected, where computation of a node of a rack associated with the detected critical error is switched to the hot spare node.
9. The system of claim 1 , wherein each processor of the plurality of processors implements a first counter, wherein the first counter of each of the plurality of processors is synchronized with those of other processors of the plurality of processors to establish the global counter for the network.
10. The system of claim 9 , wherein synchronizing the first counter of a first processor with the first counter of a second processor of the plurality of processors comprises:
transmitting, at a first time, a first value of the first counter of the first processor corresponding to the first time to the second processor, wherein the second processor is configured to, upon receiving the first value of the first counter from the first processor, reflect the first value back to the first processor;
determining a latency value of a link connecting the first processor and the second processor, based upon a difference between the first value of the first counter with a second value of the first counter of the first processor corresponding to a second time at which the reflected first value is received by the first processor;
transmitting, at a third time, a third value of the first counter of the first processor corresponding to the third time; and
responsive to receiving the third value at the second processor, adjusting a value of the first counter of the second processor based upon the third value and the determined latency value.
11. The system of claim 10 , wherein the first processor is configured to transmit a value of its first counter to the second processor periodically.
12. The system of claim 9 , wherein a first processor and a second processor of the plurality of processors are configured to perform an initial program alignment operation, comprising:
placing the second processor into a synchronization loop, wherein the second processor checks for an instruction transmitted from the first processor at each overflow boundary of the first counter of the second processor;
at a first time corresponding to an overflow boundary of the first counter of the first processor, transmitting an instruction from the first processor to the second processor;
causing the second processor to exit the synchronization loop and begin synchronized computation at an overflow boundary of the first counter of the second processor occurring after the instruction from the first processor is received; and
causing the first processor to begin synchronized computation at an overflow boundary of the first counter of the first processor aligned with the overflow boundary of the first counter of the second processor.
13. The system of claim 9 , wherein each processor of the plurality of processors further implements a second counter, and is configured to periodically delay for a target number of cycles based upon a difference between the first counter and the second counter.
14. The system of claim 1 , wherein the plurality of processors are configured to operate based upon a compiled program, wherein the compiled program includes an explicit schedule of instructions causing the plurality of processors to communicate with each other with a predetermined timing.
15. The system of claim 14 , wherein the compiled program schedules a transmission of data between a first processor and a second processor of the plurality of processors by scheduling a first instruction at the first processor configured to cause a functional unit of the first processor to read data onto a data path of the first processor at a first time, and a second instruction at the second processor configured to cause a functional unit of the second processor to consume data on a data path of the second processor at a second time.
16. The system of claim 15 , wherein the transmission occurs without the second processor sending a communication to the first processor to request the transmission of data.
17. The system of claim 14 , wherein the compiled program is generated by a compiler based on a model, and wherein the compiler schedules each instruction of the schedule of instructions to prevent routing deadlock on the local and global links due to transmission of data between the processors of the network, based on a topology of the network and determined latency values of the local and global links.
18. The system of claim 17 , where the compiler is configured to divide the compiled program into a plurality of sub-programs, and assigns each sub-program to a respective processor of the plurality of processors.
19. The system of claim 18 , wherein the compiler is configured to select the respective processors of the plurality of processors by identifying a pinch point within the model corresponding to a portion of the model with a reduced volume of data communication, and assigning sub-programs corresponding to different sides of pinch point to respective processors of different nodes of the plurality of nodes.
20. The system of claim 17 , wherein the compiler is configured to schedule a transmission of data between a first processor and a second processor of the network by dividing the data into a plurality of data portions, and scheduling instructions to cause the first processor to transmit each of the plurality of data portions via a different non-minimal path to the second processor.
21. The system of claim 20 , wherein each non-minimal path comprises at least a first link connecting the first processor to one or more intermediate processors, and a second link connecting the one or more intermediate processors to the second processor.