Techniques for efficiently synchronizing multiple program threads
Various embodiments include a parallel processing computer system that enables parallel instances of a program to synchronize at disparate addresses in memory. When the parallel program instances need to exchange data, the program instances synchronize based on a mask that identifies the program instances that are synchronizing. As each program instance reaches the point of synchronization, the program instance blocks and waits for all other program instances to reach the point of synchronization. When all program instances have reached the point of synchronization, at least one program instance executes a synchronous operation to exchange data. The program instances then continue execution at respective and disparate return addresses.
1. A computer-implemented method for synchronizing a set of threads at disparate addresses, the method comprising:
determining that a first thread included in the set of threads has blocked when executing a first synchronizing instruction located at a first address in a program instruction memory;
determining that a second thread included in the set of threads has blocked when executing a second synchronizing instruction located at a second address in the program instruction memory, wherein the second address is different from the first address;
unblocking the first thread and the second thread;
returning the first thread to the first address to synchronously execute a first instruction included in a first set of instructions subsequent to the first synchronizing instruction; and
returning the second thread to the second address to synchronously execute a second instruction included in a second set of instructions subsequent to the second synchronizing instruction.
2. The computer-implemented method of claim 1 , further comprising executing, via at least one of the first thread nor the second thread, a synchronous operation that includes the first instruction that exchanges data between the first thread and the second thread.
3. The computer-implemented method of claim 2 , wherein the synchronous operation includes the second instruction that comprises an arithmetic instruction, a breakpoint/trap instruction, a branch to relative address instruction, or a jump to absolute address instruction.
4. The computer-implemented method of claim 3 , wherein the first instruction and the second instruction are located between the first synchronizing instruction and an end instruction that terminates the synchronous operation.
5. The computer-implemented method of claim 2 , wherein the synchronous operation includes a program loop that includes the first instruction, and further comprising:
executing a first instance of a portion of the program loop that includes a first instance of the first synchronizing instruction at the first address; and
executing a second instance of the portion of the program loop that includes a second instance of the first synchronizing instruction at a third address.
6. The computer-implemented method of claim 2 , wherein the synchronous operation includes a function that includes the first instruction, and further comprising:
executing a first instance of a portion of the function that includes a first instance of the first synchronizing instruction at the first address; and
executing a second instance of the portion of the function that includes a second instance of the first synchronizing instruction at a third address.
7. The computer-implemented method of claim 1 , further comprising:
adding a first offset associated with the first synchronizing instruction to the first address to generate a first adjusted address;
adding a second offset associated with the second synchronizing instruction to the second address to generate a second adjusted address;
executing, via the first thread, the first instruction located at the first adjusted address; and
executing, via the second thread, the second instruction located at the second adjusted address.
8. The computer-implemented method of claim 1 , further comprising:
identifying a subset of threads included in the set of threads, wherein the subset of threads includes the first thread and the second thread; and
determining that all other threads included in the subset of threads have executed the first synchronizing instruction or the second synchronizing instruction.
9. The computer-implemented method of claim 8 , wherein the subset of threads is identified via a mask stored in a register specified by at least one of the first synchronizing instruction or the second synchronizing instruction.
10. The computer-implemented method of claim 1 , wherein:
the first synchronizing instruction specifies a subset of threads included in the set of threads, and
the second synchronizing instruction specifies all threads included in the set of threads.
11. One or more non-transitory computer-readable media storing program instructions that, when executed by one or more processors, cause the one or more processors to perform steps of:
determining that a first thread included in a set of threads has blocked when executing a first synchronizing instruction located at a first address in a program instruction memory;
determining that a second thread included in the set of threads has blocked when executing a second synchronizing instruction located at a second address in the program instruction memory, wherein the second address is different from the first address;
unblocking the first thread and the second thread;
returning the first thread to the first address to synchronously execute a first instruction included in a first set of instructions subsequent to the first synchronizing instruction; and
returning the second thread to the second address to synchronously execute a second instruction included in a second set of instructions subsequent to the second synchronizing instruction.
12. The one or more non-transitory computer-readable media of claim 11 , further comprising executing, via at least one of the first thread nor the second thread, a synchronous operation that includes the first instruction that exchanges data between the first thread and the second thread.
13. The one or more non-transitory computer-readable media of claim 12 , wherein the synchronous operation includes the second instruction that comprises an arithmetic instruction, a breakpoint/trap instruction, a branch to relative address instruction, or a jump to absolute address instruction.
14. The one or more non-transitory computer-readable media of claim 13 , wherein the first instruction and the second instruction are located between the first synchronizing instruction and an end instruction that terminates the synchronous operation.
15. The one or more non-transitory computer-readable media of claim 12 , wherein the synchronous operation includes a program loop that includes the first instruction, and further comprising:
executing a first instance of a portion of the program loop that includes a first instance of the first synchronizing instruction at the first address; and
executing a second instance of the portion of the program loop that includes a second instance of the first synchronizing instruction at a third address.
16. The one or more non-transitory computer-readable media of claim 12 , wherein the synchronous operation includes a function that includes the first instruction, and further comprising:
executing a first instance of a portion of the function that includes a first instance of the first synchronizing instruction at the first address; and
executing a second instance of the portion of the function that includes a second instance of the first synchronizing instruction at a third address.
17. The one or more non-transitory computer-readable media of claim 11 , further comprising:
adding a first offset associated with the first synchronizing instruction to the first address to generate a first adjusted address;
adding a second offset associated with the second synchronizing instruction to the second address to generate a second adjusted address;
executing, via the first thread, the first instruction located at the first adjusted address; and
executing, via the second thread, the second instruction located at the second adjusted address.
18. The one or more non-transitory computer-readable media of claim 11 , further comprising:
identifying a subset of threads included in the set of threads, wherein the subset of threads includes the first thread and the second thread; and
determining that all other threads included in the subset of threads have executed the first synchronizing instruction or the second synchronizing instruction.
19. The one or more non-transitory computer-readable media of claim 18 , wherein the subset of threads is identified via a mask stored in a register specified by at least one of the first synchronizing instruction or the second synchronizing instruction.
20. A system, comprising:
a memory storing instructions; and
a processor that is coupled to the memory and, when executing the instructions:
determines that a first thread included in a set of threads has blocked when executing a first synchronizing instruction located at a first address in a program instruction memory;
determines that a second thread included in the set of threads has blocked when executing a second synchronizing instruction located at a second address in the program instruction memory, wherein the second address is different from the first address;
unblocks the first thread and the second thread;
returns the first thread to the first address to synchronously execute a first instruction included in a first set of instructions subsequent to the first synchronizing instruction; and
returns the second thread to the second address to synchronously execute a second instruction included in a second set of instructions subsequent to the second synchronizing instruction.