Techniques for recovering from errors when executing software applications on parallel processors

1. A computer-implemented method for checkpointing a context associated with an execution of a software application on a parallel processor, the method comprising:

determining that a kernel executing on a plurality of parallel processing elements included in the parallel processor is tagged to indicate that the kernel is enabled for intra-kernel checkpointing and restart;

causing the plurality of parallel processing elements to stop executing a first plurality of instructions included in the kernel in accordance with the context before executing a next instruction included in the first plurality of instructions;

causing the parallel processor to collect first state data associated with the context;

generating a checkpoint based on the first state data, wherein the checkpoint is stored in a memory associated with the parallel processor; and

causing the plurality of parallel processing elements to resume executing the first plurality of instructions included in the kernel at the next instruction in accordance with the context.

2. The computer-implemented method of claim 1 , wherein causing the plurality of parallel processing elements to stop executing the first plurality of instructions comprises transmitting a control call to the parallel processor to preempt the context at an instruction level.

3. The computer-implemented method of claim 1 , wherein the first plurality of instructions comprises a kernel included in the software application.

4. The computer-implemented method of claim 1 , further comprising:

while the plurality of parallel processing elements is executing a second plurality of instructions, determining that the context is to be restarted based on the checkpoint;

causing the plurality of parallel processing elements to stop executing the second plurality of instructions; and

causing the plurality of parallel processing elements to restart executing the first plurality of instructions at the next instruction in accordance with the checkpoint.

5. The computer-implemented method of claim 1 , wherein causing the plurality of parallel processing elements to resume executing the first plurality of instructions comprises transmitting a control call to the parallel processor to re-enable the plurality of parallel processing elements.

6. The computer-implemented method of claim 1 , further comprising, prior to causing the plurality of parallel processing elements to stop executing the first plurality of instructions, preventing the parallel processor from scheduling a second plurality of instructions in accordance with a second context on the plurality of parallel processing elements.

7. The computer-implemented method of claim 1 , wherein causing the plurality of parallel processing elements to stop executing the first plurality of instructions further causes the parallel processor to collect second state data associated with the context.

8. The computer-implemented method of claim 7 , wherein generating the checkpoint comprises copying the first state data and the second state data to the checkpoint.

9. The computer-implemented method of claim 1 wherein the first state data is associated with the plurality of parallel processing elements and comprises at least one of a per-thread register state or a shared memory state.

10. The computer-implemented method of claim 1 , wherein the parallel processor comprises a parallel processing unit, a graphics processing unit, a tensor processing unit, or a multi-core central processing unit.

11. One or more non-transitory computer readable media including instructions that, when executed by one or more processors, cause the one or more processors to checkpoint a context associated with an execution of a software application on a parallel processor by performing steps of:

determining that a kernel executing on a plurality of parallel processing elements included in the parallel processor is tagged to indicate that the kernel is enabled for intra-kernel checkpointing and restart;

causing the plurality of parallel processing elements to stop executing a first plurality of instructions included in the kernel in accordance with the context before executing a next instruction included in the first plurality of instructions;

causing the parallel processor to collect first state data associated with the context;

generating a checkpoint based on the first state data, wherein the checkpoint is stored in a memory associated with the parallel processor; and

causing the plurality of parallel processing elements to resume executing the first plurality of instructions included in the kernel at the next instruction in accordance with the context.

12. The one or more non-transitory computer readable media of claim 11 , wherein causing the plurality of parallel processing elements to stop executing the first plurality of instructions comprises transmitting a control call to the parallel processor to preempt the context at an instruction level.

13. The one or more non-transitory computer readable media of claim 11 , wherein the first plurality of instructions comprises a kernel included in the software application.

14. The one or more non-transitory computer readable media of claim 11 , further comprising:

while the plurality of parallel processing elements is executing a second plurality of instructions, determining that the context is to be restarted based on the checkpoint;

causing the plurality of parallel processing elements to stop executing the second plurality of instructions; and

causing the plurality of parallel processing elements to restart executing the first plurality of instructions at the next instruction in accordance with the checkpoint.

15. The one or more non-transitory computer readable media of claim 14 , wherein the second plurality of instructions is associated with the context or a different context.

16. The one or more non-transitory computer readable media of claim 11 , further comprising, prior to causing the plurality of parallel processing elements to stop executing the first plurality of instructions, determining that the context is to be checkpointed based on a checkpoint interval.

17. The one or more non-transitory computer readable media of claim 11 , wherein causing the plurality of parallel processing elements to stop executing the first plurality of instructions further causes the parallel processor to collect second state data associated with the context.

18. The one or more non-transitory computer readable media of claim 17 , wherein generating the checkpoint comprises copying the first state data and the second state data to the checkpoint.

19. The one or more non-transitory computer readable media of claim 17 , wherein the second state data comprises at least one of a privileged register state or a flip-flop state.

20. A system comprising:

a parallel processing memory storing a kernel;

a parallel processor coupled to the parallel processing memory;

one or more memories storing instructions; and

one or more primary processors coupled to the one or more memories that, when executing the instructions:

determines that the kernel executing on a plurality of parallel processing elements included in the parallel processor is tagged to indicate that the kernel is enabled for intra-kernel checkpointing and restart;

causing the plurality of parallel processing elements to stop executing a first plurality of instructions included in the kernel in accordance with a context before executing a next instruction included in the kernel;

causing the parallel processor to collect state data associated with the context;

generating a checkpoint based on the state data, wherein the checkpoint is stored in the parallel processing memory; and

causing the plurality of parallel processing elements to resume executing the first plurality of instructions included in the kernel at the next instruction in accordance with the checkpoint.