Methods, systems and devices for restarting data processing units
Systems and methods are disclosed for maintaining a status of a respective data processing unit (DPU) of a plurality of data processing units, each coupled to non-volatile memory. In some embodiments a first DPU is configured to execute one or more persistent processes, wherein the one or more processes persist in the non-volatile memory over power cycles, generate a first broadcast message upon completion of a first persistent process, transmit the first broadcast message to a set of DPUs associated with monitoring the first DPU, receive a second broadcast message from a second DPU of the set of DPUs and assign a value indicating an active status for the second DPU in a status table to track the status of each DPU of the set of DPUs.
1. A computing system comprising:
a plurality of data processing units (DPUs) each coupled to non-volatile memory, wherein a first DPU of the plurality of DPUs is configured to:
execute one or more persistent processes, wherein the one or more processes persist in the non-volatile memory over power cycles;
generate a first broadcast message upon completion of a first persistent process;
transmit the first broadcast message to a set of DPUs associated with monitoring the first DPU;
receive a second broadcast message from a second DPU of the set of DPUs;
assign a value indicating an active status for the second DPU in a status table to track a status of each DPU of the set of DPUs;
determine a failure to receive the second broadcast message from the second DPU after a period of time exceeding a timing threshold;
assign a value indicating an inactive status for the second DPU in the status table, and
transmit a status inquiry request to the second DPU in response to determining the failure to receive the second broadcast message after a period of time exceeding a timing threshold.
2. The computing system of claim 1 , wherein the first broadcast message indicates an active state of the first DPU.
3. The computing system of claim 1 , wherein the first DPU is further configured to:
transmit a restart command to the second DPU in response to determining the failure to receive a response to the status inquiry request.
4. The computing system of claim 1 , wherein the first DPU is further configured to:
determine a failure to receive a response to the status inquiry request from the second DPU after a period of time exceeding the timing threshold; and
transmit a restart command to the second DPU in response to determining the failure to receive the response to the status inquiry request.
5. The computing system of claim 1 , wherein the first DPU is further configured to:
receive a response to the status inquiry request from the second DPU; and
assign the value indicating an active status for the second DPU in the status table to track the status of each DPU of the set of DPUs.
6. The computing system of claim 1 , wherein the first DPU is further configured to:
receive a restart command; and
perform a self-restart operation in response to receiving the restart command.
7. The computing system of claim 6 , wherein the first DPU is further configured to:
transmit a third broadcast message to the set of DPUs associated with monitoring the first DPU upon successful completion of the self-restart operation.
8. A first data processing unit (DPU) of a plurality of DPUs, each coupled to non-volatile memory, wherein the first DPU is configured to:
execute one or more persistent processes, wherein the one or more processes persist in the non-volatile memory over power cycles;
generate a first broadcast message upon completion of a first persistent process;
transmit the first broadcast message to a set of DPUs associated with monitoring the first DPU;
receive a second broadcast message from a second DPU of the set of DPUs;
assign a value indicating an active status for the second DPU in a status table to track a status of each DPU of the set of DPUs;
determine a failure to receive the second broadcast message from the second DPU after a period of time exceeding a timing threshold;
assign a value indicating an inactive status for the second DPU in the status table; and
transmit a status inquiry request to the second DPU in response to determining the failure to receive the second broadcast message after a period of time exceeding a timing threshold.
9. The first DPU of claim 8 , wherein the first broadcast message indicates an active state of the first DPU.
10. The first DPU of claim 8 further configured to: transmit a restart command to the second OPU in response to determining the failure to receive a response to the status inquiry request.
11. The first DPU of claim 8 further configured to: determine a failure to receive a response to the status inquiry request from the second DPU after a period of time exceeding the timing threshold; and transmit a restart command to the second DPU in response to determining the failure to receive the response to the status inquiry request.
12. The first DPU of claim 8 further configured to: receive a response to the status inquiry request from the second DPU; and assign the value indicating an active status for the second DPU in the status table to track the status of each DPU of the set of DPUs.
13. The first DPU of claim 8 further configured to:
receive a restart command; and
perform a self-restart operation in response to receiving the restart command.
14. The first DPU of claim 13 further configured to:
transmit a third broadcast message to the set of DPUs associated with monitoring the first DPU upon successful completion of the self-restart operation.
15. A method of maintaining status of a respective data processing unit (DPU) of a plurality of DPUs, each coupled to non-volatile memory, the method at a first DPU comprising:
executing one or more persistent processes, wherein the one or more processes persist in the non-volatile memory over power cycles;
generating a first broadcast message upon completion of a first persistent process;
transmitting the first broadcast message to a set of DPUs associated with monitoring the first DPU;
receiving a second broadcast message from a second DPU of the set of DPUs;
assigning a value indicating an active status for the second DPU in a status table to track a status of each DPU of the set of DPUs;
determining a failure to receive the second broadcast message from the second DPU after a period of time exceeding a timing threshold;
assigning a value indicating an inactive status for the second DPU in the status table; and
transmitting a status inquiry request to the second DPU in response to determining the failure to receive the second broadcast message after a period of time exceeding a timing threshold.
16. The method of claim 15 , wherein the first broadcast message indicates an active state of the first DPU.
17. The method of claim 15 further comprising:
transmitting a restart command to the second DPU in response to determining the failure to receive a response to the status inquiry request.
18. The method of claim 15 further comprising:
determining a failure to receive a response to the status inquiry request from the second DPU after a period of time exceeding the timing threshold; and
transmitting a restart command to the second DPU in response to determining the failure to receive the response to the status inquiry request.
19. The method of claim 15 further comprising:
receiving a response to the status inquiry request from the second DPU; and
assigning the value indicating an active status for the second DPU in the status table to track the status of each DPU of the set of DPUs.
20. The method of claim 15 further comprising:
receiving a restart command; and
performing a self-restart operation in response to receiving the restart command.
21. The method of claim 20 further comprising:
transmitting a third broadcast message to the set of DPUs associated with monitoring the first DPU upon successful completion of the self-restart operation.