Dynamic adaptive approach for failure detection of node in a cluster

1. A non-transitory computer readable medium comprising instructions which, when executed by one or more hardware processors, causes performance of operations comprising:

transmitting, by a first device; data to a second device at a first time;

receiving, by the first device, an acknowledgment of the data from the second device at a second time;

determining a Round Trip Time (RTT) for the first device and the second device based on the first time and the second time;

determining a first frequency for transmitting a heartbeat protocol message between the first device and the second device based on the RTT, wherein the first frequency is larger than the RTT; and

transmitting a heartbeat protocol message between the first device and the second device at the first frequency.

2. The non-transitory computer readable medium of claim 1 , wherein the operations further comprise:

at least periodically performing:

re-measuring the RTT for the first device and the second device based on current network conditions;

dynamically updating the first frequency for transmitting the heartbeat protocol message between the first device and the second device.

3. The non-transitory computer readable medium of claim 2 , wherein dynamically updating the first frequency of transmitting the heartbeat protocol message comprises increasing the first frequency of transmitting the heartbeat protocol message in response to determining a reduction in the RTT.

4. The non-transitory computer readable medium of claim 2 , wherein dynamically updating the first frequency of transmitting the heartbeat protocol message comprises decreasing the first frequency of transmitting the heartbeat protocol message in response to determining an increase in the RTT.

5. The non-transitory computer readable medium of claim 1 , wherein transmitting the heartbeat protocol message between the first device and the second device at the first frequency comprises the first device transmitting the heartbeat protocol message to the second device at the first frequency.

6. The non-transitory computer readable medium of claim 5 , wherein the operations further comprise the second device transmits the heartbeat protocol message to the first device at a second frequency different than the first frequency.

7. The non-transitory computer readable medium of claim 1 , wherein transmitting the heartbeat protocol message between the first device and the second device at the first frequency comprises the second device transmitting the heartbeat protocol message to the first device at the first frequency.

8. The non-transitory computer readable medium of claim 1 , subsequent to transmitting the heartbeat protocol message between the first device and the second device at the first frequency:

determining that no response is received in response to a particular heartbeat protocol message;

retrying transmission of the particular heartbeat protocol message in a first retry message, wherein a time period between transmitting the particular heartbeat protocol message and the first retry message is based on the RTT.

9. The non-transitory computer readable medium of claim 1 , subsequent to transmitting the heartbeat protocol message between the first device and the second device at the first frequency:

determining that no response is received in response to a particular heartbeat protocol message;

retrying transmission of the particular heartbeat protocol message via a first retry message;

determining that no response is received in response to the first retry message;

retrying transmission of the particular heartbeat protocol message via a second retry message, wherein (a) a first time difference between transmission of the particular heartbeat protocol message and the first retry message is greater than (b) a second time difference between transmission of the first retry message and the second retry message.

10. The non-transitory computer readable medium of claim 1 , wherein (a) the first frequency for transmitting the heartbeat protocol message between the first device and the second device is different than (b) a second frequency for transmitting the heartbeat protocol message between the first device and a third device.

11. A system comprising:

a first device and a second device each comprising:

a processor; and

a memory storing computer readably: instructions to cause the processor to:

transmit by the first device, data to the second device at a first time;

received by the first device, an acknowledgment of the data from the second device at a second time;

determine a Round Trip Time (RTT) for the first device and the second device based on the first time and the second time;

determine a first frequency for transmitting a heartbeat protocol message between the first device and the second device based on the RTT, wherein the first frequency is a multiple of the RTT;

transmit a heartbeat protocol message between the first device and the second device at the first frequency.

12. The system of claim 11 , comprising instructions to cause the processor to:

re-measure the RTT for the first device and the second device based on current network conditions; and

dynamically update the first frequency for transmitting the heartbeat protocol message between the first device and the second device.

13. The system of claim 12 , wherein dynamically updating the first frequency of transmitting the heartbeat protocol message comprises increasing the first frequency of transmitting the heartbeat protocol message in response to determining a reduction in the RTT.

14. The system of claim 12 , wherein dynamically updating the first frequency of transmitting the heartbeat protocol message comprises decreasing the first frequency of transmitting the heartbeat protocol message in response to determining an increase in the RTT.

15. The system of claim 11 , wherein transmitting the heartbeat protocol message between the first device and the second device at the first frequency comprises the first device transmitting the heartbeat protocol message to the second device at the first frequency.

16. The system of claim 15 , wherein the operations further comprise the second device transmits the heartbeat protocol message to the first device at a second frequency different than the first frequency.

17. The system of claim 11 , wherein transmitting the heartbeat protocol message between the first device and the second device at the first frequency comprises the second device transmitting the heartbeat protocol message to the first device at the first frequency.

18. The system of claim 11 , subsequent to transmitting the heartbeat protocol message between the first device and the second device at the first frequency:

determining that no response is received in response to the heartbeat protocol message;

retrying transmission of the heartbeat protocol message, wherein a time period between transmitting the heartbeat protocol message is based on the RTT.

19. The system of claim 11 , subsequent to transmitting the heartbeat protocol message between the first device and the second device at the first frequency:

determining that no response is received in response to the heartbeat protocol message;

retrying transmission of the heartbeat protocol message via a first retry message;

determining that no response is received in response to the first retry message;

retrying transmission of the heartbeat protocol message via a second retry message, wherein (a) a first time difference between transmission of the heartbeat protocol message and the first retry message is greater than (b) a second time difference between transmission of the first retry message and the second retry message.

20. The system of claim 11 , wherein (a) the first frequency for transmitting the heartbeat protocol message between the first device and the second device is different than (b) a second frequency for transmitting the heartbeat protocol message between the first device and a third device.