Network health services and lifecycle correctness
Monitoring health metrics of computing devices in a data system can be implemented at different levels. At a first level, local background services can be run locally on the computing devices to monitor a set of health metrics on the respective computing devices. At a second level, a central health marker service can monitor a set of health metrics for the computing devices.
1 . A system comprising:
at least one hardware processor; and
at least one memory storing instructions that cause the at least one hardware processor to perform operations comprising:
receiving one or more timestamps associated with outputs of a computing device in a network-based data system in a defined time interval;
comparing the one or more timestamps to a reference clock based on a first threshold to determine whether the computing device has a future clock drift;
based on at least one of the one or more timestamps exceeding the first threshold, triggering a recycling operation for the computing device;
based on the one or more timestamps not exceeding the first threshold, comparing the one or more timestamps to the reference clock based on a second threshold to determine whether the computing device has a past clock drift; and
based on at least one of the one or more timestamps exceeding the second threshold, triggering the recycling operation for the computing device.
2 . The system of claim 1 , wherein the second threshold is greater than the first threshold.
3 . The system of claim 1 , wherein the one or more timestamps are received from a metadata database in the network-based data system.
4 . The system of claim 1 , wherein the recycling operation comprises:
changing a state of the computing device to a kill state;
transmitting a kill command to the computing device; and
terminating pending operations at the computing device in response to receiving the kill command.
5 . The system of claim 4 , wherein the recycling operation further comprises:
changing the state of the computing device to a fail state;
triggering a recovery operation for the computing device, the recovery operation comprising cleaning metadata associated with the computing device stored in a metadata database.
6 . The system of claim 4 , wherein the kill command is transmitted from a central health service to the computing device using a remote call.
7 . A method comprising:
receiving one or more timestamps associated with outputs of a computing device in a network-based data system in a defined time interval;
comparing the one or more timestamps to a reference clock based on a first threshold to determine whether the computing device has a future clock drift;
based on at least one of the one or more timestamps exceeding the first threshold, triggering a recycling operation for the computing device;
based on the one or more timestamps not exceeding the first threshold, comparing the one or more timestamps to the reference clock based on a second threshold to determine whether the computing device has a past clock drift; and
based on at least one of the one or more timestamps exceeding the second threshold, triggering the recycling operation for the computing device.
8 . The method of claim 7 , wherein the second threshold is greater than the first threshold.
9 . The method of claim 7 , wherein the one or more timestamps are received from a metadata database in the network-based data system.
10 . The method of claim 7 , wherein the recycling operation comprises:
changing a state of the computing device to a kill state;
transmitting a kill command to the computing device; and
terminating pending operations at the computing device in response to receiving the kill command.
11 . The method of claim 10 , wherein the recycling operation further comprises:
changing the state of the computing device to a fail state;
triggering a recovery operation for the computing device, the recovery operation comprising cleaning metadata associated with the computing device stored in a metadata database.
12 . The method of claim 10 , wherein the kill command is transmitted from a central health service to the computing device using a remote call.
13 . Computer-storage media comprising instructions that, when executed by one or more processors of a machine, configure the machine to perform operations comprising:
receiving one or more timestamps associated with outputs of a computing device in a network-based data system in a defined time interval;
comparing the one or more timestamps to a reference clock based on a first threshold to determine whether the computing device has a future clock drift;
based on at least one of the one or more timestamps exceeding the first threshold, triggering a recycling operation for the computing device;
based on the one or more timestamps not exceeding the first threshold, comparing the one or more timestamps to the reference clock based on a second threshold to determine whether the computing device has a past clock drift; and
based on at least one of the one or more timestamps exceeding the second threshold, triggering the recycling operation for the computing device.
14 . The computer-storage media of claim 13 , wherein the second threshold is greater than the first threshold.
15 . The computer-storage media of claim 13 , wherein the one or more timestamps are received from a metadata database in the network-based data system.
16 . The computer-storage media of claim 13 , wherein the recycling operation comprises:
changing a state of the computing device to a kill state;
transmitting a kill command to the computing device; and
terminating pending operations at the computing device in response to receiving the kill command.
17 . The computer-storage media of claim 16 , wherein the recycling operation further comprises:
changing the state of the computing device to a fail state;
triggering a recovery operation for the computing device, the recovery operation comprising cleaning metadata associated with the computing device stored in a metadata database.
18 . The computer-storage media of claim 16 , wherein the kill command is transmitted from a central health service to the computing device using a remote call.