Quorum based aggregator detection and repair
The present system uses quorum based aggregator failure detection in which a failed aggregator is detected and configured. Rather than repair and roll-up of all metrics for a period of time associated with the failed aggregator, only the specific metrics that were to be processed by the failed aggregator are repaired. Once the failed aggregator is identified, the time range for the downed aggregator and keys processed by the aggregator are identified. Keys for replica aggregators associated with the identified time ranges and key values are then pulled, provided to a batch processor, and processed. At cluster roll-up task completion, a time rollup task for cluster rollup is then started.
1. A method comprising:
registering, at a device in a computer network, a group of collectors in the computer network and a group of aggregators in the computer network, wherein the group of collectors receive metric data from a plurality of agents on one or more remote servers, and wherein each of the aggregators receive different particular types of metrics selected by the one or more collectors based on a hash that is applied to each aggregator in the group of aggregators and aggregate the selected metrics;
determining, by the device, whether each registered aggregator in the group of aggregators has checked-in with the device within a set period of time by comparing a current list of aggregators that checked-in during the set period of time to a list of aggregators that checked-in during the previous set period of time;
in response to determining that a particular aggregator has not checked-in with the device within the set period of time, identifying the particular aggregator as failed;
in response to identifying the particular aggregator as failed, identifying a range of keys associated with the selected metrics which the particular aggregator is configured to handle and a time range associated with when the particular aggregator failed; and
based on the identified range of keys and identified time range, repairing, by the device, data associated with only the identified range of keys and identified time range, wherein only specific selected metrics that were to be processed by the failed aggregator are repaired.
2. The method of claim 1 , wherein the particular aggregator is identified by failing to check in with a logical quorum.
3. The method of claim 1 , wherein each key is generated from machine information and a time stamp.
4. The method of claim 1 , the method further comprising identifying a time range during which the group of metrics was to be processed by the particular aggregator.
5. The method of claim 4 , wherein the time range during which the group of metrics was to be processed by the particular aggregator includes a current time window and a neighboring time window.
6. The method of claim 1 , wherein the group of metrics associated with the particular aggregator are also associated with one or more replica aggregators, and wherein the one or more replica aggregators process the group of metrics not processed by the particular aggregator.
7. The method of claim 1 , wherein the repair of data includes:
causing a batch processing client to:
retrieve keys from the one or more replica aggregators that satisfy the identified range of keys and identified time range; and
process the one or more group of metrics based on the retrieved keys.
8. A tangible non-transitory computer readable medium storing program instructions that cause a computer to execute a process, the process comprising:
registering, at a device in a computer network, a group of collectors in the computer network and a group of aggregators in the computer network, wherein the group of collectors receive metric data from a plurality of agents on one or more remote servers, and wherein each of the aggregators receive different particular types of metrics selected by the one or more collectors based on a hash that is applied to each aggregator in the group of aggregators and aggregate the selected metrics;
determining, at the device, whether each registered aggregator in the group of aggregators has checked-in with the device within a set period of time by comparing a current list of aggregators that checked-in during the set period of time to a list of aggregators that checked-in during the previous set period of time;
in response to determining that a particular aggregator has not checked-in with the device within the set period of time, identifying, at the device, the particular aggregator as failed;
in response to identifying the particular aggregator as failed, identifying, at the device, a range of keys associated with the selected metrics which the particular aggregator is configured to handle and a time range associated with when the particular aggregator failed; and
based on the identified range of keys and identified time range, repair data associated with only the identified range of keys and identified time range, wherein only specific selected metrics that were to be processed by the failed aggregator are repaired.
9. The tangible non-transitory computer readable medium of claim 8 , wherein the particular aggregator is identified by failing to check in with a logical quorum.
10. The tangible non-transitory computer readable medium of claim 8 , wherein each key is generated from machine information and a time stamp.
11. The tangible non-transitory computer readable medium of claim 8 , the process further comprising identifying a time range during which the group of metrics was to be processed by the particular aggregator.
12. The tangible non-transitory computer readable medium of claim 11 , wherein the time range during which the group of metrics was to be processed by the particular aggregator includes a current time window and a neighboring time window.
13. The tangible non-transitory computer readable medium of claim 8 , wherein the group of metrics associated with the particular aggregator are also associated with one or more replica aggregators, and wherein the one or more replica aggregators process the group of metrics not processed by the particular aggregator.
14. An apparatus comprising:
one or more network interfaces to communicate with a computer network;
a processor coupled to the one or more network interfaces and configured to execute a process; and
a memory configured to store program instructions which include the process executable by the processor, the process configured to:
register a group of aggregators in the computer network, wherein each of the aggregators receive different particular types of metrics selected by the one or more collectors based on a hash that is applied to each aggregator in the group of aggregators and aggregate the selected metrics,
determine whether each registered aggregator in the group of aggregators has checked-in with the device within a set period of time by comparing a current list of aggregators that checked-in during the set period of time to a list of aggregators that checked-in during the previous set period of time,
identifying the particular aggregator as failed in response to determining that a particular aggregator has not checked-in with the device within the set period of time,
identify a range of keys associated with the selected metrics which the particular aggregator is configured to handle and a time range associated with when the particular aggregator failed in response to identifying the particular aggregator as failed, and
based on the identified range of keys and identified time range, repair data associated with only the identified range of keys and identified time range, wherein only specific selected metrics that were to be processed by the failed aggregator are repaired.
15. The apparatus of claim 14 , wherein the particular aggregator is identified by failing to check in with a logical quorum.
16. The apparatus of claim 14 , wherein each key is generated from machine information and a time stamp.
17. The apparatus of claim 14 , the process further configured to:
identify a second time range during which the group of metrics were to be processed by the particular aggregator.
18. The apparatus of claim 17 , wherein the second time range includes a current time window and a neighboring time window.
19. The apparatus of claim 17 , wherein the group of metrics associated with the particular aggregator are also associated with one or more replica aggregators, and wherein the one or more replica aggregators process the group of metrics not processed by the particular aggregator.
20. The apparatus of claim 19 , wherein the repair of data includes:
causing a batch processing client to:
retrieve keys from the one or more replica aggregators that satisfy the identified range of keys and identified time range; and
process the one or more group of metrics based on the retrieved keys.