Quorum based distributed anomaly detection and repair using distributed computing by stateless processes
Quorum based anomaly detection utilizes multiple entities to detect and attempt to configure a repair task for an anomaly. Once the repair task is generated, a system is used to assign the task to a worker entity while recording the responsibility of that task with the worker in a persistent storage. If the worker entity crashes, the degraded worker status will eventually be detected, and all tasks associated with that worker will be re-assigned. Once a worker finishes a task, the assignment information for the task is transitioned to a completed state.
1. A method comprising:
detecting, by a quorum, an anomaly among a group of collectors and aggregators in a network, the group of collectors configured to receive metric data from a plurality of agents on one or more remote servers in the computer network and the group of aggregators configured to receive one or more selected metrics from the one or more collectors;
responsive to detecting the anomaly, causing a plurality of producers to initiate a repair task generation process;
receiving, at the quorum, a query from at least one producer of the plurality of producers;
determining, by the quorum, whether the repair task has already been generated by another producer of the plurality of producers;
preventing, by the quorum, the at least one producer from generating the repair task when another producer of the plurality of producers has already generated the repair task;
causing, by the quorum, the at least one producer to generate the repair task when another producer has not already generated the repair task;
detecting that the repair task has been generated by at least one producer of the plurality of producers;
assigning the generated repair task to a worker; and
completing the generated repair task at the assigned worker.
2. The method of claim 1 , wherein the status of the repair task is maintained in a persistent scheduler.
3. The method of claim 2 , further comprising:
registering the worker by adding a worker identifier to a worker context within the persistent scheduler; and
adding the worker identifier to an assignment context, the repair task assigned to the worker in the assignment context.
4. The method of claim 3 , further comprising:
detecting that the worker is unavailable;
removing the worker identifier from the worker context;
maintaining the worker identifier and any task identifier associated with the worker identifier in the assignment context; and
searching a task assign context to determine if any tasks associated with the unavailable worker have failed; and
re-assigning any tasks that have failed to one or more other workers.
5. The method of claim 1 , wherein detecting that the assigned worker is unavailable comprises:
periodically polling the assigned worker to determine if the assigned worker is still available.
6. The method of claim 1 , wherein each of the plurality of producers are implemented on each one of the group of collectors respectively, and wherein the worker is implemented on at least one aggregator of the group of aggregators.
7. The method of claim 1 , wherein each of the plurality of producers that attempt to create the repair task are prevented from successfully generating the repair task after the repair task the quorum is notified that the repair task has been completed by the producer.
8. A non-transitory computer readable storage medium having embodied thereon a program, the program being executable by a processor to perform a method for processing metrics, the method comprising:
detecting, at a quorum, an anomaly among a group of collectors and aggregators in a network, the group of collectors configured to receive metric data from a plurality of agents on one or more remote servers in the computer network and the group of aggregators configured to receive one or more selected metrics from the one or more collectors;
responsive to detecting the anomaly, causing a plurality of producers to initiate a repair task generation process;
receiving, at the quorum, a query from at least one producer of the plurality of producers;
determining, at the quorum, whether the repair task has already been generated by another producer of the plurality of producers;
preventing, at the quorum, the at least one producer from generating the repair task when another producer of the plurality of producers has already generated the repair task;
causing, at the quorum, the at least one producer to generate the repair task when another producer has not already generated the repair task;
detecting that the repair task has been generated by at least one producer of the plurality of producers;
assigning the generated repair task to a worker; and
completing the generated repair task at the assigned worker.
9. The non-transitory computer readable storage medium of claim 8 , wherein the status of the repair task is maintained in a persistent scheduler.
10. The non-transitory computer readable storage medium of claim 9 , further comprising:
registering the worker by adding a worker identifier to a worker context within the persistent scheduler; and
adding the worker identifier to an assignment context, the repair task assigned to the worker in the assignment context.
11. The non-transitory computer readable storage medium of claim 10 , further comprising:
detecting that the worker is unavailable;
removing the worker identifier from the worker context;
maintaining the worker identifier and any task identifier associated with the worker identifier in the assignment context; and
searching a task assign context to determine if any tasks associated with the unavailable worker have failed; and
re-assigning any tasks that have failed to one or more other workers.
12. The non-transitory computer readable storage medium of claim 8 , wherein detecting that the assigned worker is unavailable comprises:
periodically polling the assigned worker to determine if the assigned worker is still available.
13. The non-transitory computer readable storage medium of claim 8 , wherein each of the plurality of producers are implemented on each one of the group of collectors respectively, and wherein the worker is implemented on at least one aggregator of the group of aggregators.
14. The non-transitory computer readable storage medium of claim 8 , wherein each of the plurality of producers that attempt to create the repair task are prevented from successfully generating the repair task after the repair task the quorum is notified that the repair task has been completed by the producer.
15. A system, comprising:
one or more network interfaces to communicate in a computer network;
a processor coupled to the network interfaces and configured to execute one or more processes; and
a memory configured to store a process executable by the processor, the process when executed operable to:
detect, at a quorum, an anomaly among a group of collectors and aggregators in a network, the group of collectors configured to receive metric data from a plurality of agents on one or more remote servers in the computer network and the group of aggregators configured to receive one or more selected metrics from the one or more collectors;
cause a plurality of producers to initiate a repair task generation process responsive to detecting the anomaly;
receive, at a quorum a query from at least one producer of the plurality of producers;
determine, at the quorum, whether the repair task has already been generated by another producer of the plurality of producers;
prevent, at the quorum, the at least one producer from generating the repair task when another producer of the plurality of producers has already generated the repair task;
cause the at least one producer to generate the repair task when another producer has not already generated the repair task;
detect that the repair task has been generated by at least one producer of the plurality of producers;
assign the generated repair task to a worker; and
complete the generated repair task at the assigned worker.
16. The system of claim 15 , wherein the status of the repair task is maintained in a persistent scheduler.
17. The system of claim 16 , the one or more modules further operable to:
register the worker by adding a worker identifier to a worker context within the persistent scheduler; and
add the worker identifier to an assignment context, the repair task assigned to the worker in the assignment context.
18. The system of claim 17 , the one or more modules further operable to:
detect that the worker is unavailable;
remove the worker identifier from the worker context;
maintain the worker identifier and any task identifier associated with the worker identifier in the assignment context; and
search a task assign context to determine if any tasks associated with the unavailable worker have failed; and
re-assign any tasks that have failed to one or more other workers.
19. The system of claim 15 , wherein one of the collectors implement each of the plurality of producers.
20. The system of claim 15 , wherein each of the plurality of producers are implemented on each one of the group of collectors respectively, and wherein the worker is implemented on at least one aggregator of the group of aggregators.
21. The system of claim 15 , wherein each of the plurality of producers that attempt to create the repair task are prevented from successfully generating the repair task after the repair task the quorum is notified that the repair task has been completed by the producer.