Monitoring service endpoints
Today, data networks are ever increasing in size and complexity. For example, a datacenter may comprise hundreds of thousands of service endpoints configured to perform work. To reduce network wide degradation, a load balancer may send work requests to healthy service endpoints, as opposed to unhealthy and/or inoperative service endpoints. Accordingly, among other things, one or more systems and/or techniques for monitoring service endpoints, which may be scalable for large scale networks, are provided. In particular, a consistent hash function may be performed to generate a monitoring scheme comprising assignments of service endpoints to monitoring groups. In this way, multiple monitoring components may monitor a subset of endpoints to ascertain health status. Additionally, the monitoring components may communicate between one another so that a monitoring component may know heath statuses of service endpoints both assigned and not assigned to the monitoring component.
1. A method for monitoring service endpoints, comprising:
performing a hash function upon identifiers of service endpoints to generate a monitoring scheme comprising an assignment of service endpoints to monitoring groups, the assignment specifying, based upon the performed hash function, that a monitoring node comprised within a monitoring group is to monitor a service endpoint, at least some of the performing implemented at least in part via a processing unit.
2. The method of claim 1 , comprising:
sending a communication from a first monitoring group to a second monitoring group, the communication indicative of a status change of a first service endpoint monitored by the first monitoring group based upon the monitoring scheme.
3. The method of claim 1 , comprising:
receiving a communication at a first monitoring group from a second monitoring group, the communication indicative of a status change of a second service endpoint monitored by the second monitoring group based upon the monitoring scheme.
4. The method of claim 3 , the second service endpoint not assigned to the first monitoring group within the monitoring scheme, the method comprising:
sending a communication from the first monitoring group to a third monitoring group, the communication sent from the first monitoring group indicative of the status change of the second service endpoint monitored by the second monitoring group based upon the monitoring scheme.
5. The method of claim 1 , comprising:
sending a communication from a first monitoring group to a distributed client not within a monitoring group, the communication indicative of a health status change of a first service endpoint monitored by the first monitoring group based upon the monitoring scheme.
6. The method of claim 1 , where at least some of the service endpoints are assigned to merely a single monitoring group within the monitoring scheme.
7. The method of claim 1 , comprising:
receiving a first status of a first service endpoint from a first monitoring node comprised within a first monitoring group; and
receiving a second status of the first service endpoint from a second monitoring node comprised within the first monitoring group.
8. The method of claim 7 , the first status different from the second status, the method comprising resolving inconsistencies between the first status and the second status.
9. The method of claim 8 , the resolving inconsistencies comprising at least one of:
if the first status indicates the first service endpoint is operating in an unhealthy state and the second status indicates the first service endpoint is operating in a healthy state, then marking the first service endpoint as unhealthy, and
if the first status indicates the first service endpoint is operating in an unhealthy state and the second status indicates the first service endpoint is operating in a healthy state, then marking the first service endpoint as healthy.
10. The method of claim 7 , the first status measured in binary measurements or scaled measurements and the second status measured in binary measurements or scaled measurements.
11. The method of claim 1 , comprising:
determining a failed monitoring group; and
redistributing one or more service endpoints assigned to the failed monitoring group within the monitoring scheme to one or more monitoring groups different from the failed monitoring group.
12. A system for monitoring service endpoints, comprising:
a grouping component configured to:
perform a hash function upon identifiers of service endpoints to generate a monitoring scheme comprising assignments of service endpoints to monitoring groups;
a first monitoring node within a first monitoring group configured to receive a first status of a first service endpoint, the first status measured in binary measurements or scaled measurements; and
a second monitoring node within the first monitoring group configured to receive a second status of the first service endpoint, the second status measured in binary measurements or scaled measurements, at least some of at least one of the grouping component, the first monitoring node and the second monitoring node implemented at least in part via a processing unit.
13. The system of claim 12 , the first monitoring node configured to monitor one or more service endpoints assigned to the first monitoring group within the monitoring scheme.
14. The system of claim 13 , the second monitoring node configured to monitor one or more service endpoints assigned to the first monitoring group within the monitoring scheme.
15. The system of claim 14 , the first monitoring group configured to:
determine whether the first status and the second status are different; and
upon determining the first status and the second status are different, resolve inconsistencies between the first status and the second status.
16. The system of claim 12 , the grouping component configured to:
redistribute one or more service endpoints assigned to a failed monitoring group within the monitoring scheme to one or more monitoring groups different from the failed monitoring group.
17. The system of claim 12 , the first monitoring group configured to send a communication to a second monitoring group, the communication indicative of a status change of the first service endpoint monitored by the first monitoring group based upon the monitoring scheme.
18. The system of claim 17 , the second monitoring group configured to send a communication to a third monitoring group, the communication indicative of the status change of the first service endpoint monitored by the first monitoring group based upon the monitoring scheme.
19. A computer readable storage medium comprising computer executable instructions that when executed via a processing unit on a computer perform a method for monitoring service endpoints, comprising:
performing a hash function upon identifiers of service endpoints to generate a monitoring scheme comprising an assignment of service endpoints to monitoring groups;
receiving a first status of a first service endpoint from a first monitoring node comprised within a first monitoring group;
receiving a second status of the first service endpoint from a second monitoring node comprised within the first monitoring group; and
resolving inconsistencies between the first status and the second status, comprising at least one of:
if the first status indicates the first service endpoint is operating in an unhealthy state and the second status indicates the first service endpoint is operating in a healthy state, then marking the first service endpoint as unhealthy, and
if the first status indicates the first service endpoint is operating in an unhealthy state and the second status indicates the first service endpoint is operating in a healthy state, then marking the first service endpoint as healthy.
20. The method of claim 19 , comprising:
determining a failed monitoring group; and
redistributing one or more service endpoints assigned to the failed monitoring group within the monitoring scheme to one or more monitoring groups different from the failed monitoring group.