Detecting systemwide service issues by using anomaly localization
Methods and systems for detecting systemwide service issues by using anomaly localization. In an example, a method includes receiving time-series monitoring data for multiple services, the time-series monitoring data including multiple dimensions and an error metric; for the monitoring data from each service, evaluating scopes within the monitoring data based on an objective function for a time-series of the error metric to identify at least one anomalous scope, each scope including at least one dimension and a value for the dimension; based on evaluating the scopes, generating a ranked list of scopes for each service based on objective function scores for the scopes; correlating the ranked lists of scopes across the multiple services to identify a cross-service anomaly; and generating an alert for the services based on the cross-service anomaly, the alert indicating at least one scope as a potential root cause for the cross-service anomaly.
1 . A computing system comprising:
a processor; and
memory storing instructions that, when executed by the processor, cause the system to perform operations comprising:
receiving time-series monitoring data for multiple services, the time-series monitoring data including multiple dimensions and an error metric;
for the time-series monitoring data from each service, evaluating scopes within the time-series monitoring data comprising:
for a current scope, calculating an objective function score for a time series of the error metric limited by a current scope;
identifying a random scope that is a not a neighbor scope of the current scope;
calculating an objective function score for the random scope;
based on the objective function scores of the random sampling of the neighbor scopes and the objective function score of the random scope, selecting one of the neighbor scopes or the random scope to be a next current scope;
based on evaluating the scopes, generating a ranked list of scopes for each service based on objective function scores for the scopes;
correlating the ranked lists of scopes across the multiple services to identify a cross-service anomaly; and
generating an alert for the multiple services based on the cross-service anomaly, the alert indicating at least one scope as a potential root cause for the cross-service anomaly.
2 . The system of claim 1 , wherein at least one of the evaluated scopes includes a combination of at least two dimensions.
3 . The system of claim 2 , wherein one of the at least two dimensions is based on a physical component and another one of the at least two dimensions is based on a logical component.
4 . The system of claim 3 , wherein evaluating the scopes comprises:
for a current scope, calculating an objective function score for a time series of the error metric limited by the current scope;
determining a random sampling of neighbor scopes of the current scope, wherein the neighbor scopes include at least a peer scope, a subset scope, or a superset scope; and
calculating objective function scores for random sampling of neighbor scopes.
5 . The system of claim 1 , wherein the objective function represents a change in the error metric before and after a point in time.
6 . The system of claim 1 , wherein the services include at least one of an email service, a messaging service, or a document storage service.
7 . The system of claim 1 , wherein the multiple services rely on at least one common dependency.
8 . A computer-implemented method for localizing anomalies across multiple services, the method comprising:
receiving time-series monitoring data for multiple services, the time-series monitoring data including multiple dimensions and an error metric;
for the time-series monitoring data from each service, evaluating scopes within the time-series monitoring data comprising:
for a current scope, calculating an objective function score for a time series of the error metric limited by a current scope;
identifying a random scope that is a not a neighbor scope of the current scope;
calculating an objective function score for the random scope;
based on the objective function scores of the random sampling of the neighbor scopes and the objective function score of the random scope, selecting one of the neighbor scopes or the random scope to be a next current scope;
based on evaluating the scopes, generating a ranked list of scopes for each service based on objective function scores for the scopes;
correlating the ranked lists of scopes across the multiple services to identify a cross-service anomaly; and
generating an alert for the multiple services based on the cross-service anomaly, the alert indicating at least one scope as a potential root cause for the cross-service anomaly.
9 . The method of claim 8 , wherein at least one of the evaluated scopes includes a combination of at least two dimensions.
10 . The method of claim 8 , wherein the objective function represents a change in the error metric before and after a point in time.
11 . The method of claim 8 , wherein the services include at least one of an email service, a messaging service, or a document storage service.
12 . The method of claim 8 , wherein the multiple services rely on at least one common dependency.
13 . The method of claim 8 , wherein the multiple services include a first a service and a second service, wherein the first service is an email service, a messaging service, or a document storage service, and the second service is an email service, a messaging service, or a document storage service, the second service being different from the first service.
14 . The method of claim 13 , wherein the first service and the second service rely on a common dependency.
15 . The method of claim 14 , wherein the alert is generated for a third service that also relies on the common dependency.
16 . A computer-implemented method for localizing anomalies across multiple services, the method comprising:
receiving time-series monitoring data for multiple services, the time-series monitoring data including multiple dimensions and an error metric;
for the monitoring data from each service, evaluating scopes within the monitoring data based on an objective function for a time-series of the error metric to identify at least one anomalous scope, wherein evaluating the scopes comprises:
for a current scope, calculating an objective function score for a time series of the error metric limited by the current scope;
determining a random sampling of neighbor scopes of the current scope, wherein the neighbor scopes include at least a peer scope, a subset scope, or a superset scope;
calculating objective function scores for random sampling of neighbor scopes;
identifying a random scope that is a not a neighbor scope of the current scope;
calculating an objective function score for the random scope; and
based on the objective function scores of the random sampling of the neighbor scopes and the objective function score of the random scope, selecting one of the neighbor scopes or the random scope to be a next current scope; and
generating an alert for the services based on a cross-service anomaly, the alert indicating at least one scope as a potential root cause for the cross-service anomaly.
17 . The method of claim 16 , wherein at least one of the evaluated scopes includes a combination of at least two dimensions.
18 . The method of claim 17 , further comprising:
based on evaluating the scopes, generating a ranked list of scopes for each service based on objective function scores for the scopes; and
correlating the ranked lists of scopes across the multiple services to identify the cross-service anomaly.
19 . The method of claim 16 , wherein the objective function represents a change in the error metric before and after a point in time.
20 . The method of claim 16 , wherein the services include at least one of an email service, a messaging service, or a document storage service.