Detecting cloud service latency issues through analysis of tenant latency signals
The techniques describe effective detection of latency-related issues for a cloud service operating in a distributed computing environment. To detect the latency-related issues, a system first determines baseline latency behavior at the tenant level (e.g., on a tenant-by-tenant basis) and compares a tenant's current latency behavior to the baseline latency behavior. If the comparison yields that the current latency behavior for the tenant is following the baseline latency behavior, the tenant is deemed healthy. However, if the comparison yields that the current latency behavior for the tenant is not closely following the baseline latency behavior, the tenant is deemed unhealthy. Once the system has made binary health determinations for various tenants on a tenant-by-tenant basis, the system is configured to aggregate the unhealthy determinations across a group of tenants to determine whether the cloud service is experiencing latency-related issues.
1 . A method comprising:
generating a tenant-specific model for a latency signal by analyzing a training dataset for a tenant over a training time period, wherein:
the training dataset includes respective percentile latency values, per a defined time bin, for each percentile in a defined set of percentiles;
the respective latency values are associated with a service offered by a cloud provider;
the tenant-specific model defines a distribution based on a mean and a standard deviation of the respective percentile latency values for each percentile in the defined set of percentiles;
accessing current percentile latency values associated with the tenant for a current time bin, wherein the current percentile latency values are respectively associated with percentiles in the defined set of percentiles;
generating a latency health score vector for the tenant and for the current time bin by determining percentile health scores for each percentile in the defined set of percentiles via a comparison of a current percentile latency value to the distribution;
calculating an overall latency health score based on a plurality of latency health scores in the latency health score vector;
determining that the overall latency health score is less than a latency health score threshold;
in response to determining that the overall latency health score is less than the latency health score threshold, designating the tenant as an unhealthy tenant due to abnormal latency;
determining that a total number of unhealthy tenants for the current time bin is greater than a predefined threshold number of unhealthy tenants; and
sending, to an owner of the service and based on the total number of unhealthy tenants being greater than the predefined threshold number of unhealthy tenants, a notification indicating a potential latency issue associated with the service.
2 . The method of claim 1 , wherein the latency signal and the tenant-specific model are associated with a resource deployed by the service and for the tenant within a defined geographic region of a cloud platform or a distributed computing environment.
3 . The method of claim 1 , wherein:
the comparison of the current percentile latency value to the distribution comprises determining a percentile rank score (PRS) for the current percentile latency value using a z-score and a z-table; and
the percentile health score (PHS) for a corresponding percentile in the defined set of percentiles is a percentage calculated as follows:
P
HS
=
(
1
-
P
RS
)
*
10
0
.
4 . The method of claim 1 , wherein the overall latency health score is calculated based on respective weights assigned to the plurality of latency health scores in the latency health score vector.
5 . The method of claim 4 , wherein the weights are defined by the tenant.
6 . The method of claim 1 , further comprising establishing the predefined threshold number of unhealthy tenants by:
calculating an average number of unhealthy tenants across time bins in a defined number N of days;
calculating a standard deviation associated with the average number of unhealthy tenants; and
setting the predefined threshold number of unhealthy tenants to be a predefined number of standard deviations above the average number of unhealthy tenants.
7 . The method of claim 1 , wherein the notification comprises information that indicates an impacted geographic region, a detection time, and a percentage of tenants impacted.
8 . A system comprising:
a processing system; and
a computer-readable medium storing instructions that, when executed by the processing system, cause the system to perform operations comprising:
generating a tenant-specific model for a latency signal by analyzing a training dataset for a tenant over a training time period, wherein:
the training dataset includes respective percentile latency values, per a defined time bin, for each percentile in a defined set of percentiles;
the respective latency values are associated with a service offered by a cloud provider;
the tenant-specific model defines a distribution based on a mean and a standard deviation of the respective percentile latency values for each percentile in the defined set of percentiles;
accessing current percentile latency values associated with the tenant for a current time bin, wherein the current percentile latency values are respectively associated with percentiles in the defined set of percentiles;
generating a latency health score vector for the tenant and for the current time bin by determining percentile health scores for each percentile in the defined set of percentiles via a comparison of a current percentile latency value to the distribution;
calculating an overall latency health score based on a plurality of latency health scores in the latency health score vector;
determining that the overall latency health score is less than a latency health score threshold;
in response to determining that the overall latency health score is less than the latency health score threshold, designating the tenant as an unhealthy tenant due to abnormal latency;
determining that a total number of unhealthy tenants for the current time bin is greater than a predefined threshold number of unhealthy tenants; and
sending, to an owner of the service and based on the total number of unhealthy tenants being greater than the predefined threshold number of unhealthy tenants, a notification indicating a potential latency issue associated with the service.
9 . The system of claim 8 , wherein the latency signal and the tenant-specific model are associated with a resource deployed by the service and for the tenant within a defined geographic region of a cloud platform or a distributed computing environment.
10 . The system of claim 8 , wherein:
the comparison of the current percentile latency value to the distribution comprises determining a percentile rank score (PRS) for the current percentile latency value using a z-score and a z-table; and
the percentile health score (PHS) for a corresponding percentile in the defined set of percentiles is a percentage calculated as follows:
P
HS
=
(
1
-
P
RS
)
*
10
0
.
11 . The system of claim 8 , wherein the overall latency health score is calculated based on respective weights assigned to the plurality of latency health scores in the latency health score vector.
12 . The system of claim 11 , wherein the weights are defined by the tenant.
13 . The system of claim 8 , wherein the operations further comprise establishing the predefined threshold number of unhealthy tenants by:
calculating an average number of unhealthy tenants across time bins in a defined number N of days;
calculating a standard deviation associated with the average number of unhealthy tenants; and
setting the predefined threshold number of unhealthy tenants to be a predefined number of standard deviations above the average number of unhealthy tenants.
14 . The system of claim 8 , wherein the notification comprises information that indicates an impacted geographic region, a detection time, and a percentage of tenants impacted.
15 . A non-transitory computer-readable storage medium storing instructions that, when executed by a processing system, cause a system to perform operations comprising:
generating a tenant-specific model for a latency signal by analyzing a training dataset for a tenant over a training time period, wherein:
the training dataset includes respective percentile latency values, per a defined time bin, for each percentile in a defined set of percentiles;
the respective latency values are associated with a service offered by a cloud provider;
the tenant-specific model defines a distribution based on a mean and a standard deviation of the respective percentile latency values for each percentile in the defined set of percentiles;
accessing current percentile latency values associated with the tenant for a current time bin, wherein the current percentile latency values are respectively associated with percentiles in the defined set of percentiles;
generating a latency health score vector for the tenant and for the current time bin by determining percentile health scores for each percentile in the defined set of percentiles via a comparison of a current percentile latency value to the distribution;
calculating an overall latency health score based on a plurality of latency health scores in the latency health score vector;
determining that the overall latency health score is less than a latency health score threshold;
in response to determining that the overall latency health score is less than the latency health score threshold, designating the tenant as an unhealthy tenant due to abnormal latency;
determining that a total number of unhealthy tenants for the current time bin is greater than a predefined threshold number of unhealthy tenants; and
sending, to an owner of the service and based on the total number of unhealthy tenants being greater than the predefined threshold number of unhealthy tenants, a notification indicating a potential latency issue associated with the service.
16 . The non-transitory computer-readable storage medium of claim 15 , wherein the latency signal and the tenant-specific model are associated with a resource deployed by the service and for the tenant within a defined geographic region of a cloud platform or a distributed computing environment.
17 . The non-transitory computer-readable storage medium of claim 15 , wherein:
the comparison of the current percentile latency value to the distribution comprises determining a percentile rank score (PRS) for the current percentile latency value using a z-score and a z-table; and
the percentile health score (PHS) for a corresponding percentile in the defined set of percentiles is a percentage calculated as follows:
P
HS
=
(
1
-
P
RS
)
*
10
0
.
18 . The non-transitory The computer-readable storage medium of claim 15 , wherein:
the overall latency health score is calculated based on respective weights assigned to the plurality of latency health scores in the latency health score vector; and
the weights are defined by the tenant.
19 . The non-transitory computer-readable storage medium of claim 15 , wherein the operations further comprise establishing the predefined threshold number of unhealthy tenants by:
calculating an average number of unhealthy tenants across time bins in a defined number N of days;
calculating a standard deviation associated with the average number of unhealthy tenants; and
setting the predefined threshold number of unhealthy tenants to be a predefined number of standard deviations above the average number of unhealthy tenants.
20 . The non-transitory computer-readable storage medium of claim 15 , wherein the notification comprises information that indicates an impacted geographic region, a detection time, and a percentage of tenants impacted.