Detecting cloud service connectivity issues through analysis of tenant network traffic signals
The techniques describe effective detection of network connectivity issues for a cloud service operating in a distributed computing environment. To detect the network connectivity issues, a system first projects network traffic patterns at the tenant level (e.g., on a tenant-by-tenant basis) and compares a tenant's current network traffic to the projected network traffic pattern. If the comparison yields that the current network traffic for the tenant is closely following the projected network traffic pattern, the tenant is deemed healthy. However, if the comparison yields that the current network traffic for the tenant is not closely following the projected network traffic pattern, the tenant is deemed unhealthy. Once the system has made these binary health determinations for various tenants on a tenant-by-tenant basis, the system is configured to aggregate the unhealthy determinations across a group of tenants to determine whether the cloud service is experiencing network connectivity issues.
1 . A method comprising:
generating a tenant-specific model that projects a network traffic pattern by analyzing a training dataset including values associated with a network traffic signal for a tenant over a training time period, wherein:
the network traffic signal is associated with a service offered by a cloud provider; and
the network traffic pattern establishes a baseline threshold value for each defined time bin in a plurality of defined time bins across the training time period;
determining that a current value associated with the network traffic signal for a current time bin is less than the baseline threshold value established for a corresponding time bin in the tenant-specific model;
in response to determining that the current value associated with the network traffic signal for the current time bin is less than the baseline threshold value, designating the tenant as an unhealthy tenant due to abnormal network traffic behavior;
determining that a total number of unhealthy tenants for the current time bin is greater than a predefined threshold number of unhealthy tenants; and
sending, to an owner of the service and based on the total number of unhealthy tenants being greater than the predefined threshold number of unhealthy tenants, a notification indicating a potential network connectivity issue associated with the service.
2 . The method of claim 1 , wherein the network traffic signal and the tenant-specific model are associated with a resource deployed by the service and for the tenant within a defined geographic region of a cloud platform or a distributed computing environment.
3 . The method of claim 1 , wherein the baseline threshold value for each defined time bin in the plurality of defined time bins across the training time period is established by:
identifying, in the training dataset, one or more patterns in one or more respective time series;
for each of the one or more patterns:
producing a residual training dataset by removing the pattern from the values associated with the network traffic signal;
producing a transformed training dataset by applying a power transformation to the residual training dataset to stabilize a variance and normalize a distribution in the residual training dataset;
projecting a normal range for each defined time bin in the plurality of defined time bins across the training time period by applying an adjusted boxplot to the transformed training dataset; and
using a lower bound of the normal range to establish the baseline threshold value for each defined time bin in the plurality of defined time bins across the training time period.
4 . The method of claim 1 , further comprising verifying, for quality purposes, that the values associated with the network traffic signal include a non-missing value or a non-zero value within a most recent defined time period within the training time period prior to analyzing the training dataset to generate the tenant-specific model.
5 . The method of claim 1 , further comprising verifying, for data quality purposes, that at least a threshold percentage of the values associated with the network traffic signal include a non-missing value or a non-zero value prior to analyzing the training dataset to generate the tenant-specific model.
6 . The method of claim 1 , further comprising verifying, for data quality purposes, that the values associated with the network traffic signal include at least a threshold number of requests for each time bin within the training time period prior to analyzing the training dataset to generate the tenant-specific model.
7 . The method of claim 1 , further comprising establishing the predefined threshold number of unhealthy tenants by:
calculating an average number of unhealthy tenants across time bins in a defined number N of days;
calculating a standard deviation associated with the average number of unhealthy tenants; and
setting the predefined threshold number of unhealthy tenants to be a predefined number of standard deviations above the average number of unhealthy tenants.
8 . The method of claim 1 , wherein the network traffic signal comprises a total number of requests received on behalf of the tenant.
9 . The method of claim 1 , wherein the notification comprises information that indicates an impacted geographic region, a detection time, and a percentage of tenants impacted.
10 . A system comprising:
a processing system; and
a computer-readable medium storing instructions that, when executed by the processing system, cause the system to perform operations comprising:
generating a tenant-specific model that projects a network traffic pattern by analyzing a training dataset including values associated with a network traffic signal for a tenant over a training time period, wherein:
the network traffic signal is associated with a service offered by a cloud provider; and
the network traffic pattern projects a baseline threshold value for each defined time bin in a plurality of defined time bins across the training time period;
determining that a current value associated with the network traffic signal for a current time bin is inconsistent with the baseline threshold value projected for the current time bin;
in response to determining that the current value associated with the network traffic signal for the current time bin is inconsistent with the baseline threshold value projected for the current time bin, designating the tenant as an unhealthy tenant due to abnormal network traffic behavior;
determining that a total number of unhealthy tenants for the current time bin is greater than a predefined threshold number of unhealthy tenants; and
sending, to an owner of the service and based on the total number of unhealthy tenants being greater than the predefined threshold number of unhealthy tenants, a notification indicating a potential network connectivity issue associated with the service.
11 . The system of claim 10 , wherein the network traffic signal and the tenant-specific model are associated with a resource deployed by the service and for the tenant within a defined geographic region of a cloud platform or a distributed computing environment.
12 . The system of claim 10 , wherein the baseline threshold value for each defined time bin in the plurality of defined time bins across the training time period is established by:
identifying, in the training dataset, one or more patterns in one or more respective time series;
for each of the one or more patterns:
producing a residual training dataset by removing the pattern from the values associated with the network traffic signal;
producing a transformed training dataset by applying a power transformation to the residual training dataset to stabilize a variance and normalize a distribution in the residual training dataset;
projecting a normal range for each defined time bin in the plurality of defined time bins across the training time period by applying an adjusted boxplot to the transformed training dataset; and
using a lower bound of the normal range to establish the baseline threshold value for each defined time bin in the plurality of defined time bins across the training time period.
13 . The system of claim 10 , wherein the operations further comprise verifying, for quality purposes, that the values associated with the network traffic signal include a non-missing value or a non-zero value within a most recent defined time period within the training time period prior to analyzing the training dataset to generate the tenant-specific model.
14 . The system of claim 10 , wherein the operations further comprise verifying, for data quality purposes, that at least a threshold percentage of the values associated with the network traffic signal include a non-missing value or a non-zero value prior to analyzing the training dataset to generate the tenant-specific model.
15 . The system of claim 10 , wherein the operations further comprise verifying, for data quality purposes, that the values associated with the network traffic signal include at least a threshold number of requests for each time bin within the training time period prior to analyzing the training dataset to generate the tenant-specific model.
16 . The system of claim 10 , wherein the operations further comprise establishing the predefined threshold number of unhealthy tenants by:
calculating an average number of unhealthy tenants across time bins in a defined number N of days;
calculating a standard deviation associated with the average number of unhealthy tenants; and
setting the predefined threshold number of unhealthy tenants to be a predefined number of standard deviations above the average number of unhealthy tenants.
17 . The system of claim 10 , wherein the network traffic signal comprises a total number of requests received on behalf of the tenant.
18 . The system of claim 10 , wherein the notification comprises information that indicates an impacted geographic region, a detection time, and a percentage of tenants impacted.
19 . A computer-readable storage medium storing instructions that, when executed by a processing system, cause a system to perform operations comprising:
generating a tenant-specific model that projects a network traffic pattern by analyzing a training dataset including values associated with a network traffic signal for a tenant over a training time period, wherein:
the network traffic signal is associated with a service offered by a cloud provider; and
the network traffic pattern establishes a baseline threshold value for each defined time bin in a plurality of defined time bins across the training time period;
determining that a current value associated with the network traffic signal for a current time bin is less than the baseline threshold value established for a corresponding time bin in the tenant-specific model;
in response to determining that the current value associated with the network traffic signal for the current time bin is less than the baseline threshold value, designating the tenant as an unhealthy tenant due to abnormal network traffic behavior;
determining that a total number of unhealthy tenants for the current time bin is greater than a predefined threshold number of unhealthy tenants; and
sending, to an owner of the service and based on the total number of unhealthy tenants being greater than the predefined threshold number of unhealthy tenants, a notification indicating a potential network connectivity issue associated with the service.
20 . The computer-readable storage medium of claim 19 , wherein the network traffic signal and the tenant-specific model are associated with a resource deployed by the service and for the tenant within a defined geographic region of a cloud platform or a distributed computing environment.