Predicting and reducing hardware related outages
Disclosed here is a system to automatically predict and reduce hardware related outages. The system can obtain a performance indicator associated with a wireless telecommunication network including a system performance indicator or an application log, along with a machine learning model trained to predict and resolve a hardware error based on the performance indicator. The machine learning model can detect an anomaly associated with the performance indicator by detecting an infrequent occurrence in the performance indicator. The machine learning model can determine whether the anomaly is similar to a prior anomaly indicating a prior hardware error. Upon determining that the anomaly is similar to the prior hardware error, the machine learning model can predict an occurrence of the hardware error.
1 . At least one tangible computer-readable medium, which when executed by at least one data processor, implements operations, comprising:
obtaining, from a hardware component of a wireless telecommunications network, a performance indicator associated with the wireless telecommunications network, wherein the performance indicator includes at least one of a system performance indicator or an application log;
providing the performance indicator to a machine learning model trained to address a hardware error,
wherein the machine learning model is trained to predict an occurrence of a hardware error and determine a resolution for the hardware error based on at least two of: prior hardware errors, a historical application log, historical system performance indicators, or a historical record of prior hardware errors, the historical record of prior hardware errors comprising a plurality of issue tickets and a plurality of issue ticket resolutions, and wherein the machine learning model is trained by:
obtaining the historical application log, the historical system performance indicators, or the historical record of prior hardware errors comprising the plurality of issue tickets and the plurality of issue ticket resolutions; and
training the machine learning model to detect an anomaly, predict the occurrence of the hardware error, and obtain the resolution to the prior hardware error based on the historical application log, the historical system performance indicators, and the historical record of prior hardware errors comprising the plurality of issue tickets and the plurality of issue ticket resolutions;
detecting, using the machine learning model, an anomaly associated with the performance indicator based on a frequency of occurrence of the performance indicator, wherein the anomaly is associated with a component of the wireless telecommunications network;
identifying, using a service registry, one or more additional components likely to be affected by the anomaly, wherein the one or more additional components comprise components of the wireless telecommunications network that are connected to the component,
wherein the service registry comprises a directed graph network, wherein each node of the directed graph network represents one or more components of the wireless telecommunications network, and wherein each edge of the directed graph network indicates dependencies between components of the wireless telecommunications network;
automatically notifying the one or more additional components of the anomaly, wherein the automatic notification prevents propagation of the hardware error to the one or more additional components;
determining, using the machine learning model, a resolution to the anomaly based on a prior anomaly indicating a prior hardware error, wherein determining the resolution comprises determining similarity between the anomaly and the prior anomaly by fitting a curve to data points in a temporal window surrounding the anomaly, fitting a prior curve to data points in a prior temporal window surrounding the prior anomaly, and calculating a difference between the curve and the prior curve; and
automatically executing the resolution to address the hardware error prior to the hardware error affecting users of the wireless telecommunications network.
2 . The at least one tangible computer-readable medium of claim 1 , wherein the performance indicator includes a CPU load or the application log associated with a piece of hardware.
3 . The at least one tangible computer-readable medium of claim 1 , wherein the hardware error is associated with a hardware element comprising a load balancer.
4 . The at least one tangible computer-readable medium of claim 1 , wherein the system performance indicator comprises a CPU load of a CPU, and wherein detecting the anomaly comprises:
determining that the CPU load of the CPU exceeds a temporally proximate CPU load of the CPU by more than 10%.
5 . The at least one tangible computer-readable medium of claim 1 , further comprising instructions to:
categorize the anomaly into a category where the anomaly occurred, the category including the application log and the system performance indicator.
6 . The at least one tangible computer-readable medium of claim 1 , further comprising instructions to:
provide an indication of the hardware error.
7 . The at least one tangible computer-readable medium of claim 1 , further comprising instructions to:
determine an anomaly in CPU load by performing a regression analysis of CPU load values, wherein the regression analysis determines at least one of a linear regression, a sinusoidal regression, a polynomial regression, or a logistic regression.
8 . A method comprising:
obtaining, from a hardware component of a wireless telecommunications network, a performance indicator associated with the wireless telecommunications network, wherein the performance indicator comprises at least one of a system performance indicator or an application log;
providing the performance indicator to a machine learning model trained to address a hardware error,
wherein the machine learning model is trained to predict an occurrence of a hardware error and determine a resolution a resolution for the hardware error based on at least two of: prior hardware errors, a historical application log, historical system performance indicators, or a historical record of prior hardware errors comprising a plurality of issue tickets and a plurality of issue ticket resolutions, and wherein the machine learning model is trained by:
obtaining the historical application log, the historical system performance indicators, or the historical record of prior hardware errors comprising the plurality of issue tickets and the plurality of issue ticket resolutions; and
training the machine learning model to detect an anomaly, predict the occurrence of the hardware error, and obtain the resolution to the prior hardware error based on the historical application log, the historical system performance indicators, and the historical record of prior hardware errors comprising the plurality of issue tickets and the plurality of issue ticket resolutions;
detecting, using the machine learning model, an anomaly associated with the performance indicator based on an infrequent occurrence in the performance indicator, wherein the anomaly is associated with a component of the wireless telecommunications network;
identifying, using a service registry, one or more additional components likely to be affected by the anomaly, wherein the one or more additional components comprise components of the wireless telecommunications network that are connected to the component,
wherein the service registry comprises a directed graph network, wherein each node of the directed graph network represents one or more components of the wireless telecommunications network, and wherein each edge of the directed graph network indicates dependencies between components of the wireless telecommunications network; and
automatically notifying the one or more additional components of the anomaly, wherein the automatic notification prevents propagation of the hardware error to the one or more additional components;
determining, using the machine learning model, a resolution to the anomaly based on a prior anomaly indicating a prior hardware error, wherein determining the resolution comprises determining similarity between the anomaly and the prior anomaly by fitting a curve to data points in a temporal window surrounding the anomaly, fitting a prior curve to data points in a prior temporal window surrounding the prior anomaly, and calculating a difference between the curve and the prior curve; and
automatically executing the resolution to address the hardware error prior to the hardware error affecting users of the wireless telecommunications network.
9 . The method of claim 8 , wherein the hardware error is associated with a hardware element comprising a load balancer.
10 . The method of claim 8 , wherein the system performance indicator comprises a CPU load of a CPU, and wherein the detecting the anomaly comprises:
detecting the CPU load of the CPU exceeding a temporally proximate CPU load of the CPU by more than 10%.
11 . The method of claim 8 , further comprising:
categorizing the anomaly into a category where the anomaly occurred, the category including the application log and the system performance indicator.
12 . The method of claim 8 , further comprising:
providing an indication of the hardware error.
13 . The method of claim 8 , further comprising:
determining an anomaly in CPU load by performing a regression analysis of CPU load values, wherein the regression analysis determines at least one of a linear regression, a sinusoidal regression, a polynomial regression, or a logistic regression.
14 . A system comprising:
one or more processors; and
a memory coupled to the one or more processors, wherein the memory stores instructions executable by the one or more processors to:
obtain, from a hardware component of a wireless telecommunications network, a performance indicator associated with the wireless telecommunications network, wherein the performance indicator includes at least one of a system performance indicator or an application log;
provide the performance indicator to a machine learning model trained to address a hardware error,
wherein the machine learning model is trained to predict an occurrence of a hardware error and determine a resolution for the hardware error based on at least two of: prior hardware errors, a historical application log, historical system performance indicators, or a historical record of prior hardware errors, the historical record of prior hardware errors comprising a plurality of issue tickets and a plurality of issue ticket resolutions, and wherein the machine learning model is trained by:
obtaining the historical application log, the historical system performance indicators, or the historical record of prior hardware errors comprising the plurality of issue tickets and the plurality of issue ticket resolutions; and
training the machine learning model to detect an anomaly, predict the occurrence of the hardware error, and obtain the resolution to the prior hardware error based on the historical application log, the historical system performance indicators, and the historical record of prior hardware errors comprising the plurality of issue tickets and the plurality of issue ticket resolutions;
detect, using the machine learning model, an anomaly associated with the performance indicator based on an infrequent occurrence in the performance indicator;
identify, using a service registry, one or more additional components likely to be affected by the anomaly, wherein the one or more additional components comprise components of the wireless telecommunications network that are connected to the component,
wherein the service registry comprises a directed graph network, wherein each node of the directed graph network represents one or more components of the wireless telecommunications network, and wherein each edge of the directed graph network indicates dependencies between components of the wireless telecommunications network; and
automatically notify the one or more additional components of the anomaly, wherein the automatic notification prevents propagation of the hardware error to the one or more additional components;
determine, using the machine learning model, a resolution to the anomaly based on a prior anomaly indicating a prior hardware error, wherein determining the resolution comprises determining similarity between the anomaly and the prior anomaly by fitting a curve to data points in a temporal window surrounding the anomaly, fitting a prior curve to data points in a prior temporal window surrounding the prior anomaly, and calculating a difference between the curve and the prior curve; and
automatically execute the resolution to address the hardware error prior to the hardware error affecting users of the wireless telecommunications network.
15 . The system of claim 14 , wherein the performance indicator includes a CPU load or the application log associated with a piece of hardware.
16 . The system of claim 14 , wherein the hardware error is associated with a hardware element comprising a load balancer.
17 . The system of claim 14 , wherein the system performance indicator comprises a CPU load of a CPU, and wherein the instructions to detect the anomaly comprise instructions to:
detect the CPU load of the CPU exceeding a temporally proximate CPU load of the CPU by more than 10%.
18 . The system of claim 14 , wherein the instructions are further configured to, when executed by the one or more processors, cause the system to:
categorize the anomaly into a category where the anomaly occurred, the category including the application log and the system performance indicator.
19 . The system of claim 14 , further comprising instructions to:
provide an indication of the hardware error and the resolution.
20 . The system of claim 14 , further comprising instructions to:
determine an anomaly in CPU load by performing a regression analysis of CPU load values, wherein the regression analysis determines at least one of a linear regression, a sinusoidal regression, a polynomial regression, or a logistic regression.