Systems and methods for self-correcting network equipment
A system and method for mitigating anomalies in a network device. One or a combination of mitigation actions may be pre-programmed into a plurality of customer premises equipment (CPE) devices. The mitigation actions may be available as temporary or permanent work arounds as problems are detected that match known or discovered patterns of log events. Unsupervised machine learning techniques may be used to identify and correlate new patterns of log events with device anomalies or malfunctions. When a new pattern of log events is discovered and correlated to an anomaly or malfunction in a group of CPE devices, the discovered pattern may be written to the group of CPE devices with instructions to use particular ones of the mitigation actions. The CPE devices can then, in examples, monitor their own logs to detect the patterns and implement one or more of the mitigation actions.
1. A method for mitigating anomalies in a network device comprising:
receiving log event information for a plurality of network devices;
determining a first pattern of log event information correlated to an anomaly of at least a first network device;
selecting a first mitigation process including one or more mitigation actions; and
instructing the first network device to monitor its own log event information for the first pattern of log event information and, when the first pattern of log event information is detected, to implement the first mitigation process.
2. The method of claim 1 , wherein determining the first pattern of log event information comprises:
receiving the log event information; and
performing string recognition for detecting log events in the log event information that match the first pattern.
3. The method of claim 2 , wherein instructing the first network device comprises instructing the plurality of network devices to each monitor its own log event information for the first pattern of log event information and, when the first pattern of log event information is detected, to implement the first mitigation process.
4. The method of claim 2 , further comprising:
when the first pattern is detected in the log event information, determining a match count;
when the match count meets a count threshold, categorizing the first pattern; and
based on the category, causing the first network device to perform an upload of detailed log information from the first network device.
5. The method of claim 4 , wherein determining the match count comprises correlating the first pattern to customer complaints and service calls information.
6. The method of claim 1 , further comprising:
using unsupervised machine learning techniques to determine a second pattern of log event information correlated to a second anomaly of at least the first network device;
selecting a second mitigation process including the one or more mitigation actions; and
instructing the first network device to monitor its own log event information for the second pattern of log event information and, when the second pattern of log event information is detected, to implement the second mitigation process.
7. The method of claim 1 , wherein receiving log event information for the plurality of network devices comprises receiving log event information for a plurality of network devices operating at a plurality of customer premises.
8. A method for mitigating anomalies in a network device, comprising:
storing a plurality of mitigation actions at the network device;
logging events associated with software, firmware, or hardware of the network device;
receiving a first pattern of log event information correlated to an anomaly and a first mitigation process to implement when the first pattern of log event information is detected;
monitoring, at the network device, logged events for the first pattern of log event information; and
when the first pattern of log event information is detected, implementing the first mitigation process at the network device, wherein the first mitigation process includes one or more of the mitigation actions.
9. The method of claim 8 , wherein receiving the first pattern of log event information and the first mitigation process comprises receiving the first pattern of log event information and the first mitigation process in a software or firmware upgrade.
10. The method of claim 8 , wherein monitoring logged events for the first pattern of log event information comprises performing string recognition for detecting log events in the log event information that match the first pattern.
11. The method of claim 10 , wherein monitoring logged events for the first pattern of log event information comprises detecting log events that match the first pattern within a limited time period.
12. The method of claim 8 , further comprising uploading logged events to a server for evaluation.
13. The method of claim 8 , further comprising:
receiving a second pattern of log event information correlated to a second anomaly and a second mitigation process to implement when the second pattern of log event information is detected;
monitoring logged events for the second pattern of log event information; and
when the second pattern of log event information is detected, implementing the second mitigation process, wherein the second mitigation process includes one or more of the mitigation actions.
14. The method of claim 8 , further comprising:
generating a log of detailed information about the logged events;
receiving a request for the detailed information about the logged events; and
providing the detailed information about the logged events.
15. A system for mitigating anomalies in a network device, comprising:
at least one processor; and
memory storing instructions that, when executed by the at least one processor, cause the system to:
receive log event information for a plurality of network devices;
determine a first pattern of log event information correlated to an anomaly of at least a first network device;
select a first mitigation process including one or more mitigation actions; and
instruct the first network device to monitor its own log event information for the first pattern of log event information and, when the first pattern of log event information is detected, to implement the first mitigation process.
16. The system of claim 15 , wherein the first pattern of log event information comprises one or more events that occur within a limited time period.
17. The system of claim 15 , wherein in determining the first pattern of log event information, the instructions cause the system to:
detect the first pattern in the log event information;
determine a match count;
when the match count meets a count threshold, categorize the first pattern; and
based on the category, trigger an upload of detailed log information from the first network device.
18. The system of claim 15 , wherein the match count includes a correlation of the first pattern to customer complaints and service calls information.
19. The system of claim 15 , wherein the instructions further cause the system to:
use unsupervised machine learning techniques to determine a second pattern of log event information correlated to a second anomaly of at least the first network device;
select a second mitigation process including the one or more mitigation actions programmed in the first network device; and
instruct the first network device to monitor its own log event information for the second pattern of log event information and, when the second pattern of log event information is detected, to implement the second mitigation process.
20. The system of claim 15 , wherein the plurality of network devices comprises a plurality of network devices operating at a plurality of customer premises.