System and method for anomaly characterization based on joint historical and time-series analysis
One embodiment provides a system for facilitating anomaly detection and characterization. During operation, the system determines, by a computing device, a first set of testing data which includes a plurality of data points, wherein the first set includes a data series for a first variable and one or more second variables. The system identifies anomalies by dividing the first set into a number of groups and performing an inter-quartile range analysis on data in each respective group. The system obtains, from the first set, a second set of testing data which includes a data series from a recent time period occurring before a current time, and which further includes a first data point from the identified anomalies. The system classifies the first data point as a first type of anomaly based on whether a magnitude of a derivative of the second set is greater than a first predetermined threshold.
1. A computer-implemented method for facilitating anomaly detection and characterization, the method comprising:
determining, by a computing device, a first set of testing data which includes a plurality of data points, wherein the first set includes a data series for a first variable and one or more second variables, and wherein the one or more second variables are dependent on the first variable;
identifying anomalies by dividing the first set of testing data into a number of groups and performing an inter-quartile range analysis on data in each respective group;
obtaining, from the first set of testing data, a second set of testing data which includes a data series from a recent time period occurring less than a predetermined period of time before a current time, and which further includes a first data point from the identified anomalies; and
responsive to determining that a magnitude of a derivative of the second set of testing data is greater than a first predetermined threshold, classifying the first data point as a first type of anomaly associated with one or more of: a surge in a control parameter; an on/off event; and a sudden transient change in the control parameter,
thereby enhancing data mining and outlier detection for the data series based on a historical analysis of the first set of testing data and a recent time-series analysis of the second set of testing data.
2. The method of claim 1 , wherein the first variable or the one or more second variables is one or more of:
a parameter associated with a physical device involved in obtaining the first set of testing data,
wherein the parameter is one or more of:
a temperature value;
a count, quantity, or other unit to measure production;
a unit or measurement of flow for a material;
a unit or measurement of pressure for a material; and
any parameter which can be used as a control parameter in measuring another parameter.
3. The method of claim 2 , wherein the physical device is one or more of a sensor, a smart meter, an Internet of Things (IoT) device, and any device which can measure the parameter.
4. The method of claim 1 , further comprising:
responsive to determining that the magnitude of the derivative of the second set of testing data is not greater than the first predetermined threshold classifying the first data point as a second type of anomaly.
5. The method of claim 4 ,
wherein the second type of anomaly is associated with a steady-state anomaly.
6. The method of claim 1 , further comprising:
detecting a trend which indicates high surges associated with a physical device involved in obtaining the first set of testing data, by determining a third set of data points which are classified as anomalies and are attributable to the surge in the control parameter, the on/off event, or the sudden transient change in the control parameter.
7. The method of claim 6 , further comprising:
in response to determining that the trend is not greater than a second predetermined threshold, determining that the physical device is experiencing normal operation; and
in response to determining that the trend is greater than the second predetermined threshold, determining that the physical device is experiencing degradation.
8. The method of claim 1 , further comprising:
classifying, by a user of the computing device, the first data point as the first type of anomaly based on a set of predetermined conditions;
performing, by the user of the computing device, an action to address the classified anomaly,
wherein the action includes one or more of:
a remedial or corrective action to ensure that the classified anomaly no longer occurs;
adjusting a physical parameter which affects the first variable or the one or more second variables; and
adjusting a physical component of a device from which the data series is obtained.
9. A computer system for facilitating anomaly detection and characterization, the computer system comprising:
a processor; and
a storage device storing instructions that when executed by the processor cause the processor to perform a method, the method comprising:
determining, by a computing device, a first set of testing data which includes a plurality of data points, wherein the first set includes a data series for a first variable and one or more second variables, and wherein the one or more second variables are dependent on the first variable;
identifying anomalies by dividing the first set of testing data into a number of groups and performing an inter-quartile range analysis on data in each respective group;
obtaining, from the first set of testing data, a second set of testing data which includes a data series from a recent time period occurring less than a predetermined period of time before a current time, and which further includes a first data point from the identified anomalies; and
responsive to determining that a magnitude of a derivative of the second set of testing data is greater than a first predetermined threshold, classifying the first data point as a first type of anomaly associated with one or more of: a surge in a control parameter; an on/off event; and a sudden transient change in the control parameter,
thereby enhancing data mining and outlier detection for the data series based on a historical analysis of the first set of testing data and a recent time-series analysis of the second set of testing data.
10. The computer system of claim 9 , wherein the first variable or the one or more second variables is one or more of:
a parameter associated with a physical device involved in obtaining the first set of testing data,
wherein the parameter is one or more of:
a temperature value;
a count, quantity, or other unit to measure production;
a unit or measurement of flow for a material;
a unit or measurement of pressure for a material; and
any parameter which can be used as a control parameter in measuring another parameter.
11. The computer system of claim 10 , wherein the physical device is one or more of a sensor, a smart meter, an Internet of Things (IoT) device, and any device which can measure the parameter.
12. The computer system of claim 9 , wherein the method further comprises:
responsive to determining that the magnitude of the derivative of the second set of testing data is not greater than the first predetermined threshold, classifying the first data point as a second type of anomaly.
13. The computer system of claim 12 ,
wherein the second type of anomaly is associated with a steady-state anomaly.
14. The computer system of claim 9 , wherein the method further comprises:
detecting a trend which indicates high surges associated with a physical device involved in obtaining the first set of testing data, by determining a third set of data points which are classified as anomalies and are attributable to the surge in the control parameter, the on/off event, or the sudden transient change in the control parameter.
15. The computer system of claim 14 , wherein the method further comprises:
in response to determining that the trend is not greater than a second predetermined threshold, determining that the physical device is experiencing normal operation; and
in response to determining that the trend is greater than the second predetermined threshold, determining that the physical device is experiencing degradation.
16. The computer system of claim 9 , wherein the method further comprises:
classifying, by a user of the computing device, the first data point as the first type of anomaly based on a set of predetermined conditions;
performing, by the user of the computing device, an action to address the classified anomaly,
wherein the action includes one or more of:
a remedial or corrective action to ensure that the classified anomaly no longer occurs;
adjusting a physical parameter which affects the first variable or the one or more second variables; and
adjusting a physical component of a device from which the data series is obtained.
17. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method, the method comprising:
determining, by a computing device, a first set of testing data which includes a plurality of data points, wherein the first set includes a data series for a first variable and one or more second variables, and wherein the one or more second variables are dependent on the first variable;
identifying anomalies by dividing the first set of testing data into a number of groups and performing an inter-quartile range analysis on data in each respective group;
obtaining, from the first set of testing data, a second set of testing data which includes a data series from a recent time period occurring less than a predetermined period of time before a current time, and which further includes a first data point from the identified anomalies; and
responsive to determining that a derivative of the second set of testing data is greater than a first predetermined threshold, classifying the first data point as a first type of anomaly associated with one or more of: a surge in a control parameter; an on/off event; and a sudden transient change in the control parameter,
thereby enhancing data mining and outlier detection for the data series based on a historical analysis of the first set of testing data and a recent time-series analysis of the second set of testing data.
18. The storage medium of claim 17 , wherein the method further comprises:
responsive to determining that the magnitude of the derivative of the second set of testing data is not greater than the first predetermined threshold classifying the first data point as a second type of anomaly.
19. The storage medium of claim 18 ,
wherein the second type of anomaly is associated with a steady-state anomaly.
20. The storage medium of claim 17 , wherein the method further comprises:
detecting a trend which indicates high surges associated with a physical device involved in obtaining the first set of testing data, by determining a third set of data points which are classified as anomalies and are attributable to the surge in the control parameter, the on/off event, or the sudden transient change in the control parameter;
in response to determining that the trend is not greater than a second predetermined threshold, determining that the physical device is experiencing normal operation; and
in response to determining that the trend is greater than the second predetermined threshold, determining that the physical device is experiencing degradation.