Efficient detection and prediction of data pattern changes in a cloud-based application acceleration as a service environment
A method implemented through a data processing device in a computing network includes sampling time series data associated with each network entity for each feature thereof into a smaller time interval as a first data series and a second data series including a maximum value and a minimum value respectively of the sampled time series data for the each feature within the smaller time interval, and generating a reference data band from predicted future data sets. The method also includes detecting, based on the reference data band, an anomaly in real-time data associated with the each network entity for the each feature thereof, and determining an event associated with a pattern of change of the real-time data associated with the each network entity based on executing an optimization algorithm to determine a series of anomalies including the detected anomaly.
1. A method of a computing network implemented in a data processing device comprising a processor communicatively coupled to a memory, comprising:
sampling time series data associated with each network entity of a plurality of network entities of the computing network for each feature thereof into a smaller time interval compared to that of the time series data as a first data series comprising a maximum value of the sampled time series data for the each feature within the smaller time interval and a second data series comprising a minimum value of the sampled time series data for the each feature within the smaller time interval;
generating a reference data band based on:
predicting a first future data set of the each network entity for the each feature based on the first data series and a second future data set of the each network entity for the each feature based on the second data series;
combining the first future data set and the second future data set for each future time interval thereof; and
transforming the combined first future data set and the second future data set for the each future time interval into the reference data band;
based on regarding a maximum of the first future data set as a maximum expected value of the reference data band and a minimum of the second future data set as a minimum expected value of the reference data band, detecting at least one anomaly in real-time data associated with the each network entity for the each feature thereof based on determining whether the real-time data falls outside the maximum expected value and the minimum expected value of the reference data band in accordance with computing a score for the at least one anomaly indicative of anomalousness thereof, the computation of the score involving both relative scoring and absolute deviation scoring, the absolute deviation scoring being based on previous data deviations from reference data bands analogous to the reference data band associated with the each network entity, and the absolute deviation scoring further comprising:
preserving a first discrete data distribution for the each network entity for the each feature for associated anomalies with values higher than the maximum expected value of the reference data band and a second discrete data distribution for the each network entity for the each feature for other associated anomalies with values lower than the minimum expected value of the reference data band, both the first discrete data distribution and the second discrete data distribution having a probability mass function of the previous data deviations from the reference data bands analogous to the reference data band associated with the each network entity; and
computing a cumulative probability utilizing a deviation value of the detected at least one anomaly from the reference data band; and
determining an event associated with a pattern of change of the real-time data associated with the each network entity based on executing an optimization algorithm to determine, among all features of the each network entity, a series of anomalies comprising the detected at least one anomaly that constitutes a sequence of patterned anomalies in accordance with scanning detected anomalies associated with the real-time data associated with the each network entity including the detected at least one anomaly.
2. The method of claim 1 , further comprising upsampling the reference data band by the smaller time interval to restore data granularity.
3. The method of claim 1 , wherein determining the event based on executing the optimization algorithm further comprises finding, based on at least one dynamic programming technique, a longest occurring sequence of anomalies as the sequence of anomalies among all the features of the each network entity that is capable of being interleaved for a specific duration.
4. The method of claim 1 , further comprising enabling predictive classification of a future event associated with the each network entity into a category of determined problems based on the determined event associated with the pattern of change of the real-time data associated with the each network entity.
5. The method of claim 1 , further comprising at least one of:
collecting feedback from another data processing device of the computing network communicatively coupled to the data processing device via the computing network; and
determining a severity indicator for the determined event based on the feedback from the another data processing device.
6. The method of claim 5 , further comprising generating an event score for the determined event utilizing the determined severity indicator.
7. The method of claim 6 , further comprising ranking the determined event with respect to a plurality of events based on the generated event score.
8. A data processing device comprising:
a memory; and
a processor communicatively coupled to the memory, the processor executing instructions to:
sample time series data associated with each network entity of a plurality of network entities of a computing network for each feature thereof into a smaller time interval compared to that of the time series data as a first data series comprising a maximum value of the sampled time series data for the each feature within the smaller time interval and a second data series comprising a minimum value of the sampled time series data for the each feature within the smaller time interval,
generate a reference data band based on:
predicting a first future data set of the each network entity for the each feature based on the first data series and a second future data set of the each network entity for the each feature based on the second data series,
combining the first future data set and the second future data set for each future time interval thereof, and
transforming the combined first future data set and the second future data set for the each future time interval into the reference data band,
based on regarding a maximum of the first future data set as a maximum expected value of the reference data band and a minimum of the second future data set as a minimum expected value of the reference data band, detect at least one anomaly in real-time data associated with the each network entity for the each feature thereof based on determining whether the real-time data falls outside the maximum expected value and the minimum expected value of the reference data band in accordance with computing a score for the at least one anomaly indicative of anomalousness thereof, the computation of the score involving both relative scoring and absolute deviation scoring, the absolute deviation scoring being based on previous data deviations from reference data bands analogous to the reference data band associated with the each network entity, and the absolute deviation scoring further comprising:
preserving a first discrete data distribution for the each network entity for the each feature for associated anomalies with values higher than the maximum expected value of the reference data band and a second discrete data distribution for the each network entity for the each feature for other associated anomalies with values lower than the minimum expected value of the reference data band, both the first discrete data distribution and the second discrete data distribution having a probability mass function of the previous data deviations from the reference data bands analogous to the reference data band associated with the each network entity, and
computing a cumulative probability utilizing a deviation value of the detected at least one anomaly from the reference data band, and
determine an event associated with a pattern of change of the real-time data associated with the each network entity based on executing an optimization algorithm to determine, among all features of the each network entity, a series of anomalies comprising the detected at least one anomaly that constitutes a sequence of patterned anomalies in accordance with scanning detected anomalies associated with the real-time data associated with the each network entity including the detected at least one anomaly.
9. The data processing device of claim 8 , wherein the processor further executes instructions to upsample the reference data band by the smaller time interval to restore data granularity.
10. The data processing device of claim 8 , wherein the processor executes instructions to determine the event based on executing the optimization algorithm in accordance with executing instructions to, based on at least one dynamic programming technique, find a longest occurring sequence of anomalies as the sequence of anomalies among all the features of the each network entity that is capable of being interleaved for a specific duration.
11. The data processing device of claim 8 , wherein the processor further executes instructions to enable predictive classification of a future event associated with the each network entity into a category of determined problems based on the determined event associated with the pattern of change of the real-time data associated with the each network entity.
12. The data processing device of claim 8 , wherein the processor further executes instructions to at least one of:
collect feedback from another data processing device of the computing network communicatively coupled to the data processing device via the computing network, and
determine a severity indicator for the determined event based on the feedback from the another data processing device.
13. The data processing device of claim 12 , wherein the processor further executes instructions to generate an event score for the determined event utilizing the determined severity indicator.
14. The data processing device of claim 13 , wherein the processor further executes instructions to rank the determined event with respect to a plurality of events based on the generated event score.
15. A computing system comprising:
a plurality of network entities of a computer network; and
a data processing device executing instructions to:
sample time series data associated with each network entity of the plurality of network entities for each feature thereof into a smaller time interval compared to that of the time series data as a first data series comprising a maximum value of the sampled time series data for the each feature within the smaller time interval and a second data series comprising a minimum value of the sampled time series data for the each feature within the smaller time interval,
generate a reference data band based on:
predicting a first future data set of the each network entity for the each feature based on the first data series and a second future data set of the each network entity for the each feature based on the second data series,
combining the first future data set and the second future data set for each future time interval thereof, and
transforming the combined first future data set and the second future data set for the each future time interval into the reference data band,
based on regarding a maximum of the first future data set as a maximum expected value of the reference data band and a minimum of the second future data set as a minimum expected value of the reference data band, detect at least one anomaly in real-time data associated with the each network entity for the each feature thereof based on determining whether the real-time data falls outside the maximum expected value and the minimum expected value of the reference data band in accordance with computing a score for the at least one anomaly indicative of anomalousness thereof, the computation of the score involving both relative scoring and absolute deviation scoring, the absolute deviation scoring being based on previous data deviations from reference data bands analogous to the reference data band associated with the each network entity, and the absolute deviation scoring further comprising:
preserving a first discrete data distribution for the each network entity for the each feature for associated anomalies with values higher than the maximum expected value of the reference data band and a second discrete data distribution for the each network entity for the each feature for other associated anomalies with values lower than the minimum expected value of the reference data band, both the first discrete data distribution and the second discrete data distribution having a probability mass function of the previous data deviations from the reference data bands analogous to the reference data band associated with the each network entity, and
computing a cumulative probability utilizing a deviation value of the detected at least one anomaly from the reference data band, and
determine an event associated with a pattern of change of the real-time data associated with the each network entity based on executing an optimization algorithm to determine, among all features of the each network entity, a series of anomalies comprising the detected at least one anomaly that constitutes a sequence of patterned anomalies in accordance with scanning detected anomalies associated with the real-time data associated with the each network entity including the detected at least one anomaly.
16. The computing system of claim 15 , wherein the data processing device executes instructions to determine the event based on executing the optimization algorithm in accordance with, based on at least one dynamic programming technique, finding a longest occurring sequence of anomalies as the sequence of anomalies among all the features of the each network entity that is capable of being interleaved for a specific duration.
17. The computing system of claim 15 , wherein the data processing device further executes instructions to enable predictive classification of a future event associated with the each network entity into a category of determined problems based on the determined event associated with the pattern of change of the real-time data associated with the each network entity.
18. The computing system of claim 15 , wherein the data processing device further executes instructions to at least one of:
upsample the reference data band by the smaller time interval to restore data granularity, and
collect feedback from another data processing device of the computer network communicatively coupled to the data processing device via the computer network.
19. The computing system of claim 18 , wherein the data processing device further executes instructions to determine a severity indicator for the determined event based on the feedback from the another data processing device.
20. The computing system of claim 19 , wherein the data processing device further executes instructions to at least one of:
generate an event score for the determined event utilizing the determined severity indicator, and
rank the determined event with respect to a plurality of events based on the generated event score.