System and method for format drift and format anomaly detection
A computerized method for detection of format drift and format anomalies is described. A format representation for each data point of a first data sample is extracted. Transformations of each format representation is conducted, resulting in a first plurality of count values (reference) and a second plurality of count values. Each count value identifies a number of occurrences of a transformed format representation within that data sample. Thereafter, a first probability distribution for the first plurality of count values and a second probability distribution for the second plurality of count values are computed. Analytics using the first and probability distributions are conducted to produce a first metric. A format drift is determined based on an evaluation of the first metric to a second metric operating as a threshold metric. Format anomalies are detected based on analytics of hashed format representation and determination of infrequent usage of a particular format representation.
1. A computerized method comprising:
extracting a format representation for a first data sample of an incoming data stream by at least accessing a data schema for each field of the first data sample to determine a data point type, wherein the first data sample comprises a plurality of data points, each data point of the plurality of data points is maintained within a field of the first data sample and corresponds to a performance measurement directed to (i) computing resource associated with a source of the incoming data stream or (ii) an operating state of the source of the incoming data stream;
conducting transformations on format representations associated with data point types of the first data sample to produce a first plurality of count values, wherein the transformed format representations associated with each data point type within the first data sample operates as a count reference;
accessing a data schema for each field of a second data sample of the incoming data stream to determine a data point type for identifying changes in field format;
conducting transformations on format representations associated with data point types of a second data sample of the incoming data stream to produce a second plurality of count values, wherein the second plurality of count values identifying a number of occurrences of the transformed format representation associated with each data point type within the second data sample;
computing a first probability distribution based on the first plurality of count values;
computing a second probability distribution based on the second plurality of count values;
conducting analytics using the first probability distribution and the second probability distribution to produce a first metric; and
determining a format drift for the data stream in response to evaluating the first metric to a second metric operating as a threshold metric signifying a format drift condition.
2. The computerized method of claim 1 further comprising:
generating the second metric, based on a bootstrap process, for detecting the format drift condition, the second metric including count values for each change in format detected for each data point type of the first data sample.
3. The computerized method of claim 2 , wherein the generating of the second metric comprises (i) clustering data points of similar types from one or more representative data samples selected from training data samples, (ii) extracting one or more format representations associated with each of the clustered data points, (iii) conducting a transformation on each of the one or more format representations to produce one or more transformed format representations for each of the data points associated with the one or more representative data samples, and (iv) computing count value for transformed format representations for each of the data points to represent a number and frequency of occurrence of the transformed format representations for each of the data points.
4. The computerized method of claim 1 , wherein the conducting of the transformations on format representations associated with each data point type comprises conducting transformations on content within input fields of the first data sample and the second data sample, wherein the transformations include conducting one or more hash operations on each format representation to produce a hashed format representation.
5. The computerized method of claim 1 , wherein the extracting of the format representation for the first data sample comprises accessing the data schema associated with each data point of the first data sample to determine the data point type for identifying changes in field format.
6. The computerized method of claim 1 further comprising:
upon determining of the format drift, performing one or more of operations including (i) utilizing a format representation having the format drift as the reference or (ii) issuing an alert to a customer to advise the customer of the format drift.
7. The computerized method of claim 1 , wherein the first metric corresponds to a result produced by a distance function with the first probability distribution and the second probability distribution as inputs.
8. The computerized method of claim 2 , wherein the determining of the format drift includes determining that the first metric identifying the first plurality of count values associated with the first format representation maintained within the second probability distribution is different from count values associated with the first format representation determined by the bootstrap process.
9. The computerized method of claim 1 further comprising:
monitoring for a prescribed number of data points of the second data sample upon which analytics are conducted and a format drift condition is determined;
including the second data sample as part of training data samples utilized to produce the second metric in response to the format drift condition being detected for at least the prescribed number of data points of the second data sample; and
refraining from providing the second data sample for inclusion as part of the training data samples in response to the data points of the second data sample detected with the format drift condition is less than the prescribed number of data points.
10. A non-transitory storage medium having stored thereon software that, when executed, is configured to perform operations comprising:
extracting a format representation for a first data sample of an incoming data stream by at least accessing a data schema for each field of the first data sample to determine a data point type, wherein the first data sample comprises a plurality of data points, each data point of the plurality of data points is maintained within a field of the first data sample and corresponds to a performance measurement directed to (i) computing resource associated with a source of the incoming data stream or (ii) an operating state of the source of the incoming data stream;
conducting transformations on format representations associated with data point types of the first data sample to produce a first plurality of count values, wherein the transformed format representations associated with each data point type within the first data sample operates as a count reference;
accessing a data schema for each field of a second data sample of the incoming data stream to determine a data point type for identifying changes in field format;
conducting transformations on format representations associated with data point types of a second data sample of the incoming data stream to produce a second plurality of count values, wherein the second plurality of count values identifying one or more detected occurrences of the transformed format representation associated with each data point type within the second data sample;
computing a first probability distribution based on the first plurality of count values;
computing a second probability distribution based on the second plurality of count values;
conducting analytics using the first probability distribution and the second probability distribution to produce a first metric; and
determining a format drift for the data stream in response to evaluating the first metric to a second metric operating as a threshold metric signifying a format drift condition.
11. The non-transitory storage medium of claim 10 including software that, when executed, is configured to further perform operations comprising:
generating the second metric, based on a bootstrap process, for detecting the format drift condition, the second metric including count values for each change in format detected for each data point type of the first data sample.
12. The non-transitory storage medium of claim 10 including software that, when executed, is configured to generate the second metric by at least (i) clustering data points of similar types from one or more representative data samples selected from training data samples, (ii) extracting one or more format representations associated with each of the clustered data points, (iii) conducting a transformation on each of the one or more format representations to produce one or more transformed format representations for each of the data points associated with the one or more representative data samples, and (iv) computing a count value for transformed format representation for each of the data points to represent a number of occurrences of the transformed format representation for each of the data points.
13. The non-transitory storage medium of claim 10 including software that, when executed, is configured to conduct the transformations on format representations associated with each data point type comprises conducting transformations on content within input fields of the first data sample and the second data sample, wherein the transformations include conducting one or more hash operations on each format representation to produce a hashed format representation as the transformed format representation.
14. The non-transitory storage medium of claim 10 including software that, when executed, is configured to extract of the format representation for the first data sample by at least accessing the data schema associated with each data point of the first data sample to determine the data point type for identifying changes in field format.
15. The non-transitory storage medium of claim 10 including software that, when executed, is further configured to perform one or more operations in response to determining the format drift, wherein the one or more of operations comprise (i) utilizing a format representation having the format drift as the reference or (ii) issuing an alert to a customer to advise the customer of the format drift.
16. The non-transitory storage medium of claim 11 including software that, when executed, is configured to determine the format drift by at least determining whether the first metric identifying the first plurality of count values associated with the first format representation maintained within the second probability distribution is different from count values associated with the first format representation determined by the bootstrap process.
17. The non-transitory storage medium of claim 10 including software that, when executed, further comprising:
monitoring for a prescribed number of data points of the second data sample upon which analytics are conducted and a format drift condition is determined;
including the second data sample as part of training data samples utilized to produce the second metric in response to the format drift condition being detected for at least the prescribed number of data points of the second data sample; and
refraining from providing the second data sample for inclusion as part of the training data samples in response to the data points of the second data sample detected with the format drift condition is less than the prescribed number of data points.
18. A computing device, comprising:
one or more streaming data processors; and
a non-transitory storage medium having stored thereon an intake system being software including a plurality of components forming a drift detection subsystem, the drift detection subsystem comprises a format anomaly component configured to perform operations including
(i) extracting a format representation for a first data sample of an incoming data stream by at least accessing a data schema for each field of the first data sample to determine a data point type, wherein the first data sample comprises a plurality of data points, each data point of the plurality of data points is maintained within a field of the first data sample and corresponds to a performance measurement directed to (a) computing resource associated with a source of the incoming data stream or (b) an operating state of the source of the incoming data stream,
(ii) conducting transformations on format representations associated with data point types of the first data sample to produce a first plurality of count values, wherein the transformed format representations associated with each data point type within the first data sample operates as a count reference,
(iii) accessing a data schema for each field of a second data sample of the incoming data stream to determine a data point type for identifying changes in field format
(iv) conducting transformations on format representations associated with data point types of a second data sample of the incoming data stream to produce a second plurality of count values, wherein the second plurality of count values identifying one or more detected occurrences of the transformed format representation associated with each data point type within the second data sample,
(v) computing a first probability distribution based on the first plurality of count values,
(vi) computing a second probability distribution based on the second plurality of count values,
(vii) conducting analytics using the first probability distribution and the second probability distribution to produce a first metric, and
(viii) detecting a format drift for the data stream in response to evaluating the first metric to a second metric operating as a threshold metric signifying a format drift condition.
19. The computing device of claim 18 , wherein the format anomaly component is configured to perform the transformation on each format representation by at least performing a hash operation on each format representation.
20. The computing device of claim 19 , wherein the format anomaly component is configured to maintain the count value for a first hashed format representation associated with the first data point of the data points by incrementing a counter assigned to the first hashed format representation.
21. The computing device of claim 19 , wherein the first data point is a timestamp or a port number.
22. The computing device of claim 19 , wherein the format anomaly component is configured to generate the output by at least generating a probability output from an information decay function that receives a first count value directed to a number of occurrences of a first hashed format representation associated with a first type of data point with the data sample under analysis and a second count value directed to a number of occurrences of a second hashed format representation associated with the first type of data point with the data sample under analysis.
23. The computing device of claim 19 , wherein the first type of data point corresponds to a timestamp.