Automated meta parameter search for invariant based anomaly detectors in log analytics

1. A computer-implemented method for automatically generating a set of meta-parameters used to train invariant-based anomaly detectors, comprising:

transforming, by a processor, data into a first set of time series data and a second set of time series data;

performing, by the processor, a fitness threshold search based on the first set of time series data to automatically generate a fitness threshold;

performing, by the processor, a time resolution search on the second set of time series data to automatically generate a time resolution; and

sending, by the processor to one or more user devices across a network, a set of meta-parameters including the fitness threshold and the time resolution to govern the training of an invariant-based anomaly detector,

wherein, in transforming the data into the first set of time series data, the processor is further configured to:

determine a maximal time resolution for the fitness threshold search; and

use time series alignment to generate the first set of time series data based on the maximal time resolution;

wherein the determination of the maximal time resolution comprises, given data spanning from a timestamp t 1 to a timestamp t 2 , a determination of a maximum k such that

t

2

-

t

1

B

⁢

⁢

2

k

≥

M

,

where B is a base integer and M is a user-set minimal number of sample points.

2. The method as recited in claim 1 , wherein the data includes log data generated by a computer system.

3. The method as recited in claim 2 , wherein the log data includes text logs.

4. The method as recited in claim 2 , wherein the log data includes performance logs.

5. The method as recited in claim 1 , wherein performing the fitness threshold search to automatically generate the fitness threshold further comprises:

performing an invariant analysis to generate all-pair invariants from the first set of time series data;

classifying the all-pair invariants into groups based on a fitness of each all-pair invariant, wherein each of the groups has a lower bound defined by a boundary value, and wherein the groups are organized in descending order of fitness such that a first one of the groups has a lower bound defined by a highest boundary value and a last one of the groups has a lower bound defined by a boundary value of zero;

determining that at least one of the groups, excluding the last group, is non-empty; and

setting the fitness threshold to a boundary value corresponding to the non-empty group having the highest boundary value.

6. The method as recited in claim 1 , wherein transforming the data into the second set of time series data further comprises:

determining a selected time resolution for the time resolution search based on a current iteration; and

using time series alignment to generate the first set of time series data based on the selected time resolution.

7. The method as recited in claim 6 , wherein:

if the current iteration is a first iteration, the selected time resolution is determined to be B2 k max ; and

if the current iteration is an (i+1)-th iteration for i>0, the selected time resolution is determined to be B2 k i+1 ;

where B is a base integer, k max is a maximum non-negative integer that satisfies

Δ

⁢

⁢

t

B

⁢

⁢

2

k

max

≥

100

,

k

i

+

1

=

k

i

-

1

,

and k i >0.

8. The method as recited in claim 1 , wherein performing the time resolution search to automatically generate the time resolution further comprises:

performing an invariant analysis to generate all-pair invariants from the second set of time series data; and

determining the time resolution based on the invariant analysis based on a current iteration.

9. The method as recited in claim 8 , wherein:

if the current iteration is a first iteration, the time resolution is determined to be a baseline time resolution T 1 used to generate a baseline set of invariants S 1 ; and

if the current iteration is an (i+1)-th iteration for i>0 associated with a time resolution T i+1 , another iteration of transforming the data into the second time series data is performed if T 1 and T i+1 are equivalent and T i+1 >B, otherwise the time resolution is determined to be 2T i+1 ;

where B is a base integer.

10. A system to automatically generate a set of meta-parameters used to train invariant-based anomaly detectors, comprising:

a computer system configured to generate data including log data; and

a processor operatively coupled to a memory, wherein the processor is configured to:

obtain the data from the computer system, and transform the obtained data into a first set of time series data and a second set of time series data;

perform a fitness threshold search based on the first set of time series data to automatically generate a fitness threshold;

perform a time resolution search on the second set of time series data to automatically generate a time resolution; and

send, to one or more user devices across a network, a set of meta-parameters including the fitness threshold and the time resolution to govern the training of an invariant-based anomaly detector,

wherein, in transforming the data into the first set of time series data, the processor is further configured to:

determine a maximal time resolution for the fitness threshold search; and

use time series alignment to generate the first set of time series data based on the maximal time resolution;

wherein the determination of the maximal time resolution comprises, given data spanning from a timestamp t 1 to a timestamp t 2 , a determination of a maximum k such that

t

2

-

t

1

B

⁢

⁢

2

k

≥

M

,

where B is a base integer and M is a user-set minimal number of sample points.

11. The system as recited in claim 10 , wherein, in performing the fitness threshold search to automatically generate the fitness threshold, the process is further configured to:

perform an invariant analysis to generate all-pair invariants from the first set of time series data;

classify the all-pair invariants into groups based on a fitness of each all-pair invariant, wherein each of the groups has a lower bound defined by a boundary value, and wherein the groups are organized in descending order of fitness such that a first one of the groups has a lower bound defined by a highest boundary value and a last one of the groups has a lower bound defined by a boundary value of zero;

determine that at least one of the groups, excluding the last group, is non-empty;

set the fitness threshold to a boundary value corresponding to the non-empty group having the highest boundary value; and

set the fitness threshold to a given value in response to determining that none of the groups excluding the last group is non-empty.

12. The system as recited in claim 10 , wherein, in transforming the data into the second set of time series data, the processor is further configured to:

determine a selected time resolution for the time resolution search based on a current iteration; and

use time series alignment to generate the first set of time series data based on the selected time resolution

wherein:

if the current iteration is a first iteration, the selected time resolution is determined to be B2 k max ; and

if the current iteration is an (i+1)-th iteration for i>0, the selected time resolution is determined to be B2 k i+1 ;

where B is a base integer, k max is a maximum non-negative integer that satisfies

Δ

⁢

⁢

t

B

⁢

⁢

2

k

max

≥

100

,

k

i

+

1

=

k

i

-

1

,

and k i >0.

13. The system as recited in claim 10 , wherein, in performing the time resolution search to automatically generate the time resolution, the processor is further configured to:

perform an invariant analysis to generate all-pair invariants from the second set of time series data; and

determine the time resolution based on the invariant analysis based on a current iteration wherein:

if the current iteration is a first iteration, the time resolution is determined to be a baseline time resolution T i used to generate a baseline set of invariants S 1 ; and

if the current iteration is an (i+1)-th iteration for i>0 associated with a time resolution T i+1 , another iteration of transforming the data into the second time series data is performed if T i and T i+1 are equivalent and T i+1 >B, otherwise the time resolution is determined to be 2T i+1 ;

where B is a base integer.

14. A computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method for automatically generating a set of meta-parameters used to train invariant-based anomaly detectors comprising:

transforming data into a first set of time series data and a second set of time series data;

performing a fitness threshold search based on the first set of time series data to automatically generate a fitness threshold;

performing a time resolution search on the second set of time series data to automatically generate a time resolution; and

send, to one or more user devices across a network, a set of meta-parameters including the fitness threshold and the time resolution to govern the training of an invariant-based anomaly detector,

wherein, in transforming the data into the first set of time series data, the processor is further configured to:

determine a maximal time resolution for the fitness threshold search; and

use time series alignment to generate the first set of time series data based on the maximal time resolution;

wherein the determination of the maximal time resolution comprises, given data spanning from a timestamp t 1 to a timestamp t 2 , a determination of a maximum k such that

t

2

-

t

1

B

⁢

⁢

2

k

≥

M

,

where B is a base integer and M is a user-set minimal number of sample points.