Data selection and sampling system for log parsing and anomaly detection in cloud microservices
A method for log data sampling is disclosed. The method includes receiving logs of a computer system. A log comprises information regarding an operation of the computer system. The method also includes determining a sample of the logs by applying a set of sampling methods to the logs. The method further includes providing the sample of the logs as an input to an anomaly detection model for the computer system. The anomaly detection model identifies a fault in the operation of the computer system based on the input.
1. A method comprising:
receiving logs of a computer system, wherein a log comprises information regarding an operation of the computer system;
determining a sample of the logs by applying a set of sampling methods to the logs to:
determine that the set of sampling methods includes a first sampling method and a second sampling method;
determine a first sub-sample by applying the first sampling method to the logs;
determine a second sub-sample by applying the second sampling method to the rest of the logs other than the first sub-sample; and
determine the first sub-sample and the second sub-sample as the sample of the logs; and
providing the sample of the logs as an input to an anomaly detection model for the computer system, wherein the anomaly detection model identifies a fault in the operation of the computer system based on the input.
2. The method of claim 1 , wherein determining the sample of the logs comprises:
determining, for each sampling method of the set, a respective sub-sample of the logs by applying a respective sampling method to the logs; and
determining a union of sub-samples as the sample of the logs.
3. The method of claim 1 , wherein determining the sample of the logs comprises:
analyzing the sample of the logs by using at least one of a bias in the sample, an error in the identified fault of the anomaly detection model based on the sample, or a statistical distribution of the sample; and
updating the set of sampling methods based on the analysis of the sample.
4. The method of claim 3 , wherein updating the set of sampling methods comprises at least one of adding or removing a sampling method from the set of sampling methods, or adjusting a weight associated with a sampling method in the set.
5. The method of claim 4 , wherein updating the set of sampling methods further comprises changing an order of applying a sampling method in the set to the logs.
6. The method of claim 1 , wherein the set of sampling methods includes at least two of a random sampling, stratified sampling, uncertainty sampling, vocabulary sampling, or topic-based sampling.
7. A system comprising:
a memory having instructions; and
at least one processor in communication with the memory, wherein the at least one processor is configured to:
receive logs of a computer system, wherein a log comprises information regarding an operation of the computer system, and wherein the computer system supports microservices;
determine a sample of the logs by applying a set of sampling methods to the logs to:
determine that the set of sampling methods includes a first sampling method and a second sampling method;
determine a first sub-sample by applying the first sampling method to the logs;
determine a second sub-sample by applying the second sampling method to the rest of the logs other than the first sub-sample; and
determine the first sub-sample and the second sub-sample as the sample of the logs; and
provide the sample of the logs as an input to an anomaly detection model for the computer system, wherein the anomaly detection model identifies a fault in the operation of the computer system based on the input.
8. The system of claim 7 , wherein to determine the sample of the logs, the at least one processor is configured to:
determine, for each sampling method of the set, a respective sub-sample of the logs by applying a respective sampling method to the logs; and
determine a union of sub-samples as the sample of the logs.
9. The system of claim 7 , wherein to determine the sample of the logs, the at least one processor is configured to:
analyze the sample of the logs by using at least one of a bias in the sample, an error in the identified fault of the anomaly detection model based on the sample, or a statistical distribution of the sample; and
update the set of sampling methods based on the analysis of the sample.
10. The system of claim 9 , wherein to update the set of sampling methods, the at least one processor is configured to perform at least one of adding or removing a sampling method from the set of sampling methods, or adjusting a weight associated with a sampling method in the set.
11. The system of claim 10 , wherein to update the set of sampling methods, the at least one processor is further configured to change an order of applying a sampling method in the set to the logs.
12. The system of claim 7 , wherein the set of sampling methods includes at least two of a random sampling, stratified sampling, uncertainty sampling, vocabulary sampling, or topic-based sampling.
13. A computer program product comprising:
a computer readable storage medium having program instructions that are executable by at least one processor to cause the at least one processor to:
receive logs of a computer system, wherein a log comprises information regarding an operation of the computer system;
determine a sample of the logs by applying a set of sampling methods to the logs to:
determine that the set of sampling methods includes a first sampling method and a second sampling method;
determine a first sub-sample by applying the first sampling method to the logs;
determine a second sub-sample by applying the second sampling method to the rest of the logs other than the first sub-sample; and
the first sub-sample and the second sub-sample as the sample of the logs; and
provide the sample of the logs as an input to an anomaly detection model for the computer system, wherein the anomaly detection model identifies a fault in the operation of the computer system based on the input.
14. The computer program product of claim 13 , wherein to determine the sample of the logs, the program instructions cause the at least one processor to:
determine, for each sampling method of the set, a respective sub-sample of the logs by applying a respective sampling method to the logs; and
determine a union of sub-samples as the sample of the logs.
15. The computer program product of claim 13 , wherein to determine the sample of the logs, the program instructions cause the at least one processor to:
analyze the sample of the logs by using at least one of a bias in the sample, an error in the identified fault of the anomaly detection model based on the sample, or a statistical distribution of the sample; and
update the set of sampling methods based on the analysis of the sample.
16. The computer program product of claim 15 , wherein to update the set of sampling methods, the program instructions cause the at least one processor to perform at least one of adding or removing a sampling method from the set of sampling methods, or adjusting a weight associated with a sampling method in the set.
17. The computer program product of claim 16 , wherein to update the set of sampling methods, the program instructions cause the at least one processor to change an order of applying a sampling method in the set to the logs.