Method and system for operation objects discovery from operation data
A method and system for operation objects discovery from operation data includes performing pattern matching of operation data with patterns in a database. Fields in the operation data are identified as having matching patterns with the database as first potential objects. Data profiling is performed on unmatched fields of the operation data to generate data profiles. The data profiles are field classified and second potential objects are generated. The first potential objects and the second potential objects are de-duplicated, and operation objects are generated.
1. A computer-implemented method of monitoring operation objects of a distributed workload multi-cloud computing platform, comprising:
receiving a workload;
distributing the received workload to the multi-cloud computing platform;
monitoring a real-time streaming operation data of the distributed workload multi-cloud computing platform by a pattern matching module of a computing device;
performing, by the pattern matching module of the computing device, pattern matching of the real-time streaming operation data with patterns in a database;
identifying each field in the operation data having matching patterns with the database as first potential objects;
performing data profiling on each of unmatched fields of the operation data to generate data profiles to profile the distribution of each field with various types of values, wherein the data profiling comprises analyzing the unmatched fields of the operation data and collecting statistics therefrom;
field classifying the data profiles to generate second potential objects based on the data profiles of the unmatched fields of the operation data, by a field classifier module operative to identify a particular class, a possible error pattern, and/or a warning pattern in the data profiles of the unmatched fields;
relieving a storage space of the multi-cloud computing platform by de-duplicating the first potential objects and the second potential objects by deleting duplicative data of the first potential objects and the second potential objects when generating operation objects in response to determining that at least two of the first potential objects and the second potential objects have a same value and different field names in different indices;
updating a knowledge base of the de-duplicated data of the first potential objects and the second potential objects; and
detecting at least one of segmented operations, errors, and/or faults in the distributed workload multi-cloud computing platform based on the generated operation objects.
2. The computer-implemented method according to claim 1 , wherein identifying fields is performed automatically without a selection of fields by using a field name or a meaning.
3. The computer-implemented method according to claim 1 , further comprising updating the database with the generated data profiles.
4. The computer-implemented method according to claim 1 , further comprising updating the database with the generated operation objects.
5. The computer-implemented method according to claim 1 , further comprising training a field classifier with the generated data profiles prior to generating the second potential objects.
6. The computer-implemented method according to claim 5 , wherein training the field classifier includes inputting one or more of patterns, rules, data profiles, and models from the database.
7. The computer-implemented method according to claim 1 , wherein a pattern matching module performs the pattern matching with operation data retrieved from a system log.
8. A computer implemented system having a computer processor coupled to a memory configured to monitor operation objects of a distributed workload multi-cloud computing platform, comprising:
receiving a workload;
distributing the received workload to the multi-cloud computing platform;
monitoring a real-time streaming operation data of the distributed workload multi-cloud computing platform by the computer processor;
a pattern matching module stored in the memory configured to perform pattern matching on the real-time streaming operation data with patterns in a database, and to identify each field in the operation data having matching patterns with the database as first potential objects;
a data profiling module stored in the memory configured to perform data profiling on each of unmatched fields of the operation data to generate data profiles to profile the distribution of each field with various types of values, wherein the data profiling comprises analyzing the unmatched fields of the operation data and collects statistics therefrom;
a field classifier module stored in the memory operative to classify the generated data profiles and to generate second potential objects based on the data profiles of the unmatched fields of the operation data and identify a particular class, a possible error pattern, and/or a warning pattern in the data profiles of the unmatched fields;
a de-duplication module stored in the memory configured to relieve a storage space of the multi-cloud computing platform by removing duplicate objects among the first potential objects and the second potential objects, and to generate operation objects in response to determining that at least two of the first potential objects and the second potential objects have a same value and different field names in different indices; and
detecting at least one of segmented operations, errors, and/or faults in the distributed workload multi-cloud computing platform based on the generated operation objects.
9. The system according to claim 8 , wherein the pattern matching module stored in the memory is further configured to perform identifying fields automatically without a selection of fields by using a field name or a meaning.
10. The system according to claim 8 , further comprising a middle events module stored in the memory configured to update the database with the generated data profiles generated by the data profiling module.
11. The system according to claim 8 , wherein the middle events module stored in the memory is configured to update the database with generated operation objects.
12. The system according to claim 8 , wherein the field classifier module stored in the memory is trained with a training set comprising the generated data profiles prior to generating the second potential objects.
13. The system according to claim 8 , wherein the field classifier module stored in the memory is trained with a training set comprising one or more of patterns, rules, data profiles or models from the database.
14. The system according to claim 8 , wherein the operation data is retrieved from a system log.
15. A non-transitory computer readable storage medium tangibly embodying a computer readable program code having computer readable instructions that, when executed, causes a computer device to perform a method of monitoring operation objects of a distributed workload multi-cloud computing platform, the method comprising:
receiving a workload;
distributing the received workload to the multi-cloud computing platform;
monitoring a real-time streaming operation data of the distributed workload multi-cloud computing platform by a pattern matching module of a computing device;
performing, by the pattern matching module of the computing device, pattern matching of the real-time streaming operation data with patterns in a database;
identifying each field in the operation data having matching patterns with the database as first potential objects;
performing data profiling on each of unmatched fields of the operation data to generate data profiles to profile the distribution of each field with various types of values, wherein the data profiling comprises analyzing the unmatched fields of the operation data and collects statistics therefrom;
field classifying the data profiles to generate second potential objects based on the data profiles of the unmatched fields of the operation data, by a field classifier module operative to identify a particular class, a possible error pattern, and/or a warning pattern in the data profiles of the unmatched fields;
relieving a storage space of the multi-cloud computing platform by de-duplicating the first potential objects and the second potential objects by deleting duplicative data of the first potential objects and the second potential objects when generating operation objects in response to determining that at least two of the first potential objects and the second potential objects have a same value and different field names in different indices;
updating a knowledge base of the de-duplicated data of the first potential objects and the second potential objects; and
detecting at least one of segmented operations, errors, and/or faults in the distributed workload multi-cloud computing platform based on the generated operation objects.
16. The non-transitory computer readable storage medium according to claim 15 , wherein identifying fields is performed automatically without a selection of fields by using a field name or a meaning.
17. The non-transitory computer readable storage medium according to claim 15 , further comprising training a field classifier with the generated data profiles prior to generating the second potential objects.