Diagnosing application problems by learning from fault injections
Identifying a likely cause of a problem in an application can include: injecting a series of a set of predetermined faults into the application; sampling a subset of a set of training features from the application during each predetermined fault injected into the application and labeling each subset with an identifier of the corresponding predetermined fault; and training a classifier to identify the likely cause by associating a set of real-time features sampled from the application that pertain to the problem to one or more of the predetermined faults in response to the training features.
1. A method for diagnosing a problem in an application, comprising:
obtaining a set of real-time features sampled from the application corresponding to a time when a set of real users of the application experienced the problem in the application; and
obtaining a likely cause of the problem in the application by applying the obtained set of real-time features to a classifier that has been pre-trained for recognizing a set of predetermined faults that may occur in the application based on a previous injection of a series of the predetermined faults into the application that occurred while a set of simulated user accesses providing at least simulated user inputs were applied to the application and while a set of training features for training the classifier were sampled from the application in a development or staging environment, wherein the pre-training is based on multiple time windows each having one or more faults injected into the application and, for each of the multiple time windows, sample sets of features from each of multiple resources within the application are extracted from corresponding logs and labeled with corresponding time window information and used as training information to generate refined sets of features corresponding to the problem in the application from multi-grain scanning to create new features corresponding to the problem in the application based on historical data utilizing a window size corresponding to a step having a pre-selected number of samples for which a feature vector is generated and multi-grain scanning is utilized to flatten each window and increase a number of features corresponding to the problem in the application for the corresponding window.
2. The method of claim 1 , further comprising generating a list of one or more of the predetermined faults having highest correlations to the real-time features of the problem.
3. The method of claim 2 , further comprising determining a respective confidence indicator for each predetermined fault on the list.
4. The method of claim 1 , further comprising refining the training features and the real-time features by interpolating one or more missing values in the training features and the real-time features before training the classifier.
5. The method of claim 1 , further comprising refining the training features and the real-time features by aggregating multiple instances of a resource type for the application into one feature in the training features and the real-time features before training the classifier.
6. The method of claim 1 , further comprising refining the training features and the real-time features by normalizing the training features and the real-time features before training the classifier.
7. The method of claim 1 , further comprising training an ensemble of local classifiers in the classifier, each local classifier adapted for the real-time features associated with a respective subset of resources upon which the application runs, and obtaining the likely cause in response to a classification vote among the local classifiers.
8. The method of claim 1 , further comprising obtaining a new set of features sampled from the application that pertain to one of the predetermined faults and updating the classifier in response to the new set of features.
9. The method of claim 1 , further comprising obtaining a new set of features sampled from the application that pertain to a new fault discovered in the application and updating the classifier in response to the new set of features.
10. A computing system to diagnose a problem in an application, the computing system comprising:
a storage device;
a hardware processor coupled with the storage device, the hardware processor and storage device to provide:
a classifier pre-trained to recognize a set of predetermined faults that may occur in the application based on an injection of a series of the predetermined faults into the application during a previous training while a set of simulated user accesses providing at least simulated user inputs were applied to the application and while a set of training features were sampled from the application in a development or staging environment, wherein the pre-training is based on multiple time windows each having one or more faults injected into the application and, for each of the multiple time windows, sample sets of features from each of multiple resources within the application are extracted from corresponding logs and labeled with corresponding time window information and used as training information to generate refined sets of features corresponding to the problem in the application from multi-grain scanning to create new features corresponding to the problem in the application based on historical data utilizing a window size corresponding to a step having a pre-selected number of samples for which a feature vector is generated and multi-grain scanning is utilized to flatten each window and increase a number of features corresponding to the problem in the application for the corresponding window; and
a diagnostic service to apply the classifier to a set of real-time features sampled from the application when a set of real users of the application experienced the problem in the application such that the classifier yields a likely cause of the problem in the application in response to the real-time features.
11. The computing system of claim 10 , wherein the classifier is further configured to generate a list of one or more of the predetermined faults having highest correlations to the real-time features of the problem.
12. The computing system of claim 11 , wherein the classifier is further configured to determine a respective confidence indicator for each predetermined fault on the list.
13. The computing system of claim 10 , further comprising a learning service that refines the training features and the real-time features by interpolating one or more missing values in the training features and the real-time features before training the classifier.
14. The computing system of claim 10 , further comprising a learning service that refines the training features and the real-time features by aggregating multiple instances of a resource type for the application into one feature in the training features and the real-time features before training the classifier.
15. The computing system of claim 10 , further comprising a learning service that refines the training features and the real-time features by normalizing the training features and the real-time features before training the classifier.
16. The computing system of claim 10 , wherein the classifier comprises an ensemble of local classifiers, each local classifier adapted for the real-time features associated with a respective subset of resources upon which the application runs such that the classifier determines the likely cause using a classification vote among the local classifiers.
17. The computing system of claim 10 , further comprising a diagnostic interface that obtains a new set of features sampled from the application that pertain to one of the predetermined faults and a learning service that updates the classifier in response to the new set of features.
18. The computing system of claim 10 , further comprising a diagnostic interface that obtains a new set of features sampled from the application that pertain to a new fault discovered in the application and a learning service that updates the classifier in response to the new set of features.
19. A computational environment for diagnosing a problem in an application, the computational environment comprising:
an application having multiple resources, wherein a set of one or more real-time features is extracted from each of the multiple resources of the application during a time window during which the problem exists;
a diagnostic service commutatively coupled with the application, the diagnostic service having a local classifier corresponding to each of the multiple resources for which real-time features are extracted, the local classifiers to generate a likely cause for each set of one or more real-time features, the diagnostic service providing a voting mechanism to select one of the likely causes from the local classifiers as an overall likely cause of the problem, wherein at least one of the local classifiers is pre-trained based on multiple time windows each having one or more faults injected into the application and, for each of the multiple time windows, sample sets of features from each of multiple resources within the application are extracted from corresponding logs and labeled with corresponding time window information and used as training information.
20. The computational environment of claim 19 wherein the diagnostic service selected a subset of real-time features and corresponding application resources for subsequent classifications.