CloudSeer: using logs to detect errors in the cloud infrastructure
View Patent ↗Systems and methods are disclosed for detecting error in a cloud infrastructure by running a plurality of training tasks on the cloud infrastructure and generating training execution logs; generating a model miner with the training execution logs to represent one or more correct task executions in the cloud infrastructure; after training, running a plurality of tasks on the cloud infrastructure and capturing live execution logs; and from the live execution logs, if a current task deviates from the correct task execution, indicating an execution error for correction in real-time.
1. A method for detecting error in a cloud infrastructure, comprising:
running a plurality of training tasks on the cloud infrastructure and generating training execution logs;
generating a model miner with the training execution logs to represent one or more correct task executions in the cloud infrastructure;
after training, running a plurality of tasks on the cloud infrastructure and capturing live execution logs; and
from the live execution logs, if a current task deviates from the correct task execution, indicating an execution error for correction in real-time.
2. The method of claim 1 , comprising generating an automaton depicting temporal dependencies between log messages for a task.
3. The method of claim 1 , comprising detecting error including performance degradation.
4. The method of claim 3 , comprising detecting unexpected time variation between two log messages.
5. The method of claim 1 , comprising providing a context for an error.
6. The method of claim 5 , comprising providing a sequence of log messages for the task where the error happened.
7. The method of claim 1 , wherein the model for a task is mined from log message sequences observed for multiple executions of the task.
8. The method of claim 1 , comprising mining temporal orders that hold for a set of log message sequences.
9. The method of claim 1 , comprising building an automaton to record the temporal orders and time intervals between messages.
10. The method of claim 2 , comprising training the automaton for different tasks to check if the log message sequences for the currently executing tasks are per their respective automata.
11. The method of claim 1 , comprising checking for errors on-the-fly as log messages for the tasks get generated.
12. The method of claim 11 , comprising monitoring multiple tasks executed concurrently in the cloud infrastructure, and interleaving log messages from the tasks.
13. The method of claim 11 , comprising applying one or more identifiers in messages to associate messages with tasks.
14. The method of claim 13 , wherein the identifier includes one of: internet protocol (IP) address, virtual machine identifier, request identifier, Uniform Resource Locator (URL).
15. The method of claim 11 , comprising checking a log message against only model(s) of predetermined task(s) associated with the log message.
16. The method of claim 11 , comprising providing execution context for an error for understanding and fixing the error.
17. The method of claim 16 , comprising determining a progression of a task according to a task automaton.
18. A system for detecting error in a cloud infrastructure, comprising:
a cloud infrastructure including a processor, a data storage device, and one or more resources;
a logger coupled to the cloud infrastructure for running a plurality of training tasks on the cloud infrastructure and generating training execution logs, and subsequent to training, the logger running a plurality of tasks on the cloud infrastructure and capturing live execution logs;
a model miner coupled to the logger and trained by the training execution logs to represent one or more correct task executions in the cloud infrastructure;
an error checker coupled to the model miner and to the logger to receive live execution logs, and if a current task deviates from the correct task execution, the error checker indicating an execution error for correction in real-time.