Performing root cause analysis in a multi-role application
A new snapshot of a storage volume is created by instructing computing nodes to suppress write requests. A snapshot of the application may be created and used to rollback or clone the application. Clones snapshots of storage volumes may be gradually populated with data from prior snapshots to reduce loading on a primary snapshot. Components of cloned applications may communicate with one another using addresses of these components in the parent application. Jobs implementing a bundled application may be referenced with a simulated file system that generates reads to hosts only when the job log file is actually read. Job logs and a job hierarchy may be used to perform root cause analysis. Job logs may be for tasks such as creating the bundled application, cloning, rolling back, backing up, scaling out, scaling in, deleting, pruning unused application images, or the like.
1. A method comprising:
providing a computer system including plurality of computing nodes;
providing an application including a plurality of jobs executing on the plurality of computing nodes, the plurality of jobs being initiated according to a manifest defining a job hierarchy;
generating, by each job of the plurality of jobs, a job log storing events generated by the each job;
detecting, by the computer system, an error in a job of the plurality of jobs;
constructing, by the computer system, a chronology that combines events from the job logs of the plurality of jobs;
determining, by the computer system, a root cause of the error according to the chronology and the job hierarchy;
wherein the job log of each job of the plurality of jobs is stored on a node of the plurality of computing nodes executing the each job; and
wherein constructing the chronology comprises:
gathering, by the computer system, the job logs of the plurality of jobs from the plurality of computing nodes, each job log including one or more events; and
interleaving, by the computer system, the events from the job logs such that the events are temporally ordered according to time stamps included in the events.
2. The method of claim 1 , wherein determining the root cause of the error comprises:
identifying, by the computer system, a chain of events in the chronology that includes the error and such that the events in the chain of events are related to one another according to the job hierarchy.
3. The method of claim 2 , wherein identifying the chain of events in the chronology comprises:
identifying a first event and a second event such that the second event follows the first event in the chronology and the second event is generated by a second job of the plurality of jobs that is an ancestor of a first job of the plurality of jobs in the job hierarchy, the first job having generated the first event.
4. The method of claim 3 , wherein identifying the chain of events in the chronology further comprises determining that a time stamp of the first event and a time stamp of the second event are within a threshold difference of one another.
5. The method of claim 3 , further comprising:
providing a dependency map specifying a plurality of dependency relationships between jobs of the plurality of jobs;
wherein identifying the chain of events in the chronology comprises:
identifying a third event and a fourth event such that the fourth event follows the third event in the chronology and the fourth event is generated by a fourth job of the plurality of jobs that has a dependency relationship on a third job of the plurality of jobs in the dependency map, the third job having generated the third event.
6. The method of claim 1 , further comprising instantiating, by the plurality of jobs, roles of a multi-role application according to the manifest.
7. The method of claim 6 , further comprising instantiating, by the plurality of jobs, containers executing the roles of the multi-role application according to the manifest.
8. A system comprising:
a plurality of computing nodes;
an orchestrator executing on the plurality of computing nodes and programmed to:
invoke execution of a plurality of jobs on the plurality of computing nodes according to a manifest defining a job hierarchy, each job of the plurality of jobs being programmed to generate a job log storing events generated by the each job;
detect an error in a job of the plurality of jobs;
obtain a lineage of the job from the job hierarchy, the lineage including ancestor jobs and descendent jobs of the job in the job hierarchy;
construct a chronology that combines events from the job logs of the ancestor jobs and descendent jobs; and
determine a root cause of the error according to the chronology and the job hierarchy;
wherein the each job of the plurality of jobs is further programmed to store the job log of the each job on a node of the plurality of computing nodes executing the each job; and
wherein the orchestrator is further programmed to construct the chronology by:
gathering the job logs of the ancestor jobs and descendent jobs from the plurality of computing nodes, each job log including one or more events; and
interleaving the events from the job logs such that the events are temporally ordered according to time stamps included in the events.
9. The system of claim 8 , wherein the orchestrator is further programmed to determine the root cause of the error by:
identifying a chain of events in the chronology that includes the error and such that the events in the chain of events are related to one another according to the job hierarchy.
10. The system of claim 9 , wherein the orchestrator is further programmed to identify the chain of events in the chronology by:
identifying a first event and a second event such that the second event follows the first event in the chronology and the second event is generated by a second job of the plurality of jobs that is an ancestor of a first job of the plurality of jobs in the job hierarchy, the first job having generated the first event.
11. The system of claim 10 , wherein the orchestration layer is further programmed to identify the chain of events in the chronology by determining that a time stamp of the first event and a time stamp of the second event are within a threshold difference of one another.
12. The system of claim 11 , wherein the orchestration layer is further programmed to identify the chain of events in the chronology by:
identifying a third event and a fourth event such that the fourth event follows the third event in the chronology and the fourth event is generated by a fourth job of the plurality of jobs that has a dependency relationship on a third job of the plurality of jobs in a dependency map, the third job having generated the third event;
wherein the dependency map specifies a plurality of dependency relationships between jobs of the plurality of jobs.
13. The system of claim 8 , wherein the plurality of jobs are programmed to instantiate roles of a multi-role application according to the manifest.
14. The system of claim 13 , wherein the plurality of jobs are programmed to instantiate containers executing the roles of the multi-role application according to the manifest.