Optimization of parallel processing using waterfall representations
Event data for an application execution is accessed from a table of logged events, the event data comprising a sequence, a hierarchy, and a start time and duration for each event. Dependency data for each event is also accessed to determine whether the start time for an event is dependent on the prior completion of at least one other event. A waterfall representation is then generated, the representation including an entry for each event in the sequence, with a start time and duration represented for each event. Based on the dependencies and hierarchy, it is determined, for each event with a start time that is later than the start time of an event which precedes it in the sequence, whether the event's start time is dependent on the prior completion of at least one preceding event. The start time for each event may then be advanced based on the determination.
1. A system, comprising:
a data access module implemented using at least one of one or more processors and configured to perform operations comprising:
accessing event data for an application from a table of logged events in the execution of the application, the event data comprising a sequence of the events, a hierarchy of the events, and a start time and duration for each event; and
accessing dependency data for each event, the dependency data indicating a dependency of a start time for a corresponding event on a prior completion of at least one other event; and
a representation module implemented using at least one of the one or more processors and configured to perform operations comprising:
generating a waterfall representation based on the accessed event data, the representation including: a first axis, a second axis orthogonal to the first axis, an entry for each event in the sequence along the first axis with a start time and duration of each event measured relative to the second axis; and
an optimization module implemented using at least one of the one or more processors and configured to perform operations comprising:
analyzing the waterfall representation and the hierarchy to identify a first event that has a first start time that is later than a start time of an event which precedes it in the sequence and is not a child of an event which precedes it in the sequence;
based on the dependency data, determining whether the first start time for the first event is dependent on a prior completion of at least one event that precedes it in the sequence; and
based on the first start time for the first event being dependent, starting execution of the first event at a start execution time based on a latest completion time of the at least one event that precedes it in the sequence, the start execution time being earlier than the first start time.
2. The system of claim 1 , the operations of the optimization module further comprising:
based on the first start time for the first event not being dependent, setting the start execution time for the first event to an earliest start time possible based on available parallel processing capacity; and
determining the earliest start time possible based on a number of processing cores available at a server hosting the application or based on a maximum number of simultaneous connections per server of the application.
3. The system of claim 1 , the operations of the optimization module further comprising:
identifying a latest completion time of all the events that precede the first event in the sequence; and
not setting the start execution time for the first event to an earlier start execution time based on the duration of the first event indicating that it will be completed prior to the latest completion time of all the events that precede it in the sequence.
4. The system of claim 1 , wherein the application includes code that:
logs the events in the execution of the application;
generates the table of logged events; and
transmits the table of logged events to a central log repository; and
the operations of the data access module further comprising accessing the event data from the central log repository.
5. The system of claim 1 , wherein the application comprises a website application and the events comprise server responses to requests from users of the application.
6. The system of claim 1 , the operations of the representation module further comprising:
showing a visualization of the waterfall representation in a user interface on a display; and
receiving input from a user via the user interface; and
the operations of the data access module further comprising accessing the dependency data from the user input.
7. The system of claim 1 , wherein:
the first axis is a vertical axis and the first axis includes a totaling event entry before an initial event in the sequence; and
the totaling event has a start time equal to the initial event in the sequence and a duration that indicates a completion time equal to a latest completion time of all the events in the sequence.
8. A method, comprising:
accessing event data for an application from a table of logged events in the execution of the application, the event data comprising a sequence of the events, a hierarchy of the events, and a start time and duration for each event; and
accessing dependency data for each event, the dependency data indicating a dependency of a start time for a corresponding event on a prior completion of at least one other event;
generating a waterfall representation based on the accessed event data, the representation including: a first axis, a second axis orthogonal to the first axis, an entry for each event in the sequence along the first axis with a start time and duration of each event measured relative to the second axis; and
analyzing the waterfall representation and the hierarchy to identify a first event that has a first start time that is later than a start time of an event which precedes it in the sequence and is not a child of an event which precedes it in the sequence;
based on the dependency data, determining whether the first start time for the first event is dependent on a prior completion of at least one event that precedes it in the sequence; and
based on the first start time for the first event being dependent, starting execution of the first event at a start execution time based on a latest completion time of the at least one event that precedes it in the sequence, the start execution time being earlier than the first start time.
9. The method of claim 8 , further comprising:
based on the first start time for the first event not being dependent, setting the start execution time for the first event to an earliest start time possible based on available parallel processing capacity; and
determining the earliest start time possible based on a number of processing cores available at a server hosting the application or based on a maximum number of simultaneous connections per server of the application.
10. The method of claim 8 , further comprising:
identifying a latest completion time of all the events that precede the event in the sequence; and
not set the start execution time for the first event to an earlier start execution time based on the duration of the first event indicating that it will be completed prior to the latest completion time of all the events that precede it in the sequence.
11. The method of claim 8 , further comprising:
logging, by the application, the events in the execution of the application;
generating, by the application, the table of logged events; and
transmitting, by the application, the table of logged events to a central log repository; and
accessing the event data for an application from the central log repository.
12. The method of claim 8 , wherein the application comprises a website application and the events comprise server responses to requests from users of the application.
13. The method of claim 8 , further comprising:
showing a visualization of the waterfall representation in a user interface on a display;
receiving input from a user via the user interface; and
accessing the dependency data from the user input.
14. The method of claim 8 , wherein:
the first axis is a vertical axis and the first axis includes a totaling event entry before an initial event in the sequence; and
the totaling event has a start time equal to the initial event in the sequence and a duration that indicates a completion time equal to the latest completion time of all the events in the sequence.
15. A non-transitory machine-readable medium storing instructions which, when executed by the at least one processor of a machine, cause the machine to perform operations comprising:
accessing event data for an application from a table of logged events in the execution of the application, the event data comprising a sequence of the events, a hierarchy of the events, and a start time and duration for each event; and
accessing dependency data for each event, the dependency data indicating a dependency of a start time for a corresponding event on a prior completion of at least one other event;
generating a waterfall representation based on the accessed event data, the representation including: a first axis, a second axis orthogonal to the first axis, an entry for each event in the sequence along the first axis with a start time and duration of each event measured relative to the second axis; and
analyzing the waterfall representation and the hierarchy to identify a first event that has a first start time that is later than a start time of an event which precedes it in the sequence and is not a child of an event which precedes it in the sequence;
based on the dependency data, determining whether the first start time for the first event is dependent on a prior completion of at least one event that precedes it in the sequence; and
based on the first start time for the first event being dependent, starting execution of the first event at a start execution time based on a latest completion time of the at least one event that precedes it in the sequence, the start execution time being earlier than the first start time.
16. The non-transitory machine-readable medium of claim 15 , the operations further comprising:
based on the first start time for the first event not being dependent, setting the start execution time for the first event to an earliest start time possible based on available parallel processing capacity; and
determining the earliest start time possible based on a number of processing cores available at a server hosting the application or based on a maximum number of simultaneous connections per server of the application.
17. The non-transitory machine-readable medium of claim 15 , the operations further comprising:
identifying a latest completion time of all the events that precede the first event in the sequence; and
not setting the start execution time for the first event to an earlier start execution time based on the duration of the first event indicating that it will be completed prior to the latest completion time of all the events that precede it in the sequence.
18. The non-transitory machine-readable medium of claim 15 , wherein the application includes code that:
logs the events in the execution of the application;
generates the table of logged events; and
transmits the table of logged events to a central log repository; and
the operations further comprise accessing the event data from the central log repository.
19. The non-transitory machine-readable medium of claim 15 , the operations further comprising:
showing a visualization of the waterfall representation in a user interface on a display;
receiving input from a user via the user interface; and
accessing the dependency data from the user input.
20. The non-transitory machine-readable medium of claim 15 , wherein:
the first axis is a vertical axis and the first axis includes a totaling event entry before an initial event in the sequence; and
the totaling event has a start time equal to the initial event in the sequence and a duration that indicates a completion time equal to the latest completion time of all the events in the sequence.