Detailed performance analysis by flow aware marker mechanism
According to aspects of the disclosure, a method is provided comprising: executing a set of threads in a storage system, the set of threads including at least a first thread; executing a plurality of performance counters of the storage system, the plurality of performance counters including at least: (i) a first performance counter that is executed when an operating state of the first thread is changed in response to the first thread accessing a synchronization object, and (ii) a second performance counter that is executed when a marker inserted in the first thread is executed; generating one or more performance data containers associated the first thread based on performance data associated with the first thread; and generating a directed graph based on the performance data containers.
1. A method comprising:
executing a set of threads in a storage system, the set of threads including at least a first thread;
executing a plurality of performance counters of the storage system, the plurality of performance counters being executed concurrently with the set of threads, the plurality of performance counters including at least: (i) a first performance counter that is executed when an operating state of the first thread is changed in response to the first thread accessing a synchronization object, and (ii) a second performance counter that is executed when a marker inserted in the first thread is executed;
generating one or more performance data containers associated the first thread based on performance data associated with the first thread; and
generating a directed graph based on the performance data containers, the directed graph including a plurality of nodes connected to one another by a plurality of edges, the plurality of nodes including a first node corresponding to the synchronization object, and a second node corresponding to the marker.
2. The method of claim 1 , wherein at least one of the plurality of edges extends between the first node and the second node, the edge corresponding to processor-executable instructions in the first thread that are executed after the first thread has accessed the respective synchronization object and before the marker is executed.
3. The method of claim 1 , wherein the marker is inserted at a location in the first thread that corresponds to one or more processor-executable instructions for accessing a resource of the storage system that is not controlled via any synchronization objects that are part of an operating system executing the set of threads.
4. The method of claim 1 , wherein the set of threads further includes a second thread having a different task identifier than the first thread, and the performance data containers include only performance data associated with threads from the set that have the same task identifier as the first thread.
5. The method of claim 1 , wherein each of the plurality of edges is identified by a three-tuple edge key based upon a task identifier associated with the edge, a first resource identifier, and a second resource identifier, the first and second resource identifiers corresponding to different ones of the plurality of nodes that are connected to one another by the edge.
6. The method of claim 1 , wherein:
the first node is associated with a first resource identifier corresponding to the synchronization object,
the second node is as associated with a second resource identifier corresponding to a resource of the storage system that is not controlled via any synchronization objects that are part of an operating system executing the set of threads; and
the marker includes one or more processor-executable instructions that are configured to provide the second performance counter with second resource identifier, and
the second performance counter includes one or more processor-executable instructions configured to map performance data associated the marker to the second resource identifier.
7. The method of claim 1 , wherein:
the operating state of the first thread is changed by a scheduler of the storage system;
the first performance counter includes one or more processor-executable instructions that are part of the scheduler of the storage system; and
the second performance counter includes one or more processor-executable instructions that are part of the of an application programming interface (API) that is separate from the scheduler.
8. A system comprising:
a memory; and
one or more processors operatively coupled to the memory, the one or more processors being configured to perform the operations:
executing a set of threads in a storage system, the set of threads including at least a first thread;
executing a plurality of performance counters of the storage system, the plurality of performance counters being executed concurrently with the set of threads, the plurality of performance counters including at least: (i) a first performance counter that is executed when an operating state of the first thread is changed in response to the first thread accessing a synchronization object, and (ii) a second performance counter that is executed when a marker inserted in the first thread is executed;
generating one or more performance data containers associated the first thread based on performance data associated with the first thread; and
generating a directed graph based on the performance data containers, the directed graph including a plurality of nodes connected to one another by a plurality of edges, the plurality of nodes including a first node corresponding to the synchronization object, and a second node corresponding to the marker.
9. The system of claim 8 , wherein at least one of the plurality of edges extends between the first node and the second node, the edge corresponding to processor-executable instructions in the first thread that are executed after the first thread has accessed the respective synchronization object and before the marker is executed.
10. The system of claim 8 , wherein the marker is inserted at a location in the first thread that corresponds to one or more processor-executable instructions for accessing a resource of the storage system that is not controlled via any synchronization objects that are part of an operating system executing the set of threads.
11. The system of claim 8 , wherein each of the plurality of edges is identified by a three-tuple edge key based upon a task identifier associated with the edge, a first resource identifier, and a second resource identifier, the first and second resource identifiers corresponding to different ones of the plurality of nodes that are connected to one another by the edge.
12. The system of claim 8 , wherein:
the first node is associated with a first resource identifier corresponding to the synchronization object,
the second node is as associated with a second resource identifier corresponding to a resource of the storage system that is not controlled via any synchronization objects that are part of an operating system executing the set of threads; and
the marker includes one or more processor-executable instructions that are configured to provide the second performance counter with second resource identifier, and
the second performance counter includes one or more processor executable instructions configured to map performance data associated the marker to the second resource identifier.
13. The system of claim 8 , wherein each of the plurality of edges is identified by a three-tuple edge key based upon a task identifier associated with the edge, a first resource identifier, and a second resource identifier, the first and second resource identifiers corresponding to different ones of the plurality of nodes that are connected to one another by the edge.
14. The system of claim 8 , wherein:
the operating state of the first thread is changed by a scheduler of the storage system;
the first performance counter includes one or more processor-executable instructions that are part of the scheduler of the storage system; and
the second performance counter includes one or more processor-executable instructions that are part of the of an application programming interface (API) that is separate from the scheduler.
15. A non-transitory computer-readable medium storing processor-executable code, which when executed by at least one processor causes the at least one processor to execute the operations of:
executing a set of threads in a storage system, the set of threads including at least a first thread;
executing a plurality of performance counters of the storage system, the plurality of performance counters being executed concurrently with the set of threads, the plurality of performance counters including at least: (i) a first performance counter that is executed when an operating state of the first thread is changed in response to the first thread accessing a synchronization object, and (ii) a second performance counter that is executed when a marker inserted in the first thread is executed;
generating one or more performance data containers associated the first thread based on performance data associated with the first thread; and
generating a directed graph based on the performance data containers, the directed graph including a plurality of nodes connected to one another by a plurality of edges, the plurality of nodes including a first node corresponding to the synchronization object, and a second node corresponding to the marker.
16. The non-transitory computer-readable medium of claim 15 , wherein at least one of the plurality of edges extends between the first node and the second node, the edge corresponding to processor-executable instructions in the first thread that are executed after the first thread has accessed the respective synchronization object and before the marker is executed.
17. The non-transitory computer-readable medium of claim 15 , wherein the marker is inserted at a location in the first thread that corresponds to one or more processor-executable instructions for accessing a resource of the storage system that is not controlled via any synchronization objects that are part of an operating system executing the set of threads.
18. The non-transitory computer-readable medium of claim 15 , wherein the set of threads further includes a second thread having a different task identifier than the first thread, and the performance data containers include only performance data associated with threads from the set that have the same task identifier as the first thread.
19. The non-transitory computer-readable medium of claim 15 , wherein each of the plurality of edges is identified by a three-tuple edge key based upon a task identifier associated with the edge, a first resource identifier, and a second resource identifier, the first and second resource identifiers corresponding to different ones of the plurality of nodes that are connected to one another by the edge.
20. The non-transitory computer-readable medium of claim 15 , wherein:
the first node is associated with a first resource identifier corresponding to the synchronization object,
the second node is as associated with a second resource identifier corresponding to a resource of the storage system that is not controlled via any synchronization objects that are part of the operating system executing the set of threads; and
the marker includes one or more processor-executable instructions that are configured to provide the second performance counter with second resource identifier, and
the second performance counter includes one or more processor-executable instructions configured to map performance data associated the marker to the second resource identifier.