Multi-core I/O trace analysis

1. For a storage system comprising a plurality of computing modules, each of the plurality of computing modules including a plurality of central processing units and a local memory dedicated to the computing module, and each of the plurality of computing modules connected to the other of the plurality of computing modules by an internal switching fabric of the storage system, wherein, within each computing module, the plurality of central processing units are grouped into a plurality of processing cores, a method comprising:

executing first code to service an I/O operation on the storage system, wherein said first code includes a plurality of trace instructions, and wherein said executing the first code includes:

for each of two or more of the plurality of computing modules, performing one or more sub-operations of the I/O operation by one or more of the plurality of cores of said each computing module, wherein the one or more sub-operations include a first sub-operation; and

executing the plurality of trace instructions and recording trace information in the respective local memory of each of the two or more computing modules for the one or more sub-operations including the first sub-operation of the I/O operation performed by the one or more of the plurality of cores of said each computing module;

a first of the plurality of processing cores on a first of the plurality of computing modules accessing first information corresponding to the recorded trace information in the respective local memory of at least a first of the two or more computing modules;

the first processing core determining a resulting form of the first information, wherein the resulting form facilitates analysis of the first information to determine functional and/or performance characteristics corresponding to the I/O operation, wherein said determining the resulting form includes:

determining, using the first processing core, a pair of recorded entries of the first information denoting the first sub-operation of the I/O operation, wherein a first entry of the pair denotes sending the first sub-operation from a sending processor core of a sending one of the plurality of computing modules at a first time, wherein a second entry of the pair denotes receiving the first sub-operation at a receiving processor core of a receiving one of the plurality of computing modules at a second time, wherein the first sub-operation includes locking, for the I/O operation, a cache slot of a cache included in the storage system, and wherein the sending processor core of the sending one of the computing modules is requesting that the receiving processor core of the receiving one of the computing modules lock the cache slot for the I/O operation;

determining, using the first processing core, a time discrepancy wherein the second time of the second entry, denoting a receiving time, is less than the first time of the first entry, denoting a sending time, by a first amount; and

responsive to determining the time discrepancy, reconciling the time discrepancy using the first processing core, wherein said reconciling includes modifying, using the first processing core, a respective time of at least one of the first entry and the second entry based on the first amount;

analyzing, using at least one of the plurality of central processing units of at least one of the plurality of computing modules, the resulting form of the first information;

determining, in response to said analyzing and using at least one of the plurality of central processing units of at least one of the plurality of computing modules, a performance issue; and

responsive to determining the performance issue, performing first processing using at least one of the plurality of central processing units of at least one of the plurality of computing modules, wherein said first processing includes requesting and retrieving additional recorded trace information from the respective memory of at least the first of the two or more computing modules.

2. The method of claim 1 , wherein the recording of the trace information includes recording the trace information in at least a first data structure including a plurality of entries, each entry representing a respective I/O sub-operation and specifying a time at which the respective I/O operation was performed according to clock utilized by the respective core that recorded the entry,

wherein the accessed first information is accessed, for each of the at least first computing module, from a second data structure of the at least first computing module derived from the at least first data structure, the second data structure including a plurality of entries, each entry in the second data structure corresponding to a respective entry in the at least first data structure and specifying the time specified in the corresponding entry in the at least first data structure, and

wherein said determining the resulting form of the first information includes sorting the first information according to the times specified in the entries of the second data structure.

3. The method of claim 2 , wherein, for each of the at least first computing module, each of the plurality of entries in the second data structure includes a pointer to a memory address of the corresponding entry in the at least first data structure, and includes less information than an amount of information recorded in the corresponding entry.

4. The method of claim 3 , wherein said requesting and said retrieving the additional recorded trace information of the first processing further comprises:

receiving a request for more information associated with the one or more sub-operations than specified by the first information; and

determining the more information by accessing one or more of the respective memory addresses pointed-to by one or more of the plurality of entries of the second data structure.

5. The method of claim 2 , wherein the first information is stored in a third data structure in the local memory of the first computing module, the third data structure derived at least in part from a respective second data structure of the at least second computing module, and the third data structure including a plurality of entries, each entry corresponding to a respective entry of the respective second data structure and including the specified time and the pointer to the memory address included in the respective entry.

6. The method of claim 5 , wherein the pair of recorded entries of the first information are included in the third data structure.

7. The method of claim 1 , wherein each execution of one of the plurality of trace instructions results in a recording of a piece of the recorded information, wherein software is compiled as part of processing that generates the first code, and the method further comprising:

after compilation of the software used to generate the first code, modifying what pieces of information are recorded.

8. The method of claim 1 , wherein said first processing further includes:

prior to performing said requesting and said retrieving, receiving first filter information including a timeframe, wherein the additional recorded trace information is determined by said requesting and said retrieving in accordance with the first filter information.

9. The method of claim 1 , wherein said executing the first code further comprises:

sending, from the sending processor core of the sending one of the plurality of computing modules, a communication regarding the first sub-operation of the I/O operation to the receiving processor core of the receiving one of the plurality of computing modules;

receiving, by the receiving processor core of the receiving one of the plurality of computing modules, the communication regarding the first sub-operation of the I/O operation;

responsive to said sending, executing a first trace instruction of the plurality of trace instructions resulting in recording first trace information corresponding to the first trace instruction in a first local memory of the sending one of the plurality of computing modules; and

responsive to said receiving, executing a second trace instruction of the plurality of trace instructions resulting in recording second trace information corresponding to the second trace instruction in a second local memory of the receiving one of the plurality of computing modules.

10. A storage system comprising:

an internal switching fabric;

a plurality of computing modules, each of the plurality of computing modules including a plurality of central processing units and a local memory dedicated to the computing module, and each of the plurality of computing modules connected to the other of the plurality of computing modules by the internal switching fabric of the storage system, wherein, within each computing module, the plurality of central processing units grouped into a plurality of processing cores; and

memory comprising code stored thereon that, when executed, performs a method including:

executing first code to service an I/O operation on the storage system, wherein said first code includes a plurality of trace instructions, and wherein said executing the first code includes:

for each of two or more of the plurality of computing modules, performing one or more sub-operations of the I/O operation by one or more of the plurality of cores of said each computing module, wherein the one or more sub-operations include a first sub-operation; and

executing the plurality of trace instructions and recording trace information in the respective local memory of each of the two or more computing modules for the one or more sub-operations including the first sub-operation of the I/O operation performed by the one or more of the plurality of cores of said each computing module;

a first of the plurality of processing cores on a first of the plurality of computing modules accessing first information corresponding to the recorded trace information in the respective local memory of at least a first of the two or more computing modules;

the first processing core determining a resulting form of the first information wherein the resulting form facilitates analysis of the first information to determine functional and/or performance characteristics corresponding to the I/O operation, wherein said determining the resulting form includes:

determining, using the first processing core, a pair of recorded entries of the first information denoting the first sub-operation of the I/O operation, wherein a first entry of the pair denotes sending the first sub-operation from a sending processor core of a sending one of the plurality of computing modules at a first time, wherein a second entry of the pair denotes receiving the first sub-operation at a receiving processor core of a receiving one of the plurality of computing modules at a second time, wherein the first sub-operation includes locking, for the I/O operation, a cache slot of a cache included in the storage system, and wherein the sending processor core of the sending one of the computing modules is requesting that the receiving processor core of the receiving one of the computing modules lock the cache slot for the I/O operation;

determining, using the first processing core, a time discrepancy wherein the second time of the second entry, denoting a receiving time, is less than the first time of the first entry, denoting a sending time, by a first amount; and

responsive to determining the time discrepancy, reconciling the time discrepancy using the first processing core, wherein said reconciling includes modifying, using the first processing core, a respective time of at least one of the first entry and the second entry based on the first amount;

analyzing, using at least one of the plurality of central processing units of at least one of the plurality of computing modules, the resulting form of the first information;

determining, in response to said analyzing and using at least one of the plurality of central processing units of at least one of the plurality of computing modules, a performance issue; and

responsive to determining the performance issue, performing first processing using at least one of the plurality of central processing units of at least one of the plurality of computing modules, wherein said first processing includes requesting and retrieving additional recorded trace information from the respective memory of at least the first of the two or more computing modules.

11. The storage system of claim 10 , wherein the recording of the trace information includes recording the trace information in at least a first data structure including a plurality of entries, each entry representing a respective I/O sub-operation and specifying a time at which the respective I/O operation was performed according to clock utilized by the respective core that recorded the entry,

wherein the accessed first information is accessed, for each of the at least first computing module, from a second data structure of the at least first computing module derived from the at least first data structure, the second data structure including a plurality of entries, each entry in the second data structure corresponding to a respective entry in the at least first data structure and specifying the time specified in the corresponding entry in the at least first data structure, and

wherein said determining the resulting form of the first information includes sorting the first information according to the times specified in the entries of the second data structure.

12. The storage system of claim 11 , wherein, for each of the at least first computing module, each of the plurality of entries in the second data structure includes a pointer to a memory address of the corresponding entry in the at least first data structure, and includes less information than an amount of information recorded in the corresponding entry.

13. The storage system of claim 12 , wherein said requesting and said retrieving the additional recorded trace information of the first processing further comprises:

receiving a request for more information associated with the one or more sub-operations than specified by the first information; and

determining the more information by accessing one or more of the respective memory addresses pointed-to by one or more of the plurality of entries of the second data structure.

14. The storage system of claim 11 , wherein the first information is stored in a third data structure in the local memory of the first computing module, the third data structure derived at least in part from a respective second data structure of the at least second computing module, and the third data structure including a plurality of entries, each entry corresponding to a respective entry of the respective second data structure and including the specified time and the pointer to the memory address included in the respective entry.

15. The storage system of claim 14 , wherein the pair of recorded entries of the first information are included in the third data structure.

16. The storage system of claim 10 , wherein each execution of one of the plurality of trace instructions results in a recording of a piece of the recorded information, wherein software is compiled as part of processing that generates the first code, and the method further comprising:

after compilation of the software used to generate the first code, modifying what pieces of information are recorded.

17. For a storage system comprising a plurality of computing modules, each of the plurality of computing modules including a plurality of central processing units and a local memory dedicated to the computing module, and each of the plurality of computing modules connected to the other of the plurality of computing modules by an internal switching fabric of the storage system, wherein, within each computing module, the plurality of central processing units are grouped into a plurality of processing cores, one or more non-transitory computer-readable media comprising:

first executable code that services an I/O operation on the storage system, wherein said first executable code includes a plurality of trace instructions, wherein said first executable code further includes:

executable code that, for each of two or more of the plurality of computing modules, performs one or more sub-operations of the I/O operation by one or more of the plurality of cores of said each computing module, wherein the one or more sub-operations include a first sub-operation; and

executable code that executes the plurality of trace instructions and controls recording trace information in the respective local memory of each of the two or more computing modules for the one or more sub-operations including the first sub-operation of the I/O operation performed by the one or more of the plurality of cores of said each computing module;

executable code that controls a first of the plurality of processing cores on a first of the plurality of computing modules accessing first information corresponding to the recorded trace information in the respective local memory of at least a first of the two or more computing modules;

executable code that controls the first processing core determining a resulting form of the first information, wherein the resulting form facilitates analysis of the first information to determine functional and/or performance characteristics corresponding to the I/O operation, wherein said executable code that controls the first processing core determining the resulting form of the first information includes:

executable code that determines a pair of recorded entries of the first information denoting the first sub-operation of the I/O operation, wherein a first entry of the pair denotes sending the first sub-operation from a sending processor core of a sending one of the plurality of computing modules at a first time, wherein a second entry of the pair denotes receiving the first sub-operation at a receiving processor core of a receiving one of the plurality of computing modules at a second time, wherein the first sub-operation includes locking, for the I/O operation, a cache slot of a cache included in the storage system, and wherein the sending processor core of the sending one of the computing modules is requesting that the receiving processor core of the receiving one of the computing modules lock the cache slot for the I/O operation;

executable code that determines a time discrepancy wherein the second time of the second entry, denoting a receiving time, is less than the first time of the first entry, denoting a sending time, by a first amount; and

executable code that, responsive to determining the time discrepancy, reconciles the time discrepancy, wherein reconciling the time discrepancy includes modifying a respective time of at least one of the first entry and the second entry based on the first amount;

executable code that analyzing, using at least one of the plurality of central processing units of at least one of the plurality of computing modules, the resulting form of the first information;

executable code that determines, in response to said analyzing the resulting form and using at least one of the plurality of central processing units of at least one of the plurality of computing modules, a performance issue; and

executable code that, responsive to determining the performance issue, performs first processing using at least one of the plurality of central processing units of at least one of the plurality of computing modules, wherein said first processing includes requesting and retrieving additional recorded trace information from the respective memory of at least the first of the two or more computing modules.

18. The one or more non-transitory computer-readable media of claim 17 , wherein the recording of the trace information includes recording the trace information in at least a first data structure including a plurality of entries, each entry representing a respective I/O sub-operation and specifying a time at which the respective I/O operation was performed according to clock utilized by the respective core that recorded the entry,

wherein the accessed first information is accessed, for each of the at least first computing module, from a second data structure of the at least first computing module derived from the at least first data structure, the second data structure including a plurality of entries, each entry in the second data structure corresponding to a respective entry in the at least first data structure and specifying the time specified in the corresponding entry in the at least first data structure, and

wherein said determining the resulting form of the first information includes sorting the first information according to the times specified in the entries of the second data structure.

19. The one or more non-transitory computer-readable media of claim 18 , wherein, for each of the at least first computing module, each of the plurality of entries in the second data structure includes a pointer to a memory address of the corresponding entry in the at least first data structure, and includes less information than an amount of information recorded in the corresponding entry.

20. The one or more non-transitory computer-readable media of claim 19 , wherein the one or more non-transitory computer-readable media further comprises:

executable code that receives a request for more information associated with the one or more sub-operations than specified by the first information; and

executable code that determines the more information by accessing one or more of the respective memory addresses pointed-to by one or more of the plurality of entries of the second data structure.

21. The one or more non-transitory computer-readable media of claim 18 , wherein the first information is stored in a third data structure in the local memory of the first computing module, the third data structure derived at least in part from a respective second data structure of the at least second computing module, and the third data structure including a plurality of entries, each entry corresponding to a respective entry of the respective second data structure and including the specified time and the pointer to the memory address included in the respective entry.

22. The one or more non-transitory computer-readable media of claim 21 , wherein the pair of recorded entries of the first information are included in the third data structure.