Method and system for clustered real-time correlation of trace data fragments describing distributed transaction executions

1. A computer-implemented method for monitoring transaction execution in a distributed computing environment, comprising:

determining, by an agent instrumented in a method, whether execution of the method is a first instance of execution caused by a given thread of a transaction executing in the distributed computing environment, where the agent is executed by a processor of a host computing device;

determining, by the agent, whether parent thread identification data is available from a parent thread that spawn the given thread;

extracting, by the agent, an identifier for a correlation server from data received by the agent from the parent thread and store the identifier for the correlation server in a data store residing on the host computing device, where the identifier for the correlation server is stored in association with an identifier for the given thread and an identifier for the transaction, and the extracting and storing of the identifier is in response to a determination that the execution of the method is a first instance of execution and a determination that the parent thread identification data is available;

selecting, by the agent, an identifier for one of a plurality of correlation servers, where the identifier is selected from a listing of the correlation servers stored on the host computing device;

storing, by the agent, the identifier for the selected correlation server in the data store, where the identifier for the selected correlation server is stored in association with an identifier for the given thread and the identifier for the transaction and the selecting and storing of the identifier is in response to a determination that the execution of the method is a first instance of execution and a determination that the parent thread identification data is not available;

retrieving, by the agent, the identifier for the correlation server from the data store in response to a determination that the execution of the method is not the first instance of execution;

creating, by the agent, a trace event that indicates start of execution of the given thread, where the trace event includes the identifier for the given thread, the identifier for the new transaction, the parent thread identification data if available, the identifier for the selected correlation server and at least one metric indicative of performance of the method, the creating of the trace event is in response to a determination that the execution of the method is a first instance of execution by the given thread; and

sending, by the agent, the trace event across a network to the selected correlation server located remotely from the host computing device.

2. The method of claim 1 further comprises

determining, by the agent, whether execution of the method triggers start of or communication with another thread; and

appending, by the agent, the identifier for the given thread and the identifier for the correlation server to a message that is used to start or communication with the another thread, where the appending is in response to a determination that the execution of the method does trigger start of or communicates with another thread.

3. The method of claim 2 further comprises

determining, by the agent, a customer from amongst a plurality of customers, where the customer controls the process that spawn the given thread; and

appending, by the agent, an identifier for the customer to the message.

4. The method of claim 3 wherein determining whether identification data is available from a parent thread further comprises

comparing, by the agent, an identifier for a customer received in data from the parent thread to a customer identifier associated with the agent;

ignoring, by the agent, the data received from the parent thread; and

creating, by the agent, a trace event that indicates start of a new monitored transaction, where the ignoring and creating occur when the identifier for a customer in the data received from the parent thread does not match the customer identifier associated with the agent.

5. The method of claim 1 further comprises selecting the identifier for one of the plurality of correlation servers randomly.

6. The method of claim 1 further comprises selecting the identifier for one of the plurality of correlation servers based on resource load currently experienced by one or more of the plurality of correlation servers.

7. The method of claim 1 wherein sending the trace event further comprises sending the trace event to an intermediate node, where the intermediate node forwards the trace event to the selected correlation server and the intermediate node is located remotely from the host computing device and the selected correlation server.

8. The method of claim 7 further comprises

receiving, by the intermediate node, the trace event from the agent;

extracting, by the intermediate node, the identifier for the selected correlation server;

retrieving, by the intermediate node, a network address for the selected correlation server from a routing table; and

sending, by the intermediate node, the trace event across a network to the selected correlation server using the network address.

9. The method of claim 8 further comprises

detecting, by the intermediate node, a failure condition with the selected correlation server;

selecting, by the intermediate node, a backup correlation server using a selection method, where the selection method is the same amongst a plurality of intermediate nodes; and

sending, by the intermediate node, the trace event across the network to the backup correlation server.

10. The method of claim 8 further comprises:

receiving, by the selected correlation server, the trace event from the intermediate node;

determining, by the selected correlation server, whether the trace event indicates a new transaction being monitored;

creating, by the selected correlation server, a transaction record using data contained in the trace event, wherein the creating is in response to the determination that the trace event indicates a new transaction being monitored; and

updating, by the selected correlation server, a transaction record using data contained in the trace event, wherein the updating is in response to the determination that the trace event is not a new transaction.

11. The method of claim 7 further comprises

periodically selecting, by the agent, another intermediate node from a listing of intermediate nodes maintained by the agent and stored on the host computing device; and

sending, by the agent, subsequent trace events to the another intermediate node.

12. A computer-implemented method for monitoring transaction execution in a distributed computing environment, comprising:

determining, by an agent instrumented in a method, whether execution of the method is a first instance of execution caused by a given thread of a transaction executing in the distributed computing environment, where the agent is executed by a processor of a host computing device;

determining, by the agent, whether parent thread identification data is available from a parent thread that spawn the given thread;

selecting, by the agent, an identifier for one of a plurality of correlation servers, where the identifier is selected from a listing of the correlation servers stored on the host computing device; and

storing, by the agent, the identifier for the selected correlation server in a data store, where the identifier for the selected correlation server is stored in association with an identifier for the given thread and an identifier for the transaction and the selecting and storing of the identifier is in response to a determination that the execution of the method is a first instance of execution and a determination that the parent thread identification data is not available;

retrieving, by the agent, the identifier for the correlation server from the data store in response to a determination that the execution of the method is not the first instance of execution and does not indicate the start of a new transaction executing in the distributed computing environment;

creating, by the agent, a trace event that indicates start of execution of the given thread, where the trace event includes the identifier for the given thread, the identifier for the transaction, the parent thread identification data if available, the identifier for the selected correlation server and at least one metric indicative of performance of the method, the creating of the trace event is in response to a determination that the execution of the method is a first instance of execution by the given thread; and

sending, by the agent, the trace event across a network to the selected correlation server located remotely from the host computing device.

13. The method of claim 12 further comprises

extracting, by the agent, an identifier for a correlation server from data received by the agent from the parent thread and store the identifier for a correlation server in the data store residing on the host computing device, where the identifier for the correlation server is stored in association with an identifier for the given thread and an identifier for the transaction, and the extracting and storing of the identifier is in response to a determination that the execution of the method is a first instance of execution and a determination that the parent thread identification data is available.

14. The method of claim 12 further comprises

determining, by the agent, whether execution of the method triggers start of or communication with another thread; and

appending, by the agent, the identifier for the given thread and the identifier for the correlation server to a message that is used to start or communication with the another thread, where the appending is in response to a determination that the execution of the method does trigger start of or communicates with another thread.

15. The method of claim 14 further comprises

determining, by the agent, a customer from amongst a plurality of customers, where the customer controls the process that spawn the given thread; and

appending, by the agent, an identifier for the customer to the message.

16. The method of claim 15 wherein determining whether identification data is available from a parent thread further comprises

comparing, by the agent, an identifier for a customer received in data from the parent thread to a customer identifier associated with the agent;

ignoring, by the agent, the data received from the parent thread; and

creating, by the agent, a trace event that indicates start of a new monitored transaction, where the ignoring and creating occur when the identifier for a customer in the data received from the parent thread does not match the customer identifier associated with the agent.

17. The method of claim 12 further comprises selecting the identifier for one of the plurality of correlation servers randomly.

18. The method of claim 12 further comprises selecting the identifier for one of the plurality of correlation servers based on resource load currently experienced by one or more of the plurality of correlation servers.

19. The method of claim 12 wherein sending the trace event further comprises sending the trace event to an intermediate node, where the intermediate node forwards the trace event to the selected correlation server and the intermediate node is located remotely from the host computing device and the selected correlation server.

20. The method of claim 19 further comprises receiving, by the intermediate node, the trace event from the agent;

extracting, by the intermediate node, the identifier for the selected correlation server;

retrieving, by the intermediate node, a network address for the selected correlation server from a routing table; and

sending, by the intermediate node, the trace event across a network to the selected correlation server using the network address.

21. The method of claim 20 further comprises

detecting, by the intermediate node, a failure condition with the selected correlation server;

selecting, by the intermediate node, a backup correlation server using a selection method, where the selection method is the same amongst a plurality of intermediate nodes; and

sending, by the intermediate node, the trace event across the network to the backup correlation server.

22. The method of claim 20 further comprises:

receiving, by the selected correlation server, the trace event from the intermediate node;

determining, by the selected correlation server, whether the trace event indicates a new transaction being monitored;

creating, by the selected correlation server, a transaction record using data contained in the trace event, wherein the creating is in response to the determination that the trace event indicates a new transaction being monitored; and

updating, by the selected correlation server, a transaction record using data contained in the trace event, wherein the updating is in response to the determination that the trace event is not a new transaction.

23. The method of claim 19 further comprises

periodically selecting, by the agent, another intermediate node from a listing of intermediate nodes maintained by the agent and stored on the host computing device; and

sending, by the agent, subsequent trace events to the another intermediate node.