Monitoring operational data in data processing systems
View Patent ↗Provided are methods, apparatus and computer programs for monitoring operational data relating to a sequence of data processing operations in a network of data processing units. In response to specified boundaries for data gathering, operational data is selectively gathered into discrete data partitions for a desired time period, a desired set of processing nodes, and/or a desired set of process executions. A set of event listeners invoke operations of a data gatherer in response to events within a processing sequence and the data gatherer allocates operational data to different ones of a set of discrete logical data containers in response to notifications from the event listeners. The data collection is responsive to system events such as a collection interval expiry to end data collection for a first partition and start data collection for a second partition.
1. A data processing apparatus for monitoring and storing operational data relating to a plurality of data processing units, comprising:
a processor configured to include:
a data gatherer for gathering and allocating operational data for a selected set of the data processing units into discrete data containers, respectively, based upon specified boundaries for data collection; and
a collector, running asynchronous to the gatherer, for accumulating data, from the data containers that correspond to a defined logical boundary, into an accumulated container.
2. The data processing apparatus according to claim 1 , wherein
the plurality of data processing units comprise a set of program code components connected to form a plurality of processing paths within a computer program computational flow.
3. The data processing apparatus according to claim 1 , wherein
the specified boundaries include physical boundaries, corresponding to a user-specified set of data processing units, which limits allocation of operational data, related to the user specified set of data process units, into a particular discrete data container.
4. The data processing apparatus according to claim 1 , wherein
the specified boundaries include temporal boundaries, corresponding to a specified data gathering period, which limits allocation of the operational data, related to the specified data gathering period, into a particular discrete data container.
5. The data processing apparatus according to claim 1 , wherein
the specified boundaries include an event, upon which allocation of operational data is switched from a first discrete data container to a second discrete data container.
6. The data processing apparatus according to claim 1 , wherein
the data gatherer allocates operational data for the selected set of data processing units into the discrete data containers, respectively, for each processing thread executed by the selected set of data processing units.
7. The data processing apparatus according to claim 1 , further comprising
a queue-based transfer mechanism for receiving data containers from the data gatherer and from which the collector dequeues data containers for accumulation.
8. The data processing apparatus according to claim 1 , wherein
each of the plurality of data processing units includes an event listener for sending notifications to the data gatherer in response to events within a monitored sequence of data processing operations;
responsive to notifications from the event listeners, the data gatherer controlling the allocating of operational data to the discrete data containers.
9. The data processing apparatus according to claim 8 , wherein
the data gatherer is responsive to a first type of notification from the event listeners to switch allocation of operational data from a first discrete data container to a second discrete data container.
10. The data processing apparatus according to claim 9 , wherein
the first type of notification corresponds to a sequence of data processing operations has progressed from a prior data processing unit to a subsequent data processing unit within the plurality of data processing units.
11. The data processing apparatus according to claim 9 , wherein
the first type of notification corresponds to completed execution of the monitored sequence of data processing operations.
12. The data processing apparatus according to claim 8 , wherein
the event listeners are configurable to:
send, to the data gatherer, a notification of the start of data gathering for a specific sequence of data processing operations in response to a first event relating to the sequence of data processing operations, and
send, to the data gatherer, a notification of the end of data gathering for the specific sequence of data processing operations in response to a second event within the sequence of data processing operations,
the data gatherer, responsive to the start notification, initiating a discrete data container, and
the data gatherer, responsive to the end notification, terminating allocation of operational data to the initiated data container.
13. The data processing apparatus according to claim 8 , wherein
each of the plurality of data processing units includes
an input connection point for receiving data,
operational data processing component for processing data, and
output connection points for receiving processed data for output from the respective data processing unit; and
for each of the plurality of data processing units, the event listener is located at the input connection point, and
the event listener, responsive to data received by the respective input connection point, sends the notification to the data gatherer.
14. A method for monitoring operational data relating to a plurality of data processing units, comprising:
in response to gathered operational data, allocating, using a computer hardware system, operational data to discrete data containers, respectively, based upon specified boundaries for data collection; and
accumulating data, asynchronously from the step of allocating operating data and using the computer hardware system, from the data containers that correspond to a defined logical boundary, into an accumulated container.
15. The method according to claim 14 , wherein
the plurality of data processing units comprise a set of program code components connected to form a plurality of processing paths within a computer program computational flow.
16. The method according to claim 14 , wherein
the step of allocating operational data is responsive to physical boundaries, corresponding to a user-specified set of data processing units, which limits the allocating of operational data, related to the user specified set of data process units, into a particular discrete data container.
17. The method according to claim 14 , wherein
the step of allocating operational data is responsive to temporal boundaries, corresponding to a specified data gathering period, which limits the allocating of the operational data, related to the specified data gathering period, into a particular discrete data container.
18. The method according to claim 14 , further comprising
switching the allocation of operational data from a first discrete data container to a second discrete data container, wherein
the specified boundaries includes an event, upon which the allocation is switched.
19. The method according to claim 14 , including the steps of:
generating, using one or more event listeners, event notifications in response to events within a monitored sequence of data processing operations; and,
in response to the event notifications from the event listeners, controlling the allocation of operational data to the discrete data containers.
20. A computer program product comprising a computer-readable storage media having stored therein computer usable program code for monitoring operational data relating to a plurality of data processing units, said computer usable program code, when executed by a computer hardware system, cause the computer hardware system to perform:
in response to gathered operational data, allocating operational data to discrete data containers, respectively, based upon specified boundaries for data collection; and
accumulating data, asynchronously from the step of allocating operating data, from the data containers that correspond to a defined logical boundary, into an accumulated container.
21. The computer program product according to claim 20 , wherein
the plurality of data processing units comprise a set of program code components connected to form a plurality of processing paths within a computer program computational flow.
22. The computer program product according to claim 20 , wherein
the step of allocating operational data is responsive to physical boundaries, corresponding to a user-specified set of data processing units, which limits the allocating of operational data, related to the user specified set of data process units, into a particular discrete data container.
23. The computer program product according to claim 20 , wherein
the step of allocating operational data is responsive to temporal boundaries, corresponding to a specified data gathering period, which limits the allocating of the operational data, related to the specified data gathering period, into a particular discrete data container.
24. The computer program product according to claim 20 , further comprising
switching the allocation of operational data from a first discrete data container to a second discrete data container, wherein
the specified boundaries includes an event, upon which the allocation is switched.
25. The computer program product according to claim 20 , including the steps of:
generating, using one or more event listeners, event notifications in response to events within a monitored sequence of data processing operations; and,
in response to the event notifications from the event listeners, controlling the allocation of operational data to the discrete data containers.