System and method for continuous low-overhead monitoring of distributed applications running on a cluster of data processing nodes
Embodiments of the present invention provide an improvement over known approaches for monitoring of and taking action on observations associated with distributed applications. Application event reporting and application resource monitoring is unified in a manner that significantly reduces storage and aggregation overhead. For example, embodiments of the present invention can employ hardware and/or software support that reduces storage and aggregation overhead. In addition to providing for fine-grained, continuous, decentralized monitoring of application activity and resource consumption, embodiments of the present invention can also provide for decentralized filtering, statistical analysis, and derived data streaming. Furthermore, embodiments of the present invention are securely implemented (e.g., for use solely under the control of an operator) and can use a separate security domain for network traffic.
1. A data processing node, comprising:
a plurality of application central processing units each having a respective application running thereon; and
a management processor unit coupled to each one of the application central processing units, wherein the management processor unit comprises an application monitoring services module including a resource assessor and an event reporter, wherein the management processor unit comprises dedicated system resources with respect to the application central processing units such that processes implemented by the application monitoring services module are out-of-band of application processes carried out on each one of the application central processing units, wherein the processes implemented by the application monitoring services module are performed on the management processor unit and do not consume resources of the application central processing units, wherein the application monitoring services module is configured to selectively implement one or more processes for assessing activity of a particular one of the application central processing units that arise from execution of the respective application running thereon and is configured to selectively implement one or more processes for outputting events generated by a particular one of the application central processing units that arise from execution of the respective application running thereon, wherein the processes implemented by the application monitoring services module include alerting an external entity of resource consumption by a particular one of the application central processing units exceeding an operating limit;
wherein:
the data processing node is one node within a cluster of interconnected nodes and implements a security domain not accessible by the external entity, wherein another node in the cluster of nodes implements a different security domain accessible by the external entity, and wherein outputting the events includes transmitting the events to the another node, thereby enabling the events to be monitored by the external entity without allowing the external entity to interact with the respective application by which the events were generated;
the application monitoring services module performs processes for assessing activity of the particular one of the application central processing units that arise from execution of the respective application running thereon;
assessing activity of the particular one of the application central processing units includes using a filter function to manipulate execution of the respective application running thereon to influence a manner in which the events are generated;
outputting the events includes applying a time stamp to each one of the events;
the data proccessing node is one node within a cluster of interconnected nodes; and
the time stamp applied to each one of the events is based upon a global time to which a local time of each one of the nodes is synchronized.
2. The data processing node of claim 1 , wherein outputting the events includes outputting the events using a messaging functionality of an application level context.