Elastic system monitoring
A monitoring system using agents to dynamically collect state information at controllable intensity levels from components of systems. The system receives state information collected by an agent at a particular intensity level, and processes the state information to determine an updated intensity level for collecting state information by the agent, or by some other agent. The state information may include data indicating the performance of one or more components, such as process response times or other metrics. The intensity level for collecting further state information can be increased when, among other things, previously collected state information indicates more detailed monitoring for the component is appropriate. The intensity level for collecting further state information can be decreased when, among other things, previously collected state information indicates continued expected behavior.
1. A method for monitoring a component of a system, comprising:
receiving, from an agent of the component of the system, first state information regarding the component, the first state information collected by the agent at a first intensity level;
determining a second intensity level for collecting second state information regarding the component by the agent based on the first state information, the first state information including a user event value indicating a number of user events processed by the component and a throughput value indicating a number of requests flowing in and out of the component, the second intensity level being different from the first intensity level;
sending an instruction to the agent that causes the agent to collect the second state information at the second intensity level; and
receiving the second state information collected by the agent at the second intensity level from the agent.
2. The method of claim 1 , further comprising:
sending instructions to a plurality of agents of a plurality of components, the instructions causing each of the plurality of agents to selectively collect state information at the first intensity level or the second intensity level; and
receiving the state information from the plurality of agents at the first intensity level or the second intensity level.
3. The method of claim 2 , wherein:
the second intensity level is higher than the first intensity level; and
the instructions cause a subset of the plurality of agents to collect the state information at the second intensity level while other agents of the plurality of agents collect the state information at the first intensity level according to rotation.
4. The method of claim 1 , wherein determining the second intensity level for collecting the second state information regarding the component by the agent is based on comparing the first state information with historical state information regarding the component.
5. The method of claim 1 , further comprising:
associating the component and a second component with a component group;
receiving, from a second agent of the second component, third state information regarding the second component;
determining a third intensity level for collecting fourth state information, regarding the second component, by the second agent based on the third state information;
sending a second instruction to the agent that causes the agent to collect fourth state information, regarding the component, at the third intensity level based on the component and the second component being associated with the component group; and
receiving the fourth state information regarding the component at the third intensity level from the agent.
6. The method of claim 1 , wherein the first state information further includes at least one of:
a network state value indicating at least one of a latency, a byte transfer rate, a number of packets lost, or a number of packets received;
a process state value indicating at least one of an amount of resource consumption or user heuristics;
a system resource value indicating at least one of available processing resources or available memory resources; or
a system health value.
7. The method of claim 6 , wherein determining the second intensity level includes determining a data intensity factor defining the second intensity level based on
E
+
T
+
N
+
P
+
SR
+
SH
Env
,
where E is the user event value, T is the throughput value, N is the network state value, P is the process state value, SR is the system resource value, and SH is the system health value, and Env is a weight value for the component.
8. The method of claim 1 , further comprising:
comparing the first state information with one or more threshold values; and
generating a notification in response to the first state information satisfying the one or more threshold values.
9. The method of claim 1 , wherein the component is one of:
a server;
an application;
a container;
a data store; or
a virtual machine.
10. A system for monitoring a component, comprising:
one or more processors configured to:
receive, from an agent of the component, first state information regarding the component, the first state information collected by the agent at a first intensity level;
determine a second intensity level for collecting second state information regarding the component by the agent based on the first state information, the first state information including a user event value indicating a number of user events processed by the component and a throughput value indicating a number of requests flowing in and out of the component, the second intensity level being different from the first intensity level;
send an instruction to the agent that causes the agent to collect the second state information at the second intensity level; and
receive the second state information collected by the agent at the second intensity level from the agent.
11. The system of claim 10 , wherein the one or more processors are further configured to:
send instructions to a plurality of agents of a plurality of components, the instructions causing each of the plurality of agents to selectively collect state information at the first intensity level or the second intensity level; and
receive the state information from the plurality of agents at the first intensity level or the second intensity level.
12. The system of claim 11 , wherein:
the second intensity level is higher than the first intensity level; and
the instructions cause a subset of the plurality of agents to collect the state information at the second intensity level while other agents of the plurality of agents collect the state information at the first intensity level according to rotation.
13. The system of claim 10 , wherein determining the second intensity level for collecting the second state information regarding the component by the agent is based on comparing the first state information with historical state information regarding the component.
14. The system of claim 10 , wherein the one or more processors are further configured to:
associate the component and a second component with a component group;
receive, from a second agent of the second component, third state information regarding the second component;
determine a third intensity level for collecting fourth state information, regarding the second component, by the second agent based on the third state information;
send a second instruction to the agent that causes the agent to collect fourth state information, regarding the component, at the third intensity level based on the component and the second component being associated with the component group; and
receive the fourth state information regarding the component at the third intensity level from the agent.
15. The system of claim 10 , wherein the first state information further includes at least one of:
a network state value indicating at least one of a latency, a byte transfer rate, a number of packets lost, or a number of packets received;
a process state value indicating at least one of an amount of resource consumption or user heuristics;
a system resource value indicating at least one of available processing resources or available memory resources; or
a system health value.
16. The system of claim 15 , wherein determining the second intensity level includes determining a data intensity factor defining the second intensity level based on
E
+
T
+
N
+
P
+
SR
+
SH
Env
,
where E is the user event, T is the throughput value, N is the network state value, P is the process state value, SR is the system resource value, and SH is the system health value, and Env is a weight value for the component.
17. The system of claim 10 , wherein the one or more processors are further configured to:
compare the first state information with one or more threshold values; and
generate a notification in response to the first state information satisfying the one or more threshold values.
18. The system of claim 10 , wherein the component is one of:
a server;
an application;
a container;
a data store; or
a virtual machine.
19. A non-transitory computer readable medium storing instructions that when executed by a processor configure the processor to:
receive, from an agent of a component of a system, first state information regarding the component, the first state information collected by the agent at a first intensity level;
determine a second intensity level for collecting second state information regarding the component by the agent based on the first state information, the first state information including a user event value indicating a number of user events processed by the component and a throughput value indicating a number of requests flowing in and out of the component, the second intensity level being different from the first intensity level;
send an instruction to the agent that causes the agent to collect the second state information at the second intensity level; and
receive the second state information collected by the agent at the second intensity level from the agent.
20. The computer readable medium of claim 19 , wherein the instructions further configure the processor to:
associate the component and a second component with a component group;
receive, from a second agent of the second component, third state information regarding the second component;
determine a third intensity level for collecting fourth state information, regarding the second component, by the second agent based on the third state information;
send a second instruction to the agent that causes the agent to collect fourth state information, regarding the component, at the third intensity level based on the component and the second component being associated with the component group; and
receive the fourth state information regarding the component at the third intensity level from the agent.