Dynamic logging
In general, techniques are described for dynamically modifying the extent of logging performed by logging information generators in response to events detected in logging information received by the collector. In some examples, a network device includes one or more processors and a collector executed by the processors to receive a log message that includes logging information from a generator. The network device also includes a rules engine to apply one or more rules that each specify a condition and a corresponding action to the logging information to identify a matching rule, wherein the rules engine, upon identifying a matching rule, executes the action of the matching rule to generate and send a logging modification message to increase an extent to which the generator generates logging information.
1. A method comprising:
sending, by a control plane virtual machine of a virtual network controller using an eXtensible Messaging and Presence Protocol (XMPP) and to a virtual network switch of a server device of a set of server devices that implement a virtual network, configuration information to configure the virtual network switch to forward, for a virtual machine of the server device that executes an application, network traffic over the virtual network by tunneling packets to one or more additional server devices having corresponding virtual network switches to facilitate overlay by the virtual network of a physical network interconnecting the set of server devices,
wherein the virtual network controller comprises:
a configuration virtual machine to store and manage at least a portion of a configuration database that stores a set of configuration information for the virtual network switches;
an analytics virtual machine to store and manage at least a portion of an analytics database that stores a set of logging information for the virtual network switches;
receiving, by a collector executing by the analytics virtual machine and via the control plane virtual machine, a log message that includes logging information from a generator of the server device;
applying, by a rules engine of the collector, one or more rules that each specifies a condition and a corresponding action to the logging information of the log message to identify a matching rule; and
upon identifying the matching rule, executing the action of the matching rule to generate and send a logging modification message to increase an extent to which the generator generates subsequent logging information, wherein the action of the matching rule specifies at least one of an active probe to trigger a diagnostic subroutine of the application and an application trace to trace contents of a memory buffer used by the application, and wherein the logging modification message includes a directive to the generator to perform the at least one of the active probe of the application and the application trace of the application;
configuring, by the control plane virtual machine, the virtual network switches using XMPP to send route data and the set of configuration information to the virtual network switches; and
receiving, by the control plane virtual machine using XMPP, the log message and routing the log message to the analytics virtual machine for storage of the logging information of the log message to the analytics database.
2. The method of claim 1 , further comprising:
receiving, by the collector, an additional log message that includes additional logging information generated by the generator in accordance with the increased extent.
3. The method of claim 2 , wherein the action of the matching rule specifies the active probe of the application, and wherein the additional logging information comprises a result of the active probe.
4. The method of claim 2 , wherein the action of the matching rule specifies the application trace of the application, and wherein the additional logging information comprises a result of the application trace.
5. The method of claim 1 , further comprising:
receiving, by the collector, a rule modification message that includes a new rule;
adding the new rule to the one or more rules; and
applying the new rule to the logging information.
6. A network device comprising:
at least one memory;
one or more processors operably coupled to the at least one memory;
a virtual network controller configured for execution by the one or more processors and comprising:
a control plane virtual machine to send, using an eXtensible Messaging and Presence Protocol (XMPP) and to a virtual network switch of a server device of a set of server devices that implement a virtual network, configuration information to configure the virtual network switch to forward, for a virtual machine of the server device that executes an application, network traffic over the virtual network by tunneling packets to one or more additional server devices having corresponding virtual network switches to facilitate overlay by the virtual network of a physical network interconnecting the set of server devices;
a configuration virtual machine to store and manage at least a portion of a configuration database that stores a set of configuration information for the virtual network switches;
an analytics virtual machine to store and manage at least a portion of an analytics database that stores a set of logging information for the virtual network switches,
wherein the analytics virtual machine is configured to execute a collector to receive, via the control plane virtual machine, a log message that includes logging information from a generator of the server device,
wherein the analytics virtual machine is further configured to execute a rules engine to apply one or more rules that each specifies a condition and a corresponding action to the logging information of the log message to identify a matching rule,
wherein the rules engine, upon identifying the matching rule, executes the action of the matching rule to generate and send a logging modification message to increase an extent to which the generator generates subsequent logging information, wherein the action of the matching rule specifies at least one of an active probe to trigger a diagnostic subroutine of the application and an application trace to trace contents of a memory buffer used by the application, and wherein the logging modification message includes a directive to the generator to perform the at least one of the active probe of the application and the application trace of the application, and
wherein the control plane virtual machine configures the virtual network switches by using XMPP to send route data and the set of configuration information to the virtual network switches, and
wherein the control plane virtual machine uses XMPP to receive the log message for the virtual network switch and routing the log message to the analytics virtual machine for storage of the logging information of the log message to the analytics database.
7. The network device of claim 6 , wherein the collector receives an additional log message that includes additional logging information generated by the generator in accordance with the increased extent.
8. The network device of claim 7 , wherein the action of the matching rule specifies the active probe of the application, and wherein the additional logging information comprises a result of the active probe.
9. The network device of claim 7 , wherein the action of the matching rule specifies the application trace of the application, and wherein the additional logging information comprises a result of the application trace.
10. The network device of claim 6 ,
wherein the control plane virtual machine is configured to receive a rule modification message that includes a new rule and add the new rule to the one or more rules,
wherein the rules engine applies the new rule to the logging information.
11. The network device of claim 6 ,
wherein the analytics database includes a log database,
wherein the collector stores the logging information to the log database.
12. The network device of claim 11 ,
wherein the analytics virtual machine is configured to analyze the analytics database to diagnose conditions of the virtual network.
13. A system comprising:
a virtual network controller comprising:
at least one memory;
first one or more processors operably coupled to the at least one memory;
a control plane virtual machine configured for execution by the first one or more processors to send, using an eXtensible Messaging and Presence Protocol (XMPP) and to a virtual network switch of a server device of a set of server devices that implement a virtual network, configuration information to configure the virtual network switch to forward, for a virtual machine of the server device that executes an application, network traffic over the virtual network by tunneling packets to one or more additional server devices having corresponding virtual network switches to facilitate overlay by the virtual network of a physical network interconnecting the set of server devices;
a configuration virtual machine configured for execution by the first one or more processors to store and manage at least a portion of a configuration database that stores a set of configuration information for the virtual network switches;
an analytics virtual machine configured for execution by the first one or more processors to store and manage at least a portion of an analytics database that stores a set of logging information for the virtual network switches,
wherein the analytics virtual machine is configured to execute a collector to receive, via the control plane virtual machine, a log message that includes logging information from a generator of the server device,
wherein the analytics virtual machine is further configured to execute a rules engine to apply one or more rules that each specifies a condition and a corresponding action to the logging information of the log message to identify a matching rule,
wherein the rules engine, upon identifying the matching rule, executes the action of the matching rule to generate and send a logging modification message to increase an extent to which the generator generates subsequent logging information, wherein the action of the matching rule specifies at least one of an active probe to trigger a diagnostic subroutine of the application and an application trace to trace contents of a memory buffer used by the application, and wherein the logging modification message includes a directive to the generator to perform the at least one of the active probe of the application and the application trace of the application, and
wherein the control plane virtual machine configures the virtual network switches by using XMPP to send route data and the set of configuration information to the virtual network switches, and
wherein the control plane virtual machine uses XMPP to receive the log message for the virtual network switch and routing the log message to the analytics virtual machine for storage of the logging information of the log message to the analytics database; and
the server device, wherein the server device comprises:
a memory;
second one or more processors operably coupled to the memory;
a virtual machine configured for execution by the second one or more processors;
the generator, wherein the generator is configured for execution by the virtual machine;
an output device to send, to the collector of the virtual network controller, the baseline logging information for the application in the log message; and
an input device to receive, from the collector of the virtual network controller, the logging modification message,
wherein the generator, responsive to the logging modification message, increases the extent to which the generator generates subsequent logging information, and
wherein the output device sends, to the collector, an additional log message that includes additional logging information generated by the generator in accordance with the increased extent.
14. The system of claim 13 , wherein the server device further comprises an active probe module of the generator to execute the active probe, wherein the additional logging information comprises a result of the application probe.
15. The system of claim 13 , wherein the server device further comprises an application trace module to execute the application trace, wherein the additional logging information comprises a result of the application trace.
16. A non-transitory computer-readable storage medium comprising instructions for causing one or more programmable processors to:
send, by a control plane virtual machine of a virtual network controller using an eXtensible Messaging and Presence Protocol (XMPP) and to a virtual network switch of a server device of a set of server devices that implement a virtual network, configuration information to configure the virtual network switch to forward, for a virtual machine of the server device that executes an application, network traffic over the virtual network by tunneling packets to one or more additional server devices having corresponding virtual network switches to facilitate overlay by the virtual network of a physical network interconnecting the set of server devices,
wherein the virtual network controller comprises:
a configuration virtual machine to store and manage at least a portion of a configuration database that stores a set of configuration information for the virtual network switches;
an analytics virtual machine to store and manage at least a portion of an analytics database that stores a set of logging information for the virtual network switches;
receive, by a collector executing by the analytics virtual machine and via the control plane virtual machine, a log message that includes logging information from a generator of the server device;
apply, by a rules engine of the collector, one or more rules that each specifies a condition and a corresponding action to the logging information of the log message to identify a matching rule; and
upon identifying the matching rule, execute the action of the matching rule to generate and send a logging modification message to increase an extent to which the generator generates subsequent logging information, wherein the action of the matching rule specifies at least one of an active probe to trigger a diagnostic subroutine of the application and an application trace to trace contents of a memory buffer used by the application, and wherein the logging modification message includes a directive to the generator to perform the at least one of the active probe of the application and the application trace of the application;
configure, by the control plane virtual machine, the virtual network switches using XMPP to send route data and the set of configuration information to the virtual network switches; and
receive, by the control plane virtual machine using XMPP, the log message and routing the log message to the analytics virtual machine for storage of the logging information of the log message to the analytics database.
17. The non-transitory computer-readable storage medium of claim 16 , wherein the instructions further cause the one or more programmable processors to:
receive, by the collector, an additional log message that includes additional logging information generated by the generator in accordance with the increased extent.
18. The non-transitory computer-readable storage medium of claim 17 , wherein the action of the matching rule specifies the active probe of the application, and wherein the additional logging information comprises a result of the active probe.