Collect and forward
Apparatus and methods are disclosed for processing messages from agents of a network environment including the use of collectors. Collectors can use configurable pipelines to improve processing of messages received from the agents. In one example of the disclosed technology, a number of networked agents are configured to gather data describing operational aspects of an agent's computing host. A collector is configured to receive reports from the agent and send the gathered data to one or more destination agent data consumers designated by augmentation information in the reports. In some examples, the collector transforms data using one or more stage selector rules.
1. A system, comprising:
a plurality of computing hosts, each computing host comprising a processor and memory, each computing host executing an agent accessible via a computer network, each agent being configured to gather data describing operational aspects of the respective agent's computing host and to augment the gathered data with augmentation information designating a plurality of destinations; and
a collector server comprising circuitry programmed to:
receive reports from the agents, the reports having a body comprising the gathered data;
select a plurality of destination agent data consumers based at least in part on the augmentation information in the reports; and
send the gathered data to the selected destination agent data consumers designated by the augmentation information in the reports,
wherein the collector server is separate and distinct from the plurality of agents, and wherein the collector server sends messages to both discover and configure the plurality of agents, and
wherein the collector server enhances data in the body of the received reports into a format selected based on the destination agent data consumer.
2. The system of claim 1 , wherein each of the agents is configured using a discovery script to configure the agent based on available resources and plugins executed by the agent.
3. The system of claim 1 , wherein the collector server includes a processing pipeline configured to select a scaled or non-scaled destination set.
4. The system of claim 1 , wherein the collector server includes a processing pipeline configured to transform data received from the agents using one or more stage selector rules.
5. The system of claim 1 , wherein the plurality of destination agent data consumers comprises at least one of the following: map-reduce computing cluster or a resilient distributed dataset cluster.
6. The system of claim 1 , wherein the collector server is configured using a programmable template.
7. The system of claim 1 , wherein the collector server includes a processing pipeline configured to transform data received from one or more of the agents using one or more stage selector rules, at least one of the stage selector rules being based on a tag associated with the one or more of the agents.
8. The system of claim 1 , wherein the augmentation information comprises at least one of the following: an IP address of a destination, a service name of a destination, data formatting rules for a destination, and contextual details about the computing host.
9. A method of transmitting data with a pipeline process, the method comprising:
receiving, by a collector server, a report from a data collecting agent, the received report including data and augmentation information designating one or more destinations, the data being gathered by the data collecting agent and describing operational aspects of a computing host that executes the data collecting agent accessible via a computer network, the data collecting agent being configured to gather data describing operational aspects of the data collecting agent's computing host and to augment the gathered data with augmentation information designating a plurality of destinations;
enhancing, by the collector server, the received data in the received report into a format selected based on the designated one or more destinations by applying one or more stage selector rules to the received data, the enhanced data including a subset of the received data defined based at least in part on the stage selector rules, the applying one or more stage selector rules including selecting one or more destination agent data consumers from a plurality of destination agent data consumers based at least in part on the augmentation information in the received report; and
sending, by the collector server, the enhanced data to the selected one or more destination agent data consumers,
wherein the sending the enhanced data comprises, based on an applied one of the stage selector rules:
if a destination agent data consumer is selected by the applied stage selector rule, then sending the enhanced data to the destination agent data consumer indicated by the stage selector rule; and
if no destination agent consumer is selected by the applied stage selector rule, then sending the enhanced data to a destination indicated by an affinity service, the affinity service selecting the destination from among multiple possible destinations based on network traffic conditions or random selection.
10. The method of claim 9 , wherein the enhancing comprises filtering the received data.
11. The method of claim 9 , wherein the enhancing comprises modifying the received data to a format selected based on the destination agent data consumer.
12. The method of claim 9 , wherein the destination agent data consumers are selected based at least in part on feedback from a load balancer.
13. The method of claim 9 , wherein the stage selector rules are arranged according to a JavaScript Object Notation (JSON) format.
14. The method of claim 9 , wherein the data is enhanced based on tag information in the received data.
15. The method of claim 9 , wherein the sending the enhanced data comprises sending the enhanced data to each destination agent data consumer in a destination set indicated by an applied one of the stage selector rules.
16. The method of claim 9 , wherein the one or more stage selector rules are for a first stage, the method further comprising, prior to the sending the enhanced data, further enhancing the received data by applying an additional one or more stage selector rules to the received data.
17. The method of claim 9 , wherein the stage selector rules are user-generated.
18. A method of generating and using augmentation data with an agent executing on a computing host, the method comprising:
configuring data gathering behavior of the agent based on a configuration template received from a collector via a computer network coupled to the computing host, the agent being configured to gather data describing operational aspects of the agent's computing host and to augment the gathered data with augmentation information designating a plurality of destinations;
generating destination information based on the configuration template, the destination information identifying multiple destinations, each of the multiple destinations being distinct from the collector;
with the agent, gathering data about the computing host according to the configured data gathering behavior defined by the configuration template;
receiving, by the collector, a report from the agent, the report having a body comprising the gathered data;
enhancing, by the collector, the gathered data into a format selected based on the identified destinations; and
sending a message to the collector, the message including the gathered data and the destination information.
19. The method of claim 18 , further comprising:
by the collector, sending the message to each of the identified destinations indicated by the destination information.
20. The method of claim 18 , wherein the configuration template includes a description of data to be collected by the agent.
21. The method of claim 18 , wherein the configuring comprises:
executing a discovery script with the agent to determine plugins of the agent for which to gather data; and
performing the configuring data gathering behavior of the agent based on the determined plugins.