Assisted visual programming for iterative message processing system
Systems and methods are described for graphical user interfaces that enable users to graphically design nodes and interconnections of pipelines for preprocessing data later ingested into an indexing system. The preprocessing can include receiving messages published to a first publish-subscribe messaging system, the messages containing raw machine data generated by one or more components in an information technology environment, performing one or more processing operations on at least some of the messages to generate preprocessed messages, republishing the preprocessed messages to a second publish-subscribe messaging system, and providing to the indexing system, a subset of the messages from the second publish-subscribe messaging system. Nodes in the pipeline can specify the preprocessing operations, and interconnections in the pipeline can represent data flow through the nodes of the pipeline.
1. A method comprising:
providing a graphical interface to receive a user input specifying a processing pipeline to be implemented on an iterative message processing system, the processing pipeline specified in the graphical interface as a series of nodes between an upstream publishing system and a downstream subscribing system and interconnections between individual nodes within the series, wherein an individual node of the individual nodes designates a manipulation of messages including machine data on the iterative message processing system, and wherein the interconnections designate a routing of messages through the iterative message processing system;
receiving specification in the graphical interface of a node in the series of nodes;
outputting, in the graphical interface, information identifying an individual node to optionally add to the series based on the received specification of the node, wherein the individual node to optionally add is identified based on a compatibility between message formats of the individual node to optionally add and the node in the received specification;
receiving, through the graphical interface, instruction to add the individual node to the processing pipeline; and
programming the iterative message processing system to implement the processing pipeline.
2. The method of claim 1 , further comprising identifying the individual node to optionally add to the series based on historical data that flowed from or into the node in the received specification.
3. The method of claim 1 , further comprising identifying the individual node to optionally add to the series based on historical data that flowed from or into the node in the received specification, and responsive to determining that a threshold percentage of the historical data contained a type of data corresponding to a particular function, identifying the individual node as a node representing the particular function.
4. The method of claim 1 , wherein the individual node is added to the pipeline prior to the node in the received specification.
5. The method of claim 1 , wherein the specification further specifies an additional node in the series of nodes, wherein the individual node to optionally add is further identified based on a compatibility between message formats of the individual node to optionally add and the additional node in the received specification.
6. The method of claim 1 , wherein the specification further specifies the upstream publishing system, and wherein the individual node to optionally add is further identified based on a compatibility between a message format of the individual node to optionally add and a message format of the upstream publishing system.
7. The method of claim 1 , wherein the specification further specifies the downstream publishing system, and wherein the individual node to optionally add is further identified based on a compatibility between a message format of the individual node to optionally add and a message format of the downstream publishing system.
8. The method of claim 1 , further comprising identifying the individual node to optionally add by using a trained machine learning model to analyze nodes currently in the series and generate a recommendation for the individual node.
9. The method of claim 1 , further comprising determining the individual node to optionally add by creating an n-gram representation of punctuation in data predicted to be output by the node in the received specification, using a trained machine learning model to analyze the n-gram representation and determine a source type, and identifying the individual node to optionally add based on the source type.
10. The method of claim 1 , further comprising outputting, based on identification of multiple source types within data expected to flow through the node, information identifying a conditional branch to insert into the pipeline, the conditional branch including a filter specifying a source type of the multiple source types.
11. The method of claim 1 , wherein programming the iterative message processing system to implement the processing pipeline comprises configuring an abstract syntax tree (AST) representation of the pipeline and implementing the AST representation to perform streaming processing between the upstream publishing system and the downstream subscribing system.
12. The method of claim 1 , further comprising configuring the graphical interface with a visual representation of the series of nodes and the interconnections.
13. The method of claim 1 , further comprising receiving specification of at least one additional upstream publishing system, wherein the series of nodes and the interconnections combines messages from the upstream publishing system and the at least one additional upstream publishing system for writing to the downstream subscribing system.
14. The method of claim 1 , further comprising receiving specification of at least one additional downstream subscribing system, wherein the series of nodes and the interconnections provides messages from the upstream publishing system to the downstream subscribing system and the at least one additional downstream subscribing system.
15. A system comprising:
a data store including computer-executable instructions; and
one or more processors configured to execute the computer-executable instructions, wherein execution of the computer-executable instructions causes the system to:
provide a graphical interface to receive a user input specifying a processing pipeline to be implemented on an iterative message processing system, the processing pipeline specified in the graphical interface as a series of nodes between an upstream publishing system and a downstream subscribing system and interconnections between individual nodes within the series, wherein an individual node of the individual nodes designates a manipulation of messages including machine data on the iterative message processing system and wherein the interconnections designate a routing of messages through the iterative message processing system;
receive specification in the graphical interface of a node in the series of nodes;
output, in the graphical interface, information identifying an individual node to optionally add to the series based on the received specification of the node, wherein the individual node to optionally add is identified based on a compatibility between message formats of the individual node to optionally add and the node in the received specification;
receive, through the graphical interface, instruction to add the individual node to the processing pipeline; and
program the iterative message processing system to implement the processing pipeline.
16. The system of claim 15 , wherein execution of the computer-executable instructions further causes the system to identify the individual node to optionally add to the series based on historical data that flowed from or into the node in the received specification, wherein identifying the individual node to optionally add to the series based on the historical comprises determining that a threshold percentage of the historical data contained a type of data corresponding to a particular function, and responsive to determining that the threshold percentage of the historical data contained the type of data corresponding to the particular function, identify the individual node as a node representing the particular function.
17. The system of claim 15 , wherein the specification further specifies an additional node in the series of nodes, wherein the individual node to optionally add is further identified based on a compatibility between message formats of the individual node to optionally add and the additional node in the received specification.
18. One or more non-transitory computer-readable media including computer-executable instructions that, when executed by a computing system, cause the computing system to:
provide a graphical interface to receive a user input specifying a processing pipeline to be implemented on an iterative message processing system, the processing pipeline specified in the graphical interface as a series of nodes between an upstream publishing system and a downstream subscribing system and interconnections between individual nodes within the series, wherein an individual node of the individual nodes designates a manipulation of messages including machine data on the iterative message processing system and wherein the interconnections designate a routing of messages through the iterative message processing system;
receive specification in the graphical interface of a node in the series of nodes;
output, in the graphical interface, information identifying an individual node to optionally add to the series based on the received specification of the node, wherein the individual node to optionally add is identified based on a compatibility between message formats of the individual node to optionally add and the node in the received specification;
receive, through the graphical interface, instruction to add the individual node to the processing pipeline; and
program the iterative message processing system to implement the processing pipeline.
19. The one or more non-transitory computer-readable media of claim 18 , wherein execution of the computer-executable instructions further causes the computing system to identify the individual node to optionally add to the series based on historical data that flowed from or into the node in the received specification, wherein identifying the individual node to optionally add to the series based on the historical comprises determining that a threshold percentage of the historical data contained a type of data corresponding to a particular function, and responsive to determining that the threshold percentage of the historical data contained the type of data corresponding to the particular function, identify the individual node as a node representing the particular function.
20. The one or more non-transitory computer-readable media of claim 18 , wherein the specification further specifies an additional node in the series of nodes, wherein the individual node to optionally add is further identified based on a compatibility between message formats of the individual node to optionally add and the additional node in the received specification.