Unified data processing across streaming and indexed data sets
Systems and methods are described for unified processing of indexed and streaming data. A system enables users to query indexed data or specify processing pipelines to be applied to streaming data. In some instances, a user may specify a query intended to be run against indexed data, but may specify criteria that includes not-yet-indexed data (e.g., a future time frame). The system may convert the query into a data processing pipeline applied to not-yet-indexed data, thus increasing the efficiency of the system. Similarly, in some instances, a user may specify a data processing pipeline to be applied to a data stream, but specify criteria including data items outside the data stream. For example, a user may wish to apply the pipeline retroactively, to data items that have already exited the data stream. The system can convert the pipeline into a query against indexed data to satisfy the users processing requirements.
1 . A method comprising:
obtaining a query for data indexed at a data processing system, the query specifying criteria for identifying search results from data items previously indexed by an indexing subsystem of the data processing system;
converting the query into a data processing pipeline, the data processing pipeline specifying a series of nodes and interconnections between individual nodes within the series, wherein the individual nodes designate a transformation of one or more data items within the data processing pipeline, wherein the interconnections designate a routing of messages through the data processing pipeline, wherein the series of nodes and interconnections logically represent the query, wherein the query is designated by a user as a recurring query, and wherein converting the query into the data processing pipeline implements the recurring query while requiring that the query be run only once against the data items previously indexed by the indexing subsystem of the data processing system;
providing a graphical user interface including the series of nodes and interconnections between the individual nodes within the series, wherein the individual nodes designate a transformation of messages on a streaming data processing subsystem and wherein the interconnections designate a routing of messages on the streaming data processing subsystem;
receiving, via the graphical user interface, a modification to the data processing pipeline to result in a modified data processing pipeline, wherein the modification comprises a change to the series of nodes and interconnections; and
applying the modified data processing pipeline to the streaming data processing subsystem of the data processing system to cause the streaming data processing subsystem to process a set of data items within a stream of new messages obtained at the data processing system according to the modified data processing pipeline.
2 . The method of claim 1 , wherein the change to the series of nodes and interconnections comprises an addition of a node to the series of nodes.
3 . The method of claim 1 , wherein the change to the series of nodes and interconnections comprises an addition of an interconnection between two nodes within the series of nodes.
4 . The method of claim 1 , further comprising:
converting the modified data processing pipeline into a modified query for data indexed at the data processing system; and
executing the modified query against the data items previously indexed by the indexing subsystem.
5 . The method of claim 1 , wherein the streaming data processing subsystem corresponds to an iterative publish-subscribe-based message processing subsystem that operates to retrieve messages from a source publish-subscribe messaging system including the stream of new messages and to publish results of processing the data items within the stream of new messages according to the data processing pipeline to a destination publish-subscribe messaging system.
6 . The method of claim 1 , wherein the stream of new messages includes data items representing raw machine data.
7 . The method of claim 1 , wherein the data processing pipeline is represented as an abstract syntax tree (AST).
8 . A system comprising:
one or more data stores including computer-executable instructions; and
one or more processors configured to execute the computer-executable instructions, wherein execution of the instructions causes the system to:
obtain a query for data indexed at a data processing system, the query specifying criteria for identifying search results from data items previously indexed by an indexing subsystem of the data processing system;
convert the query into a data processing pipeline, the data processing pipeline specifying a series of nodes and interconnections between individual nodes within the series, wherein the individual nodes designate a transformation of one or more data items within the data processing pipeline, wherein the interconnections designate a routing of messages through the data processing pipeline, wherein the series of nodes and interconnections logically represent the query, wherein the query is designated by a user as a recurring query, and wherein converting the query into the data processing pipeline implements the recurring query while requiring that the query be run only once against the data items previously indexed by the indexing subsystem of the data processing system;
provide a graphical user interface including the series of nodes and interconnections between the individual nodes within the series, wherein the individual nodes designate a transformation of messages on a streaming data processing subsystem, and wherein the interconnections designate a routing of messages on the streaming data processing subsystem;
receive, via the graphical user interface, a modification to the data processing pipeline to result in a modified data processing pipeline, wherein the modification comprises a change to the series of nodes and interconnections; and
apply the modified data processing pipeline to the streaming data processing subsystem of the data processing system to cause the streaming data processing subsystem to process a set of data items within a-stream of new messages obtained at the data processing system according to the modified data processing pipeline.
9 . The system of claim 8 , wherein the change to the series of nodes and interconnections comprises an addition of a node to the series of nodes.
10 . The system of claim 8 , wherein the change to the series of nodes and interconnections comprises an addition of an interconnection between two nodes within the series of nodes.
11 . The system of claim 8 , wherein execution of the instructions further causes the system to:
convert the modified data processing pipeline into a modified query for data indexed at the data processing system; and
execute the modified query against the data items previously indexed by the indexing subsystem.
12 . The system of claim 8 , wherein the streaming data processing subsystem corresponds to an iterative publish-subscribe-based message processing subsystem that operates to retrieve messages from a source publish-subscribe messaging system including the stream of new messages and to publish results of processing the data items within the stream of new messages according to the data processing pipeline to a destination publish-subscribe messaging system.
13 . The system of claim 8 , wherein the stream of new messages includes data items representing raw machine data.
14 . The system of claim 8 , wherein the data processing pipeline is represented as an abstract syntax tree (AST).
15 . A non-transitory computer-readable media comprising instructions executable by a computing system to:
obtain a query for data indexed at a data processing system, the query specifying criteria for identifying search results from data items previously indexed by an indexing subsystem of the data processing system;
convert the query into a data processing pipeline, the data processing pipeline specifying a series of nodes and interconnections between individual nodes within the series, wherein the individual nodes designate a transformation of one or more data items within the data processing pipeline, wherein the interconnections designate a routing of messages through the data processing pipeline, wherein the series of nodes and interconnections logically represent the query, wherein the query is designated by a user as a recurring query, and wherein converting the query into the data processing pipeline implements the recurring query while requiring that the query be run only once against the data items previously indexed by the indexing subsystem of the data processing system;
provide a graphical user interface including the series of nodes and interconnections between the individual nodes within the series, wherein the individual nodes designate a transformation of messages on a streaming data processing subsystem, and wherein the interconnections designate a routing of messages on the streaming data processing subsystem;
receive, via the graphical user interface, a modification to the data processing pipeline to result in a modified data processing pipeline, wherein the modification comprises a change to the series of nodes and interconnections; and
apply the modified data processing pipeline to the streaming data processing subsystem of the data processing system to cause the streaming data processing subsystem to process a set of data items within a stream of new messages obtained at the data processing system according to the modified data processing pipeline.
16 . The non-transitory computer-readable media of claim 15 , wherein the change to the series of nodes and interconnections comprises an addition of a node to the series of nodes.
17 . The non-transitory computer-readable media of claim 15 , wherein the change to the series of nodes and interconnections comprises an addition of an interconnection between two nodes within the series of nodes.
18 . The non-transitory computer-readable media of claim 15 , further comprising instructions executable by a computing system to:
convert the modified data processing pipeline into a modified query for data indexed at the data processing system; and
execute the modified query against the data items previously indexed by the indexing subsystem.
19 . The non-transitory computer-readable media of claim 15 , wherein the streaming data processing subsystem corresponds to an iterative publish-subscribe-based message processing subsystem that operates to retrieve messages from a source publish-subscribe messaging system including the stream of new messages and to publish results of processing the data items within the stream of new messages according to the data processing pipeline to a destination publish-subscribe messaging system.
20 . The non-transitory computer-readable media of claim 15 , wherein the stream of new messages includes data items representing raw machine data.