Tool for creating and deploying configurable pipelines
A computing system may provide an interface for creating a data processing pipeline through which the computing system may receive configuration information for a given pipeline that is configured to receive streaming messages from a given data source, process each of the streaming messages, and then output a processed version of at least a subset of the streaming messages to a given data sink. The given pipeline may comprise a chain of two or more operators, which may take the form of enrichers, routers, and/or transformers. The computing system may then use the received configuration information to create the given pipeline. In turn, the computing system may deploy the given pipeline for use in processing streaming messages received from the given data source.
1. A computing system comprising:
a network interface configured to facilitate communication with at least one data source;
at least one processor;
a tangible, non-transitory computer-readable medium; and
program instructions stored on the tangible, non-transitory computer-readable medium that are executable by the at least one processor to cause the computing system to:
provide an interface for creating a data processing pipeline;
via the interface, receive configuration information for a given pipeline that is configured to receive streaming messages from a given data source, process each of the streaming messages, and then output a processed version of at least a subset of the streaming messages to a given data sink, wherein the given pipeline comprises a chain of two or more operators, and wherein the chain of two or more operators comprises at least one router that is configured to (i) receive a streaming message, (ii) make a decision as to whether to perform a routing operation on the received streaming message, and then (iii) output the received streaming message in accordance with that decision;
use the received configuration information to create the given pipeline; and
deploy the given pipeline for use in processing streaming messages received from the given data source.
2. The computing system of claim 1 , wherein the chain of two or more operators comprises at least one enricher that is configured to receive a streaming message, produce and append a given type of enrichment to the received streaming message, and then output the received streaming message with the appended enrichment.
3. The computing system of claim 1 , wherein the routing operation comprises outputting the received streaming message to a data sink outside of the given pipeline instead of passing the received streaming message to the next operator in the chain of two or more operators.
4. The computing system of claim 1 , wherein the routing operation comprises outputting the received streaming message to a data sink outside of the given pipeline and also passing the received streaming message to the next operator in the chain of two or more operators.
5. The computing system of claim 1 , wherein the chain of two or more operators comprises at least one transformer that is configured to receive a streaming message, transform the received streaming message into a transformed version of the streaming message having a different form, and then output the transformed version of the streaming message.
6. The computing system of claim 1 , wherein the streaming messages received from the given data source comprise streaming messages produced by one or more assets.
7. The computing system of claim 1 , wherein the given data sink comprises a streaming message topic that serves as a data source for another pipeline.
8. The computing system of claim 1 , wherein the configuration information for the given pipeline comprises a selection of the given data source, a selection of the given data sink, information defining a configuration of each operator in the chain of two or more operators, and information defining a sequence of the two or more operators in the chain.
9. The computing system of claim 1 , wherein the program instructions comprise a code library that has been integrated into another application.
10. The computing system of claim 1 , wherein at least one of the operators in the chain of two or more operators comprises error-handling logic.
11. A non-transitory computer-readable medium having instructions stored thereon that are executable to cause a computing system to:
provide an interface for creating a data processing pipeline;
via the interface, receive configuration information for a given pipeline that is configured to receive streaming messages from a given data source, process each of the streaming messages, and then output a processed version of at least a subset of the streaming messages to a given data sink, wherein the given pipeline comprises a chain of two or more operators, and wherein the chain of two or more operators comprises at least one router that is configured to (i) receive a streaming message, (ii) make a decision as to whether to perform a routing operation on the received streaming message, and then (iii) output the received streaming message in accordance with that decision;
use the received configuration information to create the given pipeline; and
deploy the given pipeline for use in processing streaming messages received from the given data source.
12. The non-transitory computer-readable medium of claim 11 , wherein the chain of two or more operators comprises at least one enricher that is configured to receive a streaming message, produce and append a given type of enrichment to the received streaming message, and then output the received streaming message with the appended enrichment.
13. The non-transitory computer-readable medium of claim 11 , wherein the chain of two or more operators comprises at least one transformer that is configured to receive a streaming message, transform the received streaming message into a transformed version of the streaming message having a different form, and then output the transformed version of the streaming message.
14. The non-transitory computer-readable medium of claim 11 , wherein the given data sink comprises a streaming message topic that serves as a data source for another pipeline.
15. A computer-implemented method, the method comprising:
providing an interface for creating a data processing pipeline;
via the interface, receiving configuration information for a given pipeline that is configured to receive streaming messages from a given data source, process each of the streaming messages, and then output a processed version of at least a subset of the streaming messages to a given data sink, wherein the given pipeline comprises a chain of two or more operators, and wherein the chain of two or more operators comprises at least one router that is configured to (i) receive a streaming message, (ii) make a decision as to whether to perform a routing operation on the received streaming message, and then (iii) output the received streaming message in accordance with that decision;
using the received configuration information to create the given pipeline; and
deploying the given pipeline for use in processing streaming messages received from the given data source.
16. The computer-implemented method of claim 15 , wherein the chain of two or more operators comprises at least one enricher that is configured to receive a streaming message, produce and append a given type of enrichment to the received streaming message, and then output the received streaming message with the appended enrichment.
17. The computer-implemented method of claim 15 , wherein the chain of two or more operators comprises at least one transformer that is configured to receive a streaming message, transform the received streaming message into a transformed version of the streaming message having a different form, and then output the transformed version of the streaming message.