Guided creation interface for streaming data processing pipelines
Systems and methods are disclosed for implementing a data processing workflow user interface for a streaming data processing system. The workflow is visually represented as a series of modules along with interconnections for the modules. Each module represents an operation on a streaming data object, such as a data transformation. The user interface enables selection of a workflow template based on a user-specified data source, and then allows the user to customize the workflow template by specifying additional operations to apply to data objects. The interface may show the user a preview of output data objects processed according to the customized workflow.
1. A computer-implemented method comprising:
obtaining, by a streaming data processing system, a plurality of streaming data pipeline templates, wherein each of the plurality of streaming data pipeline templates describes steps for processing objects from data sources in a respective data source category;
obtaining, based at least in part on individual templates of the plurality of streaming data pipeline templates, a plurality of streaming data pipeline configuration user interfaces, wherein individual user interfaces of the plurality of streaming data pipeline configuration user interfaces correspond to individual templates of the plurality of streaming data pipeline templates, and wherein individual user interfaces of the plurality of streaming data pipeline configuration user interfaces enable user configuration of a streaming data pipeline in accordance with the steps for processing objects described in the corresponding streaming data pipeline template; and
displaying a data source category selection user interface, wherein the data source category selection user interface enables user selection of a data source category from a plurality of data source categories;
in response to selection of a first data source category via the data source category selection user interface, displaying a first streaming data pipeline configuration user interface of the plurality of streaming data pipeline configuration user interfaces corresponding to a first streaming data pipeline template of the plurality of streaming data pipeline templates, wherein the first streaming data pipeline template corresponds to the first data source category.
2. The computer-implemented method of claim 1 , wherein obtaining the plurality of streaming data pipeline configuration user interfaces comprises generating a streaming data pipeline configuration user interface based at least in part on a streaming data pipeline template.
3. The computer-implemented method of claim 1 , wherein the first streaming data pipeline configuration user interface enables user configuration of the streaming data pipeline to process data objects from a particular data source.
4. The computer-implemented method of claim 1 further comprising receiving, via the first streaming data pipeline configuration user interface, one or more inputs corresponding to a streaming data pipeline configuration.
5. The computer-implemented method of claim 1 further comprising configuring the streaming data pipeline to process data objects from a particular data source in accordance with one or more inputs received via the first streaming data pipeline configuration user interface.
6. The computer-implemented method of claim 1 further comprising generating a streaming data pipeline based at least in part on the first streaming data pipeline template and a streaming data pipeline configuration received via the first streaming data pipeline configuration user interface.
7. The computer-implemented method of claim 1 further comprising processing a data object from a particular data source using the streaming data pipeline in accordance with a streaming data pipeline configuration received via the first streaming data pipeline configuration user interface.
8. The computer-implemented method of claim 1 further comprising generating the data source category selection user interface based at least in part on the plurality of streaming data pipeline templates.
9. The computer-implemented method of claim 1 , wherein the data source category selection user interface comprises a plurality of user interface elements, wherein individual user interface elements of the plurality of user interface elements correspond to individual data source categories, and wherein the selection of the first data source category comprises selection of a first user interface element of the plurality of user interface elements.
10. The computer-implemented method of claim 1 , wherein the first streaming data pipeline configuration user interface enables specifying a particular data source for the streaming data pipeline.
11. The computer-implemented method of claim 1 , wherein the first streaming data pipeline configuration user interface enables configuring one or more transformations to perform on data objects that are processed using the streaming data pipeline.
12. The computer-implemented method of claim 1 , wherein the streaming data pipeline, upon processing a data object, transforms the data object from a first format associated with the data source category to a second format associated with the streaming data processing system.
13. The computer-implemented method of claim 1 , wherein the first streaming data pipeline configuration user interface comprises a streaming data pipeline input module configuration user interface.
14. The computer-implemented method of claim 1 further comprising displaying a streaming data pipeline configuration output module user interface, wherein the streaming data pipeline configuration output module user interface corresponds to an output module of the first streaming data pipeline template.
15. The computer-implemented method of claim 1 , wherein the first streaming data pipeline template comprises a plurality of modules.
16. The computer-implemented method of claim 1 , wherein the first streaming data pipeline template corresponds to a subset of the plurality of streaming data pipeline configuration user interfaces.
17. The computer-implemented method of claim 1 , wherein the first streaming data pipeline template corresponds to one or more of a streaming data pipeline input module configuration user interface, streaming data pipeline output module configuration user interface, streaming data pipeline transformation module configuration user interface, or a streaming data pipeline event identification module configuration user interface.
18. The computer-implemented method of claim 1 , wherein the first streaming data pipeline configuration user interface comprises one or more of a streaming data pipeline input module configuration user interface, streaming data pipeline output module configuration user interface, streaming data pipeline transformation module configuration user interface, or streaming data pipeline event identification module configuration user interface.
19. A system comprising:
a data store storing computer-executable instructions; and
a processor configured to execute the computer-executable instructions, wherein when executed the computer-executable instructions cause the system to:
obtain, by a streaming data processing system, a plurality of streaming data pipeline templates, wherein each of the plurality of streaming data pipeline templates describes steps for processing objects from data sources in a respective data source category;
obtain, based at least in part on individual templates of the plurality of streaming data pipeline templates, a plurality of streaming data pipeline configuration user interfaces, wherein individual user interfaces of the plurality of streaming data pipeline configuration user interfaces correspond to individual templates of the plurality of streaming data pipeline templates, and wherein individual user interfaces of the plurality of streaming data pipeline configuration user interfaces enable user configuration of a streaming data pipeline in accordance with the steps for processing objects described in the corresponding streaming data pipeline template; and
display a data source category selection user interface, wherein the data source category selection user interface enables user selection of a data source category from a plurality of data source categories;
in response to selection of a first data source category via the data source category selection user interface, display a first streaming data pipeline configuration user interface of the plurality of streaming data pipeline configuration user interfaces corresponding to a first streaming data pipeline template of the plurality of streaming data pipeline templates, wherein the first streaming data pipeline template corresponds to the first data source category.
20. One or more non-transitory computer-readable media comprising computer-executable instructions that, when executed by a computing system, cause the computing system to:
obtain, by a streaming data processing system, a plurality of streaming data pipeline templates, wherein each of the plurality of streaming data pipeline templates describes steps for processing objects from data sources in a respective data source category;
obtain, based at least in part on individual templates of the plurality of streaming data pipeline templates, a plurality of streaming data pipeline configuration user interfaces, wherein individual user interfaces of the plurality of streaming data pipeline configuration user interfaces correspond to individual templates of the plurality of streaming data pipeline templates, and wherein individual user interfaces of the plurality of streaming data pipeline configuration user interfaces enable user configuration of a streaming data pipeline in accordance with the steps for processing objects described in the corresponding streaming data pipeline template; and
display a data source category selection user interface, wherein the data source category selection user interface enables user selection of a data source category from a plurality of data source categories;
in response to selection of a first data source category via the data source category selection user interface, display a first streaming data pipeline configuration user interface of the plurality of streaming data pipeline configuration user interfaces corresponding to a first streaming data pipeline template of the plurality of streaming data pipeline templates, wherein the first streaming data pipeline template corresponds to the first data source category.