Multiple data sources for a data processing pipeline
Systems and methods are described for implementation by a streaming data processing system of a processing pipeline that obtains data items from one or more data sources. The one or more data sources may include multiple partitions of a topic or multiple topics. The one or more data sources are defined by a data pattern, a group of tenants, or a process. The implementation of the processing pipeline can include identifying the one or more data sources to obtain data items. The processing pipeline can include a plurality of processing tasks to obtain the data items. The streaming data processing system can utilize a workload coordinator to dynamically assign particular data sources to particular processing tasks. The implementation of the processing pipeline can further include executing the plurality of processing tasks to cause processing tasks to obtain data items from assigned data sources and process the data items.
1 . A method implemented by a streaming data processing system, the method comprising: obtaining a specification of a data processing pipeline, the data processing pipeline specifying a series of nodes and interconnections between individual nodes within the series, wherein the nodes designate a transformation of data items within the data processing pipeline and wherein the interconnections designate a routing of messages through the data processing pipeline; and implementing the data processing pipeline on the streaming data processing system by execution of a plurality of processing tasks, wherein implementing the data processing pipeline on the streaming data processing system comprises: identifying a plurality of topics of an intake ingestion buffer from which to obtain data items for processing by the data processing pipeline, identifying a processing task of the plurality of processing tasks as a workload coordinator, wherein each of the plurality of processing tasks are assignable to be the workload coordinator, and wherein the workload coordinator dynamically assigns at least one topic of the plurality of topics to each of the plurality of processing tasks of the data processing pipeline configured to retrieve, from a corresponding topic of the plurality of topics, and process a respective data item according to the data processing pipeline, the dynamic assignment based on monitoring data associated with the plurality of topics, using the workload coordinator, generating and storing, in a data store, a first dynamic assignment of a first processing task of the plurality of processing tasks to retrieve, from a first topic of the plurality of topics, and process a first data item, using the workload coordinator, generating and storing, in the data store, a second dynamic assignment of a second processing task of the plurality of processing tasks to retrieve, from a second topic of the plurality of topics, and process a second data item, executing the first processing task to cause the first processing task to obtain the first dynamic assignment from the data store and, based on the first dynamic assignment, retrieve, from the first topic, and process the first data item according to the series of nodes and interconnections specified by the data processing pipeline, and executing the second processing task to cause the second processing task to obtain the second dynamic assignment from the data store and, based on the second dynamic assignment, retrieve, from the second topic, and process the second data item according to the series of nodes and interconnections specified by the data processing pipeline.
2 . The method of claim 1 , wherein the plurality of topics comprises distinct network locations or partitions of at least one distinct network location of the distinct network locations.
3 . The method of claim 1 , wherein identifying the plurality of topics is based at least in part on a data pattern.
4 . The method of claim 1 , wherein the plurality of topics is associated with a plurality of tenants.
5 . The method of claim 1 , wherein implementing the data processing pipeline on the streaming data processing system further comprises, using the workload coordinator, identifying the first data item and the second data item for processing.
6 . The method of claim 1 , further comprising: using the workload coordinator, generating and storing, in the data store, a third dynamic assignment of the processing task identified as the workload coordinator to retrieve, from one or more topics of the plurality of topics, and process one or more data items.
7 . The method of claim 1 , wherein identifying the plurality of topics is based at least in part on a data pattern, wherein the data pattern identifies one or more of a tenant, a namespace, a topic, or a partition.
8 . The method of claim 1 , further comprising validating that a user account is authorized for implementation of the data processing pipeline based at least in part on the user account requesting the implementation of the data processing pipeline.
9 . The method of claim 1 , wherein the plurality of processing tasks comprises a plurality of parallel processing tasks.
10 . The method of claim 1 , further comprising: determining that the first data item has been removed; based on determining that the first data item has been removed, deallocating the first processing task from the first topic; and implementing an updated data processing pipeline on the streaming data processing system, wherein implementing the updated data processing pipeline on the streaming data processing system further comprises: identifying a third topic from which to obtain a third data item for processing by the data processing pipeline, and using the workload coordinator, generating and storing, in the data store, a third dynamic assignment of the first processing task to retrieve, from the third topic, and process the third data item.
11 . The method of claim 1 , further comprising: identifying a pipeline restart of the data processing pipeline; and in response to identifying the pipeline restart of the data processing pipeline, transmitting the first dynamic assignment to the first processing task.
12 . The method of claim 1 , wherein identifying the plurality of topics is based at least in part on a data pattern, the method further comprising: identifying an update of the data pattern, wherein generating and storing the first dynamic assignment and generating and storing the second dynamic assignment is based at least in part on identifying the update of the data pattern.
13 . The method of claim 1 , wherein the data store comprises an index, a global aggregate store, a block storage system, or a global data store.
14 . The method of claim 1 , wherein each topic of the plurality of topics comprises a publish-subscribe topic or an event namespace.
15 . The method of claim 1 , wherein a particular processing task of the plurality of processing tasks: obtains one or more dynamic assignments corresponding to the particular processing task; filters the one or more dynamic assignments based on a processing task index; identifies a set of the data items assigned for processing by the particular processing task based on filtering the one or more dynamic assignments; and processes the set of the data items.
16 . The method of claim 1 , wherein the data items comprise state information or credentials.
17 . Non-transitory computer readable media comprising computer-executable instructions that, when executed by a computing system of a streaming data processing system, cause the computing system to: obtain a specification of a data processing pipeline, the data processing pipeline specifying a series of nodes and interconnections between individual nodes within the series, wherein the nodes designate a transformation of data items within the data processing pipeline and wherein the interconnections designate a routing of messages through the data processing pipeline; and implement the data processing pipeline on the streaming data processing system by execution of a plurality of processing tasks, wherein to implement the data processing pipeline on the streaming data processing system, execution of the computer-executable instructions by the computing system, causes the computing system to: identify a plurality of topics of an intake ingestion buffer from which to obtain data items for processing by the data processing pipeline, identify a processing task of the plurality of processing tasks as a workload coordinator, wherein each of the plurality of processing tasks are assignable to be the workload coordinator, and wherein the workload coordinator dynamically assigns at least one topic of the plurality of topics to each of the plurality of processing tasks of the data processing pipeline configured to retrieve, from a corresponding topic of the plurality of topics, and process a respective data item according to the data processing pipeline, the dynamic assignment based on monitoring data associated with the plurality of topics, use the workload coordinator, generate and store, in a data store, a first dynamic assignment of a first processing task of the plurality of processing tasks to retrieve, from a first topic of the plurality of topics, and process a first data item, use the workload coordinator, generate and store, in the data store, a second dynamic assignment of a second processing task of the plurality of processing tasks to retrieve, from a second topic of the plurality of topics, and process a second data item, execute the first processing task to cause the first processing task to obtain the first dynamic assignment from the data store and, based on the first dynamic assignment, retrieve, from the first topic, and process the first data item according to the series of nodes and interconnections specified by the data processing pipeline, and execute the second processing task to cause the second processing task to obtain the second dynamic assignment from the data store and, based on the second dynamic assignment, retrieve, from the second topic, and process the second data item according to the series of nodes and interconnections specified by the data processing pipeline.
18 . The non-transitory computer readable media of claim 17 , wherein the execution of the computer-executable instructions by the computing system, further causes the computing system to: determine that the first data item has been removed; based on determining that the first data item has been removed, deallocate the first processing task from the first topic; and implement an updated data processing pipeline on the streaming data processing system, wherein to implement the updated data processing pipeline on the streaming data processing system, the execution of the computer-executable instructions by the computing system, causes the computing system to: identify a third topic from which to obtain a third data item for processing by the data processing pipeline, and using the workload coordinator, generate and store, in the data store, a third dynamic assignment of the first processing task to retrieve, from the third topic, and process the third data item.
19 . A computing system of a data ingestion system, the computing system comprising: a streaming data processing system configured to: obtain a specification of a data processing pipeline, the data processing pipeline specifying a series of nodes and interconnections between individual nodes within the series, wherein the nodes designate a transformation of data items within the data processing pipeline and wherein the interconnections designate a routing of messages through the data processing pipeline; and implement the data processing pipeline on the streaming data processing system by execution of a plurality of processing tasks, wherein to implement the data processing pipeline on the streaming data processing system, the streaming data processing system is further configured to: identify a plurality of topics of an intake ingestion buffer from which to obtain data items for processing by the data processing pipeline, identify a processing task of the plurality of processing tasks as a workload coordinator, wherein each of the plurality of processing tasks are assignable to be the workload coordinator, and wherein the workload coordinator dynamically assigns at least one topic of the plurality of topics to each of the plurality of processing tasks of the data processing pipeline configured to retrieve, from a corresponding topic of the plurality of topics, and process a respective data item according to the data processing pipeline, the dynamic assignment based on monitoring data associated with the plurality of topics, use the workload coordinator, generate and store, in a data store, a first dynamic assignment of a first processing task of the plurality of processing tasks to retrieve, from a first topic of the plurality of topics, and process a first data item, use the workload coordinator, generate and store, in the data store, a second dynamic assignment of a second processing task of the plurality of processing tasks to retrieve, from a second topic of the plurality of topics, and process a second data item, execute the first processing task to cause the first processing task to obtain the first dynamic assignment from the data store and, based on the first dynamic assignment, retrieve, from the first topic, and process the first data item according to the series of nodes and interconnections specified by the data processing pipeline, and execute the second processing task to cause the second processing task to obtain the second dynamic assignment from the data store and, based on the second dynamic assignment, retrieve, from the second topic, and process the second data item according to the series of nodes and interconnections specified by the data processing pipeline.
20 . The computing system of claim 19 , wherein the streaming data processing system is further configured to: determine that the first data item has been removed; based on determining that the first data item has been removed, deallocate the first processing task from the first topic; and implement an updated data processing pipeline on the streaming data processing system, wherein to implement the updated data processing pipeline on the streaming data processing system, the streaming data processing system is further configured to: identify a third topic from which to obtain a third data item for processing by the data processing pipeline, and using the workload coordinator, generate and store, in the data store, a third dynamic assignment of the first processing task to retrieve, from the third topic, and process the third data item.