Combining pipelines for a streaming data system
A technique includes comparing a first sequence of tasks of a first data processing pipeline of a plurality of data processing pipelines for a data streaming system to a second data processing pipeline of the plurality of data processing pipelines. The technique includes, based on the comparing, combining the first data processing pipeline and the second data processing pipeline into a single data processing pipeline.
1. A method comprising:
comparing a first sequence of tasks of a first data processing pipeline of a plurality of data processing pipelines for a data streaming system to a second sequence of tasks of a second data processing pipeline of the plurality of data processing pipelines; and
based on the comparing, combining the first data processing pipeline and the second data processing pipeline into a single data processing pipeline.
2. The method of claim 1 , wherein comparing the first sequence of tasks to the second sequence of tasks comprises:
determining a first graph representation for the first data processing pipeline based on metadata describing the first data processing pipeline;
determining a second graph representation for the second data processing pipeline based on metadata describing the second data processing pipeline; and
comparing the first graph to the second graph.
3. The method of claim 2 , wherein comparing the first graph to the second graph comprises determining whether the first and second graphs are isomorphic graphs, and combining the first data processing pipeline and the second data processing pipeline comprises combining the first data processing pipeline and the second data processing pipeline in response to determining that the first and second graphs are isomorphic graphs.
4. The method of claim 1 , further assigning a processing thread to a sequence of tasks representing the single data processing pipeline.
5. The method of claim 1 , further comprising receiving metadata describing the plurality of data processing pipelines, and comparing the data processing pipelines based on the metadata.
6. The method of claim 5 , wherein the metadata describes the first sequence of tasks of the first data processing pipeline and the second sequence of tasks of the second data processing pipeline.
7. The method of claim 6 , wherein:
the metadata describing the first data processing pipeline describes a first non-functional criterion associated with the first data processing pipeline;
the metadata describing the second data processing pipeline describes a second non-functional criterion associated with the second data processing pipeline; and
combining the first data processing pipeline and the second data processing pipeline comprises combining the first data processing pipeline and the second data processing pipeline based on the first non-functional criterion and the second non-functional criterion.
8. The method of claim 7 , wherein:
the first non-functional criterion comprises a first latency;
the second non-functional criterion comprises a second latency; and
combining the first data processing pipeline and the second data processing pipeline comprises combining the first data processing pipeline and the second data processing pipeline based on the first latency and the second latency.
9. An apparatus comprising:
a processor; and
a memory to store instructions that, when executed by the processor, cause the processor to:
receive data describing a plurality of data processing pipelines for a streaming system;
based on a graph analysis of the plurality of data processing pipelines derived from the data, combine a first data processing pipeline of the plurality of data processing pipelines with a second data processing pipeline of the plurality of data processing pipelines.
10. The apparatus of claim 9 , wherein the data describes tasks of the first data processing pipeline and a processing order of the tasks.
11. The apparatus of claim 9 , wherein the instructions when executed by the processor causes the processor to:
combine the first data processing pipeline with the second data processing pipeline to create a third data processing pipeline;
submit the third data processing pipeline to the streaming system;
remove the second data processing pipeline from the streaming system;
monitor a metric associated with third data processing pipeline; and
based on the metric:
submit the first data processing pipeline and the second data processing pipeline to the streaming system; and
remove the third data processing pipeline from the streaming system.
12. The apparatus of claim 11 , wherein the metric comprises a latency of the third data processing pipeline or a data input rate to the third data processing pipeline.
13. The apparatus of claim 11 , wherein the metric comprises a resource usage associated with the third data processing pipeline.
14. A non-transitory machine readable storage medium storing instructions that, when executed by a machine, cause the machine to:
read a first pipeline definition representing a first pipeline to be added to a streaming system;
compare the first pipeline definition with a second pipeline definition representing a second pipeline being executed by the streaming system;
based on the comparison, combine the first pipeline definition and the second pipeline definition to create a third pipeline definition representing a single pipeline; and
controlling the streaming system to replace the second pipeline with the single pipeline.
15. The non-transitory machine readable storage medium of claim 14 , wherein the instructions, when executed by the machine, cause the machine to combine the first pipeline definition with the second pipeline definition to create the single pipeline based on a first latency represented by the first pipeline definition and a second latency represented by the second pipeline definition.
16. The non-transitory machine readable storage medium of claim 14 , wherein the instructions, when executed by the machine, cause the machine to combine the first pipeline definition with the second pipeline definition to create the single pipeline based on a first criterion represented by the first pipeline definition and a second criterion represented by the second pipeline definition.
17. The non-transitory machine readable storage medium of claim 16 , wherein the first criterion comprises a latency and the second criterion comprises a latency.
18. The non-transitory machine readable storage medium of claim 16 , wherein the first criterion comprises a throughput and the second criterion comprises a throughput.
19. The non-transitory machine readable storage medium of claim 14 , wherein the instructions, when executed by the machine, cause the machine to combine the first pipeline definition with the second pipeline definition to create the single pipeline based on an available capacity of the streaming system.
20. The non-transitory machine readable storage medium of claim 14 , wherein the instructions, when executed by the machine, cause the machine to determine whether the first pipeline and the second pipeline are isomorphic graphs and combine the first pipeline and the second pipeline based on the determination.