Data stream processing instruction set previews using batch conversion
Systems and methods are described for providing previews of deployment of data stream processing instructions sets, sometimes called pipelines, to a stream data processing system. Rather than deploying such an instruction set, which may cause detrimental side effects, previews can be facilitated by conversion of a data stream processing instructions set to a batch query that is applied to an existing data set. An output of the batch query can then be provided to an end user as a preview of output of the data stream processing instructions set, when implemented.
1 . A computer-implemented method comprising:
accessing a pipeline, the pipeline comprising an instruction set indicating a data source and one or more data manipulations for processing data from a streaming data input to generate a streaming output;
executing the pipeline to generate a preview by at least:
converting the pipeline, comprising the instruction set, into a batch query at least by generating the batch query to process data from an existing data set in accordance with the one or more manipulations for processing included in the pipeline, wherein the existing data set is distinct from the streaming data input; and
applying the batch query, representing conversion of the pipeline, to the existing data set to generate a query results set; and
outputting the query results set generated from applying the batch query, representing the conversion of the pipeline, to the existing data set as the preview of the results of the pipeline when applied to the streaming data input.
2 . The computer-implemented method of claim 1 , further comprising accessing a request to preview results of the pipeline.
3 . The computer-implemented method of claim 2 , wherein the streaming output is designated by a streaming output variable, and wherein accessing the request to preview results of the pipeline comprises accessing a request to deploy the pipeline with the streaming output variable unbound.
4 . The computer-implemented method of claim 2 , wherein the request to preview results of the pipeline specifies the existing data set.
5 . The computer-implemented method of claim 2 further comprising, when the request to preview results of the pipeline does not specify the existing data set, selecting the existing data set using an association between the existing data set and the streaming data input.
6 . The computer-implemented method of claim 1 further comprising, selecting the existing data set using an association between the existing data set and the streaming data input.
7 . The computer-implemented method of claim 1 , wherein the existing data set comprises data previously read from the streaming data input.
8 . The computer-implemented method of claim 1 , wherein the pipeline and the batch query are specified in a shared query language.
9 . The computer-implemented method of claim 1 , wherein the pipeline and the batch query are specified in a shared query language, wherein a final command of the pipeline specifies the streaming output, and wherein converting the pipeline into the batch query comprises removing the final command of the pipeline and saving a result as the batch query.
10 . The computer-implemented method of claim 1 , wherein the pipeline and the batch query are specified in a common query language, and wherein converting the pipeline into the batch query comprises replacing a reference to the streaming data input in the pipeline with a reference to the existing data set.
11 . The computer-implemented method of claim 1 , wherein the pipeline further writes data to a second streaming output, wherein converting the pipeline into the batch query comprises converting the pipeline into at least two batch queries comprising a first batch query corresponding to the streaming output and a second batch query corresponding to the second streaming output, and wherein outputting the query results set as preview results of the pipeline comprises outputting query results corresponding to a combination of results from the first batch query and results from the second batch query.
12 . The computer-implemented method of claim 1 further comprising:
obtaining specification of a second pipeline, the second pipeline comprising a specification to write data to the streaming output; and
obtaining a request to preview results of the second pipeline;
wherein converting the pipeline into the batch query comprises converting both the pipeline and the second data pipeline into a single batch query; and
wherein outputting the query results set as preview results of the pipeline comprises outputting the query results as preview results of both the pipeline and the second pipeline.
13 . The computer-implemented method of claim 1 , wherein the method is implemented without reading data from the streaming data input.
14 . The computer-implemented method of claim 1 , wherein the method is implemented without writing data to the streaming data output.
15 . The computer-implemented method of claim 1 , wherein the streaming output is identified in the pipeline with an output identifier, and wherein applying the batch query to the existing data set to generate the query results set comprises associating the query results set with the output identifier.
16 . The computer-implemented method of claim 15 , wherein outputting the query results set as preview results of the pipeline is responsive to a request to read from the output identifier.
17 . A system comprising:
a processor; and
a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to:
access a pipeline, the pipeline comprising an instruction set indicating a data source and one or more data manipulations for processing data from a streaming data input to generate a streaming output;
execute the pipeline to generate a preview by at least:
converting the pipeline, comprising the instruction set, into a batch query at least by generating the batch query to process data from an existing data set in accordance with the one or more manipulations for processing included in the pipeline, wherein the existing data set is distinct from the streaming data input; and
applying the batch query, representing conversion of the pipeline, to the existing data set to generate a query results set; and
output the query results set generated from applying the batch query, representing conversion of the pipeline, to the existing data set as the preview of the results of the pipeline when applied to the streaming data input.
18 . The system of claim 17 , wherein the pipeline and the batch query are specified in a common query language, and wherein to convert the pipeline into the batch query, the instructions cause the processor to replace a reference to the streaming data input in the pipeline with a reference to the existing data set.
19 . One or more non-transitory computer-readable media having stored thereon instructions that, when executed by a computing system including one or more processors, cause the computing system to:
access a pipeline, the pipeline comprising an instruction set indicating a data source and one or more data manipulations for processing data from a streaming data input to generate a streaming output;
execute the pipeline to generate a preview by at least:
converting the pipeline, comprising the instruction set, into a batch query at least by generating the batch query to process data from an existing data set in accordance with the one or more manipulations for processing included in the pipeline, wherein the existing data set is distinct from the streaming data input; and
applying the batch query, representing conversion of the pipeline, to the existing data set to generate a query results set; and
output the query results set generated from applying the batch query, representing conversion of the pipeline, to the existing data set as the preview of the results of the pipeline when applied to the streaming data input.
20 . The one or more non-transitory computer-readable claim 19 , wherein the pipeline and the batch query are specified in a common query language, and wherein to convert the pipeline into the batch query, the instructions cause the computing system to replace a reference to the streaming data input in the pipeline with a reference to the existing data set.