Data partitioning and parallelism in a distributed event processing system

1. A method comprising:

determining a stage for a continuous query language (CQL) query being processed by an event processing system;

determining a stage type associated with the stage;

determining a transformation to be computed for the stage based at least in part on the stage type;

determining a classification for the CQL query based at least in part on a plurality of rules;

in accordance with a determination that the classification for the CQL query is stateless, marking the stage as a partitioned stage;

in accordance with a determination that the classification for the CQL query is stateful, marking the stage as a non-partitioned stage;

in response to marking the stage as a partitioned stage:

generating a first transformation in a first Directed Acyclic Graph (DAG) of a data transformation pipeline for the partitioned stage without a repartitioning transformation; and

in response to marking the stage as a non-partitioned stage:

generating a second transformation for the non-partitioned stage in a second Directed Acyclic Graph with a repartitioning transformation followed by a stage transformation; and

inputting a partitioning attribute and a number of partitions into the repartitioning transformation.

2. The method of claim 1 , wherein determining the transformation to be computed for the stage comprises parsing the CQL query into one or more tokens.

3. The method of claim 1 , wherein the plurality of rules comprise rules to classify the CQL query into at least one of a stateless query, a semi stateful query, or a fully stateful query.

4. The method of claim 1 , further comprising, responsive to marking the stage as a partitioned stage or a non-partitioned stage, applying partitioning criteria to the stage, wherein the partitioning criteria comprises one or more partitioning attributes.

5. The method of claim 4 , further comprising performing a semantic analysis of the CQL query.

6. The method of claim 5 , wherein the one or more partitioning attributes are determined based at least in part on the semantic analysis of the CQL query.

7. The method of claim 1 , further comprising traversing the data transformation pipeline from a source of the data transformation pipeline to a sink of the data transformation pipeline.

8. The method of claim 1 , wherein in response to marking the stage as a partitioned stage, determining a number of partitions for the stage based at least in part on the number of partitions from a previous stage.

9. A system, comprising:

a memory configured to store computer-executable instructions; and

a processor configured to access the memory and execute the computer-executable instructions to:

determine a stage for a continuous query language (CQL) query being processed by an event processing system;

determine a stage type associated with the stage;

determine a transformation to be computed for the stage based at least in part on the stage type;

determine a classification for the CQL query based at least in part on a plurality of rules;

in accordance with a determination that the classification for the CQL query is stateless, mark the stage as a partitioned stage;

in accordance with a determination that the classification for the CQL query is stateful, mark the stage as a non-partitioned stage;

in response to marking the stage as a partitioned stage:

determine a first transformation in a first Directed Acyclic Graph (DAG) of a data transformation pipeline for the partitioned stage without a repartitioning transformation; and

in response to marking the stage as a non-partitioned stage:

determine a second transformation for the non-partitioned stage in a second Directed Acyclic Graph with a repartitioning transformation followed by a stage transformation; and

input a partitioning attribute and a number of partitions into the repartitioning transformation.

10. The system of claim 9 , wherein determining the transformation to be computed for the stage comprises parsing the CQL query into one or more tokens.

11. The system of claim 9 , wherein the plurality of rules comprise rules to classify the CQL query into at least one of a stateless query, a semi stateful query, or a fully stateful query.

12. The system of claim 9 , further comprising, responsive to marking the stage as a partitioned stage or a non-partitioned stage, applying partitioning criteria to the stage, wherein the partitioning criteria comprises one or more partitioning attributes.

13. The system of claim 12 , wherein the computer-executable instructions are further executed to perform a semantic analysis of the CQL query.

14. The system of claim 13 , wherein the one or more partitioning attributes are determined based at least in part on the semantic analysis of the CQL query.

15. The system of claim 9 , wherein the computer-executable instructions are further executed to traverse the data transformation pipeline from a source of the data transformation pipeline to a sink of the data transformation pipeline.

16. A non-transitory computer-readable medium storing computer-executable code that, when executed by a processor, cause the processor to perform operations comprising:

determining a stage for a continuous query language (CQL) query being processed by an event processing system;

determining a stage type associated with the stage;

determining a transformation to be computed for the stage based at least in part on the stage type;

determining a classification for the CQL query based at least in part on a plurality of rules;

in accordance with a determination that the classification for the CQL query is stateless, marking the stage as a partitioned stage;

in accordance with a determination that the classification for the CQL query is stateful, marking the stage as a non-partitioned stage;

in response to marking the stage as a partitioned stage:

generating a first transformation in a Directed Acyclic Graph (DAG) of a data transformation pipeline for the partitioned stage without a repartitioning transformation; and

in response to marking the stage as a non-partitioned stage:

generating a second transformation for the non-partitioned stage in a second Directed Acyclic Graph with a repartitioning transformation followed by a stage transformation; and

inputting a partitioning attribute and a number of partitions into the repartitioning transformation.

17. The non-transitory computer-readable medium of claim 16 , wherein determining the transformation to be computed for the stage comprises parsing the CQL query into one or more tokens.

18. The non-transitory computer-readable medium of claim 16 , wherein the plurality of rules comprise rules to classify the CQL query into at least one of a stateless query, a semi stateful query, or a fully stateful query.

19. The non-transitory computer-readable medium of claim 16 , further comprising, responsive to marking the stage as a partitioned stage or a non-partitioned stage, applying partitioning criteria to the stage, wherein the partitioning criteria comprises one or more partitioning attributes.

20. The non-transitory computer-readable medium of claim 19 , wherein the operations further comprise performing a semantic analysis of the CQL query, and wherein the one or more partitioning attributes are determined based at least in part on the semantic analysis of the CQL query.