Query modification using partitioned datasets
Systems and methods are disclosed for modifying a query using partitioned datasets. A query system may receive a query that includes a data field identifier and identifies a set of data to be processed. The system may use the query to identify a partitioned dataset that is associated with the query and identify partitions of the partitioned dataset that include a data field that satisfies the data field identifier. The query system may use the identified partitions to modified the query to include filter criteria that includes a partition identifier for the identified partitions.
1 . A method, comprising:
receiving a query for execution;
identifying, in the query, a data source identifier, the data source identifier corresponding to a data source that includes a set of data to be processed;
identifying, in the query, a data field identifier associated with a command, wherein the command indicates a manner of processing at least a portion of the set of data;
identifying a partitioned dataset associated with the data source identified in the query;
identifying, using a metadata catalog, a set of partitions of the partitioned dataset that include a data field that satisfies the data field identifier in the query;
modifying the query to include, as filter criteria, a partition identifier for each of the set of partitions that include the data field represented by the data field identifier in the query; and
executing the modified query.
2 . The method of claim 1 , wherein the partitioned dataset includes a plurality of partitions, wherein each of the plurality of partitions includes data records that satisfy partition criteria for the respective partition, and wherein the set of partitions is a subset of the plurality of partitions.
3 . The method of claim 2 , wherein the partition criteria for a particular partition of the plurality of partitions includes one or more field values.
4 . The method of claim 2 , wherein the metadata catalog includes a plurality of partitioned dataset records corresponding to a plurality of partitioned datasets, wherein the plurality of partitioned datasets includes the partitioned dataset, wherein a particular partitioned dataset record corresponding to the partitioned dataset identifies the plurality of partitions of the partitioned dataset including the set of partitions.
5 . The method of claim 1 , wherein the data field identifier is a first data field identifier and the command is a first command, the method further comprising:
identifying, in the query, a second data field identifier associated with a second command;
wherein identifying, using the metadata catalog, the set of partitions of the partitioned dataset that include the data field that satisfies the data field identifier comprises identifying, using the metadata catalog, the set of partitions of the partitioned dataset that include a first data field that satisfies the first field identifier and a second data field that satisfies the second field identifier.
6 . The method of claim 1 , wherein identifying the partitioned dataset associated with the data source comprises analyzing a data source record of the metadata catalog, wherein the data source record includes a partitioned dataset identifier indicating that the data source includes at least a portion of the partitioned dataset.
7 . The method of claim 6 , wherein identifying the set of partitions of the partitioned dataset that include the data field that satisfies the data field identifier comprises analyzing a partitioned dataset record of the metadata catalog, wherein the partitioned dataset record identifies field identifiers associated with a plurality of partitions of the partitioned dataset, wherein the field identifiers indicate how data is assigned to one of the plurality of partitions.
8 . The method of claim 1 , wherein the partitioned dataset includes a plurality of partitions, wherein data of the partitioned dataset is assigned to one of the plurality of partitions at ingest.
9 . The method of claim 1 , wherein the partitioned dataset includes a plurality of partitions, wherein criteria for assigning data to one of the plurality of partitions is selected by a user.
10 . A system, comprising:
a data store; and
one or more processors configured to:
receive a query for execution;
identify, in the query, a data source identifier, the data source identifier corresponding to a data source that includes a set of data to be processed;
identify, in the query, a data field identifier associated with a command, wherein the command indicates a manner of processing at least a portion of the set of data;
identify a partitioned dataset associated with the data source identified in the query;
identify, using a metadata catalog, a set of partitions of the partitioned dataset that include a data field that satisfies the data field identifier in the query;
modify the query to include, as filter criteria, a partition identifier for each of the set of partitions that include the data field represented by the data field identifier in the query; and
execute the modified query.
11 . The system of claim 10 , wherein the partitioned dataset includes a plurality of partitions, wherein each of the plurality of partitions includes data records that satisfy partition criteria for the respective partition, and wherein the set of partitions is a subset of the plurality of partitions.
12 . The system of claim 11 , wherein the partition criteria for a particular partition of the plurality of partitions includes one or more field values.
13 . The system of claim 11 , wherein the metadata catalog includes a plurality of partitioned dataset records corresponding to a plurality of partitioned datasets, wherein the plurality of partitioned datasets includes the partitioned dataset, wherein a particular partitioned dataset record corresponding to the partitioned dataset identifies the plurality of partitions of the partitioned dataset including the set of partitions.
14 . The system of claim 10 , wherein the data field identifier is a first data field identifier and the command is a first command, wherein the one or more processors are further configured to:
identify, in the query, a second data field identifier associated with a second command;
wherein to identify, using the metadata catalog, the set of partitions of the partitioned dataset that include the data field that satisfies the data field identifier the one or more processors are configured to identify, using the metadata catalog, the set of partitions of the partitioned dataset that include a first data field that satisfies the first field identifier and a second data field that satisfies the second field identifier.
15 . The system of claim 10 , wherein to identify the partitioned dataset associated with the data source, the one or more processors are configured to analyze a data source record of the metadata catalog, wherein the data source record includes a partitioned dataset identifier indicating that the data source includes at least a portion of the partitioned dataset.
16 . The system of claim 15 , wherein to identify the set of partitions of the partitioned dataset that include the data field that satisfies the data field identifier, the one or more processors are configured to analyze a partitioned dataset record of the metadata catalog, wherein the partitioned dataset record identifies field identifiers associated with a plurality of partitions of the partitioned dataset, wherein the field identifiers indicate how data is assigned to one of the plurality of partitions.
17 . The system of claim 10 , wherein the partitioned dataset includes a plurality of partitions, wherein data of the partitioned dataset is assigned to one of the plurality of partitions at ingest.
18 . The system of claim 10 , wherein the partitioned dataset includes a plurality of partitions, wherein criteria for assigning data to one of the plurality of partitions is selected by a user.
19 . Non-transitory computer-readable media including computer-executable instructions that, when executed by a computing system, cause the computing system to:
receive a query for execution;
identify, in the query, a data source identifier, the data source identifier corresponding to a data source that includes a set of data to be processed;
identify, in the query, a data field identifier associated with a command, wherein the command indicates a manner of processing at least a portion of the set of data;
identify a partitioned dataset associated with the data source identified in the query;
identify, using a metadata catalog, a set of partitions of the partitioned dataset that include a data field that satisfies the data field identifier in the query;
modify the query to include, as filter criteria, a partition identifier for each of the set of partitions that include the data field represented by the data field identifier in the query; and
execute the modified query.
20 . The non-transitory computer-readable media of claim 19 , wherein to identify the set of partitions of the partitioned dataset that include the data field that satisfies the data field identifier, the computer-executable instructions cause the computing system to analyze a partitioned dataset record of the metadata catalog, wherein the partitioned dataset record identifies field identifiers associated with a plurality of partitions of the partitioned dataset, wherein the field identifiers indicate how data is assigned to one of the plurality of partitions.