Serverless data lake indexing subsystem and application programming interface
Methods, systems and computer program products are described herein that provide a serverless, multi-engine, multi-user data lake indexing subsystem and application programming interface. Indexes are defined as derived datasets and stored on the data lake in a universal format that enables disparate engines to create and/or discover indexes for workload optimization. Embodiment of indexes enable stateful control and management of an index via metadata included in the index and stored on the data lake.
1 . A query processing workload optimization system comprising:
a processor circuit; and
memory accessible to the processor circuit, the memory storing program code for execution by the processor circuit, the program code including:
a query processor that:
generates a query plan for a query;
searches a data lake for built indexes stored at a predetermined location;
determines, based on index metadata for found built indexes, a plurality of compatible index pairs of the found built indexes;
selects a compatible index pair from the plurality of compatible index pairs based on the selected compatible index pair having a largest number of partitions;
optimizes the query plan to use the selected compatible index pair; and
executes the optimized query plan to provide a query result.
2 . The query processing workload optimization system of claim 1 , wherein to determine the plurality of compatible index pairs of the found built indexes the query processor:
searches the query plan for a pattern that matches a pattern of an indexing rule; and
determines the plurality of compatible index pairs meet a condition of the matched pattern of the query plan.
3 . The query processing workload optimization system of claim 2 , wherein the matched pattern of the query plan corresponds to a join operator, and the query processor further replaces the matched pattern with a join operator corresponding to the selected compatible index pair.
4 . The query processing workload optimization system of claim 2 , wherein the matched pattern of the query plan corresponds to a filter operator, and the query processor further replaces the matched pattern with a filter operator corresponding to the selected compatible index pair.
5 . The query processing workload optimization system of claim 1 , wherein the index metadata for the found built indexes comprises a creating state descriptor.
6 . The query processing workload optimization system of claim 1 , wherein the partitions of the selected compatible index pair are partitioned using a hash function.
7 . The query processing workload optimization system of claim 1 , wherein the selected index pair comprises an arbitrary uniform resource identifier (URI) that refers to a chunk of data stored in the data lake, and to optimize the query plan the query processor:
prunes blocks for the query plan based on the arbitrary URI.
8 . A method comprising:
generating a query plan for a query;
searching a data lake for built indexes stored at a predetermined location;
determining, based on index metadata for found built indexes, a plurality of compatible index pairs of the found built indexes;
selecting a compatible index pair from the plurality of compatible index pairs, the selected compatible index pair comprising an arbitrary uniform resource identifier (URI) that refers to a chunk of data stored in the data lake;
optimizing the query plan to use the selected compatible index pair by pruning blocks for the query plan based on the arbitrary URI; and
executing the optimized query plan to provide a query result.
9 . The method of claim 8 , wherein said determining the plurality of compatible index pairs of the found built indexes comprises:
searching the query plan for a pattern that matches a pattern of an indexing rule; and
determining the plurality of compatible index pairs meet a condition of the matched pattern of the query plan.
10 . The method of claim 9 , wherein the matched pattern of the query plan corresponds to a join operator, and the method further comprises:
replacing the matched pattern with a join operator corresponding to the selected compatible index pair.
11 . The method of claim 9 , wherein the matched pattern of the query plan corresponds to a filter operator, and the method further comprises:
replacing the matched pattern with a filter operator corresponding to the selected compatible index pair.
12 . The method of claim 8 , wherein the index metadata for the found built indexes comprises a deleted state descriptor.
13 . The method of claim 8 , wherein partitions of the selected compatible index pair are partitioned using a hash function.
14 . The method of claim 8 , wherein said selecting the compatible index pair comprises:
arbitrarily selecting the compatible index pair from the plurality of compatible index pairs.
15 . The method of claim 8 , wherein said selecting the compatible index pair comprises:
selecting the compatible index pair from the plurality of compatible index pairs based on the selected compatible index pair having a number of portions larger than another compatible index pair.
16 . A system comprising:
a data lake storing datasets and indexes based on the datasets, the datasets comprising a chunk of data;
a processor circuit coupled to the data lake; and
memory that stores program code structured to cause the processor circuit to:
generate a query plan for a query;
search the data lake for built indexes stored at a predetermined location;
determine, based on index metadata for found built indexes, a plurality of compatible index pairs of the found built indexes;
select a compatible index pair from the plurality of compatible index pairs, the selected compatible index pair comprising an arbitrary uniform resource identifier (URI) that refers to the chunk;
optimize the query plan to use the selected compatible index pair by pruning blocks for the query plan based on the arbitrary URI; and
execute the optimized query plan to provide a query result.
17 . The system of claim 16 , wherein to determine the plurality of compatible index pairs of the found built indexes, the program code is structured to cause the processor to:
search the query plan for a pattern that matches a pattern of an indexing rule; and
determine the plurality of compatible index pairs meet a condition of the matched pattern of the query plan.
18 . The system of claim 17 , wherein the matched pattern of the query plan corresponds to a join operator, and the program code is further structured to cause the processor to:
replace the matched pattern with a join operator corresponding to the selected compatible index pair.
19 . The system of claim 17 , wherein the matched pattern of the query plan corresponds to a filter operator, and the program code is further structured to cause the processor to:
replace the matched pattern with a filter operator corresponding to the selected compatible index pair.
20 . The system of claim 16 , wherein the index metadata for the found built indexes comprises a creating state descriptor.