Serverless data lake indexing subsystem and application programming interface

1. A system comprising:

one or more processors; and

one or more memory devices accessible to the one or more processors, the one or more memory devices storing program code for execution by the one or more processors, the program code including:

a first query engine configured to be coupled to a data lake, the data lake configured to store datasets and indexes based on the datasets on the data lake, the first query engine further configured to:

receive a workload comprising a plurality of queries directed at the datasets;

generate a set of candidate indexes based on the plurality of queries;

select a predetermined number of candidate indexes from the set of candidate indexes based on a determination of an estimated performance improvement provided by the set of candidate indexes to performance of the workload; and

build an index of the selected candidate indexes according to an index specification and store the built index at a predetermined location on the data lake, the predetermined location and index metadata included in the built index conforming to the index specification, the index metadata describing a lineage of the built index that is usable by another query engine to determine whether a data source corresponding to the built index has been updated.

2. The system of claim 1 wherein the index metadata further describes one or more of:

contents of the built index;

or

state of the built index.

3. The system of claim 2 wherein the index metadata that describes the contents of the built index comprises one or more of:

a name of the built index;

a type of the built index;

a configuration of the built index including identification of indexed and included columns and types of each; or

a physical location and layout of the built index.

4. The system of claim 2 wherein the index metadata that describes the lineage of the built index comprises one or more of:

one or more identifiers each corresponding to one or more data sources being indexed;

a time at which the data source was indexed; or

a descriptive history of the built index.

5. The system of claim 4 wherein the index metadata that describes the lineage of the built index further comprises query plan information for an original query used to create the built index.

6. The system of claim 5 further comprising a second query engine different than the first query engine, the second query engine configured to:

search the data lake for built indexes stored at the predetermined location;

receive a query;

determine based on the index metadata for each built index found by the search, whether the respective built index may be used to optimize a query plan for executing the query and if so, optimizing the query plan to use the respective built index; and

execute the optimized query plan to provide a query result.

7. The system of claim 6 wherein at least one of the first query engine or the second query engine are further configured to determine, based on the index metadata describing the lineage of the built index, whether the data source has been updated since the data source corresponding to the built index was indexed and if so, to rebuild the index using the query plan information.

8. The system of claim 2 wherein the index metadata that describes the state of the built index comprises one or more state descriptors from the set of: enabled, disabled, creating or deleted.

9. A query processing workload optimization system comprising:

a first query engine comprising:

one or more processors; and

one or more memory devices accessible to the one or more processors, the one or more memory devices storing program code for execution by the one or more processors, the program code including:

a candidate index generator configured to generate a set of candidate indexes based on a plurality of queries;

an index selector configured to select a predetermined number of candidate indexes from the set of candidate indexes based on a determination of an estimated performance improvement provided by the set of candidate indexes to the performance of the workload; and

a first query processor configured to build an index of the selected candidate indexes according to an index specification and store the built index at a predetermined location on a data lake, the predetermined location and index metadata included in the built index conforming to the index specification; and

a second query engine comprising:

one or more second processors; and

one or more second memory devices accessible to the one or more second processors, the one or more second memory devices storing second program code for execution by the one or more second processors, the second program code including:

a second query processor configured to:

receive a query;

generate a query plan for the query;

search the data lake for built indexes stored at the predetermined location,

determine based on the index metadata for each built index found, whether the respective built index may be used to optimize the query plan and if so, optimize the query plan to use the respective built index; and

execute the optimized query plan to provide a query result.

10. The query processing workload optimization system of claim 9 wherein the index metadata describes one or more of:

contents of the built index;

lineage of the built index; or

state of the built index.

11. The query processing workload optimization system of claim 10 wherein the index metadata that describes the contents of the built index comprises one or more of:

a name of the built index;

a type of the built index;

a configuration of the built index including identification of indexed and included columns and types of each; or

a physical location and layout of the built index.

12. The query processing workload optimization system of claim 10 wherein the index metadata that describes the lineage of the built index comprises one or more of:

one or more identifiers each corresponding to one or more data sources being indexed;

a time at which the data source was indexed; or

a descriptive history of the built index.

13. The query processing workload optimization system of claim 12 wherein the index metadata that describes the lineage of the built index further comprises query plan information for an original query used to create the built index.

14. The query processing workload optimization system of claim 13 wherein at least one of the first query engine or the second query engine are further configured to determine, based on the index metadata describing the lineage of the built index, whether the data source has been updated since the data source corresponding to the built index was indexed and if so, to rebuild the index using the query plan information.

15. The query processing workload optimization system of claim 10 wherein the index metadata that describes the state of the built index includes one or more state descriptors from the set of: enabled, disabled, creating or deleted.

16. A query processing workload optimization system comprising:

a data lake configured to store thereon datasets and indexes based on the datasets;

at least one processor configured to be coupled to the data lake; and

at least one memory that stores program code configured to be executed by the at least one processor to perform operations, the operations comprising:

receiving a workload comprising a plurality of queries directed at the datasets;

generating a set of candidate indexes based on the plurality of queries;

selecting a predetermined number of candidate indexes from the set of candidate indexes based on a determination of an estimated performance improvement provided by the set of candidate indexes to performance of the workload; and

building an index of the selected candidate indexes according to an index specification and store the built index at a predetermined location on the data lake, the predetermined location and index metadata included in the built index conforming to the index specification, the index metadata describing a lineage of the built index that is usable by a query engine to determine whether a data source corresponding to the built index has been updated.

17. A query processing workload optimization system of claim 16 wherein the index metadata further describes one or more of:

contents of the built index;

or

state of the built index.

18. The query processing workload optimization system of claim 17 wherein the index metadata that describes the lineage of the built index comprises on or more of:

one or more identifiers each corresponding to one or more data sources being indexed;

a time at which the data source was indexed;

a descriptive history of the built index; or

query plan information for an original query used to build the index.

19. The query processing workload optimization system of claim 18 wherein the operations further comprise determining, based on the index metadata that describes the lineage of the built index, whether the data source corresponding to the built index has been updated since the data source was indexed and if so, rebuilding the index using the query plan information.

20. The query processing workload optimization system of claim 17 wherein the index metadata that describes the contents of the built index comprises one or more of:

a name of the built index;

a type of the built index;

a configuration of the built index including identification of indexed and included columns and types of each; or

a physical location and layout of the built index.