Non-partitioned field query optimization in data lakes

1. A method, comprising:

generating a mapping between values of a non-partitioning field and containing Parquet files of a data lake, the mapping correlates unique values of the non-partitioning field with specific Parquet files in which the unique values are stored, and the containing Parquet files including the values of the non-partitioning field and the data lake being organized into partitions and Parquet files, wherein generating the mapping between the values of the non-partitioning field and the Parquet files comprises:

for each Parquet file of the Parquet files of the data lake, identifying whether the each Parquet file includes the non-partitioning field as a column;

receiving a query of the data lake that includes the non-partitioning field in a filter;

identifying, based on the mapping, a subset of the containing Parquet files corresponding to the filter;

rewriting the query into a rewritten query that includes filters that are based on the subset of the containing Parquet files, the rewritten query specifies the containing Parquet files; and

transmitting or displaying results of executing the rewritten query.

2. The method of claim 1 , wherein generating the mapping between the values of the non-partitioning field and the containing Parquet files of the data lake comprises:

generating the mapping in response to receiving an initial query of the data lake that includes the non-partitioning field as a filter field.

3. The method of claim 1 , further comprising:

caching the mapping in a memory.

4. The method of claim 1 , wherein generating the mapping between the values of the non-partitioning field and the Parquet files comprises:

performing a Structured Query Language (SQL) query to generate the mapping.

5. The method of claim 1 , wherein the filter comprises a range of the values of the non-partitioning field.

6. The method of claim 1 , wherein the filter comprises a pattern of the values of the non-partitioning field.

7. The method of claim 1 , wherein the filter comprises an equality condition.

8. The method of claim 1 , wherein receiving the query of the data lake that includes the non-partitioning field as the filter comprises:

determining that a filter field included in the query is a non-partitioning field based on a directory structure of the data lake, wherein the filter field is the non-partitioning field.

9. The method of claim 1 , wherein receiving the query of the data lake that includes the non-partitioning field as the filter comprises:

determining that a filter field included in the query is a non-partitioning field in response to determining that an index is not associated with the non-partitioning field.

10. A system, comprising:

one or more memories; and

one or more processors, the one or more processors configured to execute instructions stored in the one or more memories to:

generate a mapping between values of a non-partitioning field and containing Parquet files of a data lake, the mapping correlates unique values of the non-partitioning field with specific Parquet files in which the unique values are stored, and the containing Parquet files including the values of the non-partitioning field and the data lake being organized into partitions and Parquet files, wherein to generate the mapping between the values of the non-partitioning field and the Parquet files comprises to:

for each Parquet file of the Parquet files of the data lake, identify whether the each Parquet file includes the non-partitioning field as a column;

receive a query of the data lake that includes the non-partitioning field in a filter;

identify, based on the mapping, a subset of the containing Parquet files corresponding to the filter;

rewrite the query into a rewritten query that includes filters that are based on the subset of the containing Parquet files, the rewritten query specifies the containing Parquet files; and

transmit or display results of executing the rewritten query.

11. The system of claim 10 , wherein the mapping is generated in response to receiving an initial query that is different from the query and that includes the non-partitioning field as a filter field.

12. The system of claim 10 , wherein the mapping is generated in response to receiving the query.

13. The system of claim 10 , wherein the one or more processors is further configured to execute instructions in the one or more memories to:

determine that the filter comprises a range of the values of the non-partitioning field.

14. The system of claim 10 , wherein the one or more processors is further configured to execute instructions in the one or more memories to:

determine that the filter comprises a pattern of the values of the non-partitioning field.

15. The system of claim 10 , wherein the one or more processors is further configured to execute instructions in the one or more memories to:

determine that the filter comprises an equality condition on a value of the non-partitioning field.

16. The system of claim 10 , wherein the mapping includes rows and wherein each row includes a respective value of the non-partitioning field and a respective corresponding Parquet file.

17. A non-transitory computer readable medium storing instructions operable to cause one or more processors to perform operations comprising:

generating a mapping between values of a non-partitioning field and containing Parquet files of a data lake, the mapping correlates unique values of the non-partitioning field with specific Parquet files in which the unique values are stored, and the containing Parquet files including the values of the non-partitioning field and the data lake being organized into partitions and Parquet files, wherein to generating the mapping between the values of the non-partitioning field and the Parquet files comprises:

for each Parquet file of the Parquet files of the data lake, identifying whether the each Parquet file includes the non-partitioning field as a column;

receiving a query of the data lake that includes the non-partitioning field in a filter;

identifying, based on the mapping, a subset of the containing Parquet files corresponding to the filter;

rewriting the query into a rewritten query that includes filters that are based on the subset of the containing Parquet files, the rewritten query specifies the containing Parquet files; and

transmitting or displaying results of executing the rewritten query.

18. The non-transitory computer readable medium of claim 17 , wherein the mapping is generated in response to receiving the query.

19. The non-transitory computer readable medium of claim 17 , wherein the operations further comprising:

transmitting the rewritten query to a query execution engine.

20. The non-transitory computer readable medium of claim 17 , wherein the operations further comprise:

generating the mapping in response to receiving an initial query that is different from the query and that includes the non-partitioning field as a filter field.