Utilizing metadata to prune a data set
A query directed to database data stored across a set of files is received. The query includes predicates and each file from the set of files is associated with metadata stored in a metadata store that is separate from a storage platform that stores the set of files. One or more files are removed from the set of files whose metadata does not satisfy a predicate of the plurality of predicates to generate a pruned set of files. One or more predicates are removed that are satisfied by the metadata of the pruned set of files to generate a modified query.
1. A method comprising:
receiving a query directed to database data stored across a set of files, the query comprising a plurality of predicates, wherein each file from the set of files is associated with metadata stored in a metadata store that is separate from a storage platform that stores the set of files;
removing, by a hardware resource, from the set of files, one or more files whose metadata does not satisfy a predicate of the plurality of predicates to generate a pruned set of files; and
removing, from the plurality of predicates, one or more predicates that are satisfied by the metadata of the pruned set of files to generate a modified query.
2. The method of claim 1 , further comprising:
executing the modified query against the pruned set of files to create a second set of files.
3. The method of claim 2 , further comprising:
returning the second set of files in response to the query.
4. The method of claim 1 , wherein removing from the set of files, each file whose metadata does not satisfy the predicate further comprises:
determining that the metadata for each file in the set of files does not include direct information specified by a given predicate of the plurality of predicates.
5. The method of claim 4 , wherein removing from the set of files, each file whose metadata does not satisfy the predicate further comprises:
determining, for each file in the set of files, direct information using function-specific value range propagation; and
removing from the set of files, each file whose direct information does not satisfy the given predicate of the plurality of predicates.
6. The method of claim 1 , wherein each file of the set of files comprises a block of the database data.
7. The method of claim 1 , wherein the metadata for each file of the set of files comprises at least one of:
representative values of a block of the database data of the file;
a number of null values in the block of the database data of the file; or
maximum and minimum values of the block of the database data of the file.
8. The method of claim 1 , further comprising:
loading the database data into the storage platform, wherein the metadata for each file in the set of files is obtained from the database data during the loading.
9. The method of claim 1 , wherein one or more of the plurality of predicates each comprise a set of complex expressions.
10. The method of claim 1 , wherein generating the modified query further comprises:
removing from the plurality of predicates, each predicate that is not satisfied by the metadata of any of the pruned set of files.
11. A system comprising:
a hardware resource to:
receive a query directed to database data stored across a set of files, the query comprising a plurality of predicates, wherein each file from the set of files is associated with metadata stored in a metadata store that is separate from a storage platform that stores the set of files;
remove, from the set of files, one or more files whose metadata does not satisfy a predicate of the plurality of predicates to generate a pruned set of files; and
remove, from the plurality of predicates, one or more predicates that are satisfied by the metadata of the pruned set of files to generate a modified query.
12. The system of claim 11 , wherein the hardware resource is further to:
execute the modified query against the pruned set of files to create a second set of files.
13. The system of claim 12 , wherein the hardware resource is further to:
return the second set of files in response to the query.
14. The system of claim 11 , wherein to remove from the set of files, each file whose metadata does not satisfy the predicate, the hardware resource is further to:
determine that the metadata for each file in the set of files does not include direct information specified by a given predicate of the plurality of predicates.
15. The system of claim 14 , wherein to remove from the set of files, each file whose metadata does not satisfy the predicate, the hardware resource is further to:
determine, for each file in the set of files, direct information using function-specific value range propagation; and
remove from the set of files, each file whose direct information does not satisfy the given predicate of the plurality of predicates.
16. The system of claim 11 , wherein each file of the set of files comprises a block of the database data.
17. The system of claim 11 , wherein the metadata for each file of the set of files comprises at least one of:
representative values of a block of the database data of the file;
a number of null values in the block of the database data of the file; or
maximum and minimum values of the block of the database data of the file.
18. The system of claim 11 , wherein the hardware resource is further to:
load the database data into the storage platform, wherein the metadata for each file in the set of files is obtained from the database data during the loading.
19. The system of claim 11 , wherein one or more of the plurality of predicates each comprise a set of complex expressions.
20. The system of claim 11 , wherein to generate the modified query, the hardware resource is further to:
remove, from the plurality of predicates, each predicate that is not satisfied by the metadata of any of the pruned set of files.
21. A non-transitory computer-readable medium having instructions stored thereon that, when executed by a hardware resource, cause the hardware resource to:
receive a query directed to database data stored across a set of files, the query comprising a plurality of predicates, wherein each file from the set of files is associated with metadata stored in a metadata store that is separate from a storage platform that stores the set of files;
remove, by the hardware resource, from the set of files, one or more files whose metadata does not satisfy a predicate of the plurality of predicates to generate a pruned set of files; and
remove, from the plurality of predicates, one or more predicates that are satisfied by the metadata of the pruned set of files to generate a modified query.
22. The non-transitory computer-readable medium of claim 21 , wherein the metadata for each file of the set of files comprises at least one of:
representative values of a block of the database data of the file;
a number of null values in the block of the database data of the file; or
maximum and minimum values of the block of the database data of the file.