Multi-range and runtime pruning
A system, apparatus, and method for processing queries wherein the query includes a request to access or delete data and accessing metadata associated with the set of data, the metadata defining data characteristics of the set of data and identifying at least sets of data that need or not need to be accessed or deleted based on the metadata without accessing the actual data in the set of data; also methods to optimize processing of some operations based on the collected metadata on data.
1. A method comprising:
receiving a query directed to database data stored across a set of files such that each file of the set of files comprises a block of the database data, wherein the query comprises a plurality of predicates and wherein each file from the set of files is associated with metadata stored in a metadata store that is separate from a storage platform that stores the set of files;
removing, by a processor, from the set of files, each file whose metadata does not satisfy a predicate of the plurality of predicates to generate a pruned set of files;
removing from the plurality of predicates, each predicate that is satisfied by the metadata of each file of the pruned set of files to generate a modified query; and
executing the modified query against the pruned set of files to create a second set of files.
2. The method of claim 1 , wherein removing from the set of files, each file whose metadata does not match a predicate further comprises:
determining that the metadata for each file in the set of files does not include direct information specified by a given predicate of the plurality of predicates.
3. The method of claim 2 , wherein removing from the set of files, each file whose metadata does not match a predicate further comprises:
determining, for each file in the set of files, the direct information using function-specific value range propagation; and
removing from the set of files, each file whose direct information does not satisfy the given predicate of the plurality of predicates.
4. The method of claim 1 , wherein the metadata for each file of the set of files comprises one or more of:
representative values of a block of the database data of the file;
a number of null values in the block of the database data of the file; and
maximum and minimum values of the block of the database data of the file.
5. The method of claim 1 , further comprising loading the database data into the storage platform, wherein the metadata for each file in the set of files is obtained from the database data during the loading.
6. The method of claim 1 , wherein one or more of the plurality of predicates each comprise a set of complex expressions.
7. The method of claim 1 , wherein generating the modified query further comprises removing from the plurality of predicates, each predicate that is not satisfied by the metadata of any of the pruned set of files.
8. The method of claim 1 , further comprising returning the second set of files in response to the query.
9. A system comprising:
a memory; and
a processor operatively coupled to the memory, the processor to:
receive a query directed to database data stored across a set of files such that each file of the set of files comprises a block of the database data, wherein the query comprises a plurality of predicates and wherein each file from the set of files is associated with metadata stored in a metadata store that is separate from a storage platform that stores the set of files;
remove from the set of files, each file whose metadata does not satisfy a predicate of the plurality of predicates to generate a pruned set of files;
remove from the plurality of predicates, each predicate that is satisfied by the metadata of each file of the pruned set of files to generate a modified query; and
execute the modified query against the pruned set of files to create a second set of files.
10. The system of claim 9 , wherein to remove from the set of files, each file whose metadata does not match a predicate, the processor is further to:
determine that the metadata for each file in the set of files does not include direct information specified by a given predicate of the plurality of predicates.
11. The system of claim 10 , wherein to remove from the set of files, each file whose metadata does not match a predicate, the processor is further to:
determine, for each file in the set of files, the direct information using function-specific value range propagation; and
remove from the set of files, each file whose direct information does not satisfy the given predicate of the plurality of predicates.
12. The system of claim 9 , wherein the metadata for each file of the set of files comprises one or more of:
representative values of a block of the database data of the file;
a number of null values in the block of the database data of the file; and
maximum and minimum values of the block of the database data of the file.
13. The system of claim 9 , wherein the processor is further to:
load the database data into the storage platform; and
obtain the metadata for each file in the set of files from the database data during the loading.
14. The system of claim 9 , wherein one or more of the plurality of predicates each comprise a set of complex expressions.
15. The system of claim 9 , wherein to generate the modified query, the processor is further to remove from the plurality of predicates, each predicate that is not satisfied by the metadata of any of the pruned set of files.
16. The system of claim 9 , wherein the processor is further to return the second set of files in response to the query.
17. A non-transitory computer-readable medium having instructions stored thereon that, when executed by a processor, cause the processor to:
receive a query directed to database data stored across a set of files such that each file of the set of files comprises a block of the database data, wherein the query comprises a plurality of predicates and wherein each file from the set of files is associated with metadata stored in a metadata store that is separate from a storage platform that stores the set of files;
remove, by the processor, from the set of files, each file whose metadata does not satisfy a predicate of the plurality of predicates to generate a pruned set of files;
remove from the plurality of predicates, each predicate that is satisfied by the metadata of each file of the pruned set of files to generate a modified query; and
execute the modified query against the pruned set of files to create a second set of files.
18. The non-transitory computer-readable medium of claim 17 , wherein to remove from the set of files, each file whose metadata does not match a predicate, the processor is further to:
determine that the metadata for each file in the set of files does not include direct information specified by a given predicate of the plurality of predicates.
19. The non-transitory computer-readable medium of claim 18 , wherein to remove from the set of files, each file whose metadata does not match a predicate, the processor is further to:
determine, for each file in the set of files, the direct information using function-specific value range propagation; and
remove from the set of files, each file whose direct information does not satisfy the given predicate of the plurality of predicates.
20. The non-transitory computer-readable medium of claim 17 , wherein the metadata for each file of the set of files comprises one or more of:
representative values of a block of the database data of the file;
a number of null values in the block of the database data of the file; and
maximum and minimum values of the block of the database data of the file.
21. The non-transitory computer-readable medium of claim 17 , wherein the processor is further to:
load the database data into the storage platform; and
obtain the metadata for each file in the set of files from the database data during the loading.
22. The non-transitory computer-readable medium of claim 17 , wherein one or more of the plurality of predicates each comprise a set of complex expressions.
23. The non-transitory computer-readable medium of claim 17 , wherein to generate the modified query, the processor is further to remove from the plurality of predicates, each predicate that is not satisfied by the metadata of any of the pruned set of files.