Methods and systems for performing a vectorized delete in a distributed database system
Example aspects include techniques for clustering delete targets for vectorized deletion including retrieving, from a set of delete targets in a distributed database system, a file to be deleted, scanning existing clusters of files marked for deletion to identify at least one existing cluster of files having constraints corresponding to the file, based on identifying the at least one existing cluster of files, adding the file to the at least one existing cluster of files to create a new cluster of files, based on failing to identify the at least one existing cluster of files, creating the new cluster of files including the file, and generating, for each file in the new cluster of files and based on a deletion signal, a delta array including multiple bits representing data items in each file and indicating, based on bit value, target data items to be deleted from each file.
1. A computer-implemented method for clustering target data sources for a vectorized delete operation, comprising:
retrieving, from a set of delete targets in a distributed database system, a file to be deleted in the distributed database system;
scanning existing clusters of files marked for deletion in the distributed database system to identify at least one existing cluster of files having one or more constraints corresponding to the file;
based on identifying the at least one existing cluster of files, adding the file to the at least one existing cluster of files to create a new cluster of files;
based on failing to identify the at least one existing cluster of files, creating the new cluster of files including the file; and
generating, for each file of multiple files in the new cluster of files and based on a deletion signal, a delta array including multiple bits representing data items in each file and indicating, based on bit value, target data items to be deleted from each file.
2. The computer-implemented method of claim 1 , wherein the one or more constraints include a hash function used to generate shards corresponding to one or more of the multiple files in the at least one existing cluster of files.
3. The computer-implemented method of claim 2 , wherein identifying the at least one existing cluster of files further includes verifying that a smallest common denominator can be established for a key value used in the hash function and a second key value used in the hash function to generated second shards corresponding to the file.
4. The computer-implemented method of claim 1 , wherein the one or more constraints include a range function used to generate shards corresponding to one or more of the multiple files in the at least one existing cluster of files.
5. The computer-implemented method of claim 4 , wherein identifying the at least one existing cluster of files further includes verifying that a smallest common denominator can be established for a key value used in the range function and a second key value used in the range function to generated second shards corresponding to the file.
6. The computer-implemented method of claim 1 , wherein the one or more constraints correspond to a join algorithm used for the at least one existing cluster of files.
7. The computer-implemented method of claim 6 , wherein the join algorithm is one of a sort-merge join or a hash join.
8. The computer-implemented method of claim 1 , wherein the one or more constraints include whether shards corresponding to one or more of the multiple files in the at least one existing cluster of files are partitioned using hash partitioning or range partitioning.
9. The computer-implemented method of claim 1 , wherein the one or more constraints include a shard size corresponding to one or more of the multiple files in the at least one existing cluster of files.
10. The computer-implemented method of claim 1 , wherein the one or more constraints include a type of join key used to join one or more of the multiple files in the at least one existing cluster of files.
11. A device for clustering target data sources for a vectorized delete operation, comprising:
at least one memory storing instructions; and
at least one processor coupled to the at least one memory and configured to execute the instructions to:
obtain, from a set of delete targets in a distributed database system, a file to be deleted in the distributed database system;
identify, from existing clusters of files marked for deletion in the distributed database system, at least one existing cluster of files having one or more constraints corresponding to the file;
create a new cluster of files to include the file, wherein the new cluster of files includes the at least one existing cluster of files based on identifying the at least one existing cluster of files; and
generate, for each file of multiple files in the new cluster of files and based on a deletion signal, a delta array indicating, by bit values, target data items to be deleted from each file.
12. The device of claim 11 , wherein the one or more constraints include a hash function used to generate shards corresponding to one or more of the multiple files in the at least one existing cluster of files.
13. The device of claim 12 , wherein the at least one processor is configured to identify the at least one existing cluster of files at least in part by verifying that a smallest common denominator can be established for a key value used in the hash function and a second key value used in the hash function to generated second shards corresponding to the file.
14. The device of claim 11 , wherein the one or more constraints include a range function used to generate shards corresponding to one or more of the multiple files in the at least one existing cluster of files.
15. The device of claim 14 , wherein the at least one processor is configured to identify the at least one existing cluster of files at least in part by verifying that a smallest common denominator can be established for a key value used in the range function and a second key value used in the range function to generated second shards corresponding to the file.
16. The device of claim 11 , wherein the one or more constraints correspond to a join algorithm used for the at least one existing cluster of files.
17. The device of claim 16 , wherein the join algorithm is one of a sort-merge join or a hash join.
18. The device of claim 11 , wherein the one or more constraints include whether shards corresponding to one or more of the multiple files in the at least one existing cluster of files are partitioned using hash partitioning or range partitioning.
19. One or more non-transitory computer-readable devices storing instructions thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations comprising:
retrieving, from a set of delete targets in a distributed database system, a file to be deleted in the distributed database system;
scanning existing clusters of files marked for deletion in the distributed database system to identify at least one existing cluster of files having one or more constraints corresponding to the file;
based on identifying the at least one existing cluster of files, adding the file to the at least one existing cluster of files to create a new cluster of files;
based on failing to identify the at least one existing cluster of files, creating the new cluster of files including the file; and
generating, for each file of multiple files in the new cluster of files and based on a deletion signal, a delta array including multiple bits representing data items in each file and indicating, based on bit value, target data items to be deleted from each file.
20. The one or more non-transitory computer-readable devices of claim 19 , wherein the one or more constraints include a hash function used to generate shards corresponding to one or more of the multiple files in the at least one existing cluster of files.