Partitioned bloom filter merge for massively parallel processing clustered data management
A computer-implemented method for a partitioned bloom filter merge is provided. A non-limiting example of the computer-implemented method includes partitioning, by a processing device, a bloom filter into N equal size filter partitions. The method further includes distributing, by the processing device, each of the filter partitions to an associated node. The method further includes merging, by the processing device, the filter partitions in each of the associated nodes. The method further includes redistributing, by the processing device, the merged filter partitions to each of the N nodes. The method further includes joining, by the processing device, the merged filter partitions in each of the N nodes to assemble a complete merged bloom filter.
1. A system comprising:
a memory comprising computer readable instructions; and
a processing device for executing the computer readable instructions for performing a method, the method comprising:
partitioning, by the processing device, a bloom filter at each node of a plurality of nodes into N equal size filter partitions, wherein N is the number of the plurality of nodes;
distributing, by the processing device, each of the filter partitions among the plurality of nodes such that each node of the plurality of nodes is associated with one of the filter partitions from each of the plurality of nodes;
merging, by the processing device, the filter partitions in each of the plurality of nodes;
redistributing, by the processing device, the merged filter partitions to each of the plurality of nodes; and
joining, by the processing device, the merged filter partitions in each of the plurality of nodes to assemble a complete merged bloom filter.
2. The system of claim 1 , wherein the filter partitions are bitmaps, and wherein merging the filter partitions comprises performing a bitmap merge on the bitmaps.
3. The system of claim 2 , wherein performing the bitmap merge comprises applying an OR operation on the bitmaps.
4. The system of claim 1 , wherein the method further comprises generating the bloom filter to filter data in a distributed database system prior to partitioning the bloom filter.
5. The system of claim 1 , wherein the method further comprises filtering the data using the complete merged bloom filter by applying the complete merged bloom filter to a data table to filter rows of data based on the filter.
6. The system of claim 5 , further comprising using the filtered data to perform an operation.
7. A computer program product comprising:
a computer readable storage medium having program instructions embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, the program instructions executable by a processing device to cause the processing device to perform a method comprising:
partitioning, by the processing device, a bloom filter at each node of a plurality of nodes into N equal size filter partitions, wherein N is the number of the plurality of nodes;
distributing, by the processing device, each of the filter partitions among the plurality of nodes such that each node of the plurality of nodes is associated with one of the filter partitions from each of the plurality of nodes;
merging, by the processing device, the filter partitions in each of the plurality of nodes;
redistributing, by the processing device, the merged filter partitions to each of the plurality of nodes; and
joining, by the processing device, the merged filter partitions in each of the plurality of nodes to assemble a complete merged bloom filter.
8. The computer program product of claim 7 , wherein the filter partitions are bitmaps, and wherein merging the filter partitions comprises performing a bitmap merge on the bitmaps.
9. The computer program product of claim 8 , wherein performing the bitmap merge comprises applying an OR operation on the bitmaps.
10. The computer program product of claim 7 , further comprising generating the bloom filter to filter data in a distributed database system prior to partitioning the bloom filter.
11. The computer program product of claim 7 , wherein the method further comprises:
filtering the data using the complete merged bloom filter by applying the complete merged bloom filter to a data table to filter rows of data based on the filter; and
using the filtered data to perform an operation.