Partitioned bloom filter merge for massively parallel processing clustered data management
A computer-implemented method for a partitioned bloom filter merge is provided. A non-limiting example of the computer-implemented method includes partitioning, by a processing device, a bloom filter into N equal size filter partitions. The method further includes distributing, by the processing device, each of the filter partitions to an associated node. The method further includes merging, by the processing device, the filter partitions in each of the associated nodes. The method further includes redistributing, by the processing device, the merged filter partitions to each of the N nodes. The method further includes joining, by the processing device, the merged filter partitions in each of the N nodes to assemble a complete merged bloom filter.
1. A computer-implemented method comprising:
partitioning, by a processing device, a bloom filter at each node of a plurality of nodes into N equal size filter partitions, wherein N is the number of the plurality of nodes;
distributing, by the processing device, each of the filter partitions among the plurality of nodes such that each node of the plurality of nodes is associated with one of the filter partitions from each of the plurality of nodes;
merging, by the processing device, the filter partitions in each of the plurality of nodes;
redistributing, by the processing device, the merged filter partitions to each of the plurality of nodes; and
joining, by the processing device, the merged filter partitions in each of the plurality of nodes to assemble a complete merged bloom filter.
2. The computer-implemented method of claim 1 , wherein the filter partitions are bitmaps, and wherein merging the filter partitions comprises performing a bitmap merge on the bitmaps.
3. The computer-implemented method of claim 2 , wherein performing the bitmap merge comprises applying an OR operation on the bitmaps.
4. The computer-implemented method of claim 1 , further comprising generating the bloom filter to filter data in a distributed database system prior to partitioning the bloom filter.
5. The computer-implemented method of claim 1 , further comprising filtering the data using the complete merged bloom filter by applying the complete merged bloom filter to a data table to filter rows of data based on the filter.
6. The computer-implemented method of claim 5 , further comprising using the filtered data to perform an operation.