Dynamic bloom filter for caching query results
Methods, systems, and machine-readable media are disclosed for searching a corpus of information by utilizing a Bloom filter for caching query results. According to one aspect of the present invention, a method of caching information from a corpus of information can include populating one or more Bloom filters with a plurality of bits representative of information in the corpus of information. A search request can be received identifying requested information from the corpus of information. One or more bits in the filter(s) associated with the requested information can be checked and the requested information can be retrieved from the corpus of information based on results of said checking. Furthermore, the filter(s) can be used to determine which information to make available to a particular user in a system where certain information is associated with or access is limited to certain users or groups of users.
1. A computer-implemented method for caching a corpus of information, the method comprising:
populating a dynamic Bloom filter with a plurality of virtual bits, the plurality of virtual bits representative of information comprising the corpus, a given virtual bit operative to support two or more values;
receiving a search request that identifies requested information from the corpus;
checking one or more bits in the Bloom filter associated with the requested information to determine whether the requested information is present in the corpus;
retrieving the requested information from the corpus on the basis of the result of the check;
determining whether the requested information is represented in the Bloom filter;
determining whether to represent the requested information in the Bloom filter where the requested information is not represented in the Bloom filter;
adding the requested information to the Bloom filter in response to the determining step without blocking the Bloom filter;
determining whether to continue to represent the requested information in the Bloom filter where the requested information is represented in the Bloom filter;
removing old data associated with the requested information from the Bloom filter without blocking the Bloom filter and adding new data associated with the requested information to the Bloom filter without blocking the Bloom filter in response to a determination to continue to represent the requested information in the Bloom filter;
removing the old data associated with the requested information from the Bloom filter without blocking the Bloom filter in response to a determination to not continue to represent the requested information in the Bloom filter;
wherein removing old data associated with the requested information from the Bloom filter comprises identifying one or more virtual bits associated with the requested information from the dynamic Bloom filter, determining whether a value of a given identified virtual bit is less than or equal to a minimum value, decrementing the value of the given identified virtual bit where the value of the given identified virtual bit is less than or equal to the minimum value, recording an indication that the given identified virtual bit is less than or equal to the minimum value in an underflow cache where the given identified virtual bit is less than or equal to the minimum value, determining whether the underflow cache indicates an underflow condition, and cleaning the Bloom filter in response to the determination of the existence of an underflow condition.
2. A computer-implemented method for caching a corpus of information, the method comprising:
populating a dynamic Bloom filter with a plurality of virtual bits, the plurality of virtual bits representative of information comprising the corpus, a given virtual bit operative to support two or more values;
receiving a search request that identifies requested information from the corpus;
checking one or more bits in the Bloom filter associated with the requested information to determine whether the requested information is present in the corpus; and
retrieving the requested information from the corpus on the basis of the result of the check; and
storing one or more of the plurality of virtual bits into an underflow cache to remove old data from the Bloom filter.
3. The method of claim 2 comprising:
determining whether the requested information is represented in the Bloom filter;
determining whether to represent the requested information in the Bloom filter where the requested information is not represented in the Bloom filter; and
adding the requested information to the Bloom filter in response to the determining step without blocking the Bloom filter.
4. The method of claim 3 comprising:
determining whether to continue to represent the requested information in the Bloom filter where the requested information is represented in the Bloom filter; and
removing old data associated with the requested information from the Bloom filter without blocking the Bloom filter and adding new data associated with the requested information to the Bloom filter without blocking the Bloom filter in response to a determination to continue to represent the requested information in the Bloom filter.
5. The method of claim 4 comprising removing the old data associated with the requested information from the Bloom filter without blocking the Bloom filter in response to a determination to not continue to represent the requested information in the Bloom filter.
6. The method of claim 5 wherein removing old data associated with the requested information from the Bloom filter comprises:
identifying one or more virtual bits associated with the requested information from the dynamic Bloom filter;
determining whether a value of a given identified virtual bit is less than or equal to a minimum value; and
decrementing the value of the given identified virtual bit where the value of the given identified virtual bit is less than or equal to the minimum value.
7. The method of claim 6 comprising repeating the determining and decrementing steps for each of the one or more identified virtual bits.
8. The method of claim 6 comprising recording an indication that the given identified virtual bit is less than or equal to the minimum value in the underflow cache where the given identified virtual bit is less than or equal to the minimum value.
9. The method of claim 1 wherein adding new data associated with the requested information from the Bloom filter comprises:
identifying one or more virtual bits associated with the requested information from the dynamic Bloom filter;
determining whether a value of a given identified virtual bit is greater than or equal to a maximum value; and
incrementing the value of the given identified virtual bit where the value of the given identified virtual bit is greater than or equal to the maximum value.
10. The method of claim 9 comprising repeating the determining and incrementing steps for each of the one or more identified virtual bits.
11. The method of claim 1 wherein populating a Bloom filter with a plurality of bits comprises populating with a plurality of bits representative of information in a personalization database.
12. The method of claim 1 wherein populating a Bloom filter with a plurality of bits comprises populating with a plurality of bits representative of information in a personalization database relating to a trust network.
13. The method of claim 1 wherein populating a Bloom filter with a plurality of bits comprises populating with a plurality of bits representative of information in an index of content items available on a computer network.
14. The method of claim 1 wherein populating a Bloom filter with a plurality of bits comprises populating with a plurality of bits representative of information in an index of content items available on an intranet.
15. The method of claim 1 wherein populating a Bloom filter with a plurality of bits comprises populating with a plurality of bits representative of information in an index of content items available on the Internet.
16. The method of claim 1 wherein populating a Bloom filter with a plurality of bits comprises populating a bit map with the plurality of bits.
17. The method of claim 16 wherein populating the bit map comprises defining one or more hash functions to map information comprising the corpus to one or more positions in the bit map.
18. The method of claim 1 comprising initializing the plurality of bits to zero in the Bloom filter.
19. The method of claim 1 comprising retrieving one or more items of additional information related to the requested information from the corpus on the basis of the result of the check.
20. The method of claim 1 wherein populating comprises populating a non-blocking Bloom filter.