Reducing probabilistic filter query latency
Systems and techniques for reducing probabilistic filter query latency are described herein. A query for a probabilistic filter that is stored on a first media may be received from a caller. In response to receiving the query, cached segments of the probabilistic filter stored on a second media may be obtained. Here, the probabilistic filter provides a set membership determination that is conclusive in a determination that an element is not in a set. The query may be executed on the cached segments resulting in a partial query result. Retrieval of remaining data of the probabilistic filter from the first media to the second media may be initiated without intervention from the caller. Here, the remaining data corresponds to the query and data that is not in the cached segment. The partial query results may then be returned to the caller.
1. A system for reducing probabilistic filter query latency, the system comprising processing circuitry to:
receive, from a calling software application, a query for a probabilistic filter that is stored on a first media, the query including a set of test parameters;
obtain, in response to receiving the query, cached segments of the probabilistic filter stored on a second media, wherein the cached segments are less than all of the probabilistic filter stored on the first media, wherein the probabilistic filter provides a set membership determination that is conclusive in a determination that an element is not in a set;
execute the query on the cached segments resulting in a partial query result, the partial query result including results for a subset of the test parameters performed on the cached segments;
initiate retrieval of remaining data of the probabilistic filter from the first media to the second media without intervention from the calling software application, wherein the remaining data corresponds to the query and data that is not in the cached segments; and
return to the calling software application the partial query results.
2. The system of claim 1 , wherein, to receive the query, the processing circuitry receives segment identifiers, the segment identifiers specifying which segments, of all segments of the probabilistic filter, are pertinent to the query.
3. The system of claim 2 wherein the segment identifiers are byte offsets in a file.
4. The system of claim 3 , wherein the file is a key-value set file.
5. The system of claim 4 , wherein the processing circuitry is further to:
maintain a fetch group data structure for pages of the key-value set file; and
retrieve pages in the fetch group for the key-value set file in response to a page fault on one page in the fetch group data structure.
6. The system of claim 1 , wherein the partial query results include a listing of segment identifiers that are pertinent to the query and not in the cached segments.
7. The system of claim 1 , wherein, to initiate retrieval of remaining data of the probabilistic filter, the processing circuitry identifies a stop condition of the probabilistic filter in the partial query result and abandons the retrieval of the remaining data.
8. The system of claim 1 , wherein the probabilistic filter is at least one of a Bloom filter, a Counting Bloom filter, or a Cuckoo filter.
9. A method for reducing probabilistic filter query latency, the method comprising:
receiving, from a calling software application, a query for a probabilistic filter that is stored on a first media;
obtaining, in response to receiving the query, cached segments of the probabilistic filter stored on a second media, wherein the cached segments are less than all of the probabilistic filter stored on the first media, wherein the probabilistic filter provides a set membership determination that is conclusive in a determination that an element is not in a set;
executing the query on the cached segments resulting in a partial query result;
initiating retrieval of remaining data of the probabilistic filter from the first media to the second media without intervention from the calling software application, the initiating retrieval of remaining data of the probabilistic filter including identifying a stop condition of the probabilistic filter in the partial query result and abandoning the retrieval of the remaining data, wherein the remaining data corresponds to the query and data that is not in the cached segments; and
returning to the calling software application the partial query results.
10. The method of claim 9 , wherein receiving the query includes receiving segment identifiers, the segment identifiers specifying which segments, of all segments of the probabilistic filter, are pertinent to the query.
11. The method of claim 10 , wherein the segment identifiers are byte offsets in a file.
12. The method of claim 11 , wherein the file is a key-value set file.
13. The method of claim 12 , comprising:
maintaining a fetch group data structure for pages of the key-value set file; and
retrieving pages in the fetch group for the key-value set file in response to a page fault on one page in the fetch group data structure.
14. The method of claim 9 , wherein the query includes a set of test parameters, and wherein the partial query result includes results for a subset of the test parameters performed on the cached segments.
15. The method of claim 14 , wherein the partial query results include a listing of segment identifiers that are pertinent to the query and not in the cached segments.
16. The method of claim 9 , wherein the probabilistic filter is at least one of a Bloom filter, a Counting Bloom filter, or a Cuckoo filter.
17. A non-transitory machine readable storage medium including instructions for reducing probabilistic filter query latency, the instructions, when executed by processing circuitry, cause the processing circuitry to perform operations comprising:
receiving, from a calling software application, a query for a probabilistic filter that is stored on a first media;
obtaining, in response to receiving the query, cached segments of the probabilistic filter stored on a second media, the segments of the probabilistic filter being based on a representation of the probabilistic filter in the second media, wherein the cached segments are less than all of the probabilistic filter stored on the first media, wherein the probabilistic filter provides a set membership determination that is conclusive in a determination that an element is not in a set;
executing the query on the cached segments resulting in a partial query result;
initiating retrieval of remaining data of the probabilistic filter from the first media to the second media without intervention from the calling software application, wherein the remaining data corresponds to the query and data that is not in the cached segments; and
returning to the calling software application the partial query results.
18. The machine readable medium of claim 17 , wherein receiving the query includes receiving segment identifiers, the segment identifiers specifying which segments, of all segments of the probabilistic filter, are pertinent to the query.
19. The machine readable medium of claim 18 , wherein the segment identifiers are byte offsets in a file.
20. The machine readable medium of claim 19 , wherein the query includes a file identifier for the file.
21. The machine readable medium of claim 19 , wherein the file is a key-value set file.
22. The machine readable medium of claim 21 , wherein the operations comprise:
maintaining a fetch group data structure for pages of the key-value set file; and
retrieving pages in the fetch group for the key-value set file in response to a page fault on one page in the fetch group data structure.
23. The machine readable medium of claim 21 , wherein the key-value set file includes more than one KVS tree.
24. The machine readable medium of claim 23 , wherein the probabilistic filter applies to a single KVS tree in the key-value set file.
25. The machine readable medium of claim 18 , wherein obtaining the cached segments includes obtaining segments of the probabilistic filter in the second media that correspond to the segment identifiers.
26. The machine readable medium of claim 17 , wherein the segments have a uniform size.
27. The machine readable medium of claim 26 , wherein the uniform size is a memory page size.
28. The machine readable medium of claim 27 , wherein the segment identifiers are page identifiers.
29. The machine readable medium of claim 26 , wherein the segment identifiers are memory addresses.
30. The machine readable medium of claim 17 , wherein the query includes a set of test parameters, and wherein the partial query result includes results for a subset of the test parameters performed on the cached segments.
31. The machine readable medium of claim 30 , wherein the test parameters include a location.
32. The machine readable medium of claim 31 , wherein the location is a bit offset in a segment.
33. The machine readable medium of claim 32 , wherein the subset of the test parameters includes a bit value in each location.
34. The machine readable medium of claim 30 , wherein the partial query results include a listing of segment identifiers that are pertinent to the query and not in the cached segments.
35. The machine readable medium of claim 34 , wherein the operations comprise:
receiving, by the calling software application, the partial query result;
searching the subset of test parameters in the partial query result for a stop condition defined by the probabilistic filter; and
searching for the stop condition within segments of the segment identifiers when the stop condition is not found in the subset of test parameters, the segments being resident in the second media as a result of initiating the retrieval of the remaining data of the probabilistic filter.
36. The machine readable medium of claim 17 , wherein initiating retrieval of remaining data of the probabilistic filter includes identifying a stop condition of the probabilistic filter in the partial query result and abandoning the retrieval of the remaining data.
37. The machine readable medium of claim 17 , wherein the probabilistic filter is at least one of a Bloom filter, a Counting Bloom filter, or a Cuckoo filter.