INDEXING STORED DATA OBJECTS USING PROBABILISTIC FILTERS
A system and method index data objects in an object store according to structure found in data records from which the objects are themselves formed. Embodiments order structured data records, aggregate slices of consecutively-ordered records into a single corresponding data object, store the data object in the object store, and place the associated object handle into a search tree with all the other such handles. The search tree is indexed using probabilistic set-inclusion filters, such as Bloom filters, not on the handles themselves but on the indexed fields of the records within each slice. For data sets having enough data records, and thus search trees that are deep enough, the aggregate false positive rate for depth-first searches on the tree becomes infinitesimal due to the multiplicate property of the false positive rates for independent Bloom filters. Searches are rapid on even moderately tuned filters.
1 . A system for obtaining a structured data record, the system comprising:
a storage device for storing a search tree that covers the space of possible search keys, the search tree having nodes that each provide an approximate membership query filter for a subset of the possible search keys;
a search key compositor for forming a search key as a composition of one or more values that index the structured data record, whereby the search key is uniquely associated with the structured data record;
a search processor for searching the search tree, starting from a root node and continuing until a leaf node is reached, by: for each encountered node, using the filter for that node to determine whether a subtree containing that node cannot include the search key, and if that is not the case, then traversing that subtree; and
a communication device for communicating, to a data object store, a pointer associated with the leaf node that was reached, and for responsively receiving, from the data object store, a data object having a plurality of data records, of which the structured data record is one.
2 . The system according to claim 1 , wherein the storage device stores a binary search tree.
3 . The system according to claim 1 , wherein each node in the search tree stored in the storage device provides an approximate membership query filter comprising a Bloom filter, or a quotient filter, or a cuckoo filter, according to a characteristic false positive probability between 1% and 30%.
4 . The system according to claim 1 , wherein the search key compositor is configured to form the search key as a composition of one or more database keys that index the structured data record.
5 . The system according to claim 1 , wherein the search key compositor is configured to form the search key to include a sequence number of the data record within the plurality of data records in the data object to be received.
6 . The system according to claim 1 , wherein the communication device is configured for communicating with a data object store that comprises a cloud storage service or a cloud storage device.
7 . A method of obtaining a structured data record, the method comprising:
forming a search key as a composition of one or more values that index the structured data record, whereby the search key is uniquely associated with the structured data record;
traversing a search tree that covers the space of possible search keys, starting from a root node and continuing until a leaf node is reached, by: for each encountered node, using an approximate membership query filter for that node to determine whether a subtree containing that node cannot include the search key, and if that is not the case, then traversing that subtree;
communicating, to a data object store, a pointer associated with the leaf node that was reached; and
responsively receiving, from the data object store, a data object having a plurality of data records, of which the structured data record is one.
8 . The method according to claim 7 , wherein the search tree includes a binary search tree.
9 . The method according to claim 7 , wherein using the approximate membership query filter for each encountered node comprises using a Bloom filter, or a quotient filter, or a cuckoo filter, according to a characteristic false positive probability.
10 . The method according to claim 9 , wherein the characteristic false positive probability is between 1% and 30%.
11 . The method according to claim 7 , wherein forming the search key comprises forming the search key as a composition of one or more database keys that index the structured data record.
12 . The method according to claim 7 , wherein forming the search key comprises forming the search key including a sequence number of the data record within the plurality of data records in the data object to be received.
13 . The method according to claim 7 , wherein the data object store is a cloud storage service or cloud storage device.
14 . A tangible, computer-readable storage medium, in which is non-transitorily stored computer program code for performing a method of obtaining a structured data record, the method comprising:
forming a search key as a composition of one or more values that index the structured data record, whereby the search key is uniquely associated with the structured data record;
traversing a search tree that covers the space of possible search keys, starting from a root node and continuing until a leaf node is reached, by: for each encountered node, using an approximate membership query filter for that node to determine whether a subtree containing that node cannot include the search key, and if that is not the case, then traversing that subtree;
communicating, to a data object store, a pointer associated with the leaf node that was reached; and
responsively receiving, from the data object store, a data object having a plurality of data records, of which the structured data record is one.
15 . The storage medium according to claim 14 , wherein the search tree includes a binary search tree.
16 . The storage medium according to claim 14 , wherein using the approximate membership query filter for each encountered node comprises using a Bloom filter, or a quotient filter, or a cuckoo filter, according to a characteristic false positive probability.
17 . The storage medium according to claim 16 , wherein the characteristic false positive probability is between 1% and 30%.
18 . The storage medium according to claim 14 , wherein forming the search key comprises forming the search key as a composition of one or more database keys that index the structured data record.
19 . The storage medium according to claim 14 , wherein forming the search key comprises forming the search key including a sequence number of the data record within the plurality of data records in the data object to be received.
20 . The storage medium according to claim 14 , wherein the data object store is a cloud storage service or cloud storage device.