Layered locality sensitive hashing (LSH) partition indexing for big data applications
System and method of partitioning a plurality of data objects that are each represented by a respective high dimensional feature vector is described, including performing a hashing function on each high dimensional feature vector to generate a respective lower dimensional binary compact feature vector for the data object that is represented by the high dimensional feature vector; performing a further hashing function on each compact feature vector to assign a sub-index ID to the compact feature vector; and partitioning the compact feature vectors into respective partition groups that correspond to the sub-index IDs assigned to the compact feature vectors.
1. A method of partitioning a plurality of data objects that are each represented by a respective high dimensional feature vector, comprising:
performing a locality sensitive hashing function on each high dimensional feature vector to generate a respective lower dimensional binary compact feature vector for the data object that is represented by the high dimensional feature vector;
performing on each respective lower dimensional binary compact feature vector a further locality sensitive hashing function to generate a sub-index ID for the respective lower dimensional binary compact feature vector and assigning the sub-index ID to the respective lower dimensional binary compact feature vector;
partitioning the lower dimensional binary compact feature vectors that are assigned identical sub-index IDs into respective partition groups;
generating a searchable sub-index structure for each respective partition group by:
generating a plurality of twisted compact feature vector sets for the partition group, each twisted compact feature vector set being generated by applying a respective bit order shuffling permutation to all of the lower dimensional binary compact feature vectors included in the partition group;
generating a respective index table for each twisted compact feature vector set of the partition group by indexing each of the twisted compact feature vectors included compact feature vector set in the index table based on bit order similarities,
the searchable sub-index structure for the partition group including the index tables for all of the twisted compact feature vector sets of the partition group,
and
storing the sub-index structures as independently searchable structures.
2. The method of claim 1 wherein each of the locality sensitive hashing function and the further locality sensitive hashing function are orthogonal angle hashing functions.
3. The method of claim 1 wherein
the respective bit order shuffling permutations are randomly generated.
4. A system for partitioning a plurality of data objects that are each represented by a respective high dimensional feature vector, comprising:
one or more processing units;
a system storage device coupled to each of the processing units, the system storage device tangibly storing thereon executable instructions that, when executed by the one or more processing units, cause the system to:
perform a locality sensitive hashing function on each high dimensional feature vector to generate a respective lower dimensional binary compact feature vector for the data object that is represented by the high dimensional feature vector;
perform on each respective lower dimensional binary compact feature vector a further locality sensitive hashing function to generate a sub-index ID or the respective lower dimensional binary compact feature vector and assign the sub-index ID to the respective lower dimensional binary compact feature vector;
partition the lower dimensional binary compact feature vectors that are assigned identical sub-index IDs into respective partition groups;
generate a searchable sub-index structure for each of the respective partition group by: generating a plurality of twisted compact feature vector sets for the partition group, each twisted compact feature vector set being generated by applying a respective bit order shuffling permutation to all of the lower dimensional binary compact feature vectors included in the partition group;
generating a respective index table for each twisted compact feature vector set of the partition group by indexing each of the twisted compact feature vectors included compact feature vector set in the index table based on bit order similarities,
the searchable sub-index structure for the partition group including the index tables for all of the twisted compact feature vector sets of the partition group,
and
store the sub-index structures in one or more storages as independently searchable structures.
5. The system of claim 4 wherein each of the locality sensitive hashing function and the further locality sensitive hashing function are orthogonal angle hashing functions.
6. The system of claim 4 wherein
the respective bit order shuffling permutations are random.
7. A computer program product comprising a medium tangibly storing thereon executable instructions that, when executed by a digital processing system, cause the digital processing system to:
perform a locality sensitive hashing function on each of a plurality of high dimensional feature vectors to generate a respective lower dimensional binary compact feature vector, the high dimensional feature vectors each representing a respective data object;
perform on each respective lower dimensional binary compact feature vector a further locality sensitive hashing function to generate a sub-index ID for the respective lower dimensional binary compact feature vector and assign the sub-index ID to the respective lower dimensional binary compact feature vector; and
partition the lower dimensional binary compact feature vectors that are assigned an identical sub-index ID into a respective partition group;
partition the lower dimensional binary compact feature vectors that are assigned identical sub-index IDs into respective partition groups;
generate a searchable sub-index structure for each of the respective partition group by:
generating a plurality of twisted compact feature vector sets for the partition group, each twisted compact feature vector set being generated by applying a respective bit order shuffling permutation to all of the lower dimensional binary compact feature vectors included in the partition group;
generating a respective index table for each twisted compact feature vector set of the partition group by indexing each of the twisted compact feature vectors included compact feature vector set in the index table based on bit order similarities,
the searchable sub-index structure for the partition group including the index tables for all of the twisted compact feature vector sets of the partition group,
and
store the sub-index structures in one or more storages as independently searchable structures.
8. The method of claim 1 further comprising searching the plurality of data objects for data objects that are similar to a query object, comprising:
converting the query object into a d-dimensional feature vector;
performing a locality sensitive hashing function on the d-dimensional feature vector to generate an m-dimensional binary compact query vector for the query object, where m<d;
performing a further locality sensitive hashing function on the m-dimensional binary compact query vector to determine a sub-index ID for the m-dimensional binary compact query vector; and
searching, in the sub-index structure that corresponds to the sub-index ID for the m-dimensional binary compact query vector, for lower dimensional binary compact feature vectors that are similar to the m-dimensional binary compact query vector.
9. The method of claim 8 wherein each of the locality sensitive hashing function and the further locality sensitive hashing function are orthogonal angle hashing functions.
10. The method of claim 8 further comprising:
determining a set of further sub-index IDs that fall within a similarity threshold for the sub-index ID for the m-dimensional binary compact query vector; and
searching further sub-index structures that correspond to the further sub-index IDs for compact feature vectors that are similar to the m-dimensional binary compact query vector.
11. The method of claim 10 wherein the similarity threshold is a threshold level of different bit values in the further sub-index IDs relative to the sub-index ID of the m-dimensional binary compact query vector.
12. The method of claim 10 wherein the searching of further sub-index structures is terminated if a threshold number of search results is reached before all of the sub-index structures that correspond to the further sub-index IDs have been searched.
13. The method of claim 8 comprising, concurrent with searching in a sub-index structure that corresponds to the sub-index ID: searching a further sub-index structure for compact feature vectors that are similar to a further query vector for which a further sub-index ID has been determined.
14. The method of claim 1 wherein the respective lower dimensional binary compact feature vector generated in respect of each high dimensional feature vector comprises a number m of binary values, each binary value being a hash function result of all feature values of the high dimensional feature vector with a respective hash function of an LSH hash function chain of m hash functions, the m hash functions of the LSH hash function chain each being randomly selected from a family of greater than m hash functions.