Large-scale, high-dimensional similarity clustering in linear time with error-free retrieval
A method for a processing device to determine whether to assign a data item to at least one cluster of data items is disclosed. The processing device may identify a signature of the data item, the signature including a set of elements. The processing device may select a subset of the set of elements to form at least one partial signature. The processing device may combine the selected subset of elements into at least one token. The processing device may determine whether the at least one token is present in a memory. The memory may be configured to contain an existing set of tokens. The processing device may determine whether to assign the data item to at least one cluster based on whether the at least one token is present in the memory.
1. A method for a processing device to determine whether to assign a data item to at least one cluster of data items, comprising:
identifying a signature of the data item, the signature including a set of elements;
selecting a subset of the set of elements;
combining the selected subset of elements into a token, wherein the identified signature and a second signature are similar when they have the token in common;
determining whether the token is present in a memory, the memory configured to contain an existing set of tokens; and
determining whether to assign the data item to at least one cluster based on whether the token is present in the memory.
2. The method of claim 1 , wherein the determining whether to assign the data item to at least one cluster is based on whether the token is absent from the memory.
3. The method of claim 1 , further comprising:
storing the token into the memory.
4. The method of claim 3 , further comprising:
storing a value associated with the token into the memory, wherein the value represents one or more of:
the identified signature;
the token;
an identifier of a cluster to which the data item is assigned;
a centroid of a cluster to which the data item is assigned;
a flag value indicating that the token is marked;
a probability or weight with which the data item is assigned to a cluster;
information pertaining to the data item or to the cluster to which the data item is assigned; or
at least one of a pointer, a link, an index, or a reference to a representation of one or more of said identified signature, said token, said identifier, said centroid, said flag, said probability or weight, or said information.
5. The method of claim 4 , further comprising
outputting a representation of one or more of:
a determination whether to assign the data item to at least one cluster;
an identifier of a cluster to which the data item is assigned;
a centroid of a cluster to which the data item is assigned;
a member of a cluster to which the data item is assigned; or
information pertaining to the data item or to a cluster to which the data item is assigned.
6. The method of claim 4 , wherein the storing the value associated with the token into the memory further comprises storing the token as a key with the associated value, as a key-value pair.
7. The method of claim 1 , further comprising:
receiving one or more of a second data item, a second signature, a second token, or an identifier of a cluster; and
outputting one or more values retrieved from the memory associated with the second data item, the second signature, the second token, or the identifier of a cluster.
8. The method of claim 1 , wherein at least a portion of the memory is organized as one of a hash table, a map, a symbol table, an associative array, or an unordered dictionary data structure.
9. The method of claim 1 , wherein the combining the selected subset of elements further comprises:
concatenating the selected elements with a separator to form a string.
10. The method of claim 1 , wherein the combining the selected subset of elements is based on a sorted representation of the elements.
11. The method of claim 1 , wherein the determining whether to assign the data item to at least one cluster further comprises:
determining whether to merge two or more clusters into one cluster.
12. The method of claim 1 , wherein the determining whether to assign the data item to at least one cluster further comprises:
determining whether to split a cluster into two or more clusters.
13. The method of claim 1 , wherein the determining whether to assign the data item to at least one cluster is further based on factors including at least one of:
the number of data items assigned to a cluster,
a minimum similarity between the signatures of two data items in a cluster,
a minimum similarity between the identified signature and a centroid of a cluster,
a density of a cluster,
the number of tokens in common between data items assigned to a cluster, or
the number of elements in the tokens in common between data items assigned to a cluster.
14. The method of claim 1 , wherein the determining whether to assign the data item to at least one cluster further depends on determining whether the identified signature is similar to a signature assigned to the at least one cluster.
15. The method of claim 1 , wherein the selecting the subset of the set of elements of the identified signature is based on a similarity measure of the identified signature to a second signature and on a minimum similarity threshold value.
16. The method of claim 15 , wherein the identified signature and the second signature have at least one token in common when they are similar.
17. The method of claim 15 , wherein the similarity measure is at least one of Jaccard similarity, Levenshtein similarity, a Szymkiewicz-Simpson overlap coefficient, mutual similarity, a Sorensen-Dice coefficient, or a Tversky similarity index.
18. The method of claim 1 , further comprising repeatedly performing said selecting, said combining, said determining whether the token is present in a memory, and said determining whether to assign the data item to at least one cluster, with a plurality of minimum similarity threshold values to obtain a hierarchy of clusterings.
19. The method of claim 1 , wherein the selecting the subset of the set of elements of the identified signature is based on a set of sizes of signatures, the set of sizes containing a maximum size value representing the largest number of elements in a signature.
20. The method of claim 1 , further comprising:
identifying a first size value, the first size value being the number of elements of the identified signature; and
identifying a second size value, the second size value being the number of elements of a second signature that is similar to the identified signature.
21. The method of claim 20 , further comprising segregating the memory into a plurality of sections of memory, wherein at least one section of the plurality of sections is identified by a size value; and wherein determining whether the token is present in the memory further comprises determining whether the token is present in a section of the memory identified by a size value, the size value being one of either the first size value or the second size value.
22. The method of claim 20 , further comprising:
storing the token into a section of the memory identified by the size value, the size value being one of either the first size value or the second size value.
23. The method of claim 20 , wherein the combining the selected subset of elements into a token further comprises combining a size value into the token, the size value being one of either the first size value or the second size value.
24. The method of claim 1 , wherein the selected subset of the set of elements of the identified signature has a size, the size being the smallest number of elements in common between the identified signature and a second signature that is similar to the identified signature.
25. The method of claim 1 , wherein the elements of the identified signature are equally weighted.
26. The method of claim 1 , wherein the elements of the identified signature are unequally weighted.
27. The method of claim 1 , wherein the identified signature is formed using one or more of term frequency within the data item, term frequency within a corpus, stemming, MinHash, or Bloom filtering.
28. The method of claim 1 , further comprising performing said identifying, said selecting, said combining, said determining whether the token is present in a memory, and said determining whether to assign the data item to at least one cluster in a plurality of data items, in a serial mode or a batch mode.
29. A system for determining whether to assign a data item to at least one cluster of data items, comprising:
a memory; and
a processing device, coupled to the memory, to:
identify a signature of the data item, the signature including a set of elements;
select a subset of the set of elements;
combine the selected subset of elements into a token, wherein the identified signature and a second signature are similar when they have the token in common;
determine whether the token is present in a memory, the memory configured to contain an existing set of tokens; and
determine whether to assign the data item to at least one cluster based on whether the token is present in the memory.
30. A non-transitory computer readable storage medium including instructions that, when executed by a processing device, cause the processing device to determine whether to assign a data item to at least one cluster of data items, the processing device further to:
identify a signature of the data item, the signature including a set of elements;
select a subset of the set of elements of the identified signature;
combine the selected subset of elements into a token, wherein the identified signature and a second signature are similar when they have the token in common;
determine whether the token is present in a memory, the memory configured to contain an existing set of tokens; and
determine whether to assign the data item to at least one cluster based on whether the token is present in the memory.