Information similarity and related statistical techniques for use in distributed computing environments
Embodiments of methods, systems and/or apparatuses relating to data processing in distributed computing environments are disclosed. In particular, methods, systems, and/or apparatuses for determining information similarly and/or performing related statistical techniques which may be implemented or operated in a distributed computing environment are disclosed.
1. A method, comprising:
accessing a record set comprising a plurality of records;
generating a plurality of terms for said plurality of records, individual ones of said plurality of terms comprising a q-gram of a fixed character length;
determining a plurality of respective parameter frequencies for said plurality of terms;
at least partially in response to said determining a plurality of respective parameter frequencies, assigning particular termIDs to said plurality of terms based, at least in part, on said respective parameter frequencies, wherein said particular termIDs comprise an identifier to indicate a term/record frequency within said record set;
comparing one or more positions of said termIDs for first terms of a first vector of terms for a first candidate record and for second terms of a second vector of terms for a second candidate record to determine whether there is an overlap between the first and second vectors.
2. The method of claim 1 , wherein at least one of said parameter frequencies comprises a term/record frequency.
3. The method of claim 1 , wherein said record set comprises at least two tables of records.
4. The method of claim 1 , wherein generating said plurality of terms for one or more records comprises using at least one of a parsing technique, a q-grams technique, a tokenizing technique, a truncating technique, or combinations thereof.
5. The method of claim 1 , wherein assigning said particular termIDs to said plurality of terms comprises at least one of:
assigning a lowest termID to a term associated with a lowest respective term/record frequency; and/or
assigning a highest termID to a term associated with a highest respective term/record frequency.
6. The method of claim 1 , further comprising:
determining a term weight for one or more of said plurality of terms.
7. The method of claim 6 , wherein at least one of said parameter frequencies comprises a term/record frequency associated with a particular term, and determining said term weight comprises multiplying said term/record frequency associated with said particular term with a log of the inverse of a corresponding record frequency.
8. The method of claim 1 , wherein at least one of said parameter frequencies comprises at least one weighed term/record frequency.
9. The method of claim 1 , further comprising:
sorting said plurality of terms based, at least in part, on said respective parameter frequencies.
10. The method of claim 1 , further comprising:
performing an information similarity determination process based at least in part on at least a portion of said termIDs.
11. The method of claim 10 , wherein said information similarity determination process comprises a similarity join operation.
12. The method of claim 10 , further comprising:
performing said information similarity determination process using said plurality of termIDs, wherein said plurality of termIDs are assigned to said plurality of terms based, at least in part, on their respective parameter frequencies.
13. The method of claim 10 , wherein performing said information similarity determination process comprises generating one or more statistics associated with said one or more records of said record set.
14. The method of claim 10 , further comprising:
performing one or more optimizations for said information similarity determination process operation.
15. The method of claim 14 , wherein at least one of said one or more optimizations comprises a summary statistics optimization.
16. The method of claim 10 , wherein said information similarity determination process uses one or more similarity functions based, at least in part on, or convertible to, an overlap coefficient similarity or a vector dot product.
17. The method of claim 10 , further comprising: generating one or more candidate pairs based, at least on part, on said information similarity determination process.
18. The method of claim 1 , further comprising determining term weights for said plurality of terms as a function of said respective parameter frequencies for said plurality of terms.
19. The method of claim 1 , further comprising performing a second map/reduce iteration to determine an overlap of said prefixes.
20. The method of claim 19 , further comprising performing a third map/reduce iteration to generate one or more keys for said one or more records to be emitted based at least in part on said overlap of said prefixes.
21. The method of claim 1 , wherein the multiple map/reduce iterations are further implemented to filter out terms of said plurality of terms comprising a suffix of said one or more records.
22. A method, comprising:
accessing a record set comprising a plurality of records;
generating a plurality of terms for said plurality of records, individual ones of said plurality of terms comprising a q-gram of a fixed character length;
filtering out terms comprising a suffix of said one or more records;
determining a plurality of respective parameter frequencies for said plurality of terms in response to said filtering;
at least partially in response to said determining a plurality of respective parameter frequencies, assigning particular termIDs to said plurality of terms based, at least in part, on said respective parameter frequencies, wherein said particular termIDs comprise an identifier to indicate a term/record frequency within said record set; and
comparing one or more positions of said termIDs for first terms of a first vector of terms for a first candidate record and for second terms of a second vector of terms for a second candidate record to determine whether there is an overlap between the first and second vectors.