Content based similarity detection

1. A computer implemented method comprising:

computing a hash of each word in a collection of books to produce a numerical integer token using a reduced representation;

computing an Inverse Document Frequency (IDF) vector comprising a number of books said token appears in, for every token in said collection of books;

creating a token occurrence count vector for each said book in said collection;

normalizing said token occurrence count vector using said IDF vector to create a Term Frequency-Inverse Document Frequency (TF-IDF) vector;

generating, for each said book, a first reduced TF-IDF vector by removing a TF-IDF term with highest weight from the TF-IDF vector;

generating, for each said book, a second reduced TF-IDF vector by removing two TF-IDF terms with highest weights from the TF-IDF vector;

generating, for each said book, three variants of binary vector representations by applying random projections to the TF-IDF vector, the first reduced TF-IDF vector, and the second reduced TF-IDF vector,

creating a similarity score for each variant of binary vector representations of a target book by comparing respectively with counterpart variant from each of a list of candidate books;

combining the multiple similarity scores to arrive at a final similarity measurement; and

using a trained machine learning algorithm, determining whether each of said list of candidate books is similar to said target book based at least in part on the final similarity measurement.

2. The computer implemented method of claim 1 further comprising:

creating an efficient data structure to create a reduced list of potentially matching books for each said book in said collection to reduce computational costs.

3. The computer implemented method of claim 1 further comprising:

computing a term frequency-inverse document frequency representation of a word count vector.

4. The computer implemented method of claim 3 further comprising:

computing a hash of said term frequency-inverse document frequency representation of said word count vector.

5. The computer implemented method of claim 4 wherein said hash comprises a MurmurHash function.

6. The computer implemented method of claim 1 wherein said determining comprises a Jaccard similarity.

7. The method of claim 1 , wherein a number of words in each said book is used to determine a measure of similarity such that books that are less than a predetermined number of words with respect to similarity detection are discarded.

8. An article of manufacture including a computer readable medium having instructions stored thereon that, responsive to execution by an electronic system, cause said electronic system to perform operations comprising:

computing a hash of each word in a collection of books to produce a numerical integer token using a reduced representation;

computing an Inverse Document Frequency (IDF) vector comprising a number of books said token appears in, for every token in said collection of books;

creating a token occurrence count vector for each said book in said collection;

normalizing said token occurrence count vector using said IDF vector to create a Term Frequency-Inverse Document Frequency (TF-IDF) vector;

generating, for each said book, a first reduced TF-IDF vector by removing a TF-IDF term with highest weight from the TF-IDF vector;

generating, for each said book, a second reduced TF-IDF vector by removing two TF-IDF terms with highest weights from the TF-IDF vector;

generating, for each said book, three variants of binary vector representations by applying random projections to the TF-IDF vector, the first reduced TF-IDF vector, and the second reduced TF-IDF vector,

creating a similarity score for each variant of binary vector representations of a target book by comparing respectively with counterpart variant from each of a list of candidate books;

combining the multiple similarity scores to arrive at a final similarity measurement; and

using a trained machine learning algorithm, determining whether each of said list of candidate books is similar to said target book based at least in part on the final similarity measurement.

9. The article of manufacture of claim 8 wherein said operations further comprise:

creating an efficient data structure to create a reduced list of potentially matching books for each said book in a comparison space to reduce computational costs.

10. The article of manufacture of claim 8 wherein said operations further comprise:

computing a term frequency-inverse document frequency representation of a word count vector.

11. The article of manufacture of claim 10 wherein said operations further comprise:

computing a hash of said term frequency-inverse document frequency representation of said word count vector.

12. The article of manufacture of claim 11 wherein said computing a hash comprises a MurmurHash function.

13. The article of manufacture of claim 8 wherein said determining comprises a Jaccard similarity.

14. An electronic system comprising:

one or more processors;

a memory coupled to said one or more processors, wherein said memory comprises a plurality of electronic books;

wherein said electronic system is configured to:

compute a hash of each word in a collection of books to produce a numerical integer token using a reduced representation;

compute an Inverse Document Frequency (IDF) vector comprising a number of books said token appears in, for every token in said collection of books;

create a token occurrence count vector for each said book in said collection;

normalize said token occurrence count vector using said IDF vector to create a Term Frequency-Inverse Document Frequency (TF-IDF) vector;

generate, for each said book, a first reduced TF-IDF vector by removing a TF-IDF term with highest weight from the TF-IDF vector;

generate, for each said book, a second reduced TF-IDF vector by removing two TF-IDF terms with highest weights from the TF-IDF vector;

generate, for each said book, three variants of binary vector representations by applying random projections to the TF-IDF vector, the first reduced TF-IDF vector, and the second reduced TF-IDF vector,

create a similarity score for each variant of binary vector representations of a target book by comparing respectively with counterpart variant from each of a list of candidate books;

combine the multiple similarity scores to arrive at a final similarity measurement; and

using a trained machine learning algorithm, determine whether each of said list of candidate books is similar to said target book based at least in part on the final similarity measurement.

15. The electronic system of claim 14 further configured to: compute a term frequency-inverse document frequency representation of a word count vector.

16. The electronic system of claim 15 further configured to:

compute a hash of said term frequency-inverse document frequency representation of said word count vector.

17. The electronic system of claim 16 wherein said hash comprises a MunnurHash function.

18. The electronic system of claim 14 further configured to:

determine whether two books are similar based on a comparison of more than one type of said similarity scores by using a Jaccard similarity.