METHOD AND SYSTEM FOR DISCOVERING RELATED BOOKS BASED ON BOOK CONTENT
System and method for determining book similarities based on text content and thereby discovering related books for recommending to customer-users. Each book is associated with a probability distribution on a set of topics that is derived from text content of the book against the set of topics. The pair-wise distances of the probability distributions between corresponding books are computed to derive similarities thereof. The probability distributions may be generated by leveraging a text topic model that defines a set of topics, a respective set of relevant terms under each topic, and a probability distribution on each set of relevant terms. The text topic model may be automatically generated by processing content of a corpus of training books via a training process.
1 . A computer implemented method of automatically determining relatedness between titles, said method comprising:
accessing a first probability distribution on a plurality of topics, said first probability distribution derived from a content of a first title against said plurality of topics;
accessing a second probability distribution on said plurality of topics, said second probability distribution derived from a content of a second title against said plurality of topics;
computing a similarity between said first and said second probability distributions; and
determining relatedness between said first and said second title based on said similarity.
2 . The computer implemented method of claim 1 further comprising automatically deriving a text topic model, wherein said automatically deriving comprises:
accessing content of a collection of titles;
representing content of each title in said collection by a set of terms and an occurrence frequency of each term in said title;
generating a vocabulary of said collection of titles based on said representing;
generate said plurality of topics based on said vocabulary;
allocating a respective set of terms from said vocabulary under each topic of said plurality of topics; and
assigning a probability value to each term under each topic of said plurality of topics.
3 . The computer implemented method of claim 2 , where said automatically deriving further comprises:
determining a term frequency (TF) and an inverse document frequency (IDF) of each term in said vocabulary; and
pruning stop words from said vocabulary based on term frequencies and inverse document frequencies of terms in said vocabulary.
4 . The computer implemented method of claim 2 , wherein said automatically deriving further comprises deriving said text topic model in accordance with a Latent Dirichlet Allocation (LDA) method.
5 . The computer implemented method of claim 2 further comprising:
accessing said content of said first title;
determining said first probability distribution in accordance with said text topic model;
accessing said content of said second title; and
determining said second probability distribution in accordance with said text topic model.
6 . The computer implemented method of claim 5 , wherein said determining said first probability distribution comprises determining said first probability distribution in accordance with a statistical inference method.
7 . The computer implemented method of claim 1 , wherein said first probability distribution is represented by a vector specific to said first title, and wherein further each element of said vector represents a probability of said content of said first title against a respective topic of said plurality of topics.
8 . The computer implemented method of claim 1 further comprising:
determining that a total count of words in said second title is greater than a preset threshold count before said accessing said second probability distribution.
9 . The computer implemented method of claim 1 , wherein said computing comprises computing said similarity in accordance with a Hellinger distance method.
10 . A non-transitory computer-readable storage medium embodying instructions that, when executed by a processing device of a website, cause the processing device to perform a method of creating a recommendation list of books, said method comprises:
responsive to a request for discovering books related to a first book, accessing a first probability distribution with respect to a plurality of topics, wherein said first probability distribution is derived from a content of said first book against said plurality of topics;
identifying candidate books;
accessing a plurality of probability distributions with respect to said plurality of topics, wherein a respective probability distribution of said plurality of probability distributions is derived from a content of a respective candidate book;
computing a similarity between said first probability distribution and said respective probability distribution of said respective candidate book; and
presenting said respective candidate book as a book related to said first book if said similarity satisfies a predetermined similarity threshold.
11 . The non-transitory computer-readable storage medium of claim 10 , wherein said first probability distribution and said respective probability distribution are derived in accordance with a text topic model, wherein said text topic model defines said plurality of topics, a respective plurality of terms pertinent to each topic of said plurality of topics, and a weight distribution on said respective set of terms.
12 . The non-transitory computer-readable storage medium of claim 11 , wherein said method further comprises establishing said text topic model based on content of a corpus of training books, wherein said establishing comprises:
accessing content of said corpus of training books;
reducing content of each training book to a set of terms and an occurrence frequency of each of said set of terms in said training book;
generating a vocabulary of said corpus of training books based on said reducing;
generating said plurality of topics based on said vocabulary;
allocating a respective plurality of terms from said vocabulary under each of said plurality of topics; and
assigning a respective probability value to each term under each of said plurality of topics.
13 . The non-transitory computer-readable storage medium of claim 11 , wherein said generating said plurality of topics, said allocating and said assigning are performed in accordance with a Latent Dirichlet Allocation (LDA) method.
14 . The non-transitory computer-readable storage medium of claim 11 , wherein said method further comprises: generating vectors representing and said first probability distribution and said respective probability distribution in accordance with a statistical inference method.
15 . The non-transitory computer-readable storage medium of claim 11 , wherein said identifying said candidate books comprises verifying that a total count of words in said respective candidate book is greater than a preset threshold count.
16 . The non-transitory computer-readable storage medium of claim 10 , wherein said computing comprises computing said similarity in accordance with a Hellinger distance method.
17 . A website associated system comprising:
a processor;
a memory coupled to said processor and comprising instructions that, when executed by said processor, cause the processor to perform a method of recommending books based on relevancy to a first book, said method comprising:
responsive to a request for discovering books related to said first book, accessing a first probability distribution with respect to a plurality of topics, wherein said first probability distribution is derived from a content of said first book against said plurality of topics;
accessing a second probability distribution with respect to said plurality of topics, wherein said probability distribution is derived from a content of a second book against said plurality of topics;
computing a similarity between said first and said second probability distributions; and
presenting said second book as a book related to said first book on said website if said similarity satisfies predetermined recommendation criteria.
18 . The website associated system of claim 17 , wherein said first probability and said second probability distributions are derived based on a text topic model in accordance with a Gibbs sampling and variational inference process, and wherein further said text topic model specifies said plurality of topics, a respective set of terms related to each topic, and a probability distribution associated with said respective set of terms.
19 . The website associated system of claim 18 , wherein said method further comprises establishing said text topic model, wherein said establishing comprises:
accessing content of a corpus of books;
reducing content of each book in said corpus by a set of terms and an occurrence frequency of each term in each book;
generating a vocabulary of said corpus of books based on said reducing;
generating said plurality of topics based on said vocabulary;
allocating a respective set of terms from said vocabulary under each topic of said plurality of topics; and
assigning a probability value to each term under each topic of said plurality of topics.
20 . The website associated system of claim 19 , wherein said method further comprises: removing stop words from said vocabulary in accordance with a Term Frequency-Inverse Document Frequency (IDF) method.