Quality-based scoring and inhibiting of user-generated content
View Patent ↗Methods and devices for assessing the quality of user-generated content are described. In one embodiment, a method is disclosed for measuring the quality of a user-generated answer to a question by combining various factors, including question-answer surface word vector similarity, question-answer explicit semantic analysis vector similarity, answer-answer explicit sematic analysis vector similarity, query performance predictor, sentiment analysis, textual analysis of the answer, and reputation of the answerer. The method uses a learning procedure to determine the best algorithm for measuring the overall quality of the answer based on these factors.
1. A method for displaying answers to a question, comprising:
acquiring a question text;
retrieving N answer texts corresponding to N answers to the question, wherein N is an integer greater than one;
computing a question vector representing an explicit semantic analysis vector for the question text;
computing an explicit semantic analysis vector for each of the N answer texts to obtain N answer vectors;
computing an overlap between the question vector and each of the N answer vectors to obtain a set of N measures of question-answer similarity each corresponding to a measure of similarity between the question vector and each of the N answer vectors;
determining a quality score for each of the N answer texts and a quality ranking of the N answer texts based at least partially on the set of N measures of question-answer similarity; and
displaying one or more of the N answer texts on a user interface in an order based on the quality ranking.
2. The method of claim 1 , wherein an overlap between two explicit semantic analysis vectors is determined by computing the cosine of the two explicit semantic analysis vectors.
3. The method of claim 1 , wherein retrieving N answer texts corresponding to N answers to the question comprises:
querying a database containing question-answer pairs; and
receiving, from the database, the N answer texts.
4. The method of claim 1 , wherein retrieving N answer texts corresponding to N answers to the question comprises:
querying an internet-based search engine using the question text; and
receiving, from the search engine, the N answer texts.
5. The method of claim 1 , further comprising:
for each of the N answer texts, computing a corresponding subset of N−1 measures of answer-answer similarity each corresponding to an overlap between the answer vector of the each of the N answer texts and each of the other N−1 answer vectors; and
calculating an average of the subset of N−1 measures of answer-answer similarity for each of the N answer text to obtain a set of N average measures of similarity each corresponding to a measure of similarity between each of the N answer vectors and the rest of the N−1 answer vectors,
wherein the determining of a quality score for each of the N answer texts and a quality ranking of the answer texts is further based at least partially on the set of N average measures of similarity.
6. The method of claim 5 , wherein the set of N measures of question-answer similarity are weighed more heavily than the set of N average measures of similarity in determining the quality score for each of the N answer texts and the ranking for the answer texts.
7. The method of claim 5 , wherein the first set of N measures of question-answer similarity and the set of N average measures of similarity are weighed using a predetermined proportion in determining the quality score for each the N answer texts and the quality ranking of the answer texts.
8. The method of claim 5 , wherein at least some of the set of N measures of question-answer similarity and at least some of the set of N average measures of similarity are combined to determine the quality score for each of the N answer texts in an automatic quality scoring process using a learning procedure comprising:
constructing a gold standard classification of quality for a plurality of answers of a corresponding set of questions;
fitting a model to best match the gold standard classification;
comparing a quality classification of one or more answers from a plurality of classification algorithms to the gold standard classification; and
choosing a best-performing classifier as a classifier for the automatic quality scoring process.
9. The method of claim 1 , further comprising:
using each of the N answer texts as a search term to query a search engine;
receiving a set of documents from the search engine for each of the N answer texts;
computing a first set of N general language models for the set of documents;
using answer texts from a random question-best answer pairs as search terms to query the search engine and obtain a general corps of documents from the search engine;
computing a second general language model of the general corps; and
computing a difference between the second general language model and each of the first set of N general language models to obtain a set of N language model differences,
wherein the determining of the quality score for each of the N answer texts and the quality ranking for the answer texts is further based at least partially on the set of N language model differences.
10. The method of claim 1 , further comprising:
conducting a sentiment analysis of each of the N answer texts; and
obtaining a set of N sentiment levels each corresponding to a sentiment level of one of the N answer texts,
wherein the determining of the quality score for each of the N answer texts and the quality ranking for the N answer texts is further based at least partially on the set of N sentiment levels.
11. The method of claim 1 , further comprising:
Identifying a highest quality score of the N answer texts,
wherein displaying one or more of the N answer texts on a user interface in an order based on the quality ranking comprises:
displaying, based on the quality ranking, one or more of the answer texts having at least one quality score higher than a threshold value derived from the highest quality score of the N answer texts.
12. A community question-answering server, comprising:
an input interface configured to acquire a question text from a user;
a database storing N answer texts to the question, wherein N is an integer greater than one;
a processing unit configured to:
retrieve the N answer texts;
compute a question vector representing an explicit semantic analysis vector for the question text;
compute an explicit semantic analysis vector for each of the N answer texts to obtain N answer vectors;
compute an overlap between the question vector and each of the N answer vectors to obtain a set of N measures of question-answer similarity each corresponding to a measure of similarity between the question vector and each of the N answer vectors; and
determine a quality score for each of the N answer texts and a quality ranking of the N answer texts based at least partially on the set of N measures of question-answer similarity; and
an output interface for causing a display of one or more of the N answer texts on a user device in an order based on the quality ranking.
13. The community question-answering server of claim 12 , the processing unit is further configured to:
for each of the N answer texts, compute a corresponding subset of N−1 measures of similarity each corresponding to an overlap between the answer vector of the each of the N answer texts and each of the other N−1 answer vectors; and
calculate an average of the subset of N−1 measures of answer-answer similarity for each of the N answer text to obtain a set of N average measures of similarity each corresponding to a measure of similarity between each of the N answer vectors and the rest of the N−1 answer vectors,
wherein the determining of the quality score for each of the N answer texts and the ranking of the answer texts is further based at least partially on the set of N average measures of similarity.
14. The community question-answering server of claim 12 , the processing unit is further configured to:
use each of the N answer text as a search term to query a search engine;
receive a set of documents from the search engine for each of the N answer texts;
compute a first set of N general language models for the set of documents;
use answer texts from a random question-best answer pairs as search terms to query the search engine and obtain a general corps of documents from the search engine;
compute a second general language model of the general corps; and
compute a difference between the second general language model and each of the first set of N general language models to obtain a set of N language model differences,
wherein the determining of the quality score for each of the N answer text and the quality ranking for the N answer texts is further based at least partially on the set of N language model differences.
15. The community question-answering server of claim 12 , the processing unit is further configured to:
conduct a sentiment analysis of each of the N answer texts; and
obtain a set of N sentiment levels each corresponding to a sentiment level of one of the N answer texts,
wherein the determining of the quality score for each of the N answer texts and the quality ranking for the N answer texts is further based at least partially on the set of N sentiment levels.
16. The community question-answering server of claim 12 , wherein the output interface is for causing the display, on the user device and based on the quality ranking, of one or more of the answer texts having at least one quality score higher than a threshold derived from a highest quality score among the N answer texts.
17. A non-transitory computer-readable storage medium having stored therein instructions that, when executed by a processor of a server, cause the server to:
acquire a question text;
retrieve N answer texts corresponding to N answers to the question text, wherein N is an integer greater than one;
compute a question vector representing an explicit semantic analysis vector for the question text;
compute an explicit semantic analysis vector for each of the N answer texts to obtain N answer vectors;
compute an overlap between the question vector and each of the N answer vectors to obtain a set of N measures of question-answer similarity each corresponding to a measure of similarity between the question vector and each of the N answer vectors;
determine a quality score for each of the N answer texts and a quality ranking of the N answer texts based at least partially on the set of N measures of question-answer similarity; and
display one or more of the N answer texts on a user interface in an order based on the quality ranking.
18. The non-transitory computer-readable storage medium of claim 17 , when executed by the processor of the community question-answering server, further cause the server to:
for each of the N answer texts, compute a corresponding subset of N−1 measures of answer-answer similarity each corresponding to an overlap between the answer vector of the each of the N answer texts and each of the other N−1 answer vectors; and
calculate an average of the subset of N−1 measures of answer-answer similarity for each of the N answer text to obtain a set of N average measures of similarity each corresponding to a measure of similarity between each of the N answer vectors and the rest of the N−1 answer vectors,
wherein to determine the quality score for each of the N answer texts and the quality ranking of the answer texts is further based at least partially on the set of N average measures of similarity.
19. The non-transitory computer-readable storage medium of claim 17 , when executed by the processor of the community question-answering server, further cause the server to:
use each of the N answer texts as a search term to query a search engine;
receive a set of documents from the search engine for each of the N answer texts;
compute a first set of N general language models for the set of documents;
use answer texts from a random question-best answer pairs as search terms to query the search engine and obtain a general corps of documents from the search engine;
compute a second general language model of the general corps; and
compute a difference between the second general language model and each of the first set of N general language models to obtain a set of N language model differences,
wherein to determine the quality score for each of the N answer texts and the quality ranking for the N answer texts is further based at least partially on the set of N language model differences.
20. The non-transitory computer-readable storage medium of claim 17 , when executed by the processor of the community question-answering server, further cause the server to:
conduct a sentiment analysis of each of the N answer texts; and
obtain a set of N sentiment levels each corresponding to a sentiment level of one of the N answer texts,
wherein to determine the quality score for each of the N answer texts and the quality ranking for the answer texts is further based at least partially on the set of N sentiment levels.