IP Library Granted Patent US 11,455,553
Granted Patent B2
US 11,455,553 · App. 16/693,825 · Granted Sep 27, 2022

Quality-based scoring and inhibiting of user-generated content

Inventors: Dan Pelleg (Haifa, IL); Oleg Rokhlenko (Haifa, IL); Idan Szpektor (Kfar Saba, IL); Yuval David Pinter (Binyamina-Givat Ada, IL); David Carmel (Haifa, IL); Shirin Oskooi (San Francisco, CA); Somesh Jain (Santa Clara, CA); Archit Shrivastava (Bangalore, IN)
Assignee: YAHOO ASSETS LLC
G06N5/04
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,455,553
App. No.
16/693,825
Granted
Sep 27, 2022
Kind
B2
Abstract

Methods and devices for assessing the quality of user-generated content are described. In one embodiment, a method is disclosed for measuring the quality of a user-generated answer to a question by combining various factors, including question-answer surface word vector similarity, question-answer explicit semantic analysis vector similarity, answer-answer explicit sematic analysis vector similarity, query performance predictor, sentiment analysis, textual analysis of the answer, and reputation of the answerer. The method uses a learning procedure to determine the best algorithm for measuring the overall quality of the answer based on these factors.

Claims (81)

1. A method for displaying answers to a question, comprising:

computing an overlap between a question vector associated with a question text and each of N answer vectors associated with N answer texts associated with the question text to obtain a set of N measures of question-answer similarity;

determining a quality ranking of the N answer texts based at least partially on the set of N measures of question-answer similarity; and

displaying one or more of the N answer texts on a user interface in an order based at least partially on the quality ranking.

2. The method of claim 1 , comprising determining an overlap between two explicit semantic analysis vectors by computing the cosine of the two explicit semantic analysis vectors.

3. The method of claim 1 , comprising:

querying a database containing question-answer pairs; and

receiving, from the database, the N answer texts.

4. The method of claim 1 , comprising:

querying an internet-based search engine using the question text; and

receiving, from the search engine, the N answer texts.

5. The method of claim 1 , comprising:

for each of the N answer texts, computing a corresponding subset of N−1 measures of answer-answer similarity each corresponding to an overlap between an answer vector of the each of the N answer texts and each of one or more other N−1 answer vectors; and

calculating an average of at least one subset of N−1 measures of answer-answer similarity for each of the N answer texts to obtain a set of N average measures of similarity each corresponding to a measure of similarity between each of the N answer vectors and one or more other N−1 answer vectors,

wherein the determining the quality ranking of the N answer texts is based at least partially on the set of N average measures of similarity.

6. The method of claim 5 , wherein the set of N measures of question-answer similarity are weighed more heavily than the set of N average measures of similarity in determining the quality ranking for the N answer texts.

7. The method of claim 5 , wherein the set of N measures of question-answer similarity and the set of N average measures of similarity are weighed using a predetermined proportion in determining the quality ranking of the N answer texts.

8. The method of claim 5 , wherein at least some of the set of N measures of question-answer similarity and at least some of the set of N average measures of similarity are combined to determine a quality score for each of the N answer texts in an automatic quality scoring process using a learning procedure comprising:

constructing a gold standard classification of quality for a plurality of answers of a corresponding set of questions;

fitting a model to match the gold standard classification;

comparing a quality classification of one or more answers from a plurality of classification algorithms to the gold standard classification; and

choosing a classifier for the automatic quality scoring process.

9. The method of claim 1 , comprising:

using each of the N answer texts as a search term to query a search engine;

receiving a set of documents from the search engine for each of the N answer texts;

computing a first set of N general language models for the set of documents;

using answer texts from a random question-best answer pairs as search terms to query the search engine and obtain a general corps of documents from the search engine;

computing a second general language model of the general corps; and

computing a difference between the second general language model and each of the first set of N general language models to obtain a set of N language model differences,

wherein the determining the quality ranking for the N answer texts is based at least partially on the set of N language model differences.

10. The method of claim 1 , comprising:

conducting a sentiment analysis of each of the N answer texts; and

obtaining a set of N sentiment levels each corresponding to a sentiment level of one of the N answer texts,

wherein the determining of the quality ranking for the N answer texts is based at least partially on the set of N sentiment levels.

11. The method of claim 1 , comprising:

identifying a highest quality score of the N answer texts,

wherein displaying one or more of the N answer texts on a user interface in an order based at least partially on the quality ranking comprises:

displaying, based at least partially on the quality ranking, one or more of the N answer texts having at least one quality score higher than a threshold value derived from the highest quality score of the N answer texts.

12. A community question-answering server, comprising:

a database storing N answer texts associated with a question text associated with a question vector, wherein N is an integer greater than one;

a processing unit configured to:

compute an overlap between the question vector and each of N answer vectors associated with the N answer texts to obtain a set of N measures of question-answer similarity; and

determine a quality ranking of the N answer texts based at least partially on the set of N measures of question-answer similarity; and

an output interface for causing a display of one or more of the N answer texts on a user device in an order based at least partially on the quality ranking.

13. The community question-answering server of claim 12 , the processing unit configured to:

for each of the N answer texts, compute a corresponding subset of N−1 measures of similarity each corresponding to an overlap between an answer vector of the each of the N answer texts and each of one or more other N−1 answer vectors; and

calculate an average of at least one subset of N−1 measures of answer-answer similarity for each of the N answer texts to obtain a set of N average measures of similarity each corresponding to a measure of similarity between each of the N answer vectors and one or more other N−1 answer vectors,

wherein the determining the quality ranking of the N answer texts is based at least partially on the set of N average measures of similarity.

14. The community question-answering server of claim 12 , the processing unit is configured to:

use each of the N answer texts as a search term to query a search engine;

receive a set of documents from the search engine for each of the N answer texts;

compute a first set of N general language models for the set of documents;

use answer texts from a random question-best answer pairs as search terms to query the search engine and obtain a general corps of documents from the search engine;

compute a second general language model of the general corps; and

compute a difference between the second general language model and each of the first set of N general language models to obtain a set of N language model differences,

wherein the determining the quality ranking for the N answer texts is based at least partially on the set of N language model differences.

15. The community question-answering server of claim 12 , the processing unit is configured to:

conduct a sentiment analysis of each of the N answer texts; and

obtain a set of N sentiment levels each corresponding to a sentiment level of one of the N answer texts,

wherein the determining the quality ranking for the N answer texts is based at least partially on the set of N sentiment levels.

16. The community question-answering server of claim 12 , wherein the output interface is for causing the display, on the user device and based at least partially on the quality ranking, of one or more of the N answer texts having at least one quality score higher than a threshold derived from a highest quality score among the N answer texts.

17. A non-transitory computer-readable storage medium having stored therein instructions that, when executed by a processor of a server, cause the server to:

compute an overlap between a question vector associated with a question text and each of N answer vectors associated with N answer texts associated with the question text to obtain a set of N measures of question-answer similarity;

determine a quality ranking of the N answer texts based at least partially on the set of N measures of question-answer similarity; and

display one or more of the N answer texts on a user interface in an order based at least partially on the quality ranking.

18. The non-transitory computer-readable storage medium of claim 17 , wherein the instructions, when executed by the processor of the server, cause the server to:

for each of the N answer texts, compute a corresponding subset of N−1 measures of answer-answer similarity each corresponding to an overlap between an answer vector of the each of the N answer texts and each of one or more other N−1 answer vectors; and

calculate an average of at least one subset of N−1 measures of answer-answer similarity for each of the N answer texts to obtain a set of N average measures of similarity each corresponding to a measure of similarity between each of the N answer vectors and one or more others N−1 answer vectors,

wherein the determining the quality ranking of the N answer texts is based at least partially on the set of N average measures of similarity.

19. The non-transitory computer-readable storage medium of claim 17 , wherein the instructions, when executed by the processor of the server, cause the server to:

use each of the N answer texts as a search term to query a search engine;

receive a set of documents from the search engine for each of the N answer texts;

compute a first set of N general language models for the set of documents;

use answer texts from a random question-best answer pairs as search terms to query the search engine and obtain a general corps of documents from the search engine;

compute a second general language model of the general corps; and

compute a difference between the second general language model and each of the first set of N general language models to obtain a set of N language model differences,

wherein the determining the quality ranking for the N answer texts is based at least partially on the set of N language model differences.

20. The non-transitory computer-readable storage medium of claim 17 , wherein the instructions, when executed by the processor of the server, cause the server to:

conduct a sentiment analysis of each of the N answer texts; and

obtain a set of N sentiment levels each corresponding to a sentiment level of one of the N answer texts,

wherein the determining the quality ranking for the N answer texts is based at least partially on the set of N sentiment levels.

Assignments (3)
PATENT SECURITY AGREEMENT (FIRST LIEN) Recorded Sep 29, 2022
From: YAHOO ASSETS LLC
To: ROYAL BANK OF CANADA, AS COLLATERAL AGENT
Reel/Frame 061571/0773 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 16, 2021
From: YAHOO AD TECH LLC (FORMERLY VERIZON MEDIA INC.)
To: YAHOO ASSETS LLC
Reel/Frame 058982/0282 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 26, 2020
From: OATH INC.
To: VERIZON MEDIA INC.
Reel/Frame 054258/0635 →