IP Library Granted Patent US 11,868,914
Granted Patent B2
US 11,868,914 · App. 17/942,844 · Granted Jan 9, 2024

Moderation of user-generated content

Inventors: Ashutosh Kulshreshtha (Sunnyvale, CA); Luca de Alfaro (Mountain View, CA); Mitchell Slep (San Francisco, CA); Nicu Daniel Cornea (Santa Clara, CA); Sowmya Subramanian (San Francisco, CA); Ethan G. Russell (Jersey City, NJ)
Assignee: GOOGLE LLC
G06N5/048G06F16/00G09B29/106H04W12/40
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,868,914
App. No.
17/942,844
Granted
Jan 9, 2024
Kind
B2
Abstract

A system and method for updating and correcting facts that receives proposed values for facts from users and determines a correctness score which is used to automatically accept or reject the proposed values.

Claims (38)

1. A computing system for moderating user-generated content, the computing system comprising:

one or more processors; and

one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:

obtaining a proposed user-generated content associated with a feature for inclusion in an online database;

obtaining metadata associated with a user providing the proposed user-generated content, wherein the metadata comprises information associated with previous user interactions;

accessing a reliability engine comprising one or more machine-learned models configured to determine a score indicative of a probability that unreliable information has been provided;

providing the proposed user-generated content to the one or more machine-learned models of the reliability engine;

receiving as an output of the one or more machine-learned models of the reliability engine, and in response to receipt of the proposed user-generated content, an unreliability score indicative of a probability that the user having proposed the user-generated content has provided unreliable information; and

determining to implement an action on the proposed user-generated content based on the metadata associated with the user and the unreliability score as compared to one or more thresholds.

2. The computing system of claim 1 , wherein the metadata is associated with a plurality of stored interactions of the user.

3. The computing system of claim 1 , wherein the metadata associated with the user providing the proposed user-generated content comprises data from edit logs that contain information about user interactions with the online database, each log entry of the edit logs comprising one or more of a timestamp, user ID, and type of interaction.

4. The computing system of claim 1 , wherein determining to implement an action on the proposed user-generated content based on the unreliability score as compared to one or more thresholds comprises automatically accepting or rejecting the user-generated content as part of the online database.

5. The computing system of claim 1 , wherein determining to implement an action on the proposed user-generated content based on the unreliability score as compared to one or more thresholds comprises flagging the user-generated content for manual review.

6. The computing system of claim 1 , wherein edit sessions of unreliable users are used as a training set for training the at least one of the one or more machine-learned models of the reliability engine.

7. The computing system of claim 1 , wherein the proposed user-generated content associated with the feature for inclusion in the online database comprises a new feature to be added to the online database.

8. The computing system of claim 1 , wherein the proposed user-generated content associated with the feature for inclusion in the online database comprises an updated attribute associated with an existing feature already stored in the online database.

9. The computing system of claim 1 , wherein the online database comprises edits from a plurality of sources.

10. The computing system of claim 1 , wherein the one or more machine-learned models are trained with a training dataset generated based on data from one or more manual moderation sessions.

11. The computing system of claim 10 , wherein the training dataset comprises a set of records manually identified as overclustered.

12. The computing system of claim 1 , wherein the online database comprises an online database of community pages.

13. A computer-implemented method for moderating user-generated content, the method comprising:

obtaining, by a computing system comprising one or more computing devices, a proposed user-generated value associated with a feature for inclusion in an online database of features that are accessed from a web-based application;

obtaining, by the computing system, metadata associated with a user providing the proposed user-generated value;

processing, by the computing system, the proposed user-generated value with a reliability engine to generate a value unreliability score, wherein the reliability engine comprises one or more machine-learned models, wherein the value unreliability score is indicative of a probability that unreliable information has been provided; and

determining, by the computing system, to implement an action on the proposed user-generated value based on the value unreliability score as compared to one or more thresholds and based on the metadata associated with the user.

14. The method of claim 13 , wherein the metadata comprises an edit log associated with one or more previous interactions associated with the user.

15. The method of claim 13 , wherein the value unreliability score is determined based on one or more interactions by a second user.

16. The method of claim 13 , wherein the metadata comprises accuracy data associated with one or more previous interactions with the online database.

17. The method of claim 16 , wherein the accuracy data comprises recency weighting that more heavily weights more recently proposed values.

18. A computing system for moderating user-generated content, the system comprising:

one or more processors; and

one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:

obtaining proposed user-generated content associated with a feature for inclusion in an online database of features;

obtaining metadata associated with a user providing the proposed user-generated content, wherein the metadata comprises information associated with previous user interactions with the online database of features;

processing the proposed user-generated content with a reliability engine to generate a content-based unreliability score, wherein the reliability engine comprises one or more machine-learned models, wherein the content-based unreliability score is indicative of a probability that unreliable information has been provided; and

determining to implement an action on the proposed user-generated content based on the content-based unreliability score as compared to one or more thresholds and based on the metadata associated with the user.

19. The system of claim 18 , wherein the information associated with previous user interactions comprises a frequency of editing sessions.

20. The system of claim 18 , wherein the information associated with previous user interactions comprises a number of edits made per session.

Assignments (2)
CHANGE OF NAME Recorded Sep 27, 2022
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 061552/0511 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 21, 2022
From: KULSHRESHTHA, ASHUTOSH; DE ALFARO, LUCA; SLEP, MITCHELL; CORNEA, NICU D.; SUBRAMANIAN, SOWMYA; RUSSELL, ETHAN G.
To: GOOGLE INC.
Reel/Frame 061168/0785 →
Continuity (4)
Continuation 16154377 · Oct 8, 2018
Continuation 14189937 · Feb 25, 2014
Continuation 13098342 · Apr 29, 2011
Related Publication 20230169371A1 · Jun 1, 2023