IP Library › Granted Patent US 12,657,478
Granted Patent B2
US 12,657,478 · App. 18/750,363 · Granted Jun 16, 2026

Automated dynamic data quality assessment

Inventors: Mark Thomas Daly (San Francisco, CA); Shawn Ryan Jeffery (Burlingame, CA); Matthew DeLand (San Francisco, CA); Nick Pendar (San Ramon, CA); Andrew James (Los Altos, CA); David Johnston (Portola Valley, CA)
Assignee: ByteDance Inc.
G06N5/02G06F16/215G06F16/2358G06F16/2365G06N20/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,657,478
App. No.
18/750,363
Granted
Jun 16, 2026
Kind
B2
Abstract

In general, embodiments of the present invention provide systems, methods and computer readable media for automated dynamic data quality assessment. One aspect of the subject matter described in this specification includes the actions of receiving a data quality job including a new data sample; and, if the new data sample is determined to be added to a reservoir of data samples, sending a quality verification request to an oracle; receiving a new data sample quality estimate from the oracle; and adding the new data sample and estimate to the reservoir. A second aspect of the subject matter includes the actions of receiving, from a predictive model, a judgment associated with a new data sample; analyzing the new data sample based in part on the judgment to determine whether to send a new data sample quality verification request to an oracle; and, if a new data sample quality estimate is received from the oracle, determining whether to add the new data sample and the judgment to the reservoir.

Claims (34)

1 . A system, comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to:

identify a data sample associated with a machine learning model and a corresponding oracle of a plurality of oracles, wherein the corresponding oracle is identified based at least in part on an oracle identifier included in configuration data associated with the data sample; and

send the data sample to the corresponding oracle, wherein the corresponding oracle is configured to add the data sample to a reservoir of data samples based at least in part on a data quality estimate associated with the data sample.

2 . The system of claim 1 , wherein the one or more storage devices store instructions that are operable, when executed by the one or more computers, to further cause the one or more computers to:

identify the corresponding oracle based at least in part on one or more of types of data associated with the data sample.

3 . The system of claim 1 , wherein the one or more storage devices store instructions that are operable, when executed by the one or more computers, to further cause the one or more computers to:

identify the corresponding oracle based at least in part on one or more attributes associated with the data sample.

4 . The system of claim 1 , wherein the one or more storage devices store instructions that are operable, when executed by the one or more computers, to further cause the one or more computers to:

identify the corresponding oracle based at least in part on a verified quality measure for the data sample.

5 . The system of claim 1 , wherein the one or more storage devices store instructions that are operable, when executed by the one or more computers, to further cause the one or more computers to:

obtain the configuration data from a data stream associated with the data sample.

6 . The system of claim 1 , wherein the one or more storage devices store instructions that are operable, when executed by the one or more computers, to further cause the one or more computers to:

obtain the configuration data from a data queue configured to store the data sample and the configuration data.

7 . A computer-implemented method, comprising:

identifying a data sample associated with a machine learning model and a corresponding oracle of a plurality of oracles, wherein the corresponding oracle is identified based at least in part on an oracle identifier included in configuration data associated with the data sample; and

send the data sample to the corresponding oracle, wherein the corresponding oracle is configured to add the data sample to a reservoir of data samples based at least in part on a data quality estimate associated with the data sample.

8 . The computer-implemented method of claim 7 , wherein the corresponding oracle is identified based at least in part on one or more of types of data associated with the data sample.

9 . The computer-implemented method of claim 7 , wherein the corresponding oracle is identified based at least in part on one or more attributes associated with the data sample.

10 . The computer-implemented method of claim 7 , wherein the corresponding oracle is identified based at least in part on a verified quality measure for the data sample.

11 . The computer-implemented method of claim 7 , further comprising:

obtaining the configuration data from a data stream associated with the data sample.

12 . The computer-implemented method of claim 7 , further comprising:

obtaining the configuration data from a data queue configured to store the data sample and the configuration data.

13 . A computer program product, stored on a non-transitory computer readable medium, comprising instructions that when executed by one or more computers cause the one or more computers to:

identify a data sample associated with a machine learning model and a corresponding oracle of a plurality of oracles, wherein the corresponding oracle is identified based at least in part on an oracle identifier included in configuration data associated with the data sample; and

send the data sample to the corresponding oracle, wherein the corresponding oracle is configured to add the data sample to a reservoir of data samples based at least in part on a data quality estimate associated with the data sample.

14 . The computer program product of claim 13 , further comprising instructions that when executed by the one or more computers cause the one or more computers to:

identify the corresponding oracle based at least in part on one or more of types of data associated with the data sample.

15 . The computer program product of claim 13 , further comprising instructions that when executed by the one or more computers cause the one or more computers to:

identify the corresponding oracle based at least in part on one or more attributes associated with the data sample.

16 . The computer program product of claim 13 , further comprising instructions that when executed by the one or more computers cause the one or more computers to:

identify the corresponding oracle based at least in part on a verified quality measure for the data sample.

17 . The computer program product of claim 13 , further comprising instructions that when executed by the one or more computers cause the one or more computers to:

obtain the configuration data from a data stream associated with the data sample.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 20, 2024
From: DALY, MARK THOMAS; DELAND, MATTHEW; PENDAR, NICK; JAMES, ANDREW; JOHNSTON, DAVID; JEFFERY, SHAWN RYAN
To: GROUPON, INC.
Reel/Frame 069336/0854 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 10, 2024
From: GROUPON, INC.
To: BYTEDANCE INC.
Reel/Frame 068538/0001 →
Continuity (6)
Continuation 17684935 · Mar 2, 2022
Continuation 16433762 · Jun 6, 2019
Continuation 15619786 · Jun 12, 2017
Continuation 15161495 · May 23, 2016
Continuation 14088248 · Nov 22, 2013
Related Publication 20240419981A1 · Dec 19, 2024
References Cited (47)
US 6112210A · Nori et al. · 2000 [cited by applicant]
US 6484177B1 · Van et al. · 2002 [cited by applicant]
US 7007275B1 · Hanson et al. · 2006 [cited by applicant]
US 7134024B1 · Binding et al. · 2006 [cited by applicant]
US 8285656B1 · Chang et al. · 2012 [cited by applicant]
US 8849736B2 · Miranda et al. · 2014 [cited by applicant]
US 9122710B1 · Jeffery et al. · 2015 [cited by applicant]
US 9152727B1 · Balducci et al. · 2015 [cited by applicant]
US 9158805B1 · Kalki · 2015 [cited by examiner]
US 9235652B1 · Jeffery et al. · 2016 [cited by applicant]
US 9390112B1 · Daly et al. · 2016 [cited by applicant]
US 9465857B1 · Deland et al. · 2016 [cited by applicant]
US 9600776B1 · Daly et al. · 2017 [cited by applicant]
US 9703823B2 · Daly et al. · 2017 [cited by applicant]
US 10262277B2 · Daly et al. · 2019 [cited by applicant]
US 10657457B1 · Jeffery · 2020 [cited by examiner]
US 20030149676A1 · Kasabov · 2003 [cited by applicant]
US 20040205482A1 · Basu et al. · 2004 [cited by applicant]
US 20050203978A1 · Abraham · 2005 [cited by applicant]
US 20050278309A1 · Evans et al. · 2005 [cited by applicant]
US 20060020509A1 · Strain et al. · 2006 [cited by applicant]
US 20060112130A1 · Lowson · 2006 [cited by applicant]
US 20080065630A1 · Luo et al. · 2008 [cited by applicant]
US 20090228445A1 · Gangal · 2009 [cited by applicant]
US 20100036806A1 · Lam et al. · 2010 [cited by applicant]
US 20130111005A1 · Chu et al. · 2013 [cited by applicant]
US 20130124958A1 · Mendelovich et al. · 2013 [cited by applicant]
US 20140025641A1 · Kumarasamy et al. · 2014 [cited by applicant]
US 20140047351A1 · Cui et al. · 2014 [cited by applicant]
US 20140059561A1 · Grasselt et al. · 2014 [cited by applicant]
US 20140237450A1 · Levy et al. · 2014 [cited by applicant]
US 20140279934A1 · Li et al. · 2014 [cited by applicant]
US 20150046092A1 · Chok et al. · 2015 [cited by applicant]
US 20160162507A1 · Gupta et al. · 2016 [cited by applicant]
US 20170024427A1 · Daly et al. · 2017 [cited by applicant]
US 20240419981A1 · Daly · 2024 [cited by examiner]
U.S. Appl. No. 17/684,935, filed Mar. 2, 2022, now U.S. Pat. No. 12,045,732, Issued. [cited by applicant]
U.S. Appl. No. 16/433,762, filed Jun. 6, 2019, now U.S. Pat. No. 11,295,215, Issued. [cited by applicant]
U.S. Appl. No. 15/619,786, filed Jun. 12, 2017, now U.S. Pat. No. 10,360,516, Issued. [cited by applicant]
U.S. Appl. No. 15/161,495, filed May 23, 2016, now U.S. Pat. No. 9,703,823, Issued. [cited by applicant]
U.S. Appl. No. 14/088,248, filed Nov. 22, 2013, now U.S. Pat. No. 9,930,112, Issued. [cited by applicant]
Afrati et al., “Adaptive-Sampling Algorithms for Answering Aggregation Queries on Web Sites,” ScienceDirect Data & Knowledge Engineering 64 (2008) pp. 462-490. (Year: 2008). [cited by applicant]
Burr Settles, “Active Learning Literature Survey”, Computer Sciences Technical Report 1648, University of Wisconsin-Madison, Jan. 26, 2010, pp. 1-67. [cited by applicant]
Donmex et al., “Proactive Learning: Cost-Sensetive Active Learning with Multiple Imperfect Oracles,” dl.acm.org. CIKM'08, Oct. 26-30, 2008, ACM, pp. 629-638. (Year: 2008). [cited by applicant]
Goldberg et al., “A Dynamic Oracle for Arc-Eager Dependency Parsing,” Proceedings of COLING 2012: Technical Papers, pp. 959-976, 2012. (Year: 2012). [cited by applicant]
Russel J Ryan, “Groundtruth Budgeting: A Novel Approach to Semi-Supervised Relation Extraction in Medical Language”, 2009, MIT, pp. 1-69 (Year: 2009). [cited by applicant]
Settles , :Active Learning Literature Survey, Computer Sciences Technical Report 1648, University of Wissconsin-Madison, pp. 1-67, (Jan. 26, 2010). [cited by applicant]