IP Library › Granted Patent US 12,353,968
Granted Patent B2
US 12,353,968 · App. 17/670,662 · Granted Jul 8, 2025

Methods and systems for generating training data for computer-executable machine learning algorithm within a computer-implemented crowdsource environment

Inventors: Nikita Vitalevich Pavlichenko (Moscow, RU); Valentina Pavlovna Fedorova (Sergiev Posad, RU); Valentin Andreevich Biryukov (Orenburg, RU)
Assignee: Y.E. Hub Armenia LLC
G06N20/00G06V10/764G06V10/774G06V10/776
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,353,968
App. No.
17/670,662
Granted
Jul 8, 2025
Kind
B2
Abstract

Non-limiting embodiments of the present technology are directed to a method and system for generating a training dataset. The method comprises: accessing data associated with a plurality of assessors executing digital tasks of a first type and digital tasks of a second type; generating, a first ranked list of assessors and a second ranked list of assessors based on their past performance; for a given one of the plurality of assessors: generating, a class score for the common class of digital tasks; acquiring a request for executing a digital task of a third type; ranking, the plurality of assessors based on respective class scores, the given one from the plurality of assessors being one of top ranked ones from the plurality of assessors; transmitting the digital task of the third type to the given one; generating the training data for the MLA based on a response from the given one.

Claims (48)

1. A computer-implemented method of generating training data for a computer-executable Machine Learning Algorithm (MLA), the training data being based on one or more digital tasks accessible by a plurality of assessors within a computer-implemented crowdsource environment, the method being executable by a server accessible over a communication network by electronic devices associated with the plurality of assessors, the method comprising:

accessing, by the server, assessor data associated with the plurality of assessors, the assessor data including information indicative of past performance of respective ones from the plurality of assessors when executing digital tasks of a first type and digital tasks of a second type, and wherein the assessor data comprises information indicating a difficulty of the digital tasks,

the digital tasks of the first type and the digital tasks of the second type being digital tasks of a first class of digital tasks;

generating, by the server, a first ranked list of assessors based on their past performance when executing the digital tasks of the first type, wherein generating the first ranked list comprises assigning a weighted coefficient to each previously executed task of the digital tasks of the first type based on a difficulty of the respective digital task and applying the weighted coefficient to a score of each of the previously executed digital tasks of the first type;

generating, by the server, a second ranked list of assessors based on their past performance when executing the digital tasks of the second type, wherein generating the second ranked list comprises assigning a weighted coefficient to each previously executed task of the digital tasks of the second type based on a difficulty of the respective digital task and applying the weighted coefficient to a score of each of the previously executed digital tasks of the second type;

for a given one of the plurality of assessors:

generating, by the server, a first score for the digital tasks of the first type using the first ranked list of assessors,

the first score being indicative of a past performance of the given one of the plurality of assessors when executing the digital tasks of the first type relative to the past performance of other ones from the plurality of assessors when executing the digital tasks of the first type;

generating, by the server, a second score for the digital tasks of the second type using the second ranked list of assessors,

the second score being indicative of a past performance of the given one of the plurality of assessors when executing the digital tasks of the second type relative to the past performance of other ones from the plurality of assessors when executing the digital tasks of the second type;

generating, by the server, a class score for the first class of digital tasks as a combination of the first score and the second score;

acquiring, by the server, a request for executing a digital task of a third type being different from the first type and the second type, wherein the digital task is of a second class different from the first class, and wherein the plurality of assessors have not previously completed any tasks of the second class;

ranking, by the server, the plurality of assessors based on respective class scores for the first class of digital tasks, the given one from the plurality of assessors being one of top ranked ones from the plurality of assessors;

transmitting, by the server over the communication network, the digital task of the third type to the electronic device associated with the given one from the plurality of assessors;

generating, by the server, the training data for the MLA based on a response from the given one from the plurality of assessors executing the digital task of the third type.

2. The method of claim 1 , wherein the method further comprises, for the given one of the plurality of assessors:

generating, by the server, a fourth score for the digital tasks of a fourth type, the fourth type being a digital task of a third class;

generating a class score for the third class; and

ranking the plurality of assessors comprises ranking the plurality of assessors based on the class score of the first class and the class score of the third class.

3. The method of claim 1 , wherein the first class is an image classification class, a given digital tasks of the first type being image classification of a first type of objects, a given digital tasks of the second type being image classification of a second type of objects.

4. The method of claim 1 , wherein the class score is a proxy for comparing (i) the past performance of the given one of the plurality of assessors when executing digital tasks of all types from the first class against (ii) past performance of an other given one of the plurality of assessors when executing digital tasks of all types from the first class.

5. The method of claim 1 , wherein the past performance of the given one of the plurality of assessors when executing digital tasks is a ratio of a number of correctly executed digital tasks by the given one of the plurality of assessors over a total number of digital tasks executed by the given one of the plurality of assessors.

6. The method of claim 1 , wherein the first score is a first percentile score for the past performance of the given one of the plurality of assessors relative to the past performance of other ones from the plurality of assessors when executing the digital tasks of the first type.

7. The method of claim 1 , wherein the second score is a second percentile score for the past performance of the given one of the plurality of assessors relative to the past performance of other ones from the plurality of assessors when executing the digital tasks of the second type.

8. A system for generating training data for a computer-executable machine learning algorithm (MLA), the data being based on one or more digital tasks accessible by a plurality of assessors within a computer-implemented crowdsource environment, the system comprising a server accessible over a communication network by electronic devices associated with the plurality of assessors, the server comprising a processor configured to:

access assessor data associated with the plurality of assessors, the assessor data including information indicative of past performance of respective ones from the plurality of assessors when executing digital tasks of a first type and digital tasks of a second type, and wherein the assessor data comprises information indicating a difficulty of the digital tasks,

the digital tasks of the first type and the digital tasks of the second type being digital tasks of a first class of digital tasks;

generate a first ranked list of assessors based on their past performance when executing the digital tasks of the first type, wherein generating the first ranked list comprises assigning a weighted coefficient to each previously executed task of the digital tasks of the first type based on a difficulty of the respective digital task and applying the weighted coefficient to a score of each of the previously executed digital tasks of the first type;

generate a second ranked list of assessors based on their past performance when executing the digital tasks of the second type, wherein generating the second ranked list comprises assigning a weighted coefficient to each previously executed task of the digital tasks of the second type based on a difficulty of the respective digital task and applying the weighted coefficient to a score of each of the previously executed digital tasks of the second type;

for a given one of the plurality of assessors:

generate a first score for the digital tasks of the first type using the first ranked list of assessors,

the first score being indicative of a past performance of the given one of the plurality of assessors when executing the digital tasks of the first type relative to the past performance of other ones from the plurality of assessors when executing the digital tasks of the first type;

generate a second score for the digital tasks of the second type using the second ranked list of assessors,

the second score being indicative of a past performance of the given one of the plurality of assessors when executing the digital tasks of the second type relative to the past performance of other ones from the plurality of assessors when executing the digital tasks of the second type;

generate a class score for the first class of digital tasks as a combination of the first score and the second score;

acquire a request for executing a digital task of a third type being different from the first type and the second type, wherein the digital task is of a second class different from the first class, and wherein the plurality of assessors have not previously completed any tasks of the second class;

rank the plurality of assessors based on respective class scores for the first class of digital tasks, the given one from the plurality of assessors being one of top ranked ones from the plurality of assessors;

transmit over the communication network, the digital task of the third type to the electronic device associated with the given one from the plurality of assessors;

generate the training data for the MLA based on a response from the given one from the plurality of assessors executing the digital task of the third type.

9. The system of claim 8 , wherein the processor is further configured to execute, for the given one of the plurality of assessors:

generate a fourth score for the digital tasks of a fourth type, the fourth type being a digital task of a third class;

generate a class score for the third class; and

rank the plurality of assessors comprises ranking the plurality of assessors based on the class score of the first class and the class score of the third class.

10. The system of claim 8 , wherein the first class is an image classification class, a given digital tasks of the first type being image classification of a first type of objects, a given digital tasks of the second type being image classification of a second type of objects.

11. The system of claim 8 , wherein the class score is a proxy for comparing (i) the past performance of the given one of the plurality of assessors when executing digital tasks of all types from the first class against (ii) past performance of an other given one of the plurality of assessors when executing digital tasks of all types from the first class.

12. The system of claim 8 , wherein the past performance of the given one of the plurality of assessors when executing digital tasks is a ratio of a number of correctly executed digital tasks by the given one of the plurality of assessors over a total number of digital tasks executed by the given one of the plurality of assessors.

13. The system of claim 8 , wherein the first score is a first percentile score for the past performance of the given one of the plurality of assessors relative to the past performance of other ones from the plurality of assessors when executing the digital tasks of the first type.

14. The system of claim 8 , wherein the second score is a second percentile score for the past performance of the given one of the plurality of assessors relative to the past performance of other ones from the plurality of assessors when executing the digital tasks of the second type.

Assignments (6)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 9, 2024
From: DIRECT CURSUS TECHNOLOGY L.L.C
To: Y.E. HUB ARMENIA LLC
Reel/Frame 068534/0818 →
CORRECTIVE ASSIGNMENT TO CORRECT THE PROPERTY TYPE FROM APPLICATION 11061720 TO PATENT 11061720 AND APPLICATION 11449376 TO PATENT 11449376 PREVIOUSLY RECORDED ON REEL 065418 FRAME 0705. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded Nov 8, 2023
From: YANDEX EUROPE AG
To: DIRECT CURSUS TECHNOLOGY L.L.C
Reel/Frame 065531/0493 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 1, 2023
From: YANDEX EUROPE AG
To: DIRECT CURSUS TECHNOLOGY L.L.C
Reel/Frame 065418/0705 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 16, 2023
From: PAVLICHENKO, NIKITA VITALEVICH; FEDOROVA, VALENTINA PAVLOVNA; BIRYUKOV, VALENTIN ANDREEVICH
To: YANDEX.TECHNOLOGIES LLC
Reel/Frame 062726/0050 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 16, 2023
From: YANDEX.TECHNOLOGIES LLC
To: YANDEX LLC
Reel/Frame 062726/0128 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 16, 2023
From: YANDEX LLC
To: YANDEX EUROPE AG
Reel/Frame 062726/0225 →
Priority Claims (1)
RU 2021114640 · May 24, 2021 · national
Continuity (1)
Related Publication 20220374770A1 · Nov 24, 2022
References Cited (223)
US 6560597B1 · Dhillon et al. · 2003 [cited by applicant]
US 7366705B2 · Zeng et al. · 2008 [cited by applicant]
US 7693738B2 · Guinta et al. · 2010 [cited by applicant]
US 7747083B2 · Tawde et al. · 2010 [cited by applicant]
US 8266130B2 · Jones et al. · 2012 [cited by applicant]
US 8356057B2 · Greenshpan et al. · 2013 [cited by applicant]
US 8498892B1 · Cohen et al. · 2013 [cited by applicant]
US 8554605B2 · Oleson et al. · 2013 [cited by applicant]
US 8626545B2 · Van et al. · 2014 [cited by applicant]
US 9268766B2 · Bekkerman · 2016 [cited by applicant]
US 9330071B1 · Ahmed et al. · 2016 [cited by applicant]
US 9584540B1 · Chan et al. · 2017 [cited by applicant]
US 9594944B2 · Kompalli et al. · 2017 [cited by applicant]
US 9767419B2 · Venanzi et al. · 2017 [cited by applicant]
US 9911088B2 · Nath et al. · 2018 [cited by applicant]
US 10061848B2 · Basu et al. · 2018 [cited by applicant]
US 10074200B1 · Yeturu · 2018 [cited by applicant]
US 10095688B1 · Schilling et al. · 2018 [cited by applicant]
US 10162734B1 · Podgorny et al. · 2018 [cited by applicant]
US 10192180B2 · Prabhakara et al. · 2019 [cited by applicant]
US 10445671B2 · Dubey et al. · 2019 [cited by applicant]
US 10685329B2 · Taylor et al. · 2020 [cited by applicant]
US 10762430B2 · Ehsani · 2020 [cited by examiner]
US 10978056B1 · Challa et al. · 2021 [cited by applicant]
US 11481650B2 · Bugakova et al. · 2022 [cited by applicant]
US 11900284B2 · Cuartero · 2024 [cited by examiner]
US 12148417B1 · Cardella · 2024 [cited by examiner]
US 20020032591A1 · Mahaffy et al. · 2002 [cited by applicant]
US 20030154181A1 · Liu et al. · 2003 [cited by applicant]
US 20060026240A1 · Anthony et al. · 2006 [cited by applicant]
US 20070226207A1 · Tawde · 2007 [cited by applicant]
US 20070260601A1 · Thompson et al. · 2007 [cited by applicant]
US 20080027913A1 · Chang et al. · 2008 [cited by applicant]
US 20080120129A1 · Seubert et al. · 2008 [cited by applicant]
US 20090204470A1 · Weyl et al. · 2009 [cited by applicant]
US 20100153156A1 · Guinta et al. · 2010 [cited by applicant]
US 20100293026A1 · Vojnovic et al. · 2010 [cited by applicant]
US 20110173183A1 · Dasdan et al. · 2011 [cited by applicant]
US 20110313801A1 · Biewald et al. · 2011 [cited by applicant]
US 20120005131A1 · Horvitz et al. · 2012 [cited by applicant]
US 20120130934A1 · Brillhart · 2012 [cited by examiner]
US 20120131572A1 · Shae et al. · 2012 [cited by applicant]
US 20120150579A1 · De Wit et al. · 2012 [cited by applicant]
US 20120265573A1 · Van et al. · 2012 [cited by applicant]
US 20130006717A1 · Oleson et al. · 2013 [cited by applicant]
US 20130029769A1 · Lee et al. · 2013 [cited by applicant]
US 20130096968A1 · Van Pelt et al. · 2013 [cited by applicant]
US 20130111488A1 · Gatti et al. · 2013 [cited by applicant]
US 20130132080A1 · Williams et al. · 2013 [cited by applicant]
US 20130159292A1 · Larlus et al. · 2013 [cited by applicant]
US 20130231969A1 · Van et al. · 2013 [cited by applicant]
US 20130317871A1 · Kulkarni et al. · 2013 [cited by applicant]
US 20140122188A1 · Van Pelt et al. · 2014 [cited by applicant]
US 20140172767A1 · Chen et al. · 2014 [cited by applicant]
US 20140211044A1 · Lee et al. · 2014 [cited by applicant]
US 20140214709A1 · Greaney et al. · 2014 [cited by applicant]
US 20140229413A1 · Dasgupta et al. · 2014 [cited by applicant]
US 20140278634A1 · Horvitz et al. · 2014 [cited by applicant]
US 20140321737A1 · Movellan et al. · 2014 [cited by applicant]
US 20140343984A1 · Shahabi et al. · 2014 [cited by applicant]
US 20140355835A1 · Rodriguez-Serrano et al. · 2014 [cited by applicant]
US 20150004465A1 · Basu et al. · 2015 [cited by applicant]
US 20150074033A1 · Shah et al. · 2015 [cited by applicant]
US 20150086072A1 · Kompalli et al. · 2015 [cited by applicant]
US 20150178659A1 · Dai et al. · 2015 [cited by applicant]
US 20150186784A1 · Barborak · 2015 [cited by examiner]
US 20150186951A1 · Wilson et al. · 2015 [cited by applicant]
US 20150193695A1 · Cruz Mota et al. · 2015 [cited by applicant]
US 20150213392A1 · Kittur et al. · 2015 [cited by applicant]
US 20150254593A1 · Ramos et al. · 2015 [cited by applicant]
US 20150254596A1 · Nayar et al. · 2015 [cited by applicant]
US 20150254785A1 · Yang et al. · 2015 [cited by applicant]
US 20150262111A1 · Yu et al. · 2015 [cited by applicant]
US 20150317582A1 · Nath et al. · 2015 [cited by applicant]
US 20150332188A1 · Yankelevich et al. · 2015 [cited by applicant]
US 20150347519A1 · Hornkvist et al. · 2015 [cited by applicant]
US 20150356488A1 · Eden et al. · 2015 [cited by applicant]
US 20150356489A1 · Kazai et al. · 2015 [cited by applicant]
US 20150363741A1 · Chandra et al. · 2015 [cited by applicant]
US 20160034840A1 · Venanzi et al. · 2016 [cited by applicant]
US 20160041849A1 · Naveh et al. · 2016 [cited by applicant]
US 20160100000A1 · Dey et al. · 2016 [cited by applicant]
US 20160132815A1 · Itoko et al. · 2016 [cited by applicant]
US 20160140477A1 · Karanam et al. · 2016 [cited by applicant]
US 20160162478A1 · Blassin · 2016 [cited by examiner]
US 20160162837A1 · Muntés Mulero · 2016 [cited by examiner]
US 20160180279A1 · Koerner et al. · 2016 [cited by applicant]
US 20160210570A1 · Lee et al. · 2016 [cited by applicant]
US 20160232221A1 · McCloskey et al. · 2016 [cited by applicant]
US 20160342692A1 · Bennett et al. · 2016 [cited by applicant]
US 20160035785A1 · Fan et al. · 2016 [cited by applicant]
US 20170011077A1 · Kypreos et al. · 2017 [cited by applicant]
US 20170011328A1 · Zhao · 2017 [cited by examiner]
US 20170024931A1 · Sheffer et al. · 2017 [cited by applicant]
US 20170046794A1 · Shukla et al. · 2017 [cited by applicant]
US 20170052761A1 · Gunshor et al. · 2017 [cited by applicant]
US 20170061341A1 · Haas et al. · 2017 [cited by applicant]
US 20170061356A1 · Haas et al. · 2017 [cited by applicant]
US 20170061357A1 · Dubey et al. · 2017 [cited by applicant]
US 20170076715A1 · Ohtani et al. · 2017 [cited by applicant]
US 20170103451A1 · Alipov et al. · 2017 [cited by applicant]
US 20170154313A1 · Duerr et al. · 2017 [cited by applicant]
US 20170161477A1 · Liu · 2017 [cited by examiner]
US 20170185944A1 · Volkov et al. · 2017 [cited by applicant]
US 20170200101A1 · Kumar et al. · 2017 [cited by applicant]
US 20170220973A1 · Byham et al. · 2017 [cited by applicant]
US 20170228749A1 · Larvol et al. · 2017 [cited by applicant]
US 20170293859A1 · Gusev et al. · 2017 [cited by applicant]
US 20170309193A1 · Joseph et al. · 2017 [cited by applicant]
US 20170323211A1 · Bencke et al. · 2017 [cited by applicant]
US 20170353477A1 · Faigon et al. · 2017 [cited by applicant]
US 20170364810A1 · Gusev et al. · 2017 [cited by applicant]
US 20170372225A1 · Foresti et al. · 2017 [cited by applicant]
US 20180005077A1 · Wang et al. · 2018 [cited by applicant]
US 20180143980A1 · Tanikella et al. · 2018 [cited by applicant]
US 20180144283A1 · Freitas et al. · 2018 [cited by applicant]
US 20180144654A1 · Olsen · 2018 [cited by applicant]
US 20180196579A1 · Standefer et al. · 2018 [cited by applicant]
US 20180285176A1 · Mozhaev et al. · 2018 [cited by applicant]
US 20180293325A1 · Manickavasagam · 2018 [cited by applicant]
US 20180331897A1 · Zhang et al. · 2018 [cited by applicant]
US 20180357286A1 · Wang et al. · 2018 [cited by applicant]
US 20190095801A1 · Saillet et al. · 2019 [cited by applicant]
US 20190114320A1 · Patwardhan · 2019 [cited by examiner]
US 20190138174A1 · Deets, Jr. et al. · 2019 [cited by applicant]
US 20190258985A1 · Guastella et al. · 2019 [cited by applicant]
US 20190266289A1 · Bartholomew · 2019 [cited by applicant]
US 20190318291A1 · Diriye et al. · 2019 [cited by applicant]
US 20190392547A1 · Katouzian et al. · 2019 [cited by applicant]
US 20200027052A1 · Aiyer · 2020 [cited by applicant]
US 20200089684A1 · Gotmanov et al. · 2020 [cited by applicant]
US 20200175383A1 · Blumenfeld et al. · 2020 [cited by applicant]
US 20200327582A1 · Fedorova et al. · 2020 [cited by applicant]
US 20200364282A1 · Kent et al. · 2020 [cited by applicant]
US 20200372338A1 · Woods et al. · 2020 [cited by applicant]
US 20200380410A1 · Drutsa · 2020 [cited by applicant]
US 20210042530A1 · Kim · 2021 [cited by examiner]
US 20210073596A1 · Bezzubtseva et al. · 2021 [cited by applicant]
US 20210133606A1 · Bugakova et al. · 2021 [cited by applicant]
US 20220277362A1 · Prendki · 2022 [cited by examiner]
US 20220292432A1 · Biryukov · 2022 [cited by examiner]
US 20240370793A1 · Smith · 2024 [cited by examiner]
CN 103914478A · 2014 [cited by applicant]
CN 104463424A · 2015 [cited by applicant]
CN 105608318A · 2016 [cited by applicant]
CN 106203893A · 2016 [cited by applicant]
CN 106327090A · 2017 [cited by applicant]
CN 106446287A · 2017 [cited by applicant]
CN 106557891A · 2017 [cited by applicant]
CN 107767055A · 2018 [cited by applicant]
CN 107767058A · 2018 [cited by applicant]
CN 107909262A · 2018 [cited by applicant]
CN 104794573B · 2018 [cited by applicant]
CN 109272003A · 2019 [cited by applicant]
CN 109376237A · 2019 [cited by applicant]
CN 109522545A · 2019 [cited by applicant]
CN 109544504A · 2019 [cited by applicant]
CN 109670727A · 2019 [cited by applicant]
CN 110020098A · 2019 [cited by applicant]
CN 110503396A · 2019 [cited by applicant]
CN 110909880A · 2020 [cited by applicant]
CN 110928764A · 2020 [cited by applicant]
CN 111191952A · 2020 [cited by applicant]
CN 111291973A · 2020 [cited by applicant]
CN 111723930A · 2020 [cited by applicant]
CN 109271537B · 2021 [cited by applicant]
EP 3438897A1 · 2019 [cited by applicant]
KR 102155790B1 · 2020 [cited by applicant]
KR 102156582B1 · 2020 [cited by applicant]
RU 2632143C1 · 2017 [cited by applicant]
RU 2672171C1 · 2018 [cited by applicant]
RU 2717787C1 · 2020 [cited by applicant]
RU 2744032C2 · 2021 [cited by applicant]
RU 2744038C2 · 2021 [cited by applicant]
WO 0010296A2 · 2000 [cited by applicant]
WO 2017116931A2 · 2017 [cited by applicant]
WO 2017222738A1 · 2017 [cited by applicant]
WO 2019200780A1 · 2019 [cited by applicant]
Zhiyao Li, et al., A unified task recommendation strategy for realistic mobile crowdsourcing system, Theoretical Computer Science, vol. 857, 2021, pp. 43-58, ISSN 0304-3975, https://doi.org/10.1016/j.tcs.2020.12.034, (h… [cited by examiner]
Karim et al. Learn or Earn? Intelligent Task Recommendations for Competitive Crowdsourced Software Development, Proceedings of the 51st Hawaii International Conference on System Sciences, 2018, pp. 5604-5613, URI: http:… [cited by examiner]
Vaibhav B. Sinha et al., “Fast Dawid-Skene: A Fast Vote Aggregation Scheme for Sentiment Classification”, Department of Computer Science and Engineering, Indian Institute of Technology Hyderabad, Telangana, Sep. 7, 2018… [cited by applicant]
Hongwei Li et al., “Error Rate Bounds in Crowdsourcing Models”, Department of Statistics, UC Berkeley, Department of EECS, UC Berkeley and Microsoft Research, Redmond, Jul. 10, 2013. https://arxiv.org/pdf/1307.2674.pdf. [cited by applicant]
Hongwei Li et al., “Error Rate Bounds and Iterative Weighted Majority Voting for Crowdsourcing”, University of California, Berkeley, Nov. 15, 2014; https://arxiv.org/pdf/1411.4086.pdf. [cited by applicant]
Hideaki et al., “Analysis of Minimax Error Rate for Crowdsourcing and Its Application to Worker Clustering Model”, https://arxiv.org/pdf/1802.04551.pdf Jun. 9, 2018. [cited by applicant]
Changbo et al., “Online Crowdsourcing”, https://arxiv.org/abs/1512.02393, Submitted on Dec. 8, 2015 (v1), last revised Feb. 8, 2019 (this version, v2). [cited by applicant]
Vikas et al., “Eliminating Spammers and Ranking Annotators for Crowdsourced Labeling Tasks”, Journal of Machine Learning Research 13 (2012) 491-518 ; http://www.jmlr.org/papers/volume13/raykar12a/raykar12a.pdf. [cited by applicant]
Feldman et al., “Behavior-Based Quality Assurance in Crowdsourcing Markets”, Zurich Open Repository and Archive, University of Zurich, 2014. https://www.zora.uzh.ch/id/eprint/98779/1/Feldman.pdf. [cited by applicant]
Lease, “On Quality Control and Machine Learning in Crowdsourcing”, School of Information, University of Texas at Austin; 2011, https://www.ischool.utexas.edu/˜ml/papers/lease-hcomp11.pdf. [cited by applicant]
Gadiraju, “Understanding Malicious Behavior in Crowdsourcing Platforms: The Case of Online Surveys”, http://eprints.whiterose.ac.uk/95877/1/Understanding%20malicious%20behaviour.pdf; 2015 https://doi.org/10.1145/2702123… [cited by applicant]
Carsten Eickhoff, “Cognitive Biases in Crowdsourcing”, Dept. of Computer Science, Zurich, Switzerland, 2018, https://brown.edu/Research/AI/files/pubs/wsdm18.pdf. [cited by applicant]
ECE Kamar, “Identifying and Accounting for Task-Dependent Bias in Crowdsourcing”, Microsoft Research, Redmond, WA, USA, 2015. http://erichorvitz.com/hcomp_2015_learning_bias.pdf. [cited by applicant]
D. Sánchez-Charles, “Worker ranking determination in crowdsourcing platforms using aggregation functions,” 2014 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), Beijing, 2014, pp. 1801-1808. [cited by applicant]
Khazankin, “QOS-Based Task Scheduling in Crowdsourcing Environments”, Distributed Systems Group, Vienna University of Technology, Argentinierstrasse 8/184-1, A-1040 Vienna, Austria, 2011. [cited by applicant]
Yuen, “Task recommendation in crowdsourcing systems”, Crowdkdd '12 Proceedings of the First International Workshop on Crowdsourcing and Data Mining, pp. 22-26, Beijing, China, Aug. 2012. [cited by applicant]
Ustalov “Towards the Automated Collaborative Process for Language Resource Construction”, Inzhenernyy vestnik Dona Journal, Issue No. 1(48), Published Mar. 20, 2018. [cited by applicant]
Yu, “Software Crowdsourcing Task Allocation Algorithm Based on Dynamic Utility”, IEEE Access ( vol. 7) pp. 33094-33106, Published Mar. 13, 2019. [cited by applicant]
Huang, “Enhancing reliability using peer consistency evaluation in human computation”. Published Mar. 18, 2013 in CSCW 2013—Proceedings of the 2013 ACM Conference on Computer Supported Cooperative Work (pp. 639-647). (P… [cited by applicant]
Qiu, “CrowdEval: A Cost-Efficient Strategy to Evaluate Crowdsourced Worker's Reliability”, AAMAS '18: Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, Jul. 2018, pp. 1486-149… [cited by applicant]
Hung, “An Evaluation of Aggregation Techniques in Crowdsourcing”, Web Information Systems Engineering—WISE 2013, 2013, vol. 8181, ISBN : 978-3-642-41153-3. [cited by applicant]
Office Action issued in respect of the related U.S. Appl. No. 16/852,512. [cited by applicant]
P. Radha et al. “An EREC framework for e-contract modeling, enactment and monitoring”, published on Oct. 2004, Data & Knowledge Engineering, vol. 51, Issue 1, pp. 31-58, https://doi.org/10.1016/j.datak.2004.03.006. [cited by applicant]
Thorsten et al. “A Collaborative Document Management Environment for Teaching and Learning”, published on Jan. 2000, CVE, San Francisco, pp. 197-198 DOI:10.1145/351006.351044. [cited by applicant]
Office Action issued on May 12, 2022 in respect of the related U.S. Appl. No. 16/832,095. [cited by applicant]
Notice of Allowance dated May 16, 2022 received in respect of a related U.S. Appl. No. 16/777,790. [cited by applicant]
Notice of Allowance issued on Jun. 1, 2022 in respect of the related U.S. Appl. No. 16/503,977. [cited by applicant]
Li et al.,“Crowdsourced Data Management: A Survey”, Published on Apr. 19, 2017, IEEE Transactions on Knowledge and Data Engineering, pp. 1-23, DOI:10.1109/ICDE.2017.26. [cited by applicant]
Federova et al.,“Latent Distribution Assumption for Unbiased and Consistent Consensus Modelling”, Published on Jun. 20, 2019, arXiv:1906.08776v1. [cited by applicant]
Bugakova et al.,“Aggregation of pairwise comparisons with reduction of biases”, Published on Jun. 9, 2019, arXiv:1906.03711v1. [cited by applicant]
Simpson et al., “Scalable Bayesian Preference Learning for Crowds”, Published on Dec. 11, 2019 arXiv:1912.01987v2. [cited by applicant]
Notice of Allowance dated Jun. 2, 2022 received in respect of a related U.S. Appl. No. 16/906,074. [cited by applicant]
Office Action issued on Sep. 20, 2023 in respect of the related U.S. Appl. No. 17/239,467. [cited by applicant]
Russian Search Report dated Mar. 22, 2024 issued in respect of the counterpart Russian Patent Application No. 2021106657. [cited by applicant]
Russian Search Report dated Aug. 23, 2024 issued in respect of the counterpart Russian Patent Application No. 2022129234. [cited by applicant]
Russian Search Report dated Jan. 18, 2024 issued in respect of the counterpart Russian Patent Application No. 2021127059. [cited by applicant]
Russian Search Report dated Oct. 4, 2024 issued in respect of the related Russian Patent Application No. 2023105639. [cited by applicant]
Chittilappilly et al.,“A Survey of General-Purpose Crowdsourcing Techniques”, Published on Sep. 1, 2016, IEEE Transactions on Knowledge and Data Engineering, vol. 28, Issue: 9, pp. 2246-2266 (Year: 2016). [cited by applicant]
Notice of Allowance dated Aug. 31, 2022 received in respect of a related U.S. Appl. No. 16/852,512. [cited by applicant]
Hettiachchi et al.,“A Survey on Task Assignment in Crowdsourcing”, Published on Nov. 15, 2021, pp. 1-36, https://arxiv.org/pdf/2111.08501.pdf. [cited by applicant]
List et al.,“An Evaluation of Conceptual Business Process Modelling Languages”, Published on Apr. 23, 2006, SAC'06, pp. 1532-1539, https://doi.org/10.1145/1141277.1141633. [cited by applicant]
Schnitzer et al.,“Demands on task recommendation in crowdsourcing platforms—the worker's perspective”, Sep. 2015, Crowd Rec 19, 2015, pp. 1-6. [cited by applicant]
Notice of Allowance dated Feb. 15, 2023 received in respect of a related U.S. Appl. No. 16/832,095. [cited by applicant]
Sajadmanesh et al.,“NP-GLM: A Non-Parametric Method for Temporal Link Prediction”, Published on Jun. 21, 2017, Department of Computer Engineering, Sharif University of Technology, Tehran, Iran, pp. 1-7. [cited by applicant]
Office Action dated Apr. 12, 2023 received in respect of a related U.S. Appl. No. 17/239,467. [cited by applicant]