IP Library Granted Patent US 11,475,387
Granted Patent B2
US 11,475,387 · App. 16/852,512 · Granted Oct 18, 2022

Method and system for determining productivity rate of user in computer-implemented crowd-sourced environment

Inventors: Anna Valerevna Lioznova (Sankt-Peterburg, RU); Anastasiya Aleksandrovna Bezzubtseva (Lipetsk, RU); Alexey Valerevich Drutsa (Moscow, RU); Vladimir Vladimirovich Kukushkin (Saint Petersburg, RU)
Assignee: YANDEX EUROPE AG
G06Q10/06393G06N5/04G06N20/00G06Q10/103G06Q10/1091G06Q50/2057
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,475,387
App. No.
16/852,512
Granted
Oct 18, 2022
Kind
B2
Abstract

There is disclosed a method and system for determining a productivity rate of a user in a computer-implemented crowd-sourced environment. The method comprises, in a training phase: acquiring a training project; determining, set of project-specific features indicative of one or more characteristics of the training project; acquiring, a plurality of training results; determining a set of user-task specific features; determining, a user activity history associated with the user; generating a set of training data including the set of project-specific features, the set of user-task specific features, and the user activity history; training a machine learning algorithm (MLA), the training including: determining, a set of features representative of a property of the set of training data; and generating an inferred function based on the set of features, the inferred function being configured to determine the productivity rate of the user for a given project.

Claims (71)

1. A computer-implemented method of determining a productivity rate of a user in a computer-implemented crowd-sourced environment, the method being executed by a server, the method comprising, at a training phase:

acquiring, by the server, a training project to be executed in the computer-implemented crowd-sourced environment, the training project comprising a plurality of training tasks;

determining, by the server, a set of project-specific features indicative of one or more characteristics of the training project;

acquiring, by the server, a plurality of training results responsive to the plurality of training tasks having been submitted by the user in a chronological order;

determining, by the server, a set of user-task specific features representative of characteristics of user interactions between the user and the plurality of training results, the set of user-task specific features comprising at least a respective time stamp for each training task within the plurality of training tasks, the time stamp being indicative of a time taken for the user to submit the associated training result;

determining, by the server, a user activity history associated with the user, the user activity history including previously collected characteristics of the user interactions within the computer-implemented crowd-sourced environment;

generating, by the server, a set of training data, the set of training data including the set of project-specific features, the set of user-task specific features, and the user activity history;

training a machine learning algorithm (MLA), the training including:

determining, a set of features representative of a property of the set of training data; and

generating an inferred function based on the set of features, the inferred function being configured to determine the productivity rate of the user for a given project based on the set of project-specific features associated with the given project and the user activity history, wherein

the generating the inferred function comprises determining a learning parameter associated with the user, the learning parameter being indicative of a change in efficiency of the user to submit a given training result following a number of given training results, the learning parameter being determined based at least on an analysis of the respective time stamp of each training task within the plurality of training tasks submitted in the chronological order.

2. The method of claim 1 , further comprising, at an in-use phase:

acquiring, by the server, an in-use project for execution in the computer-implemented crowd-sourced environment, the in-use project comprising a plurality of in-use tasks;

determining, by the server, the set of project-specific features associated with the in-use project;

determining, using the MLA, the productivity rate of the user for completing the project based on the set of project-specific features associated with the in-use project and the user activity history.

3. The method of claim 2 , wherein

the server is coupled, via a communication network, to a client device associated with the user;

the acquiring the in-use project is executed prior to the client device accessing the computer-implemented crowd-sourced environment; and

the method further comprising:

transmitting, the productivity rate to the client device for display in response to the client device accessing the in-use project in the computer-implemented crowd-sourced environment.

4. The method of claim 2 , wherein

each of the training tasks included within the plurality of training tasks is a first type of task;

each of the tasks included within the plurality of in-use tasks is a second type of task, the second type of task being different from the first type of task.

5. The method of claim 2 , wherein

each of the training tasks included within the plurality of training tasks is different from a plurality of old tasks previously executed by the user in the computer-implemented crowd-sourced environment.

6. The method of claim 5 , wherein the plurality of old tasks is an entirety of previously executed tasks by the user within the computer-implemented crowd-sourced environment.

7. The method of claim 6 , wherein the plurality of training tasks being different from the plurality of old tasks allows to account for a learning effect.

8. The method of claim 1 , wherein the productivity rate of the user associated with the given project is at least one of:

a predicted time for the user to complete the given project;

a predicted hourly earning for completing the given project.

9. The method of claim 1 , wherein the determining the productivity rate of the user for the given project comprises determining the productivity rate based at least on the learning parameter of the user.

10. The method of claim 1 , wherein the set of project-specific features comprises at least one of:

a number of training tasks included within the plurality of training tasks;

a time of creation of the training project;

a number of selectable results within each training task;

user earnings per each training task; and

an instruction length of the training project.

11. The method of claim 1 , wherein the user activity history comprises at least one of:

a time of a registration of the user within the computer-implemented crowd-sourced environment;

a number of projects completed since the time of registration;

a type of an operating system of a device used by the user for submitting the plurality of training results; and

a type of the device.

12. The method of claim 1 , wherein the MLA is a gradient boosting decision trees model based MLA.

13. A system for determining a productivity rate of a user in a computer-implemented crowd-sourced environment, the system comprising a server, the server comprising a processor configured to, at a training phase:

acquire a training project to be executed in the computer-implemented crowd-sourced environment, the training project comprising a plurality of training tasks in a chronological order;

determine a set of project-specific features indicative of one or more characteristics of the training project;

acquire a plurality of training results responsive to the plurality of training tasks having been submitted by the user;

determine a set of user-task specific features representative of characteristics of user interactions between the user and the plurality of training results, the set of user-task specific features comprising at least a respective time stamp for each training task within the plurality of training tasks, the time stamp being indicative of a time taken for the user to submit the associated training result;

determine a user activity history associated with the user, the user activity history including previously collected characteristics of the user interactions within the computer-implemented crowd-sourced environment;

generate a set of training data, the set of training data including the set of project-specific features, the set of user-task specific features, and the user activity history;

train a machine learning algorithm (MLA), to train the MLA, the processor being configured to:

determine, a set of features representative of a property of the set of training data; and

generate an inferred function based on the set of features, the inferred function being configured to determine the productivity rate of the user for a given project based on the set of project-specific features associated with the given project and the user activity history, wherein

to generate the inferred function, the processor being configured to determine a learning parameter associated with the user, the learning parameter being indicative of a change in efficiency of the user to submit a given training result following a number of given training results, the learning parameter being determined based at least on an analysis of the respective time stamp of each training task within the plurality of training tasks submitted in the chronological order.

14. The system of claim 13 , wherein the processor is further configured to:

at an in-use phase:

acquire an in-use project for execution in the computer-implemented crowd-sourced environment, the in-use project comprising a plurality of in-use tasks;

determine the set of project-specific features associated with the in-use project;

determine using the MLA, the productivity rate of the user for completing the project based on the set of project-specific features associated with the in-use project and the user activity history.

15. The system of claim 14 , wherein

the server is coupled, via a communication network, to a client device associated with the user;

the processor is configured to acquire the in-use project prior to the client device accessing the computer-implemented crowd-sourced environment; and

the processor is further configured to:

transmit, the productivity rate to the client device for display in response to the client device accessing the in-use project in the computer-implemented crowd-sourced environment.

16. The system of claim 14 , wherein

each of the training tasks included within the plurality of training tasks is a first type of task;

each of the tasks included within the plurality of in-use tasks is a second type of task, the second type of task being different from the first type of task.

17. The system of claim 14 , wherein

each of the training tasks included within the plurality of training tasks is different from a plurality of old tasks previously executed by the user in the computer-implemented crowd-sourced environment.

18. The system of claim 17 , wherein the plurality of old tasks is an entirety of previously executed tasks by the user within the computer-implemented crowd-sourced environment.

19. The system of claim 18 , wherein the plurality of training tasks is different from the plurality of old tasks allows to account for a learning effect.

Assignments (6)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 9, 2024
From: DIRECT CURSUS TECHNOLOGY L.L.C
To: Y.E. HUB ARMENIA LLC
Reel/Frame 068534/0619 →
CORRECTIVE ASSIGNMENT TO CORRECT THE PROPERTY TYPE FROM APPLICATION 11061720 TO PATENT 11061720 AND APPLICATION 11449376 TO PATENT 11449376 PREVIOUSLY RECORDED ON REEL 065418 FRAME 0705. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded Nov 8, 2023
From: YANDEX EUROPE AG
To: DIRECT CURSUS TECHNOLOGY L.L.C
Reel/Frame 065531/0493 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 1, 2023
From: YANDEX EUROPE AG
To: DIRECT CURSUS TECHNOLOGY L.L.C
Reel/Frame 065418/0705 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 9, 2020
From: YANDEX, LLC
To: YANDEX EUROPE AG
Reel/Frame 053165/0055 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 19, 2020
From: YANDEX.TECHNOLOGIES LLC
To: YANDEX LLC
Reel/Frame 052436/0065 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 19, 2020
From: LIOZNOVA, ANNA VALEREVNA; BEZZUBTSEVA, ANASTASIYA ALEKSANDROVNA; DRUTSA, ALEXEY VALEREVICH; KUKUSHKIN, VLADIMIR VLADIMIROVICH
To: YANDEX.TECHNOLOGIES LLC
Reel/Frame 052436/0068 →