Automated tasking and accuracy assessment systems and methods for assigning and assessing individuals and tasks
View Patent ↗Disclosed is a tasking system to assign individuals to tasks. The tasking system targets a user for the task based information in a user profile such as demographics, background and reputation of the user to perform a task. The tasking system predicts if a user will do a given task well based on a user's background, user's skill, and applied predictive algorithms. The tasking system manages the workflow of the task, assesses the probability of the user's answer being accurate, and continuously optimizes assignment and assessment of the task to improve data quality.
1. A method for determining computerized tasks for assignment, the method comprising:
targeting a user to perform a task based on targeting criteria and stored information from a user profile;
predicting, based on one or more machine-learned models, success of the user to perform the task based on a past performance of the user on a similar task, wherein the one or more machine-learned models are trained based on marked data;
assigning the task to the user if the predicted success is greater than a threshold level;
managing a workflow for the task assigned to the user;
assessing if the task as completed by the user is within a predefined acceptance range;
receiving customer feedback on the task when completed by the user;
updating the one or more machine-learned models based on the task completed by the user;
identifying common characteristics of a set of users based on previous performance of the users on a set of tasks being equal to or above an acquisition threshold value, wherein the set of users comprises the user;
acquiring additional users with the common characteristics; and
adjusting the common characteristics and the acquisition threshold value based on an assessment of additional tasks as completed by one or more of the additional users.
2. The method of claim 1 , further comprising:
receiving a login from the user through an external system; and
storing external user information in the user profile of the user based on information from the external system.
3. The method of claim 1 , wherein the stored information in the user profile further comprises information on a reputation of the user based on previous performance and activities of the user.
4. The method of claim 1 , wherein the stored information in the user profile further comprises information on a reputation of the user based on previous performance and activities of the user for a given domain.
5. The method of claim 1 , wherein the stored information in the user profile further comprises information on a reputation of the user based on previous performance and activities of the user for a given task type.
6. The method of claim 1 , wherein predicting the success of the user is further based on at least one of the following: patterns in previous tasking behavior of the user, user data from outside sources, or user surveys.
7. The method of claim 1 , wherein assigning the task further comprises:
training the user for a task set, qualifying the user to perform the task set, and giving the user access to the task after the user successfully completes the training and the qualifying steps for the task set.
8. The method of claim 7 , wherein assigning the task further comprises:
adjusting a threshold level for assigning the task to the user based on a total number of qualified users identified through the training and the qualifying steps.
9. The method of claim 1 , wherein assigning the task further comprises: setting a maximum number of tasks an individual can complete.
10. The method of claim 1 , wherein managing the workflow for the task assigned to the user further comprises varying at least one of an assignment, a qualification, or ongoing quality criteria at each step of the workflow.
11. The method of claim 1 , wherein managing the workflow for the task assigned to the user further comprises manipulating the workflow to determine a tradeoff among velocity, quality, and cost.
12. The method of claim 1 , wherein the predefined acceptance range is based on at least one of: user answers, user answers weighted by user reputation, a known answer, or assessment of the completed task by a mentor, the mentor being a user with elevated status.
13. The method of claim 12 , wherein the mentor has received positive feedback on completed tasks greater than a positive feedback threshold.
14. A non-transitory computer-readable storage medium comprising stored executable computer program instructions for assigning tasks, the computer program instructions when executed by a computer processor cause the computer processor to:
target a user to perform a task based on targeting criteria and stored information from a user profile;
predict, based on one or more machine-learned models, success of the user to perform the task based on a past performance of the user on a similar task, wherein the one or more machine-learned models are trained based on marked data;
assign the task to the user if the predicted success is greater than a threshold level;
manage a workflow for the task assigned to the user;
assess if the task as completed by the user is within a predefined acceptance range;
receive customer feedback on the task when completed by the user;
update the one or more machine-learned models based on the task completed by the user;
identify common characteristics of a set of users based on previous performance of the users on a set of tasks being equal to or above an acquisition threshold value, wherein the set of users comprises the user;
acquire additional users with the common characteristics; and
adjust the common characteristics and the acquisition threshold value based on an assessment of additional tasks as completed by one or more of the additional users.
15. The computer readable medium of claim 14 , further comprising instructions that cause the processor to:
receive a login from the user through an external system; and
store external information in the user profile of the user based on information from the external system.
16. The computer readable medium of claim 14 , wherein the stored information in the user profile further comprises information on a reputation of the user based on previous performance and activities of the user.
17. The computer readable medium of claim 14 , wherein the prediction of success of the user to perform the task is further based on at least one of the following: patterns in previous tasking behavior of the user, user data from outside sources, or user surveys.
18. The computer readable medium of claim 14 , wherein the instruction to assign the task further comprises instructions that cause the processor to:
train the user for a task set, qualify the user to perform the task set, and give the user access to the task after the user successfully completes the training and the qualifying steps for the task set.
19. The computer readable medium of claim 18 , wherein the instruction to assign the task further comprises instructions that cause the processor to:
adjust the threshold level for assigning the task to the user based on a total number of qualified users identified through the training and the qualifying steps.
20. The computer readable medium of claim 14 , wherein the instruction to assign the task further comprises instructions that cause the processor to:
set a maximum number of tasks an individual can complete.
21. The computer readable medium of claim 14 , wherein the instruction to manage the workflow of the task assigned to the user further comprises instructions to vary at least one of an assignment, qualification, or ongoing quality criteria at each step of the workflow.
22. The computer readable medium of claim 14 , wherein the predefined acceptance range is based on at least one of: user answers, user answers weighted by user reputation, a known answer, or assessment of the completed task by a mentor, the mentor being a user with elevated status.
23. The computer readable medium of claim 22 , wherein the mentor has received positive feedback on completed tasks greater than a positive feedback threshold.
24. A non-transitory computer-readable storage medium comprising stored executable computer program instructions for assigning tasks, the computer program instructions when executed by a computer processor cause the computer processor to:
target a user to perform a task based on targeting criteria and stored information from a user profile;
predict, based on one or more machine-learned models, success of the user to perform the task based on a past performance of the user on a similar task, wherein the one or more machine-learned models are trained based on marked data;
assign the task to the user if the predicted success is greater than a threshold level;
manage a workflow for the task assigned to the user;
receive customer feedback on the task when completed by the user;
update the one or more machine-learned prediction models based on the task completed by the user;
identify common characteristics of a set of users based on previous performance of the users on a set of tasks being equal to or above an acquisition threshold value, wherein the set of users comprises the user;
acquire additional users with the common characteristics; and
adjust the common characteristics and the acquisition threshold value based on an assessment of additional tasks as completed by one or more of the additional users.