Method and system for determining response for digital task executed in computer-implemented crowd-sourced environment

1. A method for determining a response to a digital task, the digital task executed in a computer-implemented crowd-sourced environment, the computer-implemented crowd-sourced environment being accessible by a plurality of crowd-sourced assessors, the method being executed by a server accessible, via a communication network, by electronic devices associated with the plurality of crowd-sourced assessors, the server executing the computer-implemented crowd-sourced environment, the method comprising:

acquiring, by the server, from a first subset of the plurality of crowd-sourced assessors a plurality of responses to the digital task;

determining, by the server, if a number of the plurality of responses to the digital task meets a pre-determined minimum answer threshold;

in response to the number of the plurality of responses to the digital task meeting the pre-determined minimum answer threshold, executing:

for each of the plurality of responses generating, by the server, a confidence parameter representing a probability of an associated one of the plurality of responses being correct;

ranking the plurality of responses based on the confidence parameter to determine a top response being associated with a highest confidence parameter;

in response to the highest confidence parameter being above a pre-determined minimum confidence threshold, assigning a value of the top response as a label for the digital task and terminating the digital task execution;

in response to the highest confidence parameter being below the pre-determined minimum confidence threshold, causing an additional response to be solicited from at least one additional crowd-sourced assessors of the of the plurality of crowd-sourced assessors.

2. The method of claim 1 , wherein in response to the highest confidence parameter being below the pre-determined minimum confidence threshold, the method further comprises:

checking if the number of the plurality of responses is below a pre-determined maximum number of solicited responses and in response to a positive determination, executing the causing the additional response to be solicited.

3. The method of claim 2 , wherein the method further comprises applying at least one Machine Learning Algorithm (MLA) to generate at least one of:

the pre-determined minimum answer threshold;

the pre-determined maximum number of solicited responses; and

pre-determined minimum confidence threshold.

4. The method of claim 3 , wherein the MLA is configured to optimize the at least one of the pre-determined minimum answer threshold; the pre-determined maximum number of solicited responses; and pre-determined minimum confidence threshold such that:

minimize a number of the plurality of responses to the digital task required to consider the digital task completed; and

maximize an accuracy parameter associated with the label assigned to the digital task.

5. The method of claim 1 , wherein in response to the highest confidence parameter being below the pre-determined minimum confidence threshold, the method further comprises:

checking if the number of the plurality of responses is above a pre-determined maximum number of solicited responses and in response to a positive determination, not executing the causing the additional response to be solicited; and

determining that the digital task can not be completed in the computer-implemented crowd-sourced environment.

6. The method of claim 1 , wherein the digital task is of a type having an infinite number of possible correct answers.

7. The method of claim 6 , wherein the digital task is an image recognition task.

8. The method of claim 7 , where the image is a CAPTCHA type image.

9. The method of claim 1 , wherein the digital task comprises assigning a label to a digital object.

10. The method of claim 9 , wherein the label is one of a binary label and a categorical label.

11. The method of claim 1 , wherein the generating, by the server, the confidence parameter comprises applying a Machine Learning Algorithm (MLA) to generate the confidence parameter.

12. The method of claim 11 , wherein the MLA generates the confidence parameter based on a feature vector, including a plurality of dimensions; a first dimension being associated with a given response; a second dimension being associated with an associated digital task; and a third dimension being associated other responses of the plurality of responses.

13. The method of claim 12 , wherein the first dimension includes features representative of at least one of:

whether the given response has capital characters;

whether the given response has punctuation;

whether the given response has Latin characters;

whether the given response has digits;

whether the given response has Cyrillic characters;

a Levenshtein distance (LD) between the given response and an OCR model prediction of a content of the digital task; and

a confidence for the OCR model prediction.

14. The method of claim 12 , wherein the second dimension includes features representative of at least one of:

a ratio of showing the digital task left or right of a control word, over the plurality of responses received till this moment

a fraction of inputs in the plurality of responses received till this moment when the digital task was to the right of the control word;

a difference between the ratio for showing the digital task to the left of the control word and to the right, over all the plurality of responses received till this moment;

a fraction of inputs in the plurality of responses when the digital task was to the left of the control word.

15. The method of claim 12 , wherein the third dimension includes features representative of at least one of:

a fraction of votes for a given answer the plurality of responses in the responses collected till the current moment;

a Levenshtein distance (LD) between the plurality of responses and a majority vote response

a fraction of votes for the plurality of responses to that for the majority vote response;

a number of responses for known digital tasks relative to a number of responses to unknown tasks;

a median input lime in the plurality of responses relative to the length of the plurality of responses in characters;

an average input lime for the plurality of responses relative to the length of the plurality of response in characters;

a maximal input time for the plurality of responses;

a 25th percentile input time for the plurality of responses;

a minimal input time for the plurality of responses;

a median input time for the plurality of responses;

a 25th percentile time of day for the plurality of responses;

a median lime of day for the plurality of responses;

a 75th percentile time of day for the plurality of responses;

a 75th percentile input time for the plurality of responses;

a minimum input time for the plurality of responses relative to an average time;

a minimum input time for the plurality of responses to a maximum time;

an average time of day for the plurality of responses;

an average input time for the plurality of responses.

16. The method of claim 1 , wherein the plurality of crowd-sourced assessors comprises at least one human assessor and at least one computer-based assessor.

17. The method of claim 1 , wherein the method results in a dynamic number of responses indicative of a number of the plurality of responses to the digital task required to consider the digital task completed.

18. The method of claim 1 , wherein the digital task comprises an unknown task and a control task, the control task associated with a known label, and wherein the method further comprises:

checking if a first given response to the control task matches the known label;

in response to a positive outcome of checking, processing a second given response to the unknown task;

in response to a negative outcome of checking, discarding the second given response to the unknown task.

19. A server for determining a response to a digital task, the digital task executed in a computer-implemented crowd-sourced environment executed by the server, the computer-implemented crowd-sourced environment being accessible by a plurality of crowd-sourced assessors, the method being executed by the server; the server being accessible, via a communication network, by electronic devices associated with the plurality of crowd-sourced assessors, the server being configured to:

acquire from a first subset of the plurality of crowd-sourced assessors a plurality of responses to the digital task;

determining if a number of the plurality of responses to the digital task meets a pre-determined minimum answer threshold;

in response to the number of the plurality of responses to the digital task meeting the pre-determined minimum answer threshold, execute:

for each of the plurality of responses generating a confidence parameter representing a probability of an associated one of the plurality of responses being correct;

ranking the plurality of responses based on the confidence parameter to determine a top response being associated with a highest confidence parameter;

in response to the highest confidence parameter being above a pre-determined minimum confidence threshold, assigning a value of the top response as a label for the digital task and terminating the digital task execution;

in response to the highest confidence parameter being below the pre-determined minimum confidence threshold, causing an additional response to be solicited from at least one additional crowd-sourced assessors of the of the plurality of crowd-sourced assessors.

20. A method for determining a response to a digital task, the digital task executed in a computer-implemented crowd-sourced environment, the computer-implemented crowd-sourced environment being accessible by a plurality of crowd-sourced assessors, the method being executed by a server accessible, via a communication network, by electronic devices associated with the plurality of crowd-sourced assessors, the server executing the computer-implemented crowd-sourced environment, the method comprising:

acquiring, by the server, from a first subset of the plurality of crowd-sourced assessors a plurality of responses to the digital task;

dynamically determining, by the server, if the plurality of responses is sufficient to determine the response to the digital task by:

for each of the plurality of responses generating, by the server, a confidence parameter representing a probability of an associated one of the plurality of responses being correct;

ranking the plurality of responses based on the confidence parameter to determine a top response being associated with a highest confidence parameter;

in response to the highest confidence parameter being above a pre-determined minimum confidence threshold, assigning a value of the top response as a label for the digital task and terminating the digital task execution;

in response to the highest confidence parameter being below the pre-determined minimum confidence threshold, causing an additional response to be solicited from at least one additional crowd-sourced assessors of the of the plurality of crowd-sourced assessors.