Method and system for selecting label from plurality of labels for task in crowd-sourced environment
There is disclosed a method system for selecting a label for a task, the method comprising: receiving a plurality of labels, each of the label included within the plurality of labels being indicative of a given assessor's perceived preference of a first object of over a second object; analyzing the comparison task to determine a set of latent biasing features; executing a MLA configured to generating a respective latent score parameter for the first object and the second object, the respective latent score parameter indicative of a probable offset between the given assessor's perceived preference and an unbiased preference parameter of the first object over the second object; generating a predicted bias degree parameter for the given assessor; generating the unbiased preference parameter; using, by the server, the unbiased preference parameter as the label for the comparison task for the given assessor.
1. A computer-implemented method for selecting a label from a plurality of labels received for a comparison task executed within a computer-implemented crowd-sourced environment, the method being executed by a server, the method comprising:
receiving, by the server, the plurality of labels having been selected by a plurality of assessors of the computer-implemented crowd-sourced environment, each of the label included within the plurality of labels being indicative of a given assessor's perceived preference of a first object of the comparison task over a second object of the comparison task;
analyzing, by the server, the comparison task to determine a set of latent biasing features associated with the first object and the second object, the set of latent biasing features comprising one or more latent features within the comparison task having a possibility of affecting preference perception of the first object over the second object by at least a portion of the plurality of human assessors;
training a Machine Learning Algorithm (MLA) to determine an unbiased preference parameter based on a respective latent score parameter for the first object and the second object, a predicted bias probability parameter for the given assessor and a predicted bias degree parameter for the given assessor;
executing, by the server, the MLA, wherein the MLA is configured to, for the given assessor of the plurality of human assessors a generative process comprising:
generating the respective latent score parameter for the first object and the second object, the respective latent score parameter indicative of a probable offset between the given assessor's perceived preference and an unbiased preference parameter of the first object over the second object, the probable offset being due to at least some of the set of latent biasing features;
generating the predicted bias probability parameter for the given assessor, the predicted bias probability parameter being indicative of a probability that the given assessor's perceived preference is one of biased and unbiased;
generating the predicted bias degree parameter for the given assessor, the predicted bias degree parameter being indicative of a degree of bias that the given assessor has towards the set of latent biasing features; wherein
the MLA is trained to generate the respective latent score parameters, the predicted bias probability parameter and the predicted bias degree parameter based on a logarithmic formula:
=
∑
∞
log
[
f
(
γ
k
)
f
(
s
i
-
s
j
)
+
(
1
-
f
(
γ
k
)
)
f
(
〈
x
kij
,
r
k
〉
)
]
wherein:
L is a likelihood of observed comparison under the generative process based on the latent score parameter of the objects, bias probability parameter and bias degree parameter for each assessor;
w k is the given assessor;
d i is the first object from the comparison task for the given assessor w k which was preferred over the second object d j ;
P is the plurality of labels selected by the plurality of assessors;
γ k is the predicted bias probability parameter;
s i and s j correspond to the latent score parameter of the first and second object respectively, the unbiased preference parameter being a logistic function of their difference;
x kij is the set of latent biasing features;
r k is the predicted bias degree parameter; wherein
maximizing the logarithmic formula by maximizing formula:
T
=
+
λ
R
=
∑
∞
log
[
f
(
γ
k
)
f
(
s
i
-
s
j
)
+
(
1
-
f
(
Y
k
)
)
f
(
〈
x
kij
,
r
k
〉
)
]
+
λ
∑
i
=
1
N
log
(
f
(
s
i
-
s
0
)
)
+
log
(
f
(
s
0
-
s
i
)
)
wherein:
T is a target function that is maximized;
λ is a regularization parameter;
R is a regularization term;
s o is a latent score parameter of a virtual object; and
generating the unbiased preference parameter based on the logistic function of the respective latent score parameters;
using, by the server, the unbiased preference parameter as the label for the comparison task for the given assessor.
2. The method of claim 1 , wherein the comparison task is a pairwise comparison task.
3. The method of claim 1 , wherein the analyzing the comparison task is executed prior to receiving the plurality of labels.
4. The method of claim 1 , wherein the set of latent biasing features include at least one of:
a font size associated with the first object and the second object respectively;
an image size associated with the first object and the second object respectively; and
a positioning associated with the first object and the second object respectively.
5. The method of claim 4 , the method further comprising:
for a given latent biasing feature included within the set of latent biasing feature, generating a latent feature vector, the latent feature vector being indicative of at least one of:
a presence of the given latent feature within the comparison task;
absence of the given latent feature within the comparison task.
6. The method of claim 5 , wherein the given latent biasing feature within the set of latent biasing features x kij corresponds to:
a first value if the given latent feature is present within the first object only;
a second value if the given latent feature is present within the second object only;
a third value if the given latent feature is present within both, or absent in both the first object and the second object.
7. The method of claim 1 , wherein the unbiased preference parameter is indicative of an actual preference of the given assessor that is not affected by the set of latent biasing features.
8. The method of claim 1 , further comprising aggregating one or more unbiased preference parameters each associated with a respective assessor of the plurality of human assessors.
9. The method of claim 8 , wherein the MLA is a first MLA, and the method further comprising using the aggregated one or more unbiased preference parameters for training a second MLA.
10. The method of claim 1 , wherein which one of the set of latent biasing feature has the possibility of affecting preference perception of the first object over the second object is a priori unknown.
11. A system for selecting a label from a plurality of labels received for a comparison task executed within a computer-implemented crowd-sourced environment, the system comprising a server, the server comprising a processor configured to:
receive the plurality of labels having been selected by a plurality of assessors of the computer-implemented crowd-sourced environment, each of the label included within the plurality of labels being indicative of a given assessor's perceived preference of a first object of the comparison task over a second object of the comparison task;
analyze the comparison task to determine a set of latent biasing features associated with the first object and the second object, the set of latent biasing features comprising one or more latent features within the comparison task having a possibility of affecting preference perception of the first object over the second object by at least a portion of the plurality of human assessors;
train a Machine Learning Algorithm (MLA) to determine an unbiased preference parameter based on a respective latent score parameter for the first object and the second object, a predicted bias probability parameter for the given assessor and a predicted bias degree parameter for the given assessor;
execute the MLA, the MLA being configured to, for the given assessor of the plurality of human assessors, a generative process to:
generate the respective latent score parameter for the first object and the second object, the respective latent score parameter indicative of a probable offset between the given assessor's perceived preference and an unbiased preference parameter of the first object over the second object, the probable offset being due to at least some of the set of latent biasing features;
generate the predicted bias probability parameter for the given assessor, the predicted bias probability parameter being indicative of a probability that the given assessor's perceived preference is one of biased and unbiased;
generate the predicted bias degree parameter for the given assessor, the predicted bias degree parameter being indicative of a degree of bias that the given assessor has towards the set of latent biasing features;
wherein:
the MLA is trained to generate the respective latent score parameters, the predicted bias probability parameter and the predicted bias degree parameter based on a logarithmic formula:
=
∑
∞
log
[
f
(
γ
k
)
f
(
s
i
-
s
j
)
+
(
1
-
f
(
γ
k
)
)
f
(
〈
x
kij
,
r
k
〉
)
]
wherein:
L is a likelihood of observed comparison under the generative process based on the latent score parameter of the objects, bias probability parameter and bias degree parameter for each assessor;
w k is the given assessor;
d i is the first object from the comparison task for the given assessor w k which was preferred over the second object d j ;
P is the plurality of labels selected by the plurality of assessors;
γ k is the predicted bias probability parameter;
s i and s j correspond to the latent score parameter of the first and second object respectively, the unbiased preference parameter being a logistic function of their difference;
x kij is the set of latent biasing features;
r k is the predicted bias degree parameter;
maximize the logarithmic formula by a maximizing formula:
T
=
+
λ
R
=
∑
∞
log
[
f
(
γ
k
)
f
(
s
i
-
s
j
)
+
(
1
-
f
(
Y
k
)
)
f
(
〈
x
kij
,
r
k
〉
)
]
+
λ
∑
i
=
j
N
log
(
f
(
s
i
-
s
0
)
)
+
log
(
f
(
s
0
-
s
i
)
)
wherein:
T is a target function that is maximized;
λ is a regularization parameter;
R is a regularization term;
S o is a latent score parameter of a virtual object; and
generate the unbiased preference parameter based on the logistic function of the respective latent score parameters;
use the unbiased preference parameter as the label for the comparison task for the given assessor.
12. The system of claim 11 , wherein the comparison task is a pairwise comparison task.
13. The system of claim 11 , wherein the processor is configured to analyze the comparison task prior to receiving the plurality of labels.
14. The system of claim 13 , wherein the set of latent biasing features include at least one of:
a font size associated with the first object and the second object respectively;
an image size associated with the first object and the second object respectively; and
a positioning associated with the first object and the second object respectively.
15. The system of claim 14 , the processor being further configure to:
for a given latent biasing feature included within the set of latent biasing feature, generate a latent feature vector, the latent feature vector being indicative of at least one of:
a presence of the given latent feature within the comparison task;
absence of the given latent feature within the comparison task.
16. The system of claim 15 , wherein the given latent biasing feature within the set of latent biasing features x kij corresponds to:
a first value if the given latent feature is present within the first object only;
a second value if the given latent feature is present within the second object only;
a third value if the given latent feature is present within both, or absent in both the first object and the second object.