Holistic student assessment framework based on multi-task learning
The present disclosure relates to a method of predicting a user's score on a question by an electronic device. The method includes: training a DP-multi tasking learning (DP-MTL) model; verifying the DP-MTL model; receiving choice selection information related to the question from the user through the terminal, and predicting 1) a probability that the user answers the question correctly and 2) the user's score related to the question using the verified DP-MTL model based on the choice selection information, and the DP-MTL model may be a model for predicting the user's score based on 1) information on whether the user answers the question correctly, 2) information on which incorrect answer is selected among choices of the question when the user selects an incorrect answer, and 3) a skill level of the user.
1 . A method of predicting a user's score on a question by an electronic device including an artificial intelligence (AI) processor and a memory storing a dichotomous-polytomous (DP)-multi tasking learning (DP-MTL) model and non-transitory computer readable instructions executable by the AI processor, the method performed by the AI processor and comprising:
training the DP-MTL model;
verifying the DP-MTL model;
receiving choice selection information related to the question from the user through a terminal; and
predicting 1) a probability that the user answers the question correctly and 2) the user's score related to the question using the verified DP-MTL model based on the choice selection information,
wherein the DP-MTL model is a model for predicting the user's score based on 1) information on whether the user answers the question correctly, 2) information on which incorrect answer is selected among choices of the question when the user selects an incorrect answer, and 3) a skill level of the user,
wherein the DP-MTL model is a multi-task learning framework that assesses an individual through tasks including (i) knowledge tracing (KT) for providing a binary classification of accuracy on a question for the individual, (ii) option tracing (OT) for predicting a choice that the individual is more likely choose among choices of the question as an answer and (iii) score prediction (SP),
wherein the training of the DP-MTL model comprises:
inputting, to the DP-MTL model, training dataset including masked data about which choice is chosen among choices of each question of a plurality of questions for each user of a plurality of users, and applying the training dataset to the DP-MTL model,
wherein the training of the DP-MTL model is based on the following Equation:
L
DP
(
θ
u
,
a
i
)
=
λ
L
D
+
(
1
-
λ
)
L
P
=
∑
i
,
u
[
λ
[
x
i
,
u
log
[
P
(
o
u
,
i
❘
θ
u
,
a
i
)
]
+
(
1
-
x
i
,
u
)
log
[
∑
o
u
,
i
≠
o
i
*
P
(
o
u
,
i
❘
θ
u
,
a
i
)
]
]
+
(
1
-
λ
)
log
[
P
(
o
u
,
i
❘
θ
u
,
a
i
)
]
]
,
the u denotes a parameter indicating the user,
the θ denotes a parameter indicating the skill level of the user, and
the a i denotes an item parameter constituting an i-th question, and
wherein the verifying of the DP-MTL model comprises training the DP-MTL model while reducing a size of the training dataset to verify how effectively the DP-MTL model processes the tasks.
2 . The method of claim 1 , wherein, in the training of the DP-MTL model, the a i and the θ are calculated to minimize the (L DP (θ u , a i )).
3 . The method of claim 1 , wherein, in the verifying of the DP-MTL model, the λ is verified to minimize the (L DP (θ u , a i )).
4 . An electronic device for predicting a user's score on a question, the electronic device comprising:
a communication module configured to communicate with a terminal;
a memory storing a dichotomous-polytomous (DP)-multi tasking learning (DP-MTL) model and non-transitory computer readable instructions; and
an artificial intelligence (AI) processor,
wherein the AI processor is configured to execute the instructions to:
train the DP-MTL model,
verify the DP-MTL model,
receive choice selection information related to the question from the user through the terminal, and
predict 1) a probability that the user answers the question correctly and 2) the user's score related to the question using the verified DP-MTL model based on the choice selection information,
wherein the DP-MTL model is a model for predicting the user's score based on 1) information on whether the user answers the question correctly, 2) information on which incorrect answer is selected among choices of the question when the user selects an incorrect answer, and 3) a skill level of the user,
wherein the DP-MTL model is a multi-task learning framework that assesses an individual through tasks including (i) knowledge tracing (KT) for providing a binary classification of accuracy on a question for the individual, (ii) option tracing (OT) for predicting a choice that the individual is more likely choose among choices of the question as an answer, and (iii) score prediction (SP),
wherein the training of the DP-MTL model comprises:
inputting, to the DP-MTL model, training dataset including masked data about which choice is chosen among choices of each question of a plurality of questions for each user of a plurality of users, and applying the training dataset to the DP-MTL model,
wherein the training of the DP-MTL model is based on the following Equation:
L
DP
(
θ
u
,
a
i
)
=
λ
L
D
+
(
1
-
λ
)
L
P
=
∑
i
,
u
[
λ
[
x
i
,
u
log
[
P
(
o
u
,
i
❘
θ
u
,
a
i
)
]
+
(
1
-
x
i
,
u
)
log
[
∑
o
u
,
i
≠
o
i
*
P
(
o
u
,
i
❘
θ
u
,
a
i
)
]
]
+
(
1
-
λ
)
log
[
P
(
o
u
,
i
❘
θ
u
,
a
i
)
]
]
,
the u denotes a parameter indicating the user,
the θ denotes a parameter indicating the skill level of the user, and
the a i denotes an item parameter constituting an i-th question, and
wherein the verifying of the DP-MTL model comprises training the DP-MTL model while reducing a size of the training dataset to verify how effectively the DP-MTL model processes the tasks.
5 . The electronic device of claim 4 , wherein the processor calculates the a i and the θ to minimize the (L DP (θ u , a i )) and trains the DP-MTL model.
6 . The electronic device of claim 4 , wherein the processor verifies the λ to minimize (L DP (θ u , a i )) in order to verify the DP-MTL model.