Electronic device for analyzing meaning of speech, and operation method therefor
An electronic device using an artificial neural network model including an attention mechanism, according to various embodiments, can comprise: a memory configured to store information including a plurality of recurrent neural network (RNN) layers; and at least one processor connected with the memory and configured to set, as a first key and a value, at least one first hidden representation acquired through at least one layer among the plurality of RNN layers, set, as a second key, at least one second hidden representation acquired through at least one second layer among the plurality of RNN layers, and acquire an attention included in an attention structure at least on the basis of data on the first key, data on the second key, or data on the value.
1. An electronic device using an artificial neural network model comprising an attention mechanism, the electronic device comprising:
a memory configured to store information comprising a plurality of recurrent neural network (RNN) layers; and
at least one processor connected with the memory,
wherein the at least one processor is configured to:
obtain first hidden representations through at least one first layer from among the plurality of RNN layers,
set the first hidden representations as a first key and a value,
obtain second hidden representations through at least one second layer from among the plurality of RNN layers,
set the second hidden representations as a second key, wherein the at least one second layer is positioned at an upper stage than the at least one first layer in the artificial neural network and the first hidden representations are used as an input of the at least one second layer, and
based on data on the first key, data on the second key, and data on the value, obtain an attention included in the attention mechanism, and
wherein a cost for the RNN is obtained based on multiplication of data indicating a cost regarding a language model regarding the RNN by a weight-decay term, in training at least one of the language model, a slot filling model regarding the RNN, or an intent detection model regarding the RNN.
2. The electronic device of claim 1 ,
wherein the at least one processor is configured to obtain the second hidden representations from the first hidden representations through the at least one second layer.
3. The electronic device of claim 2 , wherein the at least one processor is configured to obtain, from the attention, data on a slot or data on an intent, through at least one third layer positioned at a higher stage than the at least one second layer from among the plurality of RNN layers.
4. The electronic device of claim 1 , wherein the at least one processor is configured to obtain data on a weight of each of the first key and the second key, based on the data on the first key and the data on the second key, and to obtain the attention based on a weighted sum of the data on the weight and the data on the value.
5. The electronic device of claim 1 , wherein the at least one processor is configured to obtain the attention based on the following equation:
L
A
i
=
∑
j
=
1
T
x
α
ij
h
j
1
α
ij
=
exp
(
e
i
j
)
Σ
k
=
1
T
x
exp
(
e
i
k
)
e
ij
=
a
(
s
i
-
1
,
h
j
1
,
h
j
2
)
where i indicates a step of decoding, j indicates an index of an input regarding the RNN, T x indicates a length of an input sequence, s i−1 indicates a decoder state at an i-th step of decoding, LA i indicates an attention regarding the i-th step of decoding, a indicates a neural network, α ij indicates a weight regarding s i−1 , which is obtained by a, h 1 j indicates one of the first hidden representations, and h 2 j indicates one of the second hidden representations.
6. The electronic device of claim 5 , wherein the at least one processor is configured to:
obtain data on a third key in which positional encoding is applied to the first hidden representations;
obtain data on a fourth key in which the positional encoding is applied to the second hidden representations; and
obtain the attention based at least on the third key and the fourth key.
7. The electronic device of claim 1 , wherein training the language model regarding the RNN, training the slot filling model regarding the RNN, and training the intent detection model regarding the RNN share at least one of at least one layer of the plurality of RNN layers, the first hidden representations, or the second hidden representations.
8. A non-transitory computer-readable storage medium, which stores one or more programs for storing information comprising a plurality of recurrent neural network (RNN) layers, and for executing:
obtaining first hidden representations through at least one first layer from among the plurality of RNN layers;
setting the first hidden representations as a first key and a value; obtaining second hidden representations through at least one second layer from among the plurality of RNN layers;
setting the second hidden representations as a second key; and,
based on data on the first key, data on the second key, and data on the value, obtaining an attention included in an attention mechanism,
wherein the at least one second layer is positioned at an upper stage than the at least one first layer in an artificial neural network and the first hidden representations are used as an input of the at least one second layer,
and
wherein a cost for the RNN is obtained based on multiplication of data indicating a cost regarding a language model regarding the RNN by a weight-decay term, in training at least one of the language model, a slot filling model regarding the RNN, or an intent detection model regarding the RNN.
9. The non-transitory computer-readable storage medium of claim 8 ,
wherein the non-transitory computer-readable storage medium further stores one or more programs for executing obtaining the second hidden representations from the first hidden representations through the at least one second layer.
10. The non-transitory computer-readable storage medium of claim 9 , further storing one or more programs for executing obtaining, from the attention, data on a slot or data on an intent, through at least one third layer positioned at a higher stage than the at least one second layer from among the plurality of RNN layers.
11. The non-transitory computer-readable storage medium of claim 8 , further storing one or more programs for executing obtaining data on a weight of each of the first key and the second key, based on the data on the first key and the data on the second key, and obtaining the attention based on a weighted sum of the data on the weight and the data on the value.
12. The non-transitory computer-readable storage medium of claim 8 , further storing one or more programs for executing obtaining the attention based on the following equation:
L
A
i
=
Σ
j
=
1
T
x
α
ij
h
j
1
α
ij
=
exp
(
e
i
j
)
Σ
k
=
1
T
x
exp
(
e
i
k
)
e
ij
=
a
(
s
i
-
1
,
h
j
1
,
h
j
2
)
where i indicates a step of decoding, j indicates an index of an input regarding the RNN, T x indicates a length of an input sequence, s i−1 indicates a decoder state at an i-th step of decoding, LA i indicates an attention regarding the i-th step of decoding, a indicates a neural network, α ij indicates a weight regarding s i−1 , which is obtained by a, h 1 j indicates one of the first hidden representations, and h 2 j indicates one of the second hidden representations.
13. The non-transitory computer-readable storage medium of claim 12 , further storing one or more programs for executing:
obtaining data on a third key in which positional encoding is applied to the first hidden representations;
obtaining data on a fourth key in which the positional encoding is applied to the second hidden representations; and
obtaining obtain the attention based at least on the third key and the fourth key.
14. The non-transitory computer-readable storage medium of claim 8 , wherein training the language model regarding the RNN, training the slot filling model regarding the RNN, and training the intent detection model regarding the RNN share at least one of at least one layer of the plurality of RNN layers, the first hidden representations, or the second hidden representations.
15. An operation method of an electronic device using an artificial neural network model comprising an attention mechanism, the electronic device storing information comprising a plurality of recurrent neural network (RNN) layers, the method comprising:
obtaining first hidden representations through at least one first layer from among the plurality of RNN layers;
setting the first hidden representations as a first key and a value;
obtaining second hidden representations through at least one second layer from among the plurality of RNN layers;
setting the second hidden representations as a second key; and
based on data on the first key, data on the second key, and data on the value, obtaining an attention included in the attention mechanism,
wherein the at least one second layer is positioned at an upper stage than the at least one first layer in the artificial neural network and the first hidden representations are used as an input of the at least one second layer,
and
wherein a cost for the RNN is obtained based on multiplication of data indicating a cost regarding a language model regarding the RNN by a weight-decay term, in training at least one of the language model, a slot filling model regarding the RNN, or an intent detection model regarding the RNN.