Label generation device, model learning device, emotion recognition apparatus, methods therefor, program, and recording medium

1. A label generation device, wherein

K is an integer equal to or larger than 2; and

the label generation device comprises processing circuitry configured to:

reproduce an audible recording as a first utterance of an utterer before a predetermined number of listeners;

obtain an input of an emotion class corresponding to an estimate of an emotion of the utterer, as a correct emotion class, among a plurality of emotion classes C 1 , . . . , C K from the listeners who have listened to the first utterance;

determine a number of times n i that the emotion classes C i have been selected as the correct emotion classes, where i=1, . . . , K is assumed;

obtain rates of the numbers of times n k to a sum total of the numbers of times n 1 , . . . , n K or smoothed values of the rates, for each of the plurality of emotion classes, as correct emotion soft labels t k (s) corresponding to the first utterance, and

output the correct emotion soft labels t k (s) as labels of learning data for model learning,

wherein

the following formula is satisfied:

t

k

(

s

)

=

α

+

n

k

α

⁢

K

+

∑

i

=

1

K

n

i

α=α 1 is satisfied when the number of the listeners is M 1 ; α=α 2 is satisfied when the number of the listeners is M 2 ; and M 1 >M 2 and 0<α 1 <α 2 are satisfied such that a is increased when the number of listeners is M 2 , and α is decreased when the number of listeners is M 1 .

2. A model learning device comprising:

the label generation device according to claim 1 ; and

processing circuitry configured to obtain a first emotion recognition model for estimating an emotion of an utterer of an input utterance from an acoustic feature sequence of the input utterance by learning processing with a set of pairs between the correct emotion soft labels t k (s) corresponding to the first utterance and a voice feature sequence of the first utterance as learning data.

3. A model learning device, wherein

K is an integer equal to or larger than 2; and

the model learning device comprises:

processing circuitry configured to:

obtain an input of an emotion class corresponding to an estimate of an emotion of an utterer, as a correct emotion class, among a plurality of emotion classes C 1 , . . . , C K from the listeners who have listened to a first utterance of the utterer;

determine a number of times n i that the emotion classes C i have been selected as the correct emotion classes, where i=1, . . . , K is assumed;

obtain rates of the numbers of times n k to a sum total of the numbers of times n 1 , . . . , n K or smoothed values of the rates, for each of the plurality of emotion classes, as correct emotion soft labels t k (s) corresponding to the first utterance;

obtain a second emotion recognition model for estimating an emotion of an utterer of an input utterance from an acoustic feature sequence of the input utterance by learning processing with a set of pairs between correct emotion labels t k (h) corresponding to a second utterance and a voice feature sequence of the second utterance as learning data; and

update the second emotion recognition model by additional learning processing with a set of pairs between the correct emotion soft labels t k (s) corresponding to the first utterance and a voice feature sequence of the first utterance as learning data to obtain a first emotion recognition model for estimating the emotion of the utterer of the input utterance from the acoustic feature sequence of the input utterance; wherein

the correct emotion labels t k (h) corresponding to the second utterance are such that correct emotion labels t i′ (h) corresponding to emotion classes C i′ selected as correct values of an emotion of an utterer of the second utterance from among the plurality of emotion classes C 1 , . . . , C K by listeners who have listened to the second utterance the largest number of times are assumed to be positive constants, and correct emotion labels t i″ (h) corresponding to emotion classes C i″ other than the emotion classes C i′ among the emotion classes C 1 , . . . , C K are assumed to be zero.

4. The model learning device of claim 3 , wherein

the second emotion recognition model is assumed to be an emotion recognition model in a learning process;

by inputting the voice feature sequence of the first utterance to the emotion recognition model in the learning process, output values y k (s) about posterior probabilities of the emotion classes C k are obtained; and

the additional learning processing includes processing for updating the emotion recognition model in the learning process, with rates of function values for the output values y k (s) to a sum total of the function values for the output values y 1 (s) , . . . , y K (s) as posterior probabilities p k (s) of the emotion classes C k , and with a sum total of products between function values for p k (s) and the correct emotion soft labels t k (s) for k=1, . . . , K as a loss function, to obtain the first emotion recognition model.

5. A model learning device, wherein

K is an integer equal to or larger than 2; and

the model learning device comprises:

processing circuitry configured to:

obtain an input of an emotion class corresponding to an estimate of an emotion of an utterer, as a correct emotion class, among a plurality of emotion classes C 1 , . . . , C K from the listeners who have listened to a first utterance of the utterer;

determine a number of times n i that the emotion classes C i have been selected as the correct emotion classes, where i=1, . . . , K is assumed;

obtain rates of the numbers of times n k to a sum total of the numbers of times n 1 , . . . , n K or smoothed values of the rates, for each of the plurality of emotion classes, as correct emotion soft labels t k (s) corresponding to the first utterance;

obtain a first emotion recognition model for estimating an emotion of an utterer of an input utterance from an acoustic feature sequence of the input utterance by learning processing with a set of combinations among the correct emotion soft labels t k (s) corresponding to the first utterance, a voice feature sequence of the first utterance, correct emotion labels t k (h) corresponding to a second utterance and a voice feature sequence of the second utterance as leaning data; wherein

the correct emotion labels t k (h) corresponding to the second utterance are such that correct emotion labels t i′ (h) corresponding to emotion classes C i′ selected as correct values of an emotion of an utterer of the second utterance from among the plurality of emotion classes C 1 , . . . , C K by listeners who have listened to the second utterance the largest number of times are assumed to be positive constants, and correct emotion labels t i′ (h) corresponding to emotion classes C i″ other than the emotion classes C i′ among the emotion classes C 1 , . . . , C K are assumed to be zero;

by inputting the voice feature sequence of the second utterance to an emotion recognition model in a learning process, output values y k (s) about posterior probabilities of the emotion classes C k are obtained;

by inputting the voice feature sequence of the first utterance to the emotion recognition model in the learning process, output values y k (s) about posterior probabilities of the emotion classes C k are obtained; and

the learning processing includes processing for updating the emotion recognition model in the learning process, with rates of function values for output values y k (h) to a sum total of the function values for the output values y 1 (h) , . . . , y K (h) as posterior probabilities p k (h) of the emotion classes C k in the second utterance, with rates of function values for the output values y k (s) to a sum total of the function values for the output values y 1 (s) , . . . , y K (s) as posterior probabilities p k (s) of the emotion classes C k in the first utterance, and with a weighted sum of a sum total of products between the function values for p k (h) and the correct emotion labels t k (h) for k=1, . . . , K and a sum total of products between function values for p k (s) and the correct emotion soft labels t k (s) for k=1, . . . , K as a loss function, to obtain the first emotion recognition model.

6. An emotion recognition apparatus comprising:

the model learning device of any of claims 2 to 5 , and

processing circuitry configured to apply an acoustic feature sequence of the input utterance to the first emotion recognition model for obtaining output values y k about posterior probabilities of the emotion classes C k for the acoustic feature sequence of the input utterance to output the output values y k , function values for the output values y k or information indicating such emotion classes C k that the posterior probabilities indicated by the output values y k are the largest.

7. A label generation method implemented by processing circuitry of a label generation device, wherein

K is an integer equal to or larger than 2; and

the label generation method comprises:

reproducing an audible recording as a first utterance before a predetermined number of listeners;

obtaining an input of an emotion class corresponding to an estimate of an emotion of the utterer, as a correct emotion class, among a plurality of emotion classes C 1 , . . . , C K from the listeners who have listened to the first utterance;

determining a number of times n i that the emotion classes C i have been selected as the correct emotion classes, where i=1, . . . , K is assumed;

obtaining rates of the numbers of times n k to a sum total of the numbers of times n 1 , . . . , n K or smoothed values of the rates, for each of the plurality of emotion classes, as correct emotion soft labels t k (s) corresponding to the first utterance, and

outputting the correct emotion soft labels t k (s) as labels of learning data for model learning,

wherein

the following formula is satisfied:

t

k

(

s

)

=

α

+

n

k

α

⁢

K

+

∑

i

=

1

K

n

i

α=α 1 is satisfied when the number of the listeners is M 1 ; α=α 2 is satisfied when the number of the listeners is M 2 ; and M 1 >M 2 and 0<α 1 <α 2 are satisfied such that α is increased when the number of listeners is M 2 , and α is decreased when the number of listeners is M 1 .

8. A model learning method comprising:

the label generation method of claim 7 ; and

a learning step of processing circuitry of a model learning device obtaining a first emotion recognition model for estimating an emotion of an utterer of an input utterance from an acoustic feature sequence of the input utterance by learning processing with a set of pairs between the correct emotion soft labels t k (s) corresponding to the first utterance and a voice feature sequence of the first utterance as learning data.

9. A model learning method implemented by processing circuitry of a model learning device, wherein

K is an integer equal to or larger than 2; and

the model learning method comprises:

obtaining an input of an emotion class, as a correct emotion class, among a plurality of emotion classes C 1 , . . . , C K from the listeners who have listened to a first utterance;

determining a number of times n i that the emotion classes C i have been selected as the correct emotion classes, where i=1, . . . , K is assumed;

obtaining rates of the numbers of times n k to a sum total of the numbers of times n 1 , . . . , n K or smoothed values of the rates, for each of the plurality of emotion classes, as correct emotion soft labels t k (s) corresponding to the first utterance;

obtaining a second emotion recognition model for estimating an emotion of an utterer of an input utterance from an acoustic feature sequence of the input utterance by learning processing with a set of pairs between correct emotion labels t k (h) corresponding to a second utterance and a voice feature sequence of the second utterance as learning data; and

updating the second emotion recognition model by additional learning processing with a set of pairs between the correct emotion soft labels t k (s) corresponding to the first utterance and a voice feature sequence of the first utterance as learning data to obtain a first emotion recognition model for estimating the emotion of the utterer of the input utterance from the acoustic feature sequence of the input utterance; wherein

the correct emotion labels t k (h) corresponding to the second utterance are such that correct emotion labels t i′ (h) corresponding to emotion classes C i′ selected as correct values of an emotion of an utterer of the second utterance from among the plurality of emotion classes C 1 , . . . , C K by listeners who have listened to the second utterance the largest number of times are assumed to be positive constants, and correct emotion labels t i″ (h) corresponding to emotion classes C i″ other than the emotion classes C i′ among the emotion classes C 1 , . . . , C K are assumed to be zero.

10. The model learning method according to claim 9 , wherein

the second emotion recognition model is assumed to be an emotion recognition model in a learning process;

by inputting the voice feature sequence of the first utterance to the emotion recognition model in the learning process, output values y k (s) about posterior probabilities of the emotion classes C k are obtained; and

the additional learning processing includes processing for updating the emotion recognition model in the learning process, with rates of function values for the output values y k (s) to a sum total of the function values for the output values y 1 (s) , . . . , y K (s) as posterior probabilities p k (s) of the emotion classes C k , and with a sum total of products between function values for p k (s) and the correct emotion soft labels t k (s) for k=1, . . . , K as a loss function, to obtain the first emotion recognition model.

11. A model learning method, implemented by processing circuitry of a model learning device, wherein

K is an integer equal to or larger than 2; and

the model learning method comprises:

obtaining an input of an emotion class, as a correct emotion class, among a plurality of emotion classes C 1 , . . . , C K from the listeners who have listened to a first utterance;

determining a number of times n i that the emotion classes C i have been selected as the correct emotion classes, where i=1, . . . , K is assumed;

obtaining rates of the numbers of times n k to a sum total of the numbers of times n 1 , . . . , n K or smoothed values of the rates, for each of the plurality of emotion classes, as correct emotion soft labels t k (s) corresponding to the first utterance,

obtaining a first emotion recognition model for estimating an emotion of an utterer of an input utterance from an acoustic feature sequence of the input utterance by learning processing with a set of combinations among the correct emotion soft labels t k (s) corresponding to the first utterance, a voice feature sequence of the first utterance, correct emotion labels t k (h) corresponding to a second utterance and a voice feature sequence of the second utterance as leaning data; wherein

the correct emotion labels t k (h) corresponding to the second utterance are such that correct emotion labels t i′ (h) corresponding to emotion classes C i′ selected as correct values of an emotion of an utterer of the second utterance from among the plurality of emotion classes C 1 , . . . , C K by listeners who have listened to the second utterance the largest number of times are assumed to be positive constants, and correct emotion labels t i″ (h) corresponding to emotion classes C i″ other than the emotion classes C i′ among the emotion classes C 1 , . . . , C K are assumed to be zero;

by inputting the voice feature sequence of the second utterance to an emotion recognition model in a learning process, output values y k (s) about posterior probabilities of the emotion classes C k are obtained;

by inputting the voice feature sequence of the first utterance to the emotion recognition model in the learning process, output values y k (s) about posterior probabilities of the emotion classes C k are obtained; and

the learning processing includes processing for updating the emotion recognition model in the learning process, with rates of function values for output values y k (h) to a sum total of the function values for the output values y 1 (h) , . . . , y K (h) as posterior probabilities p k (h) of the emotion classes C k in the second utterance, with rates of function values for the output values y k (s) to a sum total of the function values for the output values y 1 (h) , . . . , y K (s) as posterior probabilities p k (s) of the emotion classes C k in the first utterance, and with a weighted sum of a sum total of products between the function values for p k (h) and the correct emotion labels t k (h) for k=1, . . . , K and a sum total of products between function values for p k (s) and the correct emotion soft labels t k (s) for k=1, . . . , K as a loss function, to obtain the first emotion recognition model.

12. An emotion recognition method comprising:

each step of the model learning method of any of claims 8 to 11 ; and

an emotion recognition step of processing circuitry of an emotion recognition apparatus applying an acoustic feature sequence of the input utterance to the first emotion recognition model for obtaining output values y k about posterior probabilities of the emotion classes C k for the acoustic feature sequence of the input utterance to output the output values y k , function values for the output values y k or information indicating such emotion classes C k that the posterior probabilities indicated by the output values y k are the largest.

13. A non-transitory machine-readable recording medium storing a program for causing a computer to function as any of the label generation device of claim 1 , or the model learning device of any of claims 2 to 5 .

14. A non-transitory machine-readable recording medium storing a program for causing a computer to function as the emotion recognition apparatus of claim 6 .