Method, system and computer-readable storage medium for cross-task unseen emotion class recognition
A method for unseen emotion class recognition comprises: receiving, with an emotion recognition model, a speech sample to be tested; calculating, with an encoder, a sample embedding to be tested of the speech sample to be tested; calculating a first distance metric between the sample embedding to be tested and a first registered emotion category representation, and a second distance metric between the sample embedding to be tested and a second registered emotion category representation, wherein the second registered emotion category is not included in a plurality of basic emotion categories; and determining an emotion category of the speech sample to be tested according to the first distance metric and the second distance metric.
1 . A method for unseen emotion class recognition with a speech recognition device, comprising:
receiving, by an emotion recognition model, a speech sample to be tested;
calculating, by an encoder, a sample embedding to be tested of the speech sample to be tested;
calculating a first distance metric between the sample embedding to be tested and a first registered emotion category representation, and a second distance metric between the sample embedding to be tested and a second registered emotion category representation, wherein the second registered emotion category is not included in a plurality of basic emotion categories;
comparing the first distance metric and the second distance metric; and
determining an emotion category of the speech sample to be tested based on the comparison result,
wherein the method further comprises a training the emotion recognition model by means of:
receiving, by the emotion recognition model, a first training speech sample, wherein the first training speech sample indicates a first emotion category in the basic emotion categories, and the basic emotion categories include respectively a plurality of training speech samples;
calculating, by the encoder, a first embedding of the first training speech sample;
calculating respectively, by the encoder, a plurality of sample embeddings of the training speech samples of each of the basic emotion categories;
calculating a center-of-mass representation of each of the basic emotion categories, wherein the center-of-mass representations are respectively averages of the sample embeddings of different basic emotion categories;
calculating respectively a cosine similarity of the first embedding with the center-of-mass representation of each of the basic emotion categories, wherein an angular margin is added to the cosine similarity when calculating the cosine similarity with the first emotion category, and the angular margin is negative;
calculating a loss according to a loss function; and
adjusting a plurality of parameters of the emotion recognition model based on the calculated loss, wherein the loss function is associated with the cosine similarity.
2 . The method of claim 1 , wherein the encoder comprises an acoustic feature generator to fetch an acoustic feature from the speech sample to be tested.
3 . The method of claim 2 , wherein the encoder comprises a gated recurrent unit (GRU) model and a Transformer model to transform the acoustic feature into the sample embedding to be tested.
4 . The method of claim 1 , further comprising a register procedure which comprises:
receiving, with the emotion recognition model, a plurality of first registered speech samples and a plurality of second registered speech samples, wherein the first registered speech samples indicate the first registered emotion category and the second registered speech samples indicate the second registered emotion category;
calculating, with the encoder, a plurality of first registered sample embeddings of the first registered speech samples and a plurality of second registered sample embeddings of the second registered speech samples; and
calculating respectively averages of the first registered sample embeddings and the second registered sample embeddings, to generate respectively the first registered emotion category representation and the second registered emotion category representation.
5 . The method of claim 1 , wherein the loss function includes a cross-entropy loss.
6 . The method of claim 1 , wherein determining the emotion category of the speech sample to be tested comprises: determining the emotion category as the first registered emotion category according to the first distance metric being smaller than the second distance metric, or determining the emotion category as the second registered emotion category according to the second distance metric being smaller than the first distance metric.
7 . A speech recognition device for unseen emotion class recognition, comprising:
a memory having stored thereon a plurality of instructions; and
a processor coupled to the memory, wherein the processor is configured to, when executing the instructions:
receive, by an emotion recognition model, a speech sample to be tested;
calculate, by an encoder, a sample embedding to be tested of the speech sample to be tested;
calculate a first distance metric between the sample embedding to be tested and a first registered emotion category representation, and a second distance metric between the sample embedding to be tested and a second registered emotion category representation, wherein the second registered emotion category is not included in a plurality of basic emotion categories;
comparing the first distance metric and the second distance metric; and
determine an emotion category of the speech sample to be tested based on the comparison result,
wherein the processor is further configured to:
receive, by the emotion recognition model, a first training speech sample, wherein the first training speech sample indicates a first emotion category in the basic emotion categories, and the basic emotion categories include respectively a plurality of training speech samples;
calculate, by the encoder, a first embedding of the first training speech sample;
calculate respectively, by the encoder, a plurality of sample embeddings of the training speech samples of each of the basic emotion categories;
calculate a center-of-mass representation of each of the basic emotion categories, wherein the center-of-mass representations are respectively averages of the sample embeddings of different basic emotion categories;
calculate respectively a cosine similarity of the first embedding with the center-of-mass representation of each of the basic emotion categories, wherein an angular margin is added to the cosine similarity when calculating the cosine similarity with the first emotion category, and the angular margin is negative;
calculate a loss according to a loss function; and
adjust a plurality of parameters of the emotion recognition model based on the calculated loss, wherein the loss function is associated with the cosine similarity.
8 . The speech recognition device of claim 7 , wherein the encoder comprises an acoustic feature generator to fetch an acoustic feature from the speech sample to be tested.
9 . The speech recognition device of claim 8 , wherein the encoder comprises a gated recurrent unit (GRU) model and a Transformer model to transform the acoustic feature into the sample embedding to be tested.
10 . The speech recognition device of claim 7 , wherein the processor is further configured to:
receive, with the emotion recognition model, a plurality of first registered speech samples and a plurality of second registered speech samples, wherein the first registered speech samples indicate the first registered emotion category and the second registered speech samples indicate the second registered emotion category;
calculate, with the encoder, a plurality of first registered sample embeddings of the first registered speech samples and a plurality of second registered sample embeddings of the second registered speech samples; and
calculate respectively averages of the first registered sample embeddings and the second registered sample embeddings, to generate respectively the first registered emotion category representation and the second registered emotion category representation.
11 . The speech recognition device of claim 7 ,
wherein the loss function includes a cross-entropy loss.
12 . The speech recognition device of claim 7 , wherein the processor is configured to: determine the emotion category of the speech sample to be tested as the first registered emotion category according to the first distance metric being smaller than the second distance metric, or determine the emotion category as the second registered emotion category according to the second distance metric being smaller than the first distance metric.
13 . A computer-readable non-transitory storage medium for unseen emotion class recognition loaded with a computer-readable program capable of, after being read by a speech recognition device, configure the speech recognition device to:
receive, by an emotion recognition model, a speech sample to be tested;
calculate, by an encoder, a sample embedding to be tested of the speech sample to be tested;
calculate a first distance metric between the sample embedding to be tested and a first registered emotion category representation, and a second distance metric between the sample embedding to be tested and a second registered emotion category representation, wherein the second registered emotion category is not included in a plurality of basic emotion categories;
comparing the first distance metric and the second distance metric; and
determine an emotion category of the speech sample to be tested based on the comparison result;
wherein the computer-readable program is capable of, after being read by the speech recognition device, further configure the speech recognition device to:
receive, by the emotion recognition model, a first training speech sample, wherein the first training speech sample indicates a first emotion category in the basic emotion categories, and the basic emotion categories include respectively a plurality of training speech samples;
calculate, by the encoder, a first embedding of the first training speech sample;
calculate respectively, by the encoder, a plurality of sample embeddings of the training speech samples of each of the basic emotion categories;
calculate a center-of-mass representation of each of the basic emotion categories, wherein the center-of-mass representations are respectively averages of the sample embeddings of different basic emotion categories;
calculate respectively a cosine similarity of the first embedding with the center-of-mass representation of each of the basic emotion categories, wherein an angular margin is added to the cosine similarity when calculating the cosine similarity with the first emotion category, and the angular margin is negative;
calculating a loss according to a loss function; and
adjusting a plurality of parameters of the emotion recognition model based on the calculated loss, wherein the loss function is associated with the cosine similarity.
14 . The computer-readable non-transitory storage medium of claim 13 , wherein the encoder comprises an acoustic feature generator to fetch an acoustic feature from the speech sample to be tested.
15 . The computer-readable non-transitory storage medium of claim 14 , wherein the encoder comprises a gated recurrent unit (GRU) model and a Transformer model to transform the acoustic feature into the sample embedding to be tested.
16 . The computer-readable non-transitory storage medium of claim 13 , wherein the computer-readable program is capable of, after being read by the speech recognition device, further configure the speech recognition device to:
receive, with the emotion recognition model, a plurality of first registered speech samples and a plurality of second registered speech samples, wherein the first registered speech samples indicate the first registered emotion category and the second registered speech samples indicate the second registered emotion category;
calculate, with the encoder, a plurality of first registered sample embeddings of the first registered speech samples and a plurality of second registered sample embeddings of the second registered speech samples; and
calculate respectively averages of the first registered sample embeddings and the second registered sample embeddings, to generate respectively the first registered emotion category representation and the second registered emotion category representation.
17 . The computer-readable non-transitory storage medium of claim 13 ,
wherein the loss function includes a cross-entropy loss.
18 . The computer-readable non-transitory storage medium of claim 13 , wherein the computer-readable program is capable of, after being read by the speech recognition device, configure the speech recognition device to: determine the emotion category of the speech sample to be tested as the first registered emotion category according to the first distance metric being smaller than the second distance metric, or determine the emotion category as the second registered emotion category according to the second distance metric being smaller than the first distance metric.