Method and system for fair speech emotion recognition
A computer-implemented method for speech emotion recognition is provided. The computer-implemented method includes an emotion prediction corresponding to speech data that is generated based on the speech data and a speech emotion recognition model without bias. The speech emotion recognition model without bias is trained by training a fairness-constrained adversarial network based on a labeled training set with known bias and a loss function. The fairness-constrained adversarial network includes a domain classifier for bias classification and the speech emotion recognition model. The loss function used for training the fairness-constrained adversarial network is positively related to a Wasserstein distance (WD) loss.
1 . A computer-implemented method for training a speech emotion recognition model, the method comprising:
providing a fairness-constrained adversarial network comprising a domain classifier and a speech emotion recognition model, the domain classifier configured to classify a bias in an input data, and the speech emotion recognition model configured to determine an emotion corresponding to the input data; and
training the fairness-constrained adversarial network, based on a labeled training set with known bias and a first loss function, to obtain the speech emotion recognition model without bias,
wherein;
the first loss function is positively related to a Wasserstein distance loss,
the labeled training set with known bias comprises a plurality of training data, and
each of the plurality of training data corresponds to an overall emotion label and a plurality of perspective emotion labels for a plurality of perspectives.
2 . The computer-implemented method of claim 1 , wherein the first loss function is negatively related to a first loss associated with the domain classifier and positively related to a second loss associated with the speech emotion recognition model.
3 . The computer-implemented method of claim 1 , wherein the domain classifier is configured to classify a binary bias.
4 . The computer-implemented method of claim 3 , wherein the binary bias comprises an evaluator gender bias.
5 . The computer-implemented method of claim 1 , further comprising:
generating a fairness embedding set corresponding to the labeled training set based on the speech emotion recognition model without bias; and
training a multi-perspective speech emotion recognition model based on the fairness embedding set and a second loss function,
wherein the multi-perspective speech emotion recognition model is configured to generate a plurality of perspective-specific emotion prediction results corresponding to the plurality of perspectives, and the second loss function is positively related to a metric learning loss.
6 . The computer-implemented method of claim 5 , wherein;
the metric learning loss comprises a triplet loss, and
an anchor, a positive sample, and a negative sample of the triplet loss are set based on the overall emotion label and the plurality of perspective emotion labels.
7 . The computer-implemented method of claim 5 , wherein the metric learning loss comprises at least one of a triplet loss, a quadruplet loss, an N-pair loss, a contrastive loss, or a center loss.
8 . The computer-implemented method of claim 5 , wherein the second loss function is positively related to a third loss associated with the multi-perspective speech emotion recognition model, and the third loss is associated with the plurality of perspective emotion labels.
9 . A computer-implemented method for speech emotion recognition, the method comprising:
receiving speech data; and
generating an emotion prediction result corresponding to the speech data based on the speech data and a speech emotion recognition model without bias, wherein:
the speech emotion recognition model without bias is trained by training a fairness-constrained adversarial network based on a labeled training set with known bias and a first loss function,
the first loss function is positively related to a Wasserstein distance loss,
the fairness-constrained adversarial network comprises a domain classifier for bias classification and the speech emotion recognition model, and
the labeled training set with known bias comprises a plurality of training data, and each of the plurality of training data corresponds to an overall emotion label and a plurality of perspective emotion labels for a plurality of perspectives.
10 . The computer-implemented method of claim 9 , wherein the first loss function is negatively related to a first loss associated with the domain classifier and positively related to a second loss associated with the speech emotion recognition model.
11 . The computer-implemented method of claim 9 , wherein the domain classifier is configured to classify a binary bias.
12 . The computer-implemented method of claim 11 , wherein the binary bias comprises an evaluator gender bias.
13 . The computer-implemented method of claim 9 , further comprising:
generating a fairness embedding corresponding to the speech data based on the speech data and the speech emotion recognition model without bias; and
generating at least one of a plurality of perspective-specific emotion prediction results corresponding to the speech data based on the fairness embedding and a multi-perspective speech emotion recognition model, wherein:
the multi-perspective speech emotion recognition model is trained based on a fairness embedding set corresponding to the labeled training set and a second loss function,
the fairness embedding set is generated based on the speech emotion recognition model without bias, and
the second loss function is positively related to a metric learning loss.
14 . The computer-implemented method of claim 13 , wherein;
the metric learning loss comprises a triplet loss, and
an anchor, a positive sample, and a negative sample of the triplet loss are set based on the overall emotion label and the plurality of perspective emotion labels.
15 . The computer-implemented method of claim 13 , wherein the metric learning loss comprises at least one of a triplet loss, a quadruplet loss, an N-pair loss, a contrastive loss, or a center loss.
16 . The computer-implemented method of claim 13 , wherein the second loss function is positively related to a third loss associated with the multi-perspective speech emotion recognition model, and the third loss is associated with the plurality of perspective emotion labels.
17 . A non-transitory computer-readable medium, comprising at least one instruction, when executed by a processor of an electronic device, causes the electronic device to:
receive speech data; and
generate an emotion prediction result corresponding to the speech data based on the speech data and a speech emotion recognition model without bias, wherein:
the speech emotion recognition model without bias is trained by training a fairness-constrained adversarial network based on a labeled training set with known bias and a first loss function,
the first loss function is positively related to a Wasserstein distance loss,
the fairness-constrained adversarial network comprises a domain classifier for bias classification and the speech emotion recognition model, and
the labeled training set with known bias comprises a plurality of training data, and each of the plurality of training data corresponds to an overall emotion label and a plurality of perspective emotion labels for a plurality of perspectives.