Method, device and computer program for emotion recognition from a real-time audio signal

1 . A computer implemented method for recognizing emotion from a real-time audio signal, implemented by a device comprising one or more processors and a memory, wherein said computer implemented method includes several iterations of an emotion detection phase, and wherein said computer implemented method comprises:

receiving, by the device, an acquired real-time audio signal of a voice of a person from an apparatus comprising a microphone or a camera;

transcribing, by the device, through automatic speech recognition implementing an automatic speech recognition model, into text, an audio speech signal contained in said real-time audio signal;

computing, by the device, through speech representation implementing a speech representation model, a joint representation vector as a function of said audio speech signal and said text;

computing, by the device, through emotion prediction implementing an emotion prediction model, an emotion embedding vector, as a function of said joint representation vector;

mapping, by the device, through emotion mapping implementing an emotion mapping model, said emotion embedding vector into at least one emotion representation or at least one emotional frame, according to said emotion embedding vector, by an emotion mapping model;

outputting, by the device, said at least one emotional frame for automatic use by a human computer interactive or non-interactive system during real-time execution of the method;

detecting, by the device, through novelty detection implementing a novelty detection model, one or more novel expressions of emotion in the audio speech signal based on the joint representation vector,

wherein said detecting, by the device, of said one or more novel expressions of emotion comprises

processing the joint representation vector using the novelty detection model comprising an artificial neural network trained as an autoencoder to reconstruct an input comprising the joint representation vector; and

determining that the audio speech signal includes a novel expression of emotion when an output reconstructed by the autoencoder differs from the joint representation vector by more than a threshold; and

training, by the device, the emotion prediction model in response to detection of said one or more novel expressions of emotion in the audio speech signal,

wherein said training is performed using data detected as novel by the novelty detection model.

2 . The computer implemented method according to claim 1 , further comprising detecting, by the device through voice activity detection, voice activity in the real-time audio signal, the emotion detection phase being carried out when voice activity is detected.

3 . The computer implemented method according to claim 1 , further comprising extracting, by the device, a speech signal from the real-time audio signal, wherein the emotion detection phase is carried out for said speech signal.

4 . The computer implemented method according to claim 1 , further comprising computing, by a speech embedding model, a speech embedding vector as a function of the audio speech signal, wherein said speech embedding vector is used as input data for computing the joint representation vector.

5 . The computer implemented method according to claim 1 , further comprising computing, by a textual embedding model, a textual embedding vector as a function of the text obtained during said transcribing, wherein said textual embedding vector is used as input data for computing the joint representation vector.

6 . The computer implemented method according to claim 1 , wherein the mapping comprises classifying the emotion in one class among several emotion classes, by a categorical mapping model.

7 . The computer implemented method according to claim 1 , wherein the mapping comprises locating the emotion in a coordinate system comprising several dimensions, wherein each dimension of said several dimensions corresponds to an emotional dimension, by a dimensional mapping model.

8 . A non-transitory computer-readable storage medium having a computer program comprising instructions, which when executed by a computer, cause the computer to carry out a method for recognizing emotion from a real-time audio signal, wherein said method includes several iterations of an emotion detection phase, and wherein said method comprises:

receiving an acquired real-time audio signal of a voice of a person from an apparatus comprising a microphone or a camera;

transcribing, into text, an audio speech signal contained in said real-time audio signal by an automatic speech recognition model;

computing, by a speech representation model, a joint representation vector as a function of said audio speech signal and said text;

computing, by an emotion prediction model, an emotion embedding vector, as a function of said joint representation vector;

mapping said emotion in at least one emotion representation or at least one emotional frame, according to said emotion embedding vector, by an emotion mapping model; and

outputting said at least one emotional frame for automatic use by a human computer interactive or non-interactive system during real-time execution of the method;

wherein the method further comprises

detecting one or more novel expressions of emotion in the audio speech signal, by a novelty detection model, based on the joint representation vector,

wherein said detecting, by the device, of said one or more novel expressions of emotion comprises

processing the joint representation vector using the novelty detection model comprising an artificial neural network trained as an autoencoder to reconstruct an input comprising the joint representation vector, and

determining that the audio speech signal includes a novel expression of emotion when an output reconstructed by the autoencoder differs from the joint representation vector by more than a threshold, and

training the emotion prediction model in response to detection of said one or more novel expressions of emotion in the audio speech signal,

wherein said training is performed using data detected as novel by the novelty detection model.

9 . A device for real-time recognition of emotion from a real-time audio signal, said device being configured to receive an acquired real-time audio signal of a voice of a person from an apparatus comprising a microphone or a camera, said device comprising:

one or more processors and a memory, said one or more processors implementing:

an automatic speech recognition model configured to transcribe, into text, an audio speech signal contained in said real-time audio signal;

a speech representation model configured to compute a joint representation vector as a function of said audio speech signal and said text;

an emotion prediction model configured to compute an emotion embedding vector, as a function of said joint representation vector;

an emotion mapping model configured to map said emotion embedding vector into at least one emotion representation or at least one emotional frame;

wherein said device is configured to output said at least one emotional frame for automatic use by a human computer interactive or non-interactive system during real-time execution of the device; and

a novelty detection model configured to detect one or more novel expressions of emotion in the audio speech signal, based on the joint representation vector,

wherein said novelty detection model detects said one or more novel expressions of emotion by

processing the joint representation vector using the novelty detection model comprising an artificial neural network trained as an autoencoder to reconstruct an input comprising the joint representation vector, and

determining that the audio speech signal includes a novel expression of emotion when an output reconstructed by the autoencoder differs from the joint representation vector by more than a threshold,

a training model configured to train said emotion prediction model in response to detection of said one or more novel expressions of emotion in the audio speech signal,

wherein said training model trains said emotion prediction model using data detected as novel by the novelty detection model.