Exploiting acoustic and lexical properties of phonemes to recognize valence from speech
A method includes receiving acoustic features and phonetic features associated with an utterance from a driver in a vehicle, providing the acoustic features and the phonetic features to a feature fusion sub-network, receiving a feature fusion utterance representation from the feature fusion sub-network, providing one of the acoustic features or the phonetic features to a non-fusion sub-network trained using supervised learning, receiving a non-fusion utterance representation from the non-fusion sub-network, generating an intermediate utterance representation based on the feature fusion utterance representation and the non-fusion utterance representation, providing at least a portion of the intermediate utterance representation to a fully-connected sub-network trained using supervised learning, receiving a valence vector from the fully-connected sub-network, and causing a vehicle control system to perform a vehicle maneuver based on the valence vector.
1. A method in a data processing system comprising at least one processor and at least one memory, the at least one memory comprising instructions executed by the at least one processor to implement a valence determination system, the method comprising:
receiving a plurality of acoustic features associated with an utterance from a driver in a vehicle with a vehicle control system;
receiving a plurality of phonetic features associated with the utterance;
providing the plurality of acoustic features and the plurality of phonetic features to a feature fusion sub-network trained using supervised learning;
receiving a feature fusion utterance representation from the feature fusion sub-network;
providing one of the plurality of acoustic features or the plurality of phonetic features to a non-fusion sub-network trained using supervised learning;
receiving a non-fusion utterance representation from the non-fusion sub-network;
generating an intermediate utterance representation based on the feature fusion utterance representation and the non-fusion utterance representation;
providing at least a portion of the intermediate utterance representation to a fully-connected sub-network trained using supervised learning;
receiving a valence vector from the fully-connected sub-network; and
causing the vehicle control system to perform a vehicle maneuver based on the valence vector.
2. The method of claim 1 wherein the feature fusion sub-network comprises a convolutional layer and a global pooling layer.
3. The method of claim 1 wherein the fully-connected sub-network comprises a plurality of fully-connected layers and a softmax layer, wherein a first fully-connected layer is configured to output values into the softmax layer.
4. The method of claim 3 , wherein the fully-connected sub-network further comprises a dropout layer configured to output values into a second fully-connected layer.
5. The method of claim 1 wherein the valence vector comprises a negative value, a neutral value, and a positive value.
6. The method of claim 1 wherein the phonetic features are one or more one-hot vectors and the acoustic features are one or more Mel-frequency Filterbank energy vectors.
7. The method of claim 6 , wherein each of the Mel-frequency Filterbank energy filters has a corresponding one-hot vector.
8. The method of claim 1 , wherein the non-fusion sub-network comprises a convolutional layer and a global pooling layer.
9. The method of claim 1 , wherein the intermediate utterance representation is generated by concatenating the feature fusion utterance representation and the non-fusion utterance representation.
10. The method of claim 1 , wherein the feature fusion sub-network, the non-fusion sub-network, and the fully-connected sub-network are trained using supervised learning with a learning rate no greater than 0.0001 and unweighted average recall as a performance measure, and the supervised learning is run at least twice using a plurality of different layer sizes and in the feature fusion sub-network, the non-fusion sub-network, and the fully-connected sub-network.
11. The method of claim 10 , wherein the at least two different layer sizes include 128 and 256.
12. The method of claim 1 , wherein the driving control system is associated with a fully autonomous vehicle system.
13. The method of claim 1 further comprising receiving the utterance from an audio sensor coupled to the vehicle and in communication with the processor.
14. The method of claim 13 further comprising:
extracting the acoustic features and the phonetic features from the utterance; and
aligning the phonetic features with the acoustic features.
15. A driving control system for a vehicle, the driving control system comprising:
an audio sensor coupled to a vehicle; and
a controller coupled to the vehicle and in electrical communication with the audio sensor, the controller being configured to execute a program stored in the controller to:
(i) receive a plurality of acoustic features associated with an utterance from a driver in a vehicle with a vehicle control system;
(ii) receive a plurality of phonetic features associated with the utterance;
(iii) provide the plurality of acoustic features and the plurality of phonetic features to a feature fusion sub-network trained using supervised learning;
(iv) receive a feature fusion utterance representation from the feature fusion sub-network;
(v) provide one of the plurality of acoustic features or the plurality of phonetic features to a non-fusion sub-network trained using supervised learning;
(vi) receive a non-fusion utterance representation from the non-fusion sub-network;
(vii) generate an intermediate utterance representation based on the feature fusion utterance representation and the non-fusion utterance representation;
(viii) provide at least a portion of the intermediate utterance representation to a fully-connected sub-network trained using supervised learning;
(ix) receive a valence vector from the fully-connected sub-network; and (x) cause the vehicle control system to perform a vehicle maneuver based on the valence vector.
16. The system of claim 15 , wherein the feature fusion sub-network, the non-fusion sub-network, and the fully-connected sub-network are trained using supervised learning with a learning rate no greater than 0.0001 and unweighted average recall as a performance measure, and the supervised learning is run at least twice using a plurality of different layer sizes and in the feature fusion sub-network, the non-fusion sub-network, and the fully-connected sub-network.
17. The system of claim 15 wherein the phonetic features are one or more one-hot vectors and the acoustic features are one or more Mel-frequency Filterbank energy vectors.
18. The system of claim 15 wherein the fully-connected sub-network comprises a plurality of fully-connected layers and a softmax layer, wherein a first fully-connected layer is configured to output values into the softmax layer.
19. The system of claim 18 , wherein the fully-connected sub-network further comprises a dropout layer configured to output values into a second fully-connected layer.
20. The system of claim 15 wherein the feature fusion sub-network comprises a convolutional layer and a global pooling layer.