Convolutional neural network with phonetic attention for speaker verification
Embodiments may include reception of a plurality of speech frames, determination of a multi-dimensional acoustic feature associated with each of the plurality of speech frames, determination of a plurality of multi-dimensional phonetic features, each of the plurality of multi-dimensional phonetic features determined based on a respective one of the plurality of speech frames, generation of a plurality of two-dimensional feature maps based on the phonetic features, input of the feature maps and the plurality of acoustic features to a convolutional neural network, the convolutional neural network to generate a plurality of speaker embeddings based on the plurality of feature maps and the plurality of acoustic features, aggregation of the plurality of speaker embeddings into a first speaker embedding based on respective weights determined for each of the plurality of speaker embeddings, and determination of a speaker associated with the plurality of speech frames based on the first speaker embedding.
1. A system comprising:
a processing unit; and
a storage device including program code that when executed by the processing unit causes to the system to:
determine a frame-level acoustic feature associated with each of a plurality of speech frames associated with a speaker;
determine a frame-level phonetic feature associated with each of the plurality of speech frames based on the frame-level acoustic feature associated with each of the plurality of frame-level acoustic features;
generate one or more two-dimensional feature maps based on the plurality of frame-level phonetic features;
input the one or more two-dimensional feature maps to a trained neural network to generate a plurality of frame-level speaker embeddings, the trained neural network including a convolutional neural network;
aggregate the plurality of frame-level speaker embeddings into a speaker embedding based on respective weights determined for each of the plurality of frame-level speaker embeddings; and
determine an identity of the speaker based on the speaker embedding.
2. A system according to claim 1 , wherein the respective weights determined for each of the plurality of frame-level speaker embeddings correspond to the speaker-discriminativity of the frame-level speaker embedding.
3. A system according to claim 1 , wherein input of the one or more two-dimensional feature maps to the trained neural network comprises input of the plurality of frame-level acoustic features to the trained neural network.
4. A system according to claim 3 , the program code when executed by the processing unit causes to the system to:
determine a feature map based on the frame-level acoustic features,
wherein input of the plurality of frame-level acoustic features to the trained neural network comprises input of the feature map based on the frame-level acoustic features.
5. A system of claim 4 , wherein generation of the one or more two-dimensional feature maps based on the plurality of frame-level phonetic features comprises:
input of the plurality of frame-level phonetic features to a trained deep neural network including a plurality of fully-connected layers,
wherein the one or more two-dimensional feature maps and the feature map based on the frame-level acoustic features exhibit substantially similar contiguous spectral patterns.
6. A system according to claim 1 , wherein determination of a frame-level phonetic feature associated with each of the plurality of speech frames comprises:
input of the plurality of frame-level acoustic features to a feature extraction network; and
output of the plurality of frame-level phonetic features from the feature extraction network.
7. A system according to claim 6 , wherein the feature extraction network comprises a trained speech recognition network, and
wherein output of the plurality of frame-level phonetic features from the feature extraction network comprises extraction of the plurality of frame-level phonetic features from one or more hidden layers of the trained speech recognition network.
8. A non-transient, computer-readable medium storing program code to be executed by a processing unit to provide:
a bi-directional long short-term memory network to receive a plurality of multi-dimensional acoustic features and to determine a plurality of multi-dimensional long-range contextual features based on the plurality of multi-dimensional acoustic features;
a projection layer to generate one or more two-dimensional feature maps based on the plurality of multi-dimensional long-range contextual features;
a convolutional neural network to receive the one or more two-dimensional feature maps and to generate a plurality of speaker embeddings based on the one or more two-dimensional feature maps;
an attentive pooling network to determine respective weights for each of the plurality of speaker embeddings and to aggregate the plurality of speaker embeddings into a speaker embedding based on the respective weights; and
a speaker identifier to determine a speaker associated with the plurality of speech frames based on the speaker embedding.
9. A non-transient, computer-readable medium according to claim 8 , wherein the respective weights determined for each of the plurality of embeddings correspond to the speaker-discriminativity of the speaker embedding.
10. A non-transient, computer-readable medium according to claim 8 , wherein the convolutional neural network is to receive the one or more two-dimensional feature maps and the plurality of multi-dimensional acoustic features and is to generate a plurality of speaker embeddings based on the one or more two-dimensional feature maps and the plurality of multi-dimensional acoustic features.
11. A non-transient, computer-readable medium according to claim 10 , the program code to be executed by the processing unit to:
determine a feature map based on the multi-dimensional acoustic features,
wherein receipt of the one or more two-dimensional feature maps and the plurality of multi-dimensional acoustic features comprises receipt of the one or more two-dimensional feature maps and the feature map based on the multi-dimensional features.
12. A non-transient, computer-readable medium of claim 11 ,
wherein the one or more two-dimensional feature maps and the feature map based on the multi-dimensional acoustic features exhibit substantially similar contiguous spectral patterns.
13. A non-transient, computer-readable medium according to claim 8 , wherein the bi-directional long short-term memory network, the projection layer and the convolutional neural network comprise a bi-directional long short-term memory-ResNet network architecture.
14. A non-transient, computer-readable medium according to claim 8 , wherein the bi-directional long short-term memory network, the projection layer, the convolutional neural network and the attentive pooling network are jointly trained.
15. A non-transient, computer-readable medium storing program code to be executed by a processing unit to provide:
a feature extractor to receive a plurality of multi-dimensional acoustic features and to determine a plurality of multi-dimensional phonetic features based on the plurality of multi-dimensional acoustic features;
a deep neural network to generate a plurality of two-dimensional feature maps based on the plurality of multi-dimensional phonetic features;
a convolutional neural network to receive the plurality of two-dimensional feature maps and the plurality of multi-dimensional acoustic features and to generate a plurality of speaker embeddings based on the plurality of two-dimensional feature maps and the plurality of multi-dimensional acoustic features;
an attentive pooling network to determine respective weights for each of the plurality of speaker embeddings and to aggregate the plurality of speaker embeddings into a first speaker embedding based on the respective weights; and
a speaker identifier to determine a speaker associated with the plurality of speech frames based on the speaker embedding.
16. A non-transient, computer-readable medium according to claim 15 , wherein the respective weights determined for each of the plurality of embeddings correspond to the speaker-discriminativity of each of the plurality of embeddings.
17. A non-transient, computer-readable medium according to claim 15 , the program code to be executed by the processing unit to:
determine a feature map based on the multi-dimensional acoustic features,
wherein inputting the plurality of two-dimensional feature maps and the plurality of multi-dimensional acoustic features to the convolutional neural network comprises inputting the plurality of two-dimensional feature maps and the feature map based on the multi-dimensional features.
18. A non-transient, computer-readable medium of claim 17 , wherein the deep neural network comprises a plurality of fully-connected layers, and
wherein the plurality of two-dimensional feature maps and the feature map based on the multi-dimensional acoustic features exhibit substantially similar contiguous spectral patterns.
19. A non-transient, computer-readable medium according to claim 15 , wherein the feature extraction network comprises a trained speech recognition network, and
wherein determination of the plurality multi-dimensional phonetic features comprises extraction of the plurality of multi-dimensional phonetic features from one or more hidden layers of the trained speech recognition network.
20. A non-transient, computer-readable medium according to claim 15 , wherein the speaker identifier is further to compare the speaker embedding to each of a plurality set of speaker embeddings associated with respective speakers.