Systems and methods for a triplet network with attention for speaker diartzation

1. A method for the diarization of speakers in a speech audio sample, comprising:

segmenting an audio recording featuring one or more speakers in the time-domain into a plurality of audio samples, wherein a set of temporal sequence features or a set of mel-frequency cepstral coefficients are extracted from each of the plurality of audio samples;

learning a set of hidden representations from a set of hidden embeddings using an attention model, wherein the attention model is implemented using a neural network and wherein a set of encoded ordering information are used to learn the set of hidden representations and wherein the set of mel-frequency cepstral coefficients are used to learn the set of hidden embeddings;

learning a similarity metric between each of the plurality of audio samples using a metric learner, wherein the metric learner computes the triplet ranking loss using the set of hidden representations and the set of temporal sequence features, and wherein the metric learner is implemented using the neural network; and

performing speaker clustering using the similarity metric, wherein the result of speaker clustering diarizes the one or more speakers featured in the audio recording,

wherein the neural network is iteratively trained using a labeled corpus,

wherein a set of semi-hard triplets are used to learn the similarity metric, and wherein a sampling space is restricted to one of a set of mini-batches while training the neural network, and

wherein the set of mini-batches are constructed using a batch size B and an allowance of M speakers per batch, wherein the M speakers are first sampled and a set of B/M audio samples are sampled for each of the M speakers.

2. The method of claim 1 , wherein the mel-frequency cepstral coefficients are extracted from each of the plurality of audio samples using Hamming windows.

3. The method of claim 2 , wherein a set of delta and double delta coefficients are added to the set of mel-frequency cepstral coefficients and the set of mel-frequency cepstral coefficients are embodied in the form of 60-dimensional feature vectors for every one of a plurality of frames.

4. The method of claim 3 , further comprising:

encoding a set of ordering information contained in the audio recording by mapping a plurality of relative frame positions in each audio sample to fixed locations in a random lookup table;

incorporating the encoded ordering information with the set of hidden embeddings and obtaining the set of hidden representations using a multi-head self-attention module; and

reducing the set of hidden representations into a D-dimensional vector using a temporal pooling neural network layer, wherein D is representative of the size of a hidden layer in the neural network.

5. The method of claim 1 , wherein a set of attention model parameters and the metric learner are jointly optimized using back-propagation.

6. A method for learning a set of hidden representations from an audio recording for speaker diarization, the method comprising:

extracting a set of mel-frequency cepstral coefficients from an audio sample of an audio recording;

learning a set of hidden embeddings from the set of mel-frequency cepstral coefficients;

encoding a set of ordering information contained in the audio recording by mapping a plurality of relative frame positions in each audio sample to fixed locations in a random lookup table;

executing a multi-head self-attention mechanism, wherein a set of hidden representations are learned from the set of hidden embeddings and the set of ordering information using a feed-forward neural network; and

performing temporal pooling on the set of hidden representations to reduce the set of hidden representations to a D-dimensional vector, wherein D is representative of a size of a hidden layer in the neural network;

wherein each of the hidden representations are computed using a plurality of different network parameterizations.

7. The method of claim 6 , wherein weighted contributions from other temporal positions are included during the computation of each of the set of hidden representations.

8. The method of claim 6 , wherein the audio recording is segmented using non-overlapping temporal segmentation.

9. The method of claim 6 , wherein an attention module is stacked L times to learn deeper hidden representations.

10. The method of claim 6 , further comprising encoding the set of ordering information and learning the set of hidden embeddings simultaneously.

11. The method of claim 10 , wherein the set of ordering information and the set of hidden embeddings are incorporated together before being operated on by the multi-head self-attention mechanism.

12. A method for clustering samples of an audio recording for speaker diarization, the method comprising:

executing a multi-head self-attention network, wherein a set of hidden representations are learned from a set of mel-frequency cepstral coefficients representative of an audio sample using a feed-forward neural network;

constructing a triplet network using a set of samples, wherein each of the set of samples are operated on by the multi-head self-attention network, and wherein each input comprises three samples;

determining a similarity metric by calculating a triplet loss for each of the three samples of the input, wherein the similarity metric is used to cluster samples based on proximity to each other in an embedding space; and

performing temporal pooling on the set of hidden representations to reduce the set of hidden representations to a D-dimensional vector, wherein D is representative of a size of a hidden layer in the neural network.

13. The method of claim 12 , wherein a set of multi-head self-attention network parameters and a set of metric learner parameters are jointly optimized using back-propagation.

14. The method of claim 12 , wherein one of the set of three samples is an anchor sample, wherein one of a set of three samples is a positive sample from the same class as the anchor sample, and wherein one of the set of three samples is a negative sample from a different class from the anchor sample.

15. The method of claim 12 , wherein the result of clustering based on the similarity metric is speaker diarization, wherein samples in the same distinct cluster can be linked to one particular speaker.