Progressive contrastive learning framework for self-supervised speaker verification
A method includes receiving, by a first encoder, an original speech segment, receiving, by a second encoder, an augmented speech segment of the original speech segment, generating, by the first encoder, a first speaker representation based on the original speech segment, generating, by the second encoder, a second speaker representation based on the augmented speech segment, and generating a contrastive loss based on the first speaker representation and the second speaker representation.
1. A method for speaker identification and verification, the method being executed by at least one processor, and the method comprising:
training a dynamic clustering model for speaker identification and verification, the training comprising:
receiving, by a first encoder, a plurality of original speech segments;
receiving, by a second encoder, a plurality of augmented speech segments based on the plurality of the original speech segments,
generating, by the first encoder, first speaker representations based on the plurality of original speech segments;
generating, by the second encoder, second speaker representations based on the plurality of augmented speech segments;
prior to estimating a cluster for the second speaker representations, inputting the second speaker representations into a memory queue;
dynamically determining a number of clusters available for classification of the plurality of original speech segments based statistical characteristics of the second speaker representations in the memory queue;
assigning a respective cluster to each second speaker representation among the second speaker representations, the cluster being one from among the dynamically determined number of clusters; and
generating a contrastive loss based on the first speaker representations and the assigned second speaker representations, wherein the contrastive loss is based on queue-level centroids associated with the dynamically determined number of clusters instead of dataset-level centroids; and
verifying an identity of a speaker of a first audio by applying the first audio as input into the trained dynamic clustering model for speaker identification and verification.
2. The method of claim 1 , wherein the second encoder comprises a momentum encoder.
3. The method of claim 1 , wherein the dynamically determining the number of clusters available for classification comprises updating the memory queue based on the second speaker representations.
4. The method of claim 3 , wherein the dynamically determining the number of clusters available for classification further comprises generating an affinity matrix of the updated memory queue.
5. The method of claim 4 , wherein the dynamically determining the number of clusters available for classification further comprises decomposing the affinity matrix into eigenvalues.
6. The method of claim 5 , wherein the dynamically determining the number of clusters available for classification further comprises estimating a cluster number based on a maximum eigengap of the eigenvalues.
7. The method of claim 3 , wherein updating the memory queue comprises removing an oldest mini-batch from the memory queue.
8. An apparatus for speaker identification and verification, comprising:
at least one memory configured to store computer program code; and
at least one processor configured to access said computer program code and operate as instructed by said computer program code, said computer program code comprising:
training code configured to cause the at least one processor to train a dynamic clustering model for speaker identification and verification, the training code comprising:
first receiving code configured to cause the at least one processor to receive, by a first encoder, a plurality of original speech segments;
second receiving code configured to cause the at least one processor to receive, by a second encoder, a plurality of augmented speech segments based on the plurality of the original speech segments,
first generating code configured to cause the at least one processor to generate, by the first encoder, first speaker representations based on the plurality of original speech segments;
second generating code configured to cause the at least one processor to generate, by the second encoder, second speaker representations based on the plurality of augmented speech segments;
prior to estimating a cluster for the second speaker representations, inputting the second speaker representations into a memory queue;
dynamically determining code configured to cause the at least one processor to dynamically determine a number of clusters available for classification of the plurality of original speech segments based statistical characteristics of the second speaker representations in the memory queue;
assigning code configured to cause the at least one processor to assign each second speaker representation among the second speaker representations to a cluster from among the dynamically determined number of clusters; and
third generating code configured to cause the at least one processor to generate a contrastive loss based on the first speaker representations and the assigned second speaker representations, wherein the contrastive loss is based on queue-level centroids associated with the dynamically determined number of clusters instead of dataset-level centroids; and
verifying code configured to cause the at least one processor to verify an identity of a speaker of a first audio by applying the first audio as input into the trained dynamic clustering model for speaker identification and verification.
9. The apparatus of claim 8 , wherein the second encoder comprises a momentum encoder.
10. The apparatus of claim 8 , wherein the dynamically determining code further comprises updating code configured to cause the at least one processor to update the memory queue based on the second speaker representation.
11. The apparatus of claim 10 , wherein the dynamically determining code is further configured to cause the at least one processor to generate an affinity matrix of the updated memory queue.
12. The apparatus of claim 11 , wherein the dynamically determining code is further configured to cause the at least one processor to generate the contrastive loss by decomposing the affinity matrix into eigenvalues.
13. The apparatus of claim 12 , wherein the dynamically determining code is further configured to cause the at least one processor to generate the contrastive loss by estimating a cluster number based on a maximum eigengap of the eigenvalues.
14. The apparatus of claim 10 , wherein the updating code is further configured to cause the at least one processor to update the memory queue removing an oldest mini-batch from the memory queue.
15. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor for speaker identification and verification, cause the at least one processor to:
train a dynamic clustering model for speaker identification and verification, the training comprising:
receive, by a first encoder, a plurality of original speech segments;
receive, by a second encoder, a plurality of augmented speech segments based on the plurality of the original speech segments,
generate, by the first encoder, first speaker representations based on the plurality of original speech segments;
generate, by the second encoder, second speaker representations based on the plurality of augmented speech segments;
prior to estimating a cluster for the second speaker representations, input the second speaker representations into a memory queue;
dynamically determine a number of clusters available for classification of the plurality of original speech segments based statistical characteristics of the second speaker representations in the memory queue;
assign a respective cluster to each second speaker representation among the second speaker representations, the cluster being one from among the dynamically determined number of clusters; and
generate a contrastive loss based on the first speaker representations and the assigned second speaker representations, wherein the contrastive loss is based on queue-level centroids associated with the dynamically determined number of clusters instead of dataset-level centroids; and
verify an identity of a speaker of a first audio by applying the first audio as input into the trained dynamic clustering model for speaker identification and verification.
16. The storage medium of claim 15 , wherein the second encoder comprises a momentum encoder.
17. The storage medium of claim 15 , wherein the instructions, when executed, cause the at least one processor to update the memory queue based on the second speaker representation.
18. The storage medium of claim 17 , wherein the instructions, when executed, cause the at least one processor to generate an affinity matrix of the updated memory queue.
19. The storage medium of claim 18 , wherein the instructions, when executed, cause the at least one processor to generate the contrastive loss by decomposing the affinity matrix into eigenvalues.
20. The storage medium of claim 19 , wherein the instructions, when executed, cause the at least one processor to generate the contrastive loss by estimating a cluster number based on a maximum eigengap of the eigenvalues.