Granted Patent B2

US 12,198,677 · App. 17/826,908 · Granted Jan 14, 2025

Inventors: Chunlei Zhang (Bellevue, WA); Dong Yu (Palo Alto, CA)

Assignee: TENCENT AMERICA LLC

G10L15/063G10L15/16G10L2015/0631

Abstract

Claims (68)

1. A method of end-to-end speaker diarization (EESD) using neural speaker clustering, performed by at least one processor and comprising:

generating a set of speech labels corresponding to a set of speakers based on an input stream, wherein a speech label indicates whether dialogue of a speaker is speech or non-speech;

generating dialogue based on the set of speakers;

extracting speaker embeddings from the dialogue;

mapping the speaker embeddings to a cluster identification (ID), an overlapped speech value, or a non-speech value, based on a neural network; and

outputting EESD labels based on the mapping,

wherein the method further comprises:

sampling the set of speakers with N utterances for each speaker to generate a speaker subset including the speaker and a corresponding speech label;

adding a first utterance of the speaker and the corresponding speech label to the input stream and the output EESD labels; and

controlling an overlap length between a current utterance and a corresponding segment of the input stream based on a random variable L, where L is determined by a minimum value from among d or 0 (min (d,0)), where d follows a Gaussian distribution parameterized by mean μ and standard deviation σ.

2. The method of claim 1 , further comprising:

determining a speech label in the set of speech labels to be non-speech when a value of the speech label is 0; and

determining a speech label in the set of speech labels to be speech when the value of the speech label is 1.

3. The method of claim 2 , further comprising:

converting the speech label in the set of speech labels determined to be speech into an absolute speaker label.

4. The method of claim 3 , wherein a speaker embedding corresponding to the absolute speaker label is mapped to the cluster ID,

a speaker embedding corresponding to the speech label determined to be speech is mapped to the overlapped speech value, the overlapped speech value being 1, and

a speaker embedding corresponding to the speech label determined to be non-speech is mapped to the non-speech value, the non-speech value being 0.

5. The method of claim 1 , further comprising training the neural network by:

splitting the input stream into segments;

performing neural clustering on each of the segments; and

assigning each of the segments an existing cluster ID or a new cluster ID based on a predetermined distance threshold.

6. A apparatus for end-to-end speaker diarization (EESD) using neural speaker clustering, the apparatus comprising:

at least one memory configured to store program code; and

at least one processor configured to read the program code and operate as instructed by the program code, the program code configured to case the at least one processor to:

generate a set of speech labels corresponding to a set of speakers based on an input stream, wherein a speech label indicates whether dialogue of a speaker is speech or non-speech;

generate dialogue based on the set of speakers;

extract speaker embeddings from the dialogue;

map the speaker embeddings to a cluster identification (ID), an overlapped speech value, or a non-speech value, based on a neural network; and

output EESD labels based on the mapping,

wherein the at least one processor is further configured to:

sample the set of speakers with N utterances for each speaker to generate a speaker subset including the speaker and a corresponding speech label;

add a first utterance of the speaker and the corresponding speech label to the input stream and the output EESD labels; and

control an overlap length between a current utterance and a corresponding segment of the input stream based on a random variable L, where L is determined by a minimum value from among d or 0 (min (d,0)), where d follows a Gaussian distribution parameterized by mean μ and standard deviation σ.

7. The apparatus of claim 6 , wherein the at least one processor is configured to read the program code and operate as instructed by the program code to:

determine a speech label in the set of speech labels to be non-speech when a value of the speech label is 0; and

determine a speech label in the set of speech labels to be speech when the value of the speech label is 1.

8. The apparatus of claim 7 , wherein the at least one processor is configured to read the program code and operate as instructed by the program code to:

convert the speech label in the set of speech labels determined to be speech into an absolute speaker label.

9. The apparatus of claim 8 , wherein a speaker embedding corresponding to the absolute speaker label is mapped to the cluster ID,

a speaker embedding corresponding to the speech label determined to be speech is mapped to the overlapped speech value, the overlapped speech value being 1, and

a speaker embedding corresponding to the speech label determined to be non-speech is mapped to the non-speech value, the non-speech value being 0.

10. The apparatus of claim 6 , wherein the at least one processor is configured to read the program code and operate as instructed by the program code to train the neural network by:

splitting the input stream into segments;

performing neural clustering on each of the segments; and

assigning each of the segments an existing cluster ID or a new cluster ID based on a predetermined distance threshold.

11. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor of an apparatus for end-to-end speaker diarization (EESD) using neural speaker clustering, cause the at least one processor to:

generate a set of speech labels corresponding to a set of speakers based on an input stream, wherein a speech label indicates whether dialogue of a speaker is speech or non-speech;

generate dialogue based on the set of speakers;

extract speaker embeddings from the dialogue;

map the speaker embeddings to a cluster identification (ID), an overlapped speech value, or a non-speech value, based on a neural network; and

output EESD labels based on the mapping,

wherein the non-transitory computer-readable medium storing instructions cause the at least one processor to:

sample the set of speakers with N utterances for each speaker to generate a speaker subset including the speaker and a corresponding speech label;

add a first utterance of the speaker and the corresponding speech label to the input stream and the output EESD labels; and

control an overlap length between a current utterance and a corresponding segment of the input stream based on a random variable L, where L is determined by a minimum value from among d or 0 (min (d,0)), where d follows a Gaussian distribution parameterized by mean μ and standard deviation σ.

12. The non-transitory computer-readable medium of claim 11 , wherein the instructions further cause the at least one processor to:

determine a speech label in the set of speech labels to be non-speech when a value of the speech label is 0; and

determine a speech label in the set of speech labels to be speech when the value of the speech label is 1.

13. The non-transitory computer-readable medium of claim 12 , wherein the instructions further cause the at least one processor to:

convert the speech label in the set of speech labels determined to be speech into an absolute speaker label.

14. The non-transitory computer-readable medium of claim 13 , wherein a speaker embedding corresponding to the absolute speaker label is mapped to the cluster ID,

a speaker embedding corresponding to the speech label determined to be speech is mapped to the overlapped speech value, the overlapped speech value being 1, and

a speaker embedding corresponding to the speech label determined to be non-speech is mapped to the non-speech value, the non-speech value being 0.

15. The non-transitory computer-readable medium of claim 11 , wherein the instructions further cause the at least one processor to train the neural network by:

splitting the input stream into segments;

performing neural clustering on each of the segments; and

assigning each of the segments an existing cluster ID or a new cluster ID based on a predetermined distance threshold.

Assignments (1)

Continuity (1)

References Cited (16)