Speaker separation based on real-time latent speaker state characterization
Systems, methods, and non-transitory computer-readable media can obtain a stream of audio waveform data that represents speech involving a plurality of speakers. As the stream of audio waveform data is obtained, a plurality of audio chunks can be determined. An audio chunk can be associated with one or more identity embeddings. The stream of audio waveform data can be segmented into a plurality of segments based on the plurality of audio chunks and respective identity embeddings associated with the plurality of audio chunks. A segment can be associated with a speaker included in the plurality of speakers. Information describing the plurality of segments associated with the stream of audio waveform data can be provided.
1. A method comprising:
obtaining a stream of audio waveform data that represents speech involving a plurality of speakers;
determining a plurality of audio chunks, wherein the plurality of audio chunks is associated with one or more identity embeddings;
generating one or more identity-based pretrained markers from the one or more identity embeddings; and
segmenting the stream of audio waveform data into a plurality of segments, the segmenting including assigning a first speaker of the plurality of speakers to a first audio chunk of the plurality of audio chunks or adding a new speaker to the plurality of speakers to assign to the first audio chunk, wherein the assigning is based on a degree of matching of an identity-based pretrained marker of the one or more identity-based pretrained markers to a previously-generated identity-based pretrained marker, the previously-generated identity-based pretrained marker being stored in a speaker inventory.
2. The method of claim 1 , wherein the segmenting is performed based on a computational graph.
3. The method of claim 1 , wherein the one or more identity-based pretrained markers include speaker state information, and wherein the degree of matching is based in part on the speaker state information.
4. The method of claim 1 , wherein the one or more identity embeddings are generated by a temporal convolutional network that pre-processes the plurality of audio chunks and outputs the one or more identity embeddings.
5. The method of claim 1 , wherein the degree of matching is based on output from a temporal convolutional network that takes the identity-based pretrained marker of the one or more identity-based pretrained markers and the previously-generated identity-based pretrained marker as inputs.
6. The method of claim 1 , wherein the speaker inventory maintains associations between the plurality of speakers, the plurality of audio chunks, and the one or more identity embeddings.
7. The method of claim 1 , wherein the speaker inventory is refreshed at regular time intervals to reconcile a first speaker in the speaker inventory and a second speaker in the speaker inventory as a same speaker.
8. The method of claim 1 , wherein the adding of the new speaker is based on a determination that the identity-based pretrained marker of the one or more identity-based pretrained markers does not match any of a plurality of previously-generated identity-based pretrained markers.
9. The method of claim 2 , further comprising providing information describing the plurality of segments associated with the stream of audio waveform data.
10. The method of claim 9 , wherein the information includes a label for a segment of the plurality of segments indicating that the segment represents a particular speaker of the plurality of speakers.
11. A system comprising:
obtaining a stream of audio waveform data that represents speech involving a plurality of speakers;
determining a plurality of audio chunks, wherein the plurality of audio chunks is associated with one or more identity embeddings;
generating one or more identity-based pretrained markers from the one or more identity embeddings; and
segmenting the stream of audio waveform data into a plurality of segments, the segmenting including assigning a first speaker of the plurality of speakers to a first audio chunk of the plurality of audio chunks or adding a new speaker to the plurality of speakers to assign to the first audio chunk, wherein the assigning is based on a degree of matching of an identity-based pretrained marker of the one or more identity-based pretrained markers to a previously-generated identity-based pretrained marker, the previously-generated identity-based pretrained marker being stored in a speaker inventory.
12. The system of claim 11 , wherein the one or more identity-based pretrained markers include speaker state information, and wherein the degree of matching is based in part on the speaker state information.
13. The system of claim 11 , wherein the speaker inventory maintains associations between the plurality of speakers, the plurality of audio chunks, and the one or more identity embeddings.
14. The system of claim 11 , wherein the speaker inventory is refreshed at regular time intervals to reconcile a first speaker in the speaker inventory and a second speaker in the speaker inventory as a same speaker.
15. The system of claim 11 , wherein the adding of the new speaker is based a determination that the identity-based pretrained marker of the one or more identity-based pretrained markers does not match any of a plurality of previously-generated identity-based pretrained markers.
16. A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to perform:
obtaining a stream of audio waveform data that represents speech involving a plurality of speakers;
determining a plurality of audio chunks, wherein the plurality of audio chunks is associated with one or more identity embeddings;
generating one or more identity-based pretrained markers from the one or more identity embeddings; and
segmenting the stream of audio waveform data into a plurality of segments, the segmenting including assigning a first speaker of the plurality of speakers to a first audio chunk of the plurality of audio chunks or adding a new speaker to the plurality of speakers to assign to the first audio chunk, wherein the assigning is based on a degree of matching of an identity-based pretrained marker of the one or more identity-based pretrained markers to a previously-generated identity-based pretrained marker, the previously-generated identity-based pretrained marker being stored in a speaker inventory.
17. The non-transitory computer-readable storage medium of claim 16 , wherein the one or more identity-based pretrained markers include speaker state information, and wherein the degree of matching is based in part on the speaker state information.
18. The non-transitory computer-readable storage medium of claim 16 , wherein the speaker inventory maintains associations between the plurality of speakers, the plurality of audio chunks, and the one or more identity embeddings.
19. The non-transitory computer-readable storage medium of claim 16 , wherein the speaker inventory is refreshed at regular time intervals to reconcile a first speaker in the speaker inventory and a second speaker in the speaker inventory as a same speaker.
20. The non-transitory computer-readable storage medium of claim 16 , wherein the adding of the new speaker is based a determination that the identity-based pretrained marker of the one or more identity-based pretrained markers does not match any of a plurality of previously-generated identity-based pretrained markers.