Granted Patent B2

US 12,412,588 · App. 18/528,244 · Granted Sep 9, 2025

Inventors: William Carter Huffman (Cambridge, MA); Michael Pappas (Cambridge, MA)

Assignee: Modulate, Inc.

G10L21/013G10L15/02G10L15/063G10L15/22G10L19/018G10L2015/025G10L2021/0135G10L25/30

Abstract

Claims (46)

1. A method of training a speech conversion system, the method comprising:

receiving timbre data relating to a plurality of different voices, each of the different voices associated with a particular speaker;

mapping the plurality of different voices in a timbre vector space as a function of the timbre data;

using generative machine learning to refine the mapping of each of the plurality of different voices in the timbre vector space.

2. The method as defined by claim 1 , wherein the plurality of different voices include a first speaker and a second speaker, the method further comprising:

receiving timbre data relating to the first speaker;

receiving second timbre data relating to the second speaker;

using the generative machine learning system to produce, as a function of the first timbre data and the second timbre data, first candidate data that is a function of a first candidate speech segment in a first candidate voice;

receiving inconsistency data relating to a difference between the first candidate data and data relating to a plurality of other voices mapped in the timbre vector space;

feeding back the inconsistency data to the generative machine learning system;

refining the second timbre data in the timbre space as a function of said feeding back to produce refined second timbre data.

3. The method as defined by claim 1 , wherein the first timbre data is from an audio input of the first speaker.

4. The method as defined by claim 1 , further comprising:

using a generative machine learning system to produce second candidate data in a second candidate voice as a function of the first timbre data and the refined second timbre data;

receiving second inconsistency data, the second inconsistency data being a function of a plurality of voices, the second inconsistency data having information relating to a difference between the second candidate data and data relating to the second speaker.

5. The method as defined by claim 1 , further comprising transforming the first timbre data to into the second timbre.

6. The method as defined by claim 1 , wherein the second timbre data is received from an audio input in the second speaker.

7. The method as defined by claim 1 , further comprising using discriminative machine learning to refine the mapping of each of the plurality of different voices in the timbre vector space.

8. The method as defined by claim 1 , further comprising:

mapping a representation of the plurality of voices and the first candidate voice in a vector space as a function of a frequency distribution in the speech segment provided by each voice.

9. The method as defined by claim 8 , further comprising:

adjusting a representation of the first candidate voice relative to representations of the plurality of voices in the vector space to reflect the second candidate voice as a function of the inconsistency message.

10. The method as defined by claim 1 , wherein the inconsistency message is produced when the discriminative neural network has less than a 95 percent confidence interval that the first candidate voice is the voice.

11. A system for training a speech conversion system, the system comprising:

first timbre data that represents a first speech segment of a first speaker;

second timbre data that relates to a second speaker;

a generative machine learning system configured to produce first candidate data that represents a first candidate voice as a function of the first timbre data and the second timbre data;

an inconsistency message having information relating to a distinction between the first candidate data and data relating to the second speaker, the inconsistency message being a function of a plurality of voices, wherein the system uses the inconsistency message to refine the second timbre data in the timbre space to produce refined second timbre data.

12. The system as defined by claim 11 , wherein the first timbre data is from an audio input of the first speaker.

13. The system as defined by claim 11 , wherein the generative machine learning system is configured to produce second candidate data in a second candidate voice as a function of the first timbre data and the refined second timbre data.

14. The system as defined by claim 13 , further comprising second inconsistency data, the second inconsistency data being a function of a plurality of voices, the second inconsistency data having information relating to a difference between the second candidate data and data relating to the second speaker.

15. The system as defined by claim 11 , wherein the second timbre data is obtained from an audio input in the second speaker.

16. The system as defined by claim 11 , wherein the machine learning system is a neural network.

17. A method of building a speech conversion system using second speaker information from a second speaker, and timbre data that represents a speech segment of a first speaker, the method comprising:

receiving first timbre data that is a function of a first speech segment of a first speaker;

receiving second timbre data relating to the second speaker, the second timbre data being within a timbre space;

using a generative machine learning system to produce first candidate data that is a function of a first candidate speech segment in a first candidate voice as a function of the first timbre data and the second timbre data;

receiving inconsistency data, the inconsistency data being a function of a plurality of voices, the inconsistency data having information relating to a difference between the first candidate data and data relating to the second speaker;

feeding back the inconsistency data to the generative machine learning system; and

refining the generative machine learning system as a result of said feeding back.

18. The method as defined by claim 17 , wherein the inconsistency data is a function of a plurality of timbre data.

19. The method as defined by claim 17 , further comprising:

using the generative machine learning system to produce second candidate data in a second candidate voice as a function of the first timbre data and the feeding back;

receiving second inconsistency data, the second inconsistency data being a function of a plurality of voices, the second inconsistency data having information relating to a difference between the second candidate data and data relating to the second speaker.

20. The method as defined by claim 17 , further comprising:

using the generative machine learning system to produce sequential candidate data in a sequential candidate voice as a function of the first timbre data and the feeding back until the inconsistency data indicates no difference between the sequential candidate data and the data relating to the second speaker.

Assignments (2)

Continuity (5)

References Cited (294)

Cited By (1)

US 12,609,111