Granted Patent B2

US 12,367,890 · App. 17/925,025 · Granted Jul 22, 2025

Inventors: Stefan Uhlich (Stuttgart, DE); Giorgio Fabbro (Stuttgart, DE); Marc Ferras Font (Stuttgart, DE); Falk-Martin Hoffmann (Stuttgart, DE); Thomas Kemp (Stuttgart, DE)

Assignee: Sony Group Corporation

G10L21/028G10H1/361G10L13/04G10L15/08G10L15/22G10L25/18G10H2210/005G10L2015/088H04S3/008H04S2400/01

Abstract

Claims (25)

1. An electronic device, comprising:

circuitry configured to

perform audio source separation on an audio input signal to obtain a separated source and a residual signal, wherein the residual signal includes audio content from the audio input signal other than the separated source,

delay the separated source by a first delay amount to obtain a delayed separated source, wherein the first delay amount corresponds to a processing latency of phrase detection,

delay the residual signal by a second delay amount to obtain a delayed residual signal, wherein the second delay amount corresponds to a combined processing latency of the phrase detection and audio dubbing,

perform audio dubbing on the separated source based on replacement conditions to obtain a personalized separated source, and

mix the personalized separated source with the delayed residual signal to obtain a personalized audio signal.

2. The electronic device of claim 1 , wherein the circuitry is further configured to perform lyrics recognition on the separated source to obtain lyrics and to perform lyrics replacement on the lyrics based on the replacement conditions to obtain personalized lyrics.

3. The electronic device of claim 2 , wherein the circuitry is further configured to perform text-to-vocals synthesis on the personalized lyrics based on the separated source to obtain the personalized separated source.

4. The electronic device of claim 2 , wherein the circuitry is further configured to apply a Seq2Seq Model on the personalized lyrics based on the separated source to obtain a Mel-Spectrogram, and to apply a MelGAN generator on the Mel-Spectrogram to obtain the personalized separated source.

5. The electronic device of claim 1 , wherein the circuitry is further configured to perform the source separation on the audio input signal to obtain the separated source and a residual signal, and to perform mixing of the personalized separated source with the residual signal, to obtain a personalized audio signal.

6. The electronic device of claim 1 , wherein the circuitry is further configured to perform the audio dubbing on the separated source based on a trigger signal to obtain the personalized separated source.

7. The electronic device of claim 6 , wherein the circuitry is further configured to perform phrase detection on the separated source based on the replacement conditions obtain the trigger signal.

8. The electronic device of claim 7 , wherein the circuitry is further configured to perform speech recognition on the separated source to obtain transcript/lyrics.

9. The electronic device of claim 8 , wherein the circuitry is further configured to perform target phrase detection on the transcript/lyrics based on the replacement conditions to obtain the trigger signal.

10. The electronic device of claim 1 , wherein the separated source comprises vocals and the residual signal comprises an accompaniment.

11. The electronic device of claim 1 , wherein the separated source comprises speech and the residual signal comprises background noise.

12. The electronic device of claim 1 , wherein the replacement conditions are age dependent replacement conditions.

13. A method comprising:

performing audio source separation on an audio input signal to obtain a separated source and a residual signal, wherein the residual signal includes audio content from the audio input signal other than the separated source;

delaying the separated source by a first delay amount to obtain a delayed separated source, wherein the first delay amount corresponds to a processing latency of phrase detection;

delaying the residual signal by a second delay amount to obtain a delayed residual signal, wherein the second delay amount corresponds to a combined processing latency of the phrase detection and audio dubbing;

performing dubbing on the separated source based on replacement conditions to obtain a personalized separated source; and

mixing the personalized separated source with the delayed residual signal to obtain a personalized audio signal.

14. A computer program comprising instructions, the instructions when executed on a processor causing the processor to perform the method of claim 13 .

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (24)