IP Library › Granted Patent US 11,114,085
Granted Patent B2
US 11,114,085 · App. 16/235,776 · Granted Sep 7, 2021

Text-to-speech from media content item snippets

Inventors: Rohit Kumar (Austin, TX); Henrik Lindström (Stockholm, SE); Henriette Cramer (San Francisco, CA); Sarah Mennicken (San Francisco, CA); Sravana Reddy (Cambridge, MA); Jennifer Thom-Santelli (Boston, MA)
Assignee: Spotify AB
G10L13/00G06F16/685G10L13/04
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,114,085
App. No.
16/235,776
Filed
Dec 28, 2018
Granted
Sep 7, 2021
Kind
B2
Examiner
HE, JIALONG
Art Unit
2659
USPC
704/258
Abstract

A text-to-speech engine creates audio output that includes synthesized speech and one or more media content item snippets. The input text is obtained and partitioned into text sets. A track having lyrics that match a part of one of the text sets is identified. The location of the track's audio that contains the lyric is extracted based on forced alignment data. The extracted audio is combined with synthesized speech corresponding to the remainder of the input text to form audio output.

Claims (135)

1. A system providing text-to-speech functionality, comprising:

a forced alignment data store having stored thereon forced alignment data for tracks;

a combining engine configured to:

obtain input text;

portion the input text into a first text set, a second text set, and a third text set;

identify a first track having first track lyrics that include the first text set;

identify, using the forced alignment data, a first audio location of the first track corresponding to the first text set;

create a first audio snippet containing the first audio location;

identify a second track based on the second track having second track lyrics containing the third text set;

create a second audio snippet from the second track based on the third text set;

using a speech synthesizer, create a synthesized utterance based on the second text set;

combine the first audio snippet, the second audio snippet, and the synthesized utterance to form combined audio; and

provide an audio output that includes the combined audio.

2. The system of claim 1 , wherein the forced alignment data describes alignment between audio data of the tracks and lyrics data of the tracks.

3. The system of claim 2 , wherein the forced alignment data, for each of the tracks, describes lyrics data and time data of where lyrics occur in the audio data.

4. The system of claim 1 , wherein the identifying the second track is further based on audio characteristics of the first track and the second track.

5. The system of claim 4 , wherein identifying the second track includes:

selecting the second track based on similarities in musical style between the first track and the second track.

6. Then system of claim 5 , wherein the similarities in musical style between the first track and the second track is determined based on a distance between the first track and the second track in a vector space.

7. The system of claim 1 , further comprising a forced alignment engine for aligning input track audio and input track lyrics.

8. The system of claim 7 , wherein the forced alignment engine is configured to find a Viterbi path through the input track lyrics and the input track audio under an acoustic model.

9. The system of claim 7 , wherein the forced alignment engine is configured to align the input track lyrics and input track audio on a line-by-line basis.

10. The system of claim 7 , wherein the forced alignment engine is configured to:

receive a third track having track metadata;

receive third track lyrics associated with the third track;

select an acoustic model from an acoustic model data store based on the track metadata;

using the acoustic model, generate third-track forced alignment data that aligns audio data of the third track and the third track lyrics; and

add the third-track forced alignment data to the forced alignment data store.

11. The system of claim 1 , wherein the combining engine is further configured to:

determine a quality described in the input text,

wherein identifying the first track is based at least in part on the quality.

12. The system of claim 11 , wherein the quality is a music genre or music artist.

13. A method comprising:

obtaining input text;

portioning the input text into a first text set, a second text set, and a third text set;

determine a quality described in the first text set,

identifying a first track based on:

the first track having first track lyrics that include the first text set; and

the first track having the quality;

identifying, using forced alignment data, a first audio location of the first track corresponding to the first text set;

creating a first audio snippet containing the first audio location;

identifying a second track based on the second track having second track lyrics containing the third text set based on similarities in musical style between the first track and the second track;

creating a second audio snippet from the second track based on the second text set;

using a speech synthesizer, creating a synthesized utterance based on the second text set;

combining the first audio snippet, the second audio snippet, and the synthesized utterance to form combined audio; and

providing an audio output that includes the combined audio.

14. A system providing text-to-speech functionality, comprising:

a forced alignment engine for aligning input track audio and input track lyrics, wherein the forced alignment engine is configured to:

receive a third track having track metadata;

receive third track lyrics associated with the third track;

select an acoustic model from an acoustic model data store based on the track metadata;

using the acoustic model, generate third-track forced alignment data that aligns audio data of the third track and the third track lyrics; and

add the third-track forced alignment data to a forced alignment data store;

the forced alignment data store having stored thereon forced alignment data for tracks;

a combining engine configured to:

obtain input text;

portion the input text into a first text set and a second text set;

identify a first track having first track lyrics that include the first text set;

identify, using the forced alignment data, a first audio location of the first track corresponding to the first text set;

create a first audio snippet containing the first audio location;

using a speech synthesizer, create a synthesized utterance based on the second text set;

combine the first audio snippet and the synthesized utterance to form combined audio; and

provide an audio output that includes the combined audio.

15. The system of claim 14 , wherein the forced alignment data describes alignment between audio data of the tracks and lyrics data of the tracks.

16. The system of claim 15 , wherein the forced alignment data, for each of the tracks, describes lyrics data and time data of where lyrics occur in the audio data.

17. The system of claim 14 ,

wherein the input text has a third text set; and

wherein the combining engine is further configured to:

identify a second track based on the second track having second track lyrics containing the third text set; and

create a second audio snippet from the second track based on the third text set,

wherein the combined audio further includes the second audio snippet.

18. The system of claim 17 , wherein the identifying the second track is further based on audio characteristics of the first track and the second track.

19. The system of claim 18 , wherein identifying the second track includes:

selecting the second track based on similarities in musical style between the first track and the second track.

20. Then system of claim 19 , wherein the similarities in musical style between the first track and the second track is determined based on a distance between the first track and the second track in a vector space.

21. The system of claim 14 , wherein the forced alignment engine is configured to find a Viterbi path through the input track lyrics and the input track audio under an acoustic model.

22. The system of claim 14 , wherein the forced alignment engine is configured to align the input track lyrics and input track audio on a line-by-line basis.

23. The system of claim 14 , wherein the combining engine is further configured to:

determine a quality described in the input text,

wherein identifying the first track is based at least in part on the quality.

24. The system of claim 23 , wherein the quality is a music genre or music artist.

25. The method of claim 13 , wherein the forced alignment data describes alignment between audio data and lyrics data of tracks.

26. The method of claim 25 , wherein the forced alignment data, for each of the tracks, describes lyrics data and time data of where lyrics occur in the audio data.

27. The method of claim 13 , wherein the identifying the second track is further based on audio characteristics of the first track and the second track.

28. The method of claim 27 , wherein identifying the second track includes:

selecting the second track based on similarities in musical style between the first track and the second track.

29. Then method of claim 28 , wherein the similarities in musical style between the first track and the second track is determined based on a distance between the first track and the second track in a vector space.

30. The method of claim 13 , further comprising:

using a forced alignment engine to find a Viterbi path through input track lyrics and input track audio under an acoustic model.

31. The method of claim 13 , further comprising:

using a forced alignment engine to align input track lyrics and input track audio on a line-by-line basis.

32. The method of claim 13 , further comprising:

with a forced alignment engine:

receiving a third track having track metadata;

receiving third track lyrics associated with the third track;

selecting an acoustic model from an acoustic model data store based on the track metadata;

with the acoustic model, generating third-track forced alignment data that aligns audio data of the third track and the third track lyrics; and

adding the third-track forced alignment data to a forced alignment data store.

33. The method of claim 13 , further comprising:

determining a quality described in the input text,

wherein identifying the first track is based at least in part on the quality.

34. The method of claim 33 , wherein the quality is a music genre or music artist.

35. A method comprising:

with a forced alignment engine for aligning input track audio and input track lyrics:

receiving a third track having track metadata;

receiving third track lyrics associated with the third track;

selecting an acoustic model from an acoustic model data store based on the track metadata;

using the acoustic model, generating third-track forced alignment data that aligns audio data of the third track and the third track lyrics; and

adding the third-track forced alignment data to the forced alignment data store obtaining input text;

portioning the input text into a first text set a second text set;

determine a quality described in the first text set,

identifying a first track based on:

the first track having first track lyrics that include the first text set; and

the first track having the quality;

identifying, using forced alignment data, a first audio location of the first track corresponding to the first text set;

creating a first audio snippet containing the first audio location;

using a speech synthesizer, creating a synthesized utterance based on the second text set;

combining the first audio snippet and the synthesized utterance to form combined audio; and

providing an audio output that includes the combined audio.

36. The method of claim 35 , wherein the forced alignment data describes alignment between audio data and lyrics data of tracks.

37. The method of claim 36 , wherein the forced alignment data, for each of the tracks, describes lyrics data and time data of where lyrics occur in the audio data.

38. The method of 35 , further comprising:

identifying a second track based on audio characteristics of the first track and the second track,

wherein the combined audio includes a second audio snipped from the second track.

39. The method of claim 38 , wherein identifying the second track includes:

selecting the second track based on similarities in musical style between the first track and the second track.

40. Then method of claim 39 , wherein the similarities in musical style between the first track and the second track is determined based on a distance between the first track and the second track in a vector space.

41. The method of claim 35 , further comprising:

with the forced alignment engine, finding a Viterbi path through input track lyrics and input track audio under an acoustic model.

42. The method of claim 35 , further comprising:

with the forced alignment engine, aligning input track lyrics and input track audio on a line-by-line basis.

43. The method of claim 35 , further comprising:

determining a quality described in the input text,

wherein identifying the first track is based at least in part on the quality.

44. The method of claim 43 , wherein the quality is a music genre or music artist.

Assignments (4)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 27, 2023
From: SPOTIFY USA INC.
To: SPOTIFY AB
Reel/Frame 063105/0815 →
CORRECTIVE ASSIGNMENT TO CORRECT THE RECEIVING PARTY'S NAME ON THE COVER SHEET PREVIOUSLY RECORDED AT REEL: 055025 FRAME: 0669. ASSIGNOR(S) HEREBY CONFIRMS THE EMPLOYMENT AGREEMENT. Recorded Feb 6, 2023
From: KUMAR, ROHIT
To: SPOTIFY USA INC.
Reel/Frame 062662/0199 →
EMPLOYMENT AGREEMENT Recorded Jan 19, 2021
From: KUMAR, ROHIT
To: SPOTIFY AB
Reel/Frame 055025/0669 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 6, 2021
From: LINDSTROM, HENRIK; CRAMER, HENRIETTE; MENNICKEN, SARAH; REDDY, SRAVANA; THOM-SANTELLI, JENNIFER
To: SPOTIFY AB
Reel/Frame 054828/0573 →
Continuity (1)
Related Publication 20200211531A1 · Jul 2, 2020
Cited By (1)
US 12,437,744