IP Library Granted Patent US 11,238,839
Granted Patent B2
US 11,238,839 · App. 16/575,889 · Granted Feb 1, 2022

Audio stem identification systems and methods

Inventors: François Pachet (Paris, FR); Pierre Roy (Paris, FR); Mathieu Ramona (Paris, FR); Tristan Jehan (Brooklyn, NY); Juan José Bosch Vicente (Paris, FR)
Assignee: Spotify AB
G10H1/0025G06F16/632G06F16/635G06K9/6256G06N20/00G10H2250/221
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,238,839
App. No.
16/575,889
Granted
Feb 1, 2022
Kind
B2
Abstract

Methods, systems and computer program products are provided for identifying an audio stem. Audio stems (t 1 , . . . , t N ) are stored on a stem database and songs (S 1 , . . . , S P ) made with at least a subset of the plurality of the audio stems (t 1 , . . . , t N ) are stored on a song database. At least partially composed song (S*) having a predetermined number of pre-selected stems (k) are received. In turn, a probability vector (or relevance value or ranking) is produced for each stem (t 1 , . . . , t N ) to be complementary to the at least partially composed song (S*).

Claims (94)

1. A method for identifying an audio stem, comprising the steps of:

storing, on a stem database, a plurality of audio stems (t 1 , . . . , t N );

storing, on a song database, a plurality of songs (S 1 , . . . , S P ) made with at least a subset of the plurality of the audio stems (t 1 , . . . , t N );

receiving an at least partially composed song (S*) having a predetermined number of pre-selected stems (k); and

for each audio stem (t) of the plurality of audio stems (t 1 , . . . , t N ):

for each song (S) of the plurality of songs (S 1 , . . . , S P ) containing at least one of the audio stems (t 1 , . . . , t N ):

computing, using a similarity function, a similarity value of the song (S) to the at least partially composed song (S*);

determining, using a quantitative sociometric function, a sociometric status value of the song (S); and

computing a probability vector based on the similarity value and the sociometric status value, wherein the probability vector for each stem (t 1 , . . . , t N ) is complementary to the at least partially composed song (S*).

2. The method according to claim 1 , further comprising the steps of:

predicting, using a trained machine learning model, at least one stem from the stem database having a probability value above a predetermined threshold, thereby defining the at least one stem as being complementary to the predetermined number of preselected stems(k).

3. The method according to claim 2 , further comprising the steps of:

training, using a machine learning processor, the machine learning model based on any one of (i) a measure of success of a stem, (ii) a rank order of sociometric status values corresponding to a plurality of songs including at least the stem, (iii) a plurality of stems obtained from plural songs, or any combination of (i), (ii) and (iii).

4. The method according to claim 1 , further comprising:

for each audio stem (t) of the plurality of audio stems (t 1 , . . . , t N ):

initializing a corresponding relevance value (R 1 , . . . , R N );

for each song (S) of the plurality of songs (S 1 , . . . , S P ) containing at least one of the audio stems (t 1 , . . . , t N ):

computing a relevance value based on the similarity value and the sociometric status value; and

ranking the audio stem (t) relative to the other plurality of audio stems (t 1 , . . . , t N ).

5. The method according to claim 1 , wherein the similarity function is based on a cosine function.

6. The method according to claim 1 , wherein the quantitative sociometric function is based on a number of streams of the song (S).

7. The method according to claim 1 , wherein at least one of the pre-selected stems (k) is not in the stem database.

8. The method according to claim 1 , wherein at least one of the pre-selected stems (k) is in the stem database.

9. The method according to claim 1 , determining at least one of the plurality of stems (t 1 , . . . , t N ) having a relevance value greater than a predetermined threshold.

10. The method according to claim 1 , further comprising the steps of:

determining a distance between two songs d cos (s,s′) by calculating a cosine distance between vectors, wherein s is song vector according to sε{0,1} N , wherein s is a vector with a value of one (1) at position i if stem t i is used in s;

receiving an audio similarity measure between audio stems, such that a distance for any two stems is according to d audio (ti,tj)ε[0,1];

determining a distance between songs, based on d audio by defining, first, a distance between a song and a stem, according to:

d audio ( s,t i )=min{ d audio ( t i ,t j ), for t j εs }; and

determining a distance between two songs (s,s′) according to:

d audio ( s,s ′)=avg t′εs′ d audio ( s,t′ ).

11. A stem identification system, comprising:

a stem database configured to store a plurality of audio stems (t 1 , . . . , t N );

a song database configured to store a plurality of songs (S 1 , . . . , S P ) made with at least a subset of the plurality of the audio stems (t 1 , . . . , t N ); and

a stem processor configured to:

receive an at least partially composed song (S*) having a predetermined number of pre-selected stems (k),

for each audio stem (t) of the plurality of audio stems (t 1 , . . . , t N ):

for each song (S) of the plurality of songs (S 1 , . . . , S P ) containing at least one of the audio stems (t 1 , . . . , t N ):

compute, using a similarity function, a similarity value of the song (S) to the at least partially composed song (S*);

determine, using a quantitative sociometric function, a sociometric status value of the song (S); and

compute a probability vector based on the similarity value and the sociometric status value, wherein the probability vector for each stem (t 1 , . . . , t N ) is complementary to the at least partially composed song (S*).

12. The stem identification system according to claim 11 , the stem processor further configured to:

predict, using a trained machine learning model, at least one stem from the stem database having a probability value above a predetermined threshold, thereby defining the at least one stem as being complementary to the predetermined number of preselected stems(k).

13. The stem identification system according to claim 12 , further comprising:

a machine learning processor configured to train the machine learning model based on any one of (i) a measure of success of a stem, (ii) a rank order of sociometric status values corresponding to a plurality of songs including at least the stem, (iii) a plurality of stems obtained from plural songs, or any combination of (i), (ii) and (iii).

14. The stem identification system according to claim 11 , wherein the stem processor is further configured to:

for each audio stem (t) of the plurality of audio stems (t 1 , . . . , t N ):

initialize a corresponding relevance value (R 1 , . . . , R N );

for each song (S) of the plurality of songs (S 1 , . . . , S P ) containing at least one of the audio stems (t 1 , . . . , t N ):

compute a relevance value based on the similarity value and the sociometric status value; and

rank the audio stem (t) relative to the other plurality of audio stems (t 1 , . . . , t N ).

15. The stem identification system according to claim 11 , wherein the similarity function is based on a cosine function.

16. The stem identification system according to claim 11 , wherein the quantitative sociometric function is based on a number of streams of the song (S).

17. The stem identification system according to claim 11 , wherein at least one of the pre-selected stems (k) is not in the stem database.

18. The stem identification system according to claim 11 , wherein at least one of the pre-selected stems (k) is in the stem database.

19. The stem identification system according to claim 11 , wherein the stem processor is further configured to determine at least one of the plurality of stems (t 1 , . . . , t N ) having a relevance value greater than a predetermined threshold.

20. The stem identification system according to claim 11 , wherein the stem processor is further configured to:

determine a distance between two songs d cos (s,s′) by calculating a cosine distance between vectors, wherein s is song vector according to sε{0,1} N , wherein s is a vector with a value of one (1) at position i if stem t i is used in s;

receive an audio similarity measure between audio stems, such that a distance for any two stems is according to d audio (ti,tj)ε[0,1];

determine a distance between songs, based on d audio by defining, first, a distance between a song and a stem, according to:

d audio ( s,t i )=min{ d audio ( t i ,t j ), for t j εs }; and

determine a distance between two songs (s,s′) according to:

d audio ( s,s ′)=avg t′εs′ d audio ( s,t ′).

21. A non-transitory computer-readable medium having stored thereon one or more sequences of instructions for causing one or more processors to perform:

storing, on a stem database, a plurality of audio stems (t 1 , . . . , t N );

storing, on a song database, a plurality of songs (S 1 , . . . , S P ) made with at least a subset of the plurality of the audio stems (t 1 , . . . , t N );

receiving an at least partially composed song (S*) having a predetermined number of pre-selected stems (k);

for each audio stem (t) of the plurality of audio stems (t, . . . , t N ):

for each song (S) of the plurality of songs (S 1 , . . . , S P ) containing at least one of the audio stems (t 1 , . . . , tN):

computing, using a similarity function, a similarity value of the song (S) to the at least partially composed song (S*);

determining, using a quantitative sociometric function, a sociometric status value of the song (S); and

computing a probability vector based on the similarity value and the sociometric status value, wherein the probability vector for each stem (t 1 , . . . , t N ) is complementary to the at least partially composed song (S*).

22. The non-transitory computer-readable medium of claim 21 , further having stored thereon a sequence of instructions for causing the one or more processors to:

predict, using a trained machine learning model, at least one stem from the stem database having a probability value above a predetermined threshold, thereby defining the at least one stem as being complementary to the predetermined number of preselected stems(k).

23. The non-transitory computer-readable medium of claim 22 , further having stored thereon a sequence of instructions for causing the one or more processors to:

train the machine learning model based on any one of (i) a measure of success of a stem, (ii) a rank order of sociometric status values corresponding to a plurality of songs including at least the stem, (iii) a plurality of stems obtained from plural songs, or any combination of (i), (ii) and (iii).

24. The non-transitory computer-readable medium of claim 21 , further having stored thereon a sequence of instructions for causing the one or more processors to:

for each audio stem (t) of the plurality of audio stems (t 1 , . . . , t N ):

initialize a corresponding relevance value (R 1 , . . . , R N );

for each song (S) of the plurality of songs (S 1 , . . . , S P ) containing at least one of the audio stems (t 1 , . . . , t N ):

compute a relevance value based on the similarity value and the sociometric status value; and

rank the audio stem (t) relative to the other plurality of audio stems (t 1 , . . . , t N ).

25. The non-transitory computer-readable medium according to claim 21 , wherein the similarity function is based on a cosine function.

26. The non-transitory computer-readable medium according to claim 21 , wherein the quantitative sociometric function is based on a number of streams of the song (S).

27. The non-transitory computer-readable medium according to claim 21 , wherein at least one of the pre-selected stems (k) is not in the stem database.

28. The non-transitory computer-readable medium according to claim 21 , wherein at least one of the pre-selected stems (k) is in the stem database.

29. The non-transitory computer-readable medium of claim 21 , further having stored thereon a sequence of instructions for causing the one or more processors to determine at least one of the plurality of stems (t 1 , . . . , t N ) having a relevance value greater than a predetermined threshold.

30. The non-transitory computer-readable medium of claim 21 , further having stored thereon a sequence of instructions for causing the one or more processors to:

determine a distance between two songs d cos (s,s′) by calculating a cosine distance between vectors, wherein s is song vector according to sε{0,1} N , wherein s is a vector with a value of one (1) at position i if stem t i is used in s;

receive an audio similarity measure between audio stems, such that a distance for any two stems is according to d audio (ti,tj)ε[0,1];

determine a distance between songs, based on d audio by defining, first, a distance between a song and a stem, according to:

d audio ( s, t i )=min { d audio ( t i ,t j ), for t j εs}; and

determine a distance between two songs (s,s′) according to:

d audio ( s,s′ )=avg t′εs′ d audio ( s,t′ ).

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 19, 2020
From: PACHET, FRANCOIS; ROY, PIERRE; RAMONA, MATHIEU; JEHAN, TRISTAN; VICENTE, JUAN JOSE BOSCH
To: SPOTIFY AB
Reel/Frame 052993/0393 →
Continuity (1)
Related Publication 20210090536A1 · Mar 25, 2021
Cited By (2)
US 12,394,399 US 12,394,400