IP Library Granted Patent US 12,347,409
Granted Patent B1
US 12,347,409 · App. 17/342,400 · Granted Jul 1, 2025

Systems and methods for converting music into segmented digital assets for dynamic uses in digital experiences

Inventors: Nicholas L. Venti (La Mesa, CA); Tyson S. McDowell (La Mesa, CA); Markus Cremer (La Mesa, CA)
G10H1/0025G10G1/00G10H2210/061G10H2240/131
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,347,409
App. No.
17/342,400
Granted
Jul 1, 2025
Kind
B1
Abstract

Systems and methods for converting music into segmented digital assets for dynamic uses in digital experiences and discovery thereto are provided. In one example, a digital music file having at least one song is processed to identify segmented portions of the song, which are separated. The segmented portions of the song are analyzed to identify at least one musical quality of each of the segmented portions. An emotion, style, or vibe attribute is correlated to each of the segmented portions based on the musical quality identified. A composite soundtrack is constructed from the segmented portions by arranging at least a portion of the segmented portions based on the emotion, style, or vibe attribute correlated with each of the segmented portions. The composite soundtrack may be used within a digital entertainment experience.

Claims (44)

1. A method for converting music into segmented digital assets for dynamic uses in a digital entertainment experience, the method comprising:

uploading at least one digital music file, or pointing to a database having the at least one digital music file, to a computerized system, the at least one digital music file having at least one song;

processing, with a computerized processor of the computerized system, the at least one digital music file to identify segmented portions of the at least one song;

separating the at least one song into the segmented portions;

analyzing the segmented portions of the at least one song to identify at least one musical quality of each of the segmented portions, wherein an emotion, style, or vibe attribute is correlated to each of the segmented portions based on the at least one musical quality identified;

analyzing a digital entertainment environment having a visual component to identify at least one event occurring in and coming from the digital entertainment environment and depicted in the visual component;

constructing a composite soundtrack from the segmented portions by arranging at least a portion of the segmented portions in a composite order, wherein the composite order is irrespective of an original chronological order of the segmented portions of the at least one song, and is based on the emotion, style, or vibe attribute correlated with each of the segmented portions arranged relative to the at least one event occurring in and coming from the digital entertainment environment having the visual component; and

playing at least a portion of the composite soundtrack and the visual component of the digital entertainment environment, wherein the portion of the composite soundtrack accompanies and is played based on the at least one event occurring in the digital entertainment environment.

2. The method of claim 1 , further comprising constructing at least one additional composite soundtrack from the segmented portions by re-arranging at least a portion of the segmented portions of the at least one song based on the emotion, style, or vibe attribute correlated with each of the segmented portions, wherein the at least one additional composite soundtrack is used with the digital entertainment environment in a period of time that is different from when the composite soundtrack is used within the digital entertainment environment.

3. The method of claim 1 , further comprising dynamically adjusting the composite soundtrack, whereby one of a plurality of segmented portions correlated to the emotion, style, or vibe attribute is playable within the composite soundtrack in substantially real-time.

4. The method of claim 1 , wherein the digital entertainment environment further comprises an electronic game.

5. The method of claim 1 , further comprising providing at least one link within the digital entertainment environment, the link being associated with the composite soundtrack, wherein at least one link is electronically connectable to a third party database having the at least one song.

6. The method of claim 1 , wherein using the composite soundtrack within the digital entertainment environment further comprises streaming, in real-time, the composite soundtrack through a network connection, wherein the composite soundtrack is stored in a remote location from the digital entertainment environment.

7. The method of claim 1 , wherein processing the at least one digital music file to identify the segmented portions of the at least one song further comprises segmenting the at least one song based on at least one of: a beat of the at least one song; a bar of the at least one song; a tempo of the at least one song; or a musical transition within the at least one song.

8. The method of claim 1 , wherein analyzing the segmented portions of the at least one song to identify at least one musical quality of each of the segmented portions further comprising identifying at least one of:

a musical style of the segmented portion;

specific or composite intensity including density of onsets, volume level or composition complexity score;

a vocal presence within the segmented portion;

an instrumental presence within the segmented portion;

a volume level within the segmented portion; or

a tempo of the segmented portion.

9. A method of providing a composite soundtrack for use in a digital entertainment experience, the method comprising:

providing a computerized system having a database storing a plurality of digital music files' time-index attributes, each of the plurality of digital music files having at least one song;

selecting, by a user, one or more songs to be used to generate a soundtrack for a visual digital entertainment environment;

mapping a plurality of events within the visual digital entertainment environment to a plurality of emotion, style, or vibe attributes, respectively, wherein the plurality of events are occurring in and coming from the digital entertainment environment;

constructing a composite soundtrack from segmented portions of the one or more songs, the segmented portions being selected based on at least one musical quality therein, wherein at least one of the segmented portions is correlated to each of the plurality of emotion, style, or vibe attributes to form the composite soundtrack from an arrangement of the segmented portions in a composite order, wherein the composite order is irrespective of an original chronological order of the segmented portions of the one or more songs, wherein the segmented portions are arranged relative to at least one of the events of the plurality of events occurring in and coming from the digital entertainment environment having the visual component; and

outputting at least a portion of the composite soundtrack, which accompanies and is played based on at least one event of the plurality of events within the visual digital entertainment environment.

10. The method of claim 9 , wherein the visual digital entertainment environment further comprises an electronic game.

11. The method of claim 9 , wherein outputting the composite soundtrack to be played within the visual digital entertainment environment further comprises at least one of:

streaming, in real-time, the composite soundtrack through a network connection, wherein the composite soundtrack is stored in a remote location from the visual digital entertainment environment; or

downloading the composite soundtrack through the network connection to the visual digital entertainment environment.

12. The method of claim 9 , wherein mapping the plurality of events within the visual digital entertainment environment to the plurality of emotion, style, or vibe attributes, respectively, is determined based on a predetermined correlation of the plurality of events within the visual digital entertainment environment to the plurality of emotion, style, or vibe attributes, wherein the predetermined correlation is defined by a developer of the visual digital entertainment environment.

13. The method of claim 9 , wherein mapping the plurality of events within the visual digital entertainment environment to the plurality of emotion, style, or vibe attributes, respectively, is determined based on analysis of a preexisting soundtrack for the visual digital entertainment environment, wherein the plurality of emotional, style, or vibe attributes is identified within the preexisting soundtrack.

14. The method of claim 9 , wherein the composite soundtrack is dynamically adjustable, whereby one of a plurality of segmented portions correlated to one of the plurality of emotion, style, or vibe attributes is playable within the composite soundtrack in substantially real-time.

15. The method of claim 9 , further comprising providing at least one link within the visual digital entertainment environment, the link being associated with the composite soundtrack, wherein at least one link is electronically connectable to a third party database having the at least one song.

16. A method for converting music into segmented digital assets for dynamic uses in a digital entertainment experience, the method comprising:

processing, with a computerized processor of the computerized system, at least a first digital music file or a stream of the first digital music file to identify segmented portions of at least one song within the first digital music file;

separating the at least one song into the segmented portions;

analyzing the segmented portions of the at least one song to identify at least one musical quality of each of the segmented portions, wherein an emotion, style, or vibe attribute is correlated to each of the segmented portions based on the at least one musical quality identified;

processing, with the computerized processor of the computerized system, at least a second digital music file or a stream of the second digital music file to identify segmented portions of at least one song within the second digital music file which matches an emotion, style, or vibe attribute of at least one segmented portion of the first digital music file;

analyzing a digital entertainment environment having a visual component to identify at least one event occurring in and coming from the digital entertainment environment and depicted in the visual component;

constructing a composite soundtrack from the segmented portions of the first and second digital music files by arranging the segmented portions in a composite order, wherein the composite order is irrespective of an original chronological order of the segmented portions of the at least one song, and is based on the emotion, style, or vibe attribute correlated with each of the segmented portions arranged relative to the at least one event occurring in and coming from the digital entertainment environment having the visual component; and

playing at least a portion of the composite soundtrack and the visual component of the digital entertainment environment, wherein the portion of the composite soundtrack accompanies and is played based on the at least one event occurring in the digital entertainment environment.

17. The method of claim 16 , further comprising the step of iteratively searching for additional digital music files which have segments which match the emotion, style, or vibe attribute of at least one segmented portion of the first or second digital music files, wherein the composite soundtrack is constructed from segmented portions of the first, second, and additional digital music files.

References Cited (110)
US 7637424B2 · Silverbrook · 2009 [cited by applicant]
US 7730216B1 · Issa · 2010 [cited by applicant]
US 8819030B1 · Freed et al. · 2014 [cited by applicant]
US 8928727B1 · Milligan · 2015 [cited by applicant]
US 9021088B2 · Bilinski et al. · 2015 [cited by applicant]
US 9992316B2 · Hardi · 2018 [cited by applicant]
US 10037780B1 · Roberts et al. · 2018 [cited by applicant]
US 10048931B2 · Vartakavi et al. · 2018 [cited by applicant]
US 10178365B1 · Singh et al. · 2019 [cited by applicant]
US 10333876B2 · Guthery et al. · 2019 [cited by applicant]
US 10474422B1 · Venti et al. · 2019 [cited by applicant]
US 10692537B2 · Eppolito · 2020 [cited by applicant]
US 10891103B1 · Venti et al. · 2021 [cited by applicant]
US 10956945B1 · Lewis · 2021 [cited by applicant]
US 11169770B1 · Venti et al. · 2021 [cited by applicant]
US 11449306B1 · Venti et al. · 2022 [cited by applicant]
US 11481434B1 · Venti et al. · 2022 [cited by applicant]
US 20050147256A1 · Peters · 2005 [cited by applicant]
US 20080270138A1 · Knight · 2008 [cited by applicant]
US 20090013263A1 · Fortnow · 2009 [cited by applicant]
US 20090128335A1 · Leung · 2009 [cited by applicant]
US 20090177303A1 · Logan · 2009 [cited by applicant]
US 20090187624A1 · Brownholtz · 2009 [cited by applicant]
US 20090300670A1 · Barish · 2009 [cited by applicant]
US 20100023578A1 · Brant · 2010 [cited by applicant]
US 20100031299A1 · Harrang · 2010 [cited by applicant]
US 20100164956A1 · Hyndman · 2010 [cited by applicant]
US 20100241711A1 · Ansari · 2010 [cited by applicant]
US 20110161409A1 · Nair · 2011 [cited by applicant]
US 20110225417A1 · Maharajh · 2011 [cited by applicant]
US 20110261149A1 · Anuar · 2011 [cited by applicant]
US 20120069131A1 · Abelow · 2012 [cited by applicant]
US 20120144343A1 · Tseng · 2012 [cited by applicant]
US 20120144979A1 · Tansley · 2012 [cited by applicant]
US 20120278387A1 · Garcia · 2012 [cited by applicant]
US 20120304087A1 · Walkin · 2012 [cited by applicant]
US 20130066964A1 · Odio · 2013 [cited by applicant]
US 20130173742A1 · Thomas · 2013 [cited by applicant]
US 20130297686A1 · Bilinski · 2013 [cited by applicant]
US 20140123041A1 · Morse · 2014 [cited by applicant]
US 20140137144A1 · Jarvenpaa · 2014 [cited by applicant]
US 20140181110A1 · Imbruce et al. · 2014 [cited by applicant]
US 20140214848A1 · Devkar et al. · 2014 [cited by applicant]
US 20140280498A1 · Frankel · 2014 [cited by applicant]
US 20140289330A1 · Liu · 2014 [cited by applicant]
US 20140301573A1 · Kiely · 2014 [cited by examiner]
US 20140337761A1 · Glass · 2014 [cited by applicant]
US 20150089397A1 · Gorod et al. · 2015 [cited by applicant]
US 20150169747A1 · Hume · 2015 [cited by applicant]
US 20150193516A1 · Harb · 2015 [cited by applicant]
US 20150195620A1 · Buchner · 2015 [cited by applicant]
US 20150215597A1 · Xu · 2015 [cited by applicant]
US 20150242525A1 · Perlegos · 2015 [cited by applicant]
US 20150318020A1 · Pribula · 2015 [cited by applicant]
US 20150339300A1 · Stoddard, II et al. · 2015 [cited by applicant]
US 20150373065A1 · Holmquist · 2015 [cited by applicant]
US 20160149956A1 · Birnbaum · 2016 [cited by applicant]
US 20160173683A1 · Abreu · 2016 [cited by applicant]
US 20160196105A1 · Vartakavi et al. · 2016 [cited by applicant]
US 20160217328A1 · Yanai · 2016 [cited by applicant]
US 20160224311A1 · Touch · 2016 [cited by applicant]
US 20160227115A1 · Bin Mahfooz et al. · 2016 [cited by applicant]
US 20160246452A1 · Bockhold · 2016 [cited by applicant]
US 20160248840A1 · Bockhold · 2016 [cited by applicant]
US 20160300594A1 · Allen · 2016 [cited by applicant]
US 20170024399A1 · Boyle · 2017 [cited by applicant]
US 20170154615A1 · Serletic et al. · 2017 [cited by applicant]
US 20170289202A1 · Krasadakis · 2017 [cited by applicant]
US 20170372525A1 · Rosenthal et al. · 2017 [cited by applicant]
US 20180053510A1 · Kofman · 2018 [cited by applicant]
US 20180103292A1 · Zheng · 2018 [cited by applicant]
US 20180152736A1 · Alexander · 2018 [cited by applicant]
US 20190045252A1 · Lyons et al. · 2019 [cited by applicant]
US 20190051272A1 · Lewis · 2019 [cited by examiner]
US 20190163830A1 · DeLuca · 2019 [cited by applicant]
US 20190197589A1 · Singh et al. · 2019 [cited by applicant]
US 20190286720A1 · Agrawal et al. · 2019 [cited by applicant]
US 20220319478A1 · Lyske · 2022 [cited by examiner]
CN 107076631 · 2017 [cited by applicant]
WO WO2013166140 · 2013 [cited by applicant]
WO WO2014100893 · 2014 [cited by applicant]
U.S. Appl. No. 17/138,695, filed Dec. 30, 2020, Venti et al. [cited by applicant]
U.S. Appl. No. 16/699,330, filed Nov. 29, 2019, Venti et al. [cited by applicant]
Notice of Allowance issued in U.S. Appl. No. 15/490,800, dated Aug. 29, 2019 (9 pgs). [cited by applicant]
Office Action issued in U.S. Appl. No. 15/490,800, dated Jun. 27, 2019 (18 pgs). [cited by applicant]
Office Action issued in U.S. Appl. No. 15/490,800, dated Sep. 7, 2018 (18 pgs). [cited by applicant]
Wu et al., Bridging Music and Image via Cross-Modal Ranking Analysis; IEEE, Jul. 2016; 14 pgs. [cited by applicant]
Notice of Allowance issued in U.S. Appl. No. 16/592,549, dated Sep. 9, 2020 (10 pgs). [cited by applicant]
Office Action issued in U.S. Appl. No. 16/592,549, dated Aug. 21, 2020 (9 pgs). [cited by applicant]
Office Action issued in U.S. Appl. No. 16/592,549, dated May 29, 2020 (35 pgs). [cited by applicant]
Maestre et al., Enriched Multimodal Representations of Music Performances: Online Access and Visualization; IEEE; 2017; 11 pages. [cited by applicant]
Segmentino, https://code.soundsoftware.ac.uk/projects/segmenter-vamp-plugin, accessed Jul. 8, 2021, 3 pages. [cited by applicant]
Ren et al., “Automatic Music Mood Classification Based on Timbre and Modulation Features” IEEE, pp. 236-246, Apr. 29, 2015, abstract, 3 pages. [cited by applicant]
Zhao, “Explore Music World: Categorize Music by Mood”, CCTP 607 Spring 2020, Apr. 26, 2020, https://blogs.commons.georgetown.edu/cctp-607-spring2020/2020/04/26/explore-music-world-categorize-music-by-mood 10 pages. [cited by applicant]
Notice of Allowance issued in U.S. Appl. No. 17/138,695, dated Jul. 9, 2021 (18 pgs). [cited by applicant]
Oyang et al., Characterizing the Service Usage of Online Video Sharing System: Uploading vs. Playback; IEEE; 2016, 7 pages. [cited by applicant]
Xing et al., Proximiter: Enabling Mobile Proximity-Based Content Sharing on Portable Devices; 2009 IEEE, 3 pages. [cited by applicant]
Belimpasakis et al., Home Media Atomizer: Remote Sharing of Home Content—without Semi-trusted Proxies; 2008 IEEE, 9 pages. [cited by applicant]
Pering et al., Face-to-Face Media Sharing Using Wireless Mobile Device; 2005, IEEE 8 pages. [cited by applicant]
Notice of Allowance issued in U.S. Appl. No. 17/520,579, dated May 11, 2022 (19 pgs). [cited by applicant]
Lee et al., Face-to-Face Media Sharing Using Wireless Mobile Devices; 2013, IEEE, 10 pages. [cited by applicant]
Office Action issued in U.S. Appl. No. 16/699,330, dated Feb. 17, 2022 (13 pgs). [cited by applicant]
Notice of Allowance issued in U.S. Appl. No. 16/699,330, dated Jun. 17, 2022 (33 pgs). [cited by applicant]
Notice of Allowance issued in U.S. Appl. No. 17/947,799, dated May 26, 2023 (11 pgs). [cited by applicant]
Hopmann et al., Virtual Shelf: Sharing music between people and devices, 2010, IEEE, 7 pages. [cited by applicant]
Office Action issued in U.S. Appl. No. 18/049,255, dated Aug. 18, 2023 (16 pgs). [cited by applicant]
U.S. Appl. No. 15/490,800, filed Apr. 18, 2017. [cited by applicant]
U.S. Appl. No. 16/592,549, filed Oct. 3, 2019. [cited by applicant]
U.S. Appl. No. 17/138,695, filed Dec. 30, 2020. [cited by applicant]
Office Action issued in U.S. Appl. No. 17/947,799, dated Feb. 2, 2023 (12 pgs). [cited by applicant]
Cited By (1)
US 1,108,482