IP Library › Granted Patent US 12,236,160
Granted Patent B2
US 12,236,160 · App. 18/306,169 · Granted Feb 25, 2025

Audio techniques for music content generation

Inventors: Edward Balassanian (Austin, TX); Patrick E. Hutchings (Melbourne, AU); Toby Gifford (Melbourne, AU)
Assignee: AiMi Inc.
G06F3/165G05B13/027G05B15/02G06F21/105G06F21/16G10H1/0025G10H1/0066G10H1/06G10L21/12G10L25/06H04L9/0637H04L9/0643G10H2210/076G10H2220/126H04L9/50
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,236,160
App. No.
18/306,169
Granted
Feb 25, 2025
Kind
B2
Abstract

Techniques are disclosed relating to implementing audio techniques for real-time audio generation. For example, a music generator system may generate new music content from playback music content based on different parameter representations of an audio signal. In some cases, an audio signal can be represented by both a graph of the signal (e.g., an audio signal graph) relative to time and a graph of the signal relative to beats (e.g., a signal graph). The signal graph is invariant to tempo, which allows for tempo invariant modification of audio parameters of the music content in addition to tempo variant modifications based on the audio signal graph.

Claims (46)

1. A method, comprising:

accessing, by a computer system, a first graph of an audio signal, wherein the first graph is a graph of audio parameters relative to time;

accessing, by the computer system, a second graph of the audio signal, wherein the second graph is a signal graph of the audio parameters relative to beat;

accessing, by the computer system, playback music content; and

modifying, by the computer system, the audio parameters in the playback music content to generate new music content, wherein the audio parameters are modified based on a combination of the first graph and the second graph.

2. The method of claim 1 , wherein the first graph and the second graph are accessed from a cloud-based server.

3. The method of claim 1 , wherein the audio parameters in the first graph and the second graph are defined by nodes in the graphs that determine changes in properties of the audio signal.

4. The method of claim 3 , wherein modifying the audio parameters in the playback music content to generate the new music content includes:

determining a first node in the first graph corresponding to an audio signal in the playback music content;

determining a second node in the second graph that corresponds to the first node;

determining one or more specified audio parameters based on the second node; and

modifying one or more properties of an audio signal in the playback music content by modifying the specified audio parameters.

5. The method of claim 4 , further comprising:

determining one or more additional specified audio parameters based on the first node; and

modifying one or more properties of an additional audio signal in the playback music content by modifying the additional specified audio parameters.

6. The method of claim 4 , wherein determining the one or more specified audio parameters includes:

determining a portion of the second graph to implement for the audio parameters based on a position of the second node in the second graph; and

selecting the audio parameters from the determined portion of the second graph as the one or more specified audio parameters.

7. The method of claim 6 , wherein modifying the one or more specified audio parameters modifies a portion of the playback music content that corresponds to the determined portion of the second graph.

8. The method of claim 4 , wherein the modified properties of the audio signal in the playback music content include signal amplitude, signal frequency, or a combination thereof.

9. The method of claim 1 , further comprising applying one or more automations to the audio parameters, wherein at least one of the automations is a pre-programmed temporal manipulation of at least one of the audio parameters.

10. The method of claim 9 , further comprising applying one or more modulations to the audio parameters, wherein at least one of the modulations modifies at least one of the audio parameters multiplicatively on top of at least one automation.

11. The method of claim 1 , wherein the first graph is a tempo variant graph of the audio signal.

12. The method of claim 1 , wherein the second graph is a tempo invariant graph of the audio signal.

13. A non-transitory computer-readable medium having instructions stored thereon that are executable by a computing device to perform operations comprising:

accessing a first graph of an audio signal, wherein the first graph is a graph of audio parameters relative to time;

accessing a second graph of the audio signal, wherein the second graph is a signal graph of the audio parameters relative to beat;

accessing playback music content; and

modifying the audio parameters in the playback music content to generate new music content, wherein the audio parameters are modified based on a combination of the first graph and the second graph.

14. The non-transitory computer-readable medium of claim 13 , wherein the audio parameters in the first graph and the second graph are defined by nodes in the graphs that determine changes in properties of the audio signal, and wherein modifying the audio parameters in the playback music content to generate the new music content includes:

determining a first node in the first graph corresponding to an audio signal in the playback music content;

determining a second node in the second graph that corresponds to the first node;

determining one or more specified audio parameters based on the second node; and

modifying one or more properties of an audio signal in the playback music content by modifying the specified audio parameters.

15. The non-transitory computer-readable medium of claim 14 , wherein the modified properties of the audio signal in the playback music content include signal amplitude, signal frequency, or a combination thereof.

16. The non-transitory computer-readable medium of claim 13 , wherein at least one of: the audio signal, the first graph, and the second graph are access as one or more objects associated with a set of music content stored in a heap allocated memory.

17. The non-transitory computer-readable medium of claim 13 , further comprising adding an identifier to the new music content, and storing the new music content in at least one circular buffer in a static array of circular buffers.

18. The non-transitory computer-readable medium of claim 17 , wherein the static array of circular buffers is accessible by a single user.

19. An apparatus, comprising:

one or more processors; and

one or more memories having program instructions stored thereon that are executable by the one or more processors to:

access a first graph of an audio signal, wherein the first graph is a graph of audio parameters relative to time;

access a second graph of the audio signal, wherein the second graph is a signal graph of the audio parameters relative to beat;

access playback music content; and

modify the audio parameters in the playback music content to generate new music content, wherein the audio parameters are modified based on a combination of the first graph and the second graph.

20. The apparatus of claim 19 , further comprising a static array of circular buffers, wherein the program instructions stored on the one or more memories are executable by the one or more processors to store the new music content in at least one circular buffer in the static array of circular buffers.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 24, 2023
From: BALASSANIAN, EDWARD; HUTCHINGS, PATRICK E.; GIFFORD, TOBY
To: AIMI INC.
Reel/Frame 063422/0570 →
Continuity (6)
Continuation 17174066 · Feb 11, 2021
Provisional Application 63068433 · Aug 21, 2020
Provisional Application 63068431 · Aug 21, 2020
Provisional Application 63028233 · May 21, 2020
Provisional Application 62972711 · Feb 11, 2020
Related Publication 20230259327A1 · Aug 17, 2023
References Cited (106)
US 5410100A · Kim · 1995 [cited by applicant]
US 5589947A · Sato et al. · 1996 [cited by applicant]
US 6297439B1 · Browne · 2001 [cited by applicant]
US 6307139B1 · Iwamura · 2001 [cited by applicant]
US 6506969B1 · Baron · 2003 [cited by applicant]
US 6542869B1 · Foote · 2003 [cited by examiner]
US 7491878B2 · Orr · 2009 [cited by applicant]
US 8115090B2 · Miyajima et al. · 2012 [cited by applicant]
US 8751030B2 · You et al. · 2014 [cited by applicant]
US 8812144B2 · Balassanian · 2014 [cited by applicant]
US 9367613B1 · Tome · 2016 [cited by examiner]
US 9471271B2 · Georges et al. · 2016 [cited by applicant]
US 9799312B1 · Cabral et al. · 2017 [cited by applicant]
US 9977645B2 · Herlöfsson · 2018 [cited by applicant]
US 10446126B1 · Kaye · 2019 [cited by examiner]
US 20040199395A1 · Schulz · 2004 [cited by examiner]
US 20050267819A1 · Kaplan · 2005 [cited by applicant]
US 20070022867A1 · Yamashita · 2007 [cited by examiner]
US 20080092721A1 · Schnepel et al. · 2008 [cited by applicant]
US 20080249644A1 · Jehan · 2008 [cited by applicant]
US 20080276793A1 · Yamashita · 2008 [cited by examiner]
US 20080314228A1 · Dreyfuss et al. · 2008 [cited by applicant]
US 20090069917A1 · Yamashita · 2009 [cited by examiner]
US 20090107320A1 · Willacy · 2009 [cited by applicant]
US 20090157203A1 · Bregar et al. · 2009 [cited by applicant]
US 20090164034A1 · Cohen et al. · 2009 [cited by applicant]
US 20090231968A1 · Ochi et al. · 2009 [cited by applicant]
US 20100071535A1 · McKinney et al. · 2010 [cited by applicant]
US 20100146583A1 · Prehofer et al. · 2010 [cited by applicant]
US 20110054646A1 · Hernandez et al. · 2011 [cited by applicant]
US 20120192701A1 · Watanabe et al. · 2012 [cited by applicant]
US 20140052282A1 · Balassanian · 2014 [cited by applicant]
US 20140260909A1 · Matusiak · 2014 [cited by applicant]
US 20140314391A1 · Kim et al. · 2014 [cited by applicant]
US 20150179156A1 · Uemura et al. · 2015 [cited by applicant]
US 20150179157A1 · Chon et al. · 2015 [cited by applicant]
US 20150234564A1 · Snibbe et al. · 2015 [cited by applicant]
US 20150347971A1 · D'Amore et al. · 2015 [cited by applicant]
US 20160379611A1 · Georges et al. · 2016 [cited by applicant]
US 20170062006A1 · Plom · 2017 [cited by examiner]
US 20170092247A1 · Silverstein · 2017 [cited by applicant]
US 20170287135A1 · Pozos et al. · 2017 [cited by applicant]
US 20180226063A1 · Wood et al. · 2018 [cited by applicant]
US 20180247624A1 · Elkins · 2018 [cited by applicant]
US 20190362696A1 · Balassanian et al. · 2019 [cited by applicant]
US 20200021966A1 · Wylie et al. · 2020 [cited by applicant]
US 20200090632A1 · Rein et al. · 2020 [cited by applicant]
US 20200265817A1 · Elkins · 2020 [cited by applicant]
US 20200402488A1 · Williams · 2020 [cited by applicant]
US 20210247954A1 · Balassanian et al. · 2021 [cited by applicant]
US 20210247955A1 · Balassanian et al. · 2021 [cited by applicant]
US 20210248213A1 · Balassanian et al. · 2021 [cited by applicant]
US 20210248983A1 · Balassanian et al. · 2021 [cited by applicant]
US 20210319774A1 · Elkins · 2021 [cited by applicant]
US 20220059062A1 · Balassanian et al. · 2022 [cited by applicant]
US 20220156863A1 · Shimizu et al. · 2022 [cited by applicant]
CN 110089137A · 2019 [cited by applicant]
CN 110444185A · 2019 [cited by applicant]
CN 115331648A · 2022 [cited by applicant]
JP 63184875A · 1988 [cited by applicant]
JP H08293039A · 1996 [cited by applicant]
JP 2559390B2 · 1996 [cited by applicant]
JP 2006084749A · 2006 [cited by applicant]
JP 2009020387A · 2009 [cited by applicant]
JP 2015011146A · 2015 [cited by applicant]
JP 2015099358A · 2015 [cited by applicant]
WO 2017087333 · 2017 [cited by applicant]
Office Action in Japanese Appl. No. 2022-548683 mailed Aug. 15, 2023, 6 pages. [cited by applicant]
Office Action in European Appl. No. 21710719.2 mailed Oct. 2, 2023, 7 pages. [cited by applicant]
Office Action in U.S. Appl. No. 17/174,084 mailed Apr. 24, 2024, 24 pages. [cited by applicant]
Jehan, “Creating Music by Listening,” (Year: 2005), 137 pages. [cited by applicant]
Kudumakis, et al., “MPEG IPR Ontologies for Media Trading and Personalization,” (Year: 2019), 11 pages. [cited by applicant]
Notice of Allowance in U.S. Appl. No. 17/174,052 mailed Nov. 29, 2023, 8 pages. [cited by applicant]
Office Action in U.S. Appl. No. 17/174,084 mailed Nov. 30, 2023, 23 pages. [cited by applicant]
Kudumakis et al., “The Challenge: From MPEG Intellectual Property Rights Ontologies to Smart Contracts and Blockchains,” IEEE Signal Processing Magazine, Mar. 2020, pp. 89-95. [cited by applicant]
Yakov Vorobyev and Eric Coomes, “Beyond Beatmatching—Take your DJ Career to the Next Level,” Mixed in Key LLC, 2012, 129 pages. [cited by applicant]
International Search Report and Written Opinion in Application No. PCT/US13/55445 dated Jan. 17, 2014, 9 pages. [cited by applicant]
International Search Report and Written Opinion in Application No. PCT/US2019/033686 dated Aug. 16, 2019, 26 pages. [cited by applicant]
Christopher Anderson et al., “The Generative Electronic Dance Music Algorithmic System (GEDMAS),” Musical Metacreation: Papers from the 2013 AIIDE Workshop (WS-13-22), pp. 5-8. [cited by applicant]
Andrew R. Brown et al., “Adaptive Music Techniques,” Improvise: The Australasian Computer Music Conference, pp. 26-31. [cited by applicant]
Carlos Castellanos, et al. “Biometric Tendency Recognition and Classification System: An Artistic Approach,” Conference Paper, Sep. 2008, 8 pages. [cited by applicant]
Nick Collins, “INFNO: Generating Synth Pop and Electronic Dance Music on Demand,” International Computer Music Conference, ICMC Aug. 2008, 4 pages. [cited by applicant]
Arne Eigenfeldt et al., “Evolving Structures for Electronic Dance Music,” Genetic and Evolutionary Computation Conference (GECCO), Jul. 2013, 8 pages. [cited by applicant]
Toby Gifford, “Tuning Into the Task: Sonic Environmental Cues and Mental Task Switching,” The 22nd International Conference on Auditory Display (ICAD-2016), Jul. 2-8, 2016, 6 pages. [cited by applicant]
Jon McCormack et al., “In a Silent Way: Communication Between AI and Improvising Musicians Beyond Sound,” Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems; Paper No. 38, May 4-9, 2019, 11 pag… [cited by applicant]
Steven R. Livingstone et al., “Dynamic Response: Real-Time Adaptation for Music Emotion,” Australasian conference on Interactive Entertainment, Nov. 2005, 7 pages. [cited by applicant]
Steven R. Livingstone et al., “Controlling musical emotionality: an affective computational architecture for influencing musical emotions,” Digital Creativity, 2007, vol. 18, No. 1, pp. 43-53. [cited by applicant]
Aengus Martin et al., “Implementation of a Real-Time Musical Decision-Maker,” ACMC 2012 Conference Proceedings, pp. 44-50. [cited by applicant]
Jack Stockholm et al., “Reinforcement Learning of Listener Response for Mood Classification of Audio,” 12th IEEE International Conference on Computational Science and Engineering, Aug. 29-31, 2009, 5 pages. [cited by applicant]
International Preliminary Report in PCT Appl. No. PCT/US2019/033686 mailed Aug. 18, 2020, 14 pages. [cited by applicant]
International Search Report and Written Opinion in PCT Appl. No. PCT/US2021/017722 mailed May 26, 2021, 12 pages. [cited by applicant]
Modrzejewski et al., “Application of Deep Neural Networks to Music Composition Based on MIDI Datasets and Graphical Representation,” May 24, 2019, Advances Indatabases and Information Systems; pp. 143-152, XP047508521. [cited by applicant]
Guan et al., “A GAN Model With Self-attention Mechanism to Generate Multi-instruments Symbolic Music,” 2019 International Joint Conference on Neural Networks (IJCNN), IEEE, Jul. 14, 2019, pp. 1-6, XP033622129. [cited by applicant]
Casini et al., “Some Reflections on the Potential and Limitations of Deep Learning for Automated Music Generation,” 2018 IEEE 29th Annual Internationalsymposium on Personal, Indoor and Mobileradio Communications (PIMRC)… [cited by applicant]
ProTools Reference Guide (Year: 2019). [cited by applicant]
Open Music Initiative—Minimum Viable Interoperability Meeting Notes (Year: 2017). [cited by applicant]
Open Music Initiative—Minimum Viable Interoperability requirements (Year: 2017). [cited by applicant]
ID3 draft specification (Year: 2000). [cited by applicant]
MusicXML manual (Year: 2011). [cited by applicant]
Office Action in U.S. Appl. No. 17/174,052 mailed May 24, 2023, 10 pages. [cited by applicant]
Office Action in U.S. Appl. No. 17/174,075 mailed May 25, 2023, 20 pages. [cited by applicant]
Notice of Allowance in Japanese Appl. No. 2022-548683 mailed Jan. 16, 2024, 3 pages. [cited by applicant]
Office Action in Chinese Appl. No. 202180013939.X mailed Jun. 8, 2024, 16 pages. [cited by applicant]
Office Action in U.S. Appl. No. 17/174,084 mailed Aug. 30, 2024, 23 pages. [cited by applicant]
Notice of Allowance in European Appl. No. 21710719.2 mailed Aug. 30, 2024, 8 pages. [cited by applicant]
Matthew J. Dovery, “A technique for ”regular expression“ style searching in polyphonic music,” http://www.lib.ox.ac.uk/jafer and http://www.jafer.org; Dec. 2001, 7 pages. [cited by applicant]