Virtual music experiences
Techniques for generating a virtual music experience. The techniques include source separating an arbitrary digital audio input into a plurality of source-separated tracks. Sets of music features are determined from the plurality of source-separated tracks and provided to a video presentation system at a video frame rate of the video presentation system. The providing the sets of music features to the video presentation system causes the video presentation system to animate one or more graphical assets based on the provided sets of music features.
1. A method for generating a virtual music experience, the method performed by a computing system comprising one or more processors and memory storing one or more programs executed by the one or more processors to perform the method, the method comprising:
source separating a digital music input into a plurality of source-separated tracks;
determining sets of one or more music features from each source-separated track of the plurality of source-separated tracks including determining sets of one or more labeled onset music features from a particular source-separated track of the plurality of source-separated tracks; and
on each event tick of a plurality of event ticks that occur at or approximately at a video frame rate of a video presentation system, providing a set of one or more music features, of the sets of one or more music features, in a data frame for the event tick including providing a set of one or more labeled onset music features, of the sets of one or more labeled onset music features, in the data frame for the event tick to the video presentation system thereby causing the video presentation system to change one or more visual properties of a graphical asset computer animated by the video presentation system at or approximately at the video frame rate.
2. The method of claim 1 , further comprising:
source separating the particular source-separated track of the plurality of source-separated tracks into a plurality of track-specific source separated tracks; and
determining the set of one or more labeled onset music features from the plurality of track-specific source separated tracks.
3. The method of claim 1 , wherein the digital music input comprises Pulse Code Modulation (PCM) samples.
4. The method of claim 1 , wherein the set of one or more music features determined from the particular source-separated track of the plurality of source-separated tracks comprises one or more of spectrum, cestrum, amplitude, envelope, panorama, or formants.
5. The method of claim 1 , wherein the set of one or more music features determined from the particular source-separated track of the plurality of source-separated tracks corresponds to a window of time of the digital music input, the window of time having a length, the length of the window of time being or approximately being eleven and one-half milliseconds.
6. The method of claim 1 , wherein each music feature in the set of one or more music features determined from the particular source-separated track of the plurality of source-separated tracks is determined based, at least in part, on a window of pulse code modulation (PCM) samples from the particular source-separated track.
7. The method of claim 6 , wherein the window of pulse code modulation (PCM) samples consists of 512 pulse code modulation (PCM) samples.
8. The method of claim 1 , wherein each music feature in the set of one or more music features determined from the particular source-separated track of the plurality of source-separated tracks comprises a tag or label specifying the particular source-separated track that the music feature is determined from.
9. The method of claim 1 , wherein the video frame rate of the video presentation system is or is approximately ninety frames per second.
10. A system comprising:
one or more processors;
storage media;
one or more programs stored in the storage media and configured for execution by the one or more processors, the one or more programs comprising instructions configured for:
source separating a digital music input into a plurality of source-separated tracks;
determining sets of one or more music features from each source-separated track of the plurality of source-separated tracks including determining sets of one or more labeled onset music features from a particular source-separated track of the plurality of source-separated tracks; and
on each event tick of a plurality of event ticks that occur at or approximately at a video frame rate of a video presentation system, providing a set of one or more music features, of the sets of one or more music features, in a data frame for the event tick including providing a set of one or more labeled onset music features, of the sets of one or more labeled onset music features, in the data frame for the event tick to the video presentation system thereby causing the video presentation system to change one or more visual properties of a graphical asset computer animated by the video presentation system at or approximately at the video frame rate.
11. The system of claim 10 , the instructions further configured for:
source separating the particular source-separated track of the plurality of source-separated tracks into a plurality of track-specific source separated tracks; and
determining the set of one or more labeled onset music features from the plurality of track-specific source separated tracks.
12. The system of claim 10 , wherein the graphical asset is a particle system.
13. The system of claim 10 , wherein the plurality of source-separated tracks comprises a rhythm track, a bass track, a melody track, and a lead track.
14. The system of claim 10 , the instructions further configured for:
receiving the plurality of event ticks from the video presentation system at or approximately at the video frame rate of the video presentation system.
15. The system of claim 10 , wherein each source-separated track of the plurality of source-separated tracks comprises a plurality of pulse code modulation (PCM) samples.