Method, device and software for applying an audio effect
The present invention provides a method for processing music audio data, comprising the steps of providing input audio data representing a first piece of music containing a mixture of predetermined musical timbres, decomposing the input audio data to generate at least a first audio track representing a first musical timbre selected from the predetermined musical timbres, and a second audio track representing a second musical timbre selected from the predetermined musical timbres, applying a predetermined first audio effect to the first audio track, applying no audio effect or a predetermined second audio effect, which is different from the first audio effect, to the second audio track, and obtaining recombined audio data by recombining the first audio track with the second audio track.
1 . A method for processing music audio data, comprising:
providing input audio data representing a first piece of music, the input audio data comprising a mixture of predetermined musical timbres;
decomposing the input audio data to generate at least a first audio track representing a first musical timbre selected from the predetermined musical timbres for a time period represented by the input audio data, and a second audio track representing a second musical timbre selected from the predetermined musical timbres for the time period represented by the input audio data;
receiving, via user input to an effect control unit, input (A) controlling an application of a first audio effect to the first audio track or (B) actively switching on/off or changing the first audio effect;
applying, to the second audio track, (A) no audio effect or (B) a second audio effect, wherein the second audio effect is different from the first audio effect; and
obtaining recombined audio data by at least recombining the first audio track having the applied predetermined first audio effect with the second audio track having (A) no audio effect applied or (B) the applied predetermined second audio effect.
2 . The method of claim 1 , wherein the predetermined first audio effect is a pitch scaling effect that changes a pitch of audio data of the first audio track while maintaining a playback duration of the audio data of the first audio track.
3 . The method of claim 1 , wherein the first audio track and the second audio track generated from the decomposed input audio data are complements, such that a sum of the first audio track and the second audio track generated from the decomposed input audio data is substantially equal to the input audio data.
4 . The method of claim 1 , wherein one or more of:
the first musical timbre is a harmonic vocal timbre or a harmonic instrumental timbre; or the second musical timbre is a non-harmonic vocal timbre or a non-harmonic instrumental timbre.
5 . The method of claim 1 ,
wherein decomposing the input audio data further comprises generating a third audio track representing a third musical timbre, wherein the first audio track, the second audio track and the third audio track generated from the input audio data are complements, such that a sum of the first audio track, the second audio track and the third audio track generated from the input audio data substantially equals to the input audio data,
wherein the predetermined first audio effect is applied to the first audio track, but not to the second audio track and not to the third audio track, and
wherein obtaining the recombined audio data comprises at least recombining:
(1) the first audio track having the applied predetermined first audio effect,
(2) the second audio track having (A) no audio effect applied or (B) the applied predetermined second audio effect, and
(3) the third audio track.
6 . The method of claim 1 , wherein decomposing the input audio data further comprises processing the input audio data by an artificial intelligence (“AI”) system comprising a trained neural network, wherein the neural network is trained to decompose the input audio data to generate at least the first audio track and the second audio track.
7 . The method of claim 1 , further comprising:
determining output data from the recombined audio data; and further processing the output data.
8 . The method of claim 7 , wherein further processing the output data comprises one or more of (a) storing the output data in a storage unit, (b) playing back the output data by a playback unit, or (c) mixing the output data with second-song output data.
9 . The method of claim 1 , wherein the user input further controls a selection of at least one audio effect from a plurality of audio effects as the first audio effect to be applied to the first audio track.
10 . A device for processing music audio data, comprising:
an input unit for receiving input audio data representing a first piece of music comprising a mixture of predetermined musical timbres;
a decomposition unit for decomposing the input audio data received from the input unit to generate at least a first audio track representing a first musical timbre selected from the predetermined musical timbres and a second audio track representing a second musical timbre selected from the predetermined musical timbres;
an effect unit for applying a first audio effect to the first audio track, but not to the second audio track;
a recombination unit for obtaining recombined audio data by recombining the first audio track with the second audio track; and
an effect control unit for allowing a user to (A) control operation of the effect unit, including to apply the first audio effect to the first audio track, or (B) actively switch on/off or change the first audio effect.
11 . The device of claim 10 , wherein the effect unit comprises a pitch scaling unit for changing a pitch of audio data of the first audio track while maintaining a playback duration of the audio data of the first audio track.
12 . The device of claim 10 , wherein the decomposition unit includes an artificial intelligence (AI) system comprising a trained neural network, wherein the neural network is trained to decompose the input audio data to generate at least the first audio track and the second audio track.
13 . The device of claim 10 , further comprising one or more of:
a storage unit for storing output data determined from the recombined audio data;
a playback unit for playing back the output data; or
a mixing unit for mixing the output data with second-song output data.
14 . The device of claim 10 , wherein the effect unit controls a plurality of audio effects, and wherein the effect control unit comprises an effect control element, wherein the effect control element allows the user to select at least one audio effect from the plurality of audio effects as the first audio effect to be applied to the first audio track.
15 . The device of claim 10 , wherein the effect control unit comprises a parameter control element, wherein the parameter control element allows control of at least one effect parameter of the first audio effect.
16 . The device of claim 10 ,
wherein the decomposition unit is adapted to decompose the input audio data to generate a plurality of decomposed audio tracks, the plurality of decomposed audio tracks comprising at least a first decomposed audio track and a second decomposed audio track, wherein each of the plurality of decomposed audio tracks each represents a respective different timbre selected from the predetermined musical timbres of the same piece of music, and
wherein the effect control unit comprises a combo effect control element, wherein the combo effect control element is adapted to control an application of at least a first audio effect to the first decomposed audio track and a second audio effect to the second decomposed audio track, wherein the second audio effect is different from the first audio effect.
17 . The device of claim 16 , wherein the combo effect control element is adapted to control the application of at least the first audio effect to the first decomposed audio track and the second audio effect to the second decomposed audio track by a single control operation of a user.
18 . The device of claim 10 , further comprising:
a computer comprising a microprocessor, a storage unit, an input interface, and an output interface, wherein at least the input unit, the decomposition unit, the effect unit and the recombination unit are formed by a software executed by the microprocessor, wherein the software is configured to control the computer to perform operations of the input unit, the decomposition unit, the effect unit, and the recombination unit.
19 . A non-transitory computer-readable storage medium comprising computer readable program instructions stored therein that when executed by a computer cause the computer to perform operations comprising:
providing input audio data representing a first piece of music, the input audio data comprising a mixture of predetermined musical timbres;
decomposing the input audio data to generate at least a first audio track representing a first musical timbre selected from the predetermined musical timbres for an entire time of the input audio data, and a second audio track representing a second musical timbre selected from the predetermined musical timbres for the entire time of the input audio data;
receiving, via user input to an effect control unit, input (A) controlling an application of a first audio effect to the first audio track or (B) actively switching on/off or changing the first audio effect;
applying, to the second audio track, (A) no audio effect or (B) a predetermined second audio effect, wherein the predetermined second audio effect is different from the predetermined first audio effect; and
obtaining recombined audio data by at least recombining the first audio track having the applied predetermined first audio effect with the second audio track having (A) no audio effect applied or (B) the applied predetermined second audio effect.
20 . The non-transitory computer-readable storage medium of claim 19 , wherein the user input further controls a selection of at least one audio effect from a plurality of audio effects as the first audio effect to be applied to the first audio track.