Method and apparatus for generating haptic signal with auditory saliency estimation
A method and apparatus for generating a haptic signal with auditory saliency estimation from an audio signal is disclosed. The method includes detecting audio characteristic information of a bass component and audio characteristic information of a treble component from a received audio signal, estimating auditory saliency with respect to the audio signal based on the two types of audio characteristic information, and calculating a tactile signal based on the auditory saliency.
1. A method of generating a haptic signal with auditory saliency estimation from an audio signal, comprising:
detecting audio characteristic information of a bass component and audio characteristic information of a treble component from a received audio signal;
estimating auditory saliency with respect to the audio signal based on the two types of audio characteristic information; and
calculating a tactile signal based on the auditory saliency,
wherein the calculating the tactile signal includes calculating a peak pitch in the auditory saliency for each subband of the two types of audio characteristic information, calculating a perceptual magnitude of vibration of each subband in the peak pitch, and converting the perceptual magnitude into a physical vibration amplitude, and
wherein the perceptual magnitude of vibration includes spectral components of each of a subband of a treble component and a subband of a bass component which are played in a vibration actuator at the same time.
2. The method of claim 1 , wherein each of the two types of audio characteristic information includes a total amplitude obtained by summing every component of each subband corresponding to the two types of audio characteristic information, a peak loudness obtained by estimating the audio signal of each subband as a predetermined scale, and a peak energy of the audio signal of each subband.
3. The method of claim 2 , further comprising, before the estimating of the auditory saliency, performing normalization by applying a predetermined weight value to at least one among the total amplitude, the peak loudness, and the peak energy.
4. The method of claim 1 , wherein the estimating of the auditory saliency includes estimating the auditory saliency Si(n) based on the following Equation 3,
S i ( n )= A i ( n )· L i ( n )·log E i ( n ) [Equation 3]
where n represents an arbitrary natural number, i represents an i-th subband among subbands of an n-th loop in which i=1 represents the lowest frequency band and i=2 or 6 represents the highest frequency band, Ai(n) represents the total amplitude of the n-th loop, Li(n) represents a peak loudness in the n-th loop, and Ei(n) represents a peak energy in the n-th loop.
5. The method of claim 4 , wherein each of the amplitudes of all components of each subband corresponding to the two types of audio characteristic information has a value which is linearly multiplied or divided, and the peak energy is estimated based on Teager-Kaiser energy according to the following Equation 2,
E=a 2 sin 2 ω [Equation 2]
where “a” represents a spectral magnitude, and w represents a frequency of each component.
6. The method of claim 1 , further comprising, before the detecting of the two types of audio characteristic information, dividing a frequency band of the received audio signal into two sub frequency bands, and generating the audio signal for each sub frequency band.
7. The method of claim 1 , wherein the vibration amplitude includes perceptual characteristics of superimposed dual frequency vibration emphasizing a bass component and an attenuating treble component of the two types of audio characteristic information, and is converted into voltage command amplitudes through I/O mapping of a vibration actuator.
8. The method of claim 1 , further comprising, after the generating of the tactile signal, selecting a designed intensity or mode of a haptic stimulus by the tactile signal or the perceptual magnitude of the vibration.
9. An apparatus for generating a haptic signal with auditory saliency estimation from an audio signal, comprising:
an audio signal receiver configured to receive an audio signal;
an auditory characteristics information calculator configured to calculate audio characteristics information from the received audio signal;
an auditory saliency calculator configured to calculate auditory saliency according to the audio characteristics information based on an auditory saliency weight; and
a tactile signal calculator configured to calculate a tactile signal based on the auditory saliency,
wherein the tactile signal calculator calculates a peak pitch In the auditory saliency for each subband of the two types of audio characteristic Information, calculates a perceptual magnitude of vibration of each subband in the peak pitch, and converts the perceptual magnitude into a physical vibration amplitude, and
wherein the perceptual magnitude of vibration includes spectral components of each of a subband of a treble component and a subband of a bass component which are played in a vibration actuator at the same time.
10. The apparatus of claim 9 , wherein the audio characteristics information includes at least one among an intensity, a pitch, a rhythm, an energy, a temporal contrast, a melody, a harmony, and a timbre, of the received audio signal.
11. The apparatus of claim 9 , wherein the audio characteristics information calculator divides a frequency band of the received audio signal into a predetermined unit, and calculates the audio characteristics for each sub frequency band.
12. The apparatus of claim 11 , wherein the auditory saliency calculator calculates the auditory saliency based on the audio characteristics for each sub frequency band.
13. The apparatus of claim 9 , wherein the auditory saliency weight is predetermined or is calculated from the audio characteristics information.
14. The apparatus of claim 9 , wherein the tactile signal is generated by converting a frequency of the received audio signal, or is generated as a reference vibration frequency corresponding to each sub frequency band in which a frequency band of the received audio signal is divided into the predetermine unit.
15. The apparatus of claim 14 , wherein the tactile signal is generated as the reference vibration frequency by determining an amplitude according to the auditory saliency.
16. The apparatus of claim 9 , wherein the tactile signal generator determines a pattern of a haptic stimulus based on the auditory saliency and a sound source configuring the received audio signal.
17. The apparatus of claim 9 , wherein the audio signal is generated from an element of an operating system controlling an audio driver.
18. The apparatus of claim 9 , further comprising:
a haptic stimulus generator configured to receive the calculated tactile signal and generate a haptic stimulus.