Sound signal refinement method, sound signal decode method, apparatus thereof program, and storage medium
There is provided a technology that improves, in a case where there is a sound signal obtained from a different code that is different from a code from which a decoded sound signal is obtained and that is derived from the same sound signal, the decoded sound signal by using the sound signal obtained from the different code. A signal (hereinafter, referred to as an upmixed common signal) obtained by upmixing a decoded sound common signal obtained by downmixing a decoded sound signal of each channel is subjected to signal purification using a signal (hereinafter, referred to as an upmixed monaural decoded sound signal) obtained by upmixing a monaural decoded sound signal to thereby generate a purified upmixed signal, and in each channel, the upmixed common signal is subtracted from the decoded sound signal and the purified upmixed signal is added thereto, to thereby generate a purified decoded sound signal.
1 . A sound signal purification method for obtaining, for each frame, an n-th channel purified decoded sound signal ˜ X n that is a sound signal of each channel of stereo by using at least an n-th channel decoded sound signal {circumflex over ( )}X n (n is each integer of 1 or more and 2 or less) that is a decoded sound signal of the each channel of the stereo obtained by decoding a stereo code CS and a monaural decoded sound signal {circumflex over ( )}X M that is a monaural decoded sound signal obtained by decoding a monaural code CM that is a code different from the stereo code CS, wherein
the n-th channel decoded sound signal {circumflex over ( )}X n is obtained by decoding the stereo code CS without using either information obtained by decoding the monaural code CM or the monaural code CM, and
the sound signal purification method comprises
a decoded sound common signal estimation step of obtaining, for the each frame, a decoded sound common signal {circumflex over ( )}Y M that is a signal common to all channels of the stereo by using at least all of one or more and two or less n-th channel decoded sound signals {circumflex over ( )}X n ,
a decoded sound common signal upmixing step of obtaining, for the each frame, an n-th channel upmixed common signal {circumflex over ( )}Y Mn that is a signal obtained by upmixing the decoded sound common signal {circumflex over ( )}Y M for the each channel by an upmixing process using the decoded sound common signal {circumflex over ( )}Y M and inter-channel relationship information that is information indicating a relationship between the channels of the stereo,
a monaural decoded sound upmixing step of obtaining, for the each frame, an n-th channel upmixed monaural decoded sound signal {circumflex over ( )}X Mn that is a signal obtained by upmixing the monaural decoded sound signal {circumflex over ( )}X M for the each channel by an upmixing process using the monaural decoded sound signal {circumflex over ( )}X M and information indicating a relationship between the channels of the stereo,
an n-th channel signal purification step of obtaining, for the each frame and for each corresponding sample t with respect to the each channel n, a sequence based on a value ˜ y Mn (t)=(1-α Mn )×{circumflex over ( )}y Mn (t)+α Mn ×{circumflex over ( )}x Mn (t) obtained by adding a value α Mn ×{circumflex over ( )}x Mn (t) obtained by multiplying an n-th channel purification weight α Mn by a sample value {circumflex over ( )}x Mn (t) of the n-th channel upmixed monaural decoded sound signal {circumflex over ( )}X Mn and a value (1−α Mn )×{circumflex over ( )}y Mn (t) obtained by multiplying a value (1−α Mn ) obtained by subtracting the n-th channel purification weight α Mn from 1 by a sample value {circumflex over ( )}y Mn (t) of the n-th channel upmixed common signal {circumflex over ( )}Y Mn , as an n-th channel purified upmixed signal ˜Y Mn ,
an n-th channel separation combination weight estimation step of obtaining, for the each frame with respect to the each channel n, a normalized inner product value for the n-th channel upmixed common signal {circumflex over ( )}Y Mn of the n-th channel decoded sound signal {circumflex over ( )}X n as an n-th channel separation combination weight β n , and
an n-th channel separation combination step of obtaining, for the each frame and for each corresponding sample t with respect to the each channel n, a sequence based on a value ˜ x n (t)={circumflex over ( )}x n (t)−β n ×{circumflex over ( )}y Mn (t)+β n × ˜ y Mn (t) obtained by subtracting a value β n ×{circumflex over ( )}y Mn (t) obtained by multiplying the n-th channel separation combination weight β n by the sample value {circumflex over ( )}y Mn (t) of the n-th channel upmixed common signal {circumflex over ( )}Y Mn from a sample value {circumflex over ( )}X n (t) of the n-th channel decoded sound signal {circumflex over ( )}X n and adding a value β n × ˜ y Mn (t) obtained by multiplying the n-th channel separation combination weight β n by a sample value ˜ y Mn (t) of the n-th channel purified upmixed signal ˜ Y Mn , as the n-th channel purified decoded sound signal ˜ X n , and
the decoded sound common signal estimation step
uses a sample value {circumflex over ( )}x 1 (t) of the first channel decoded sound signal {circumflex over ( )}X 1 , a sample value {circumflex over ( )}x 2 (t) of the second channel decoded sound signal {circumflex over ( )}X 2 , and an inter-channel correlation coefficient γ of a first channel and a second channel to obtain,
on a basis of information indicating which of the first channel and the second channel is preceding,
a sequence based on {circumflex over ( )}y M (t)=((1+γ)/2)×{circumflex over ( )}X 1 (t)+((1−γ)/2)×{circumflex over ( )}x 2 (t) as the decoded sound common signal {circumflex over ( )}Y M in a case where the first channel is preceding,
a sequence based on {circumflex over ( )}y M (t)=((1−γ)/2)×{circumflex over ( )}x 1 (t)+((1+γ)/2)×{circumflex over ( )}x 2 (t) as the decoded sound common signal {circumflex over ( )}Y M in a case where the second channel is preceding, or
a sequence based on {circumflex over ( )}y M (t)=({circumflex over ( )}x 1 (t)+{circumflex over ( )}x 2 (t))/2 as the decoded sound common signal {circumflex over ( )}Y M in a case where none of the channels is preceding.
2 . The sound signal purification method according to claim 1 , wherein
the inter-channel relationship information includes information indicating a number of samples |τ| corresponding to a time difference between channels of the first channel and the second channel and the information indicating which of the first channel and the second channel is preceding, and
the decoded sound common signal upmixing step
obtains the decoded sound common signal without change as the first channel upmixed common signal {circumflex over ( )}Y M1 and obtains a signal obtained by delaying the decoded sound common signal by |τ| samples as the second channel upmixed common signal {circumflex over ( )}Y M2 in a case where the first channel is preceding, and
obtains a signal obtained by delaying the decoded sound common signal by |τ| samples as the first channel upmixed common signal {circumflex over ( )}Y M1 and obtains the decoded sound common signal without change as the second channel upmixed common signal {circumflex over ( )}Y M2 in a case where the second channel is preceding.
3 . The sound signal purification method according to claim 1 , wherein
the inter-channel relationship information includes information indicating a number of samples |τ| corresponding to a time difference between channels of the first channel and the second channel, the information indicating which of the first channel and the second channel is preceding, and an inter-channel correlation coefficient γ that is a correlation coefficient between the first channel decoded sound signal and the second channel decoded sound signal, and
the decoded sound common signal upmixing step
uses the decoded sound common signal without change as a temporary first channel upmixed common signal Y′ M1 and uses a signal obtained by delaying the decoded sound common signal by |τ| samples as a temporary second channel upmixed common signal Y′ M2 in a case where the first channel is preceding,
uses a signal obtained by delaying the decoded sound common signal by |τ| samples as a temporary first channel upmixed common signal Y′ M1 and uses the decoded sound common signal without change as a temporary second channel upmixed common signal Y′ M2 in a case where the second channel is preceding, and
obtains, with respect to the each channel n, a sequence based on {circumflex over ( )}y MN (t)=(1−γ)×{circumflex over ( )}x n (t) +γy′ Mn (t) based on a sample value y′ Mn (t) of the temporary n-th channel upmixed common signal Y′ Mn , a sample value {circumflex over ( )}x n (t) of the n-th channel decoded sound signal {circumflex over ( )}X n , and the inter-channel correlation coefficient γ as the n-th channel upmixed common signal {circumflex over ( )}Y Mn .
4 . The sound signal purification method according to claim 1 , further comprising
an n-th channel purification weight estimation step of obtaining,
for the each frame with respect to the each channel n,
the n-th channel purification weight α Mn by
[
Math
.
50
]
α
Mn
=
2
-
2
b
m
T
2
-
2
b
m
T
+
2
-
2
b
m
T
using a number of samples T per frame, a number of bits b m corresponding to a common signal in a number of bits of the stereo code CS, and a number of bits b M of the monaural code CM.
5 . The sound signal purification method according to claim 1 , further comprising
an n-th channel purification weight estimation step of obtaining,
for the each frame with respect to the each channel n,
a value that is larger than 0 and smaller than 1, 0.5 when b m and b M are equal, closer to 0 than 0.5 as b m is larger than b M , and closer to 1 than 0.5 as b M is larger than b m by using at least a number of bits b m corresponding to a common signal in a number of bits of the stereo code CS, and a number of bits b M of the monaural code CM, as the n-th channel purification weight α Mn .
6 . The sound signal purification method according to claim 1 , further comprising
an n-th channel purification weight estimation step of obtaining,
for the each frame with respect to the each channel n,
a value c n ×r n obtained by multiplying
a normalized inner product value r n for the n-th channel upmixed monaural decoded sound signal {circumflex over ( )}X Mn of the n-th channel upmixed common signal {circumflex over ( )}Y Mn by
a correction coefficient c n obtained by
[
Math
.
51
]
c
n
=
2
-
2
b
m
T
2
-
2
b
m
T
+
2
-
2
b
m
T
using a number of samples T per frame, a number of bits b m corresponding to a common signal in a number of bits of the stereo code CS, and a number of bits b M of the monaural code CM, as the n-th channel purification weight α Mn .
7 . The sound signal purification method according to claim 1 , further comprising
an n-th channel purification weight estimation step of obtaining,
for the each frame with respect to the each channel n,
with a number of bits corresponding to a common signal in a number of bits of the stereo code CS as b m and a number of bits of the monaural code CM as b M ,
a value c n ×r n obtained by multiplying
r n that is a value closer to 1 as a correlation between the n-th channel upmixed common signal {circumflex over ( )}Y Mn and the n-th channel upmixed monaural decoded sound signal {circumflex over ( )}X Mn is higher, and closer to 0 as the correlation is lower by
a correction coefficient c n that is a value larger than 0 and smaller than 1, 0.5 when b m and b M are equal, closer to 0 than 0.5 as b m is larger than b M , and closer to 1 than 0.5 as b m is smaller than b M , as the n-th channel purification weight α Mn .
8 . The sound signal purification method according to claim 1 , wherein
T is a number of samples per frame and each of ε n and ε Mn is a value larger than 0 and smaller than 1, and
the sound signal purification method further comprises
an n-th channel purification weight estimation step of obtaining,
for the each frame with respect to the each channel n,
a value c n ×r n obtained by multiplying
a normalized inner product value r n obtained by
r n =E n (0)/ E Mn (0) [Math. 54]
using an inner product value E n (0) obtained by
[
Math
.
52
]
E
n
(
0
)
=
ϵ
n
E
n
(
-
1
)
+
(
1
-
ϵ
n
)
T
∑
t
=
1
T
y
^
Mn
(
t
)
x
^
Mn
(
t
)
using each sample value {circumflex over ( )}y Mn (t) of the n-th channel upmixed common signal {circumflex over ( )}Y Mn , each sample value {circumflex over ( )}x Mn (t) of the n-th channel upmixed monaural decoded sound signal {circumflex over ( )}X Mn , and an inner product value E n (−1) of a previous frame, and
energy E Mn (0) of the n-th channel upmixed monaural decoded sound signal obtained by
[
Math
.
53
]
E
Mn
(
0
)
=
ϵ
Mn
E
Mn
(
-
1
)
+
(
1
-
ϵ
Mn
)
T
∑
t
=
1
T
x
^
Mn
(
t
)
x
^
Mn
(
t
)
using the each sample value {circumflex over ( )}x Mn (t) of the n-th channel upmixed monaural decoded sound signal {circumflex over ( )}X Mn and energy E Mn (−1) of the n-th channel upmixed monaural decoded sound signal of the previous frame, by
a correction coefficient c n obtained by
[
Math
.
55
]
c
n
=
2
-
2
b
m
T
2
-
2
b
m
T
+
2
-
2
b
m
T
using a number of samples T per frame, a number of bits b m corresponding to a common signal in a number of bits of the stereo code CS, and a number of bits b M of the monaural code CM, as the n-th channel purification weight α Mn .
9 . The sound signal purification method according to claim 6 , wherein
the n-th channel purification weight estimation step
obtains a value λ×c n ×r n obtained by multiplying the normalized inner product value r n , the correction coefficient c n , and λ that is a predetermined value larger than 0 and smaller than 1 as the n-th channel purification weight α Mn .
10 . The sound signal purification method according to claim 6 , wherein
the n-th channel purification weight estimation step
obtains a value γ×c n ×r n obtained by multiplying the normalized inner product value r n , the correction coefficient c n , and an inter-channel correlation coefficient γ that is a correlation coefficient between the first channel decoded sound signal and the second channel decoded sound signal as the n-th channel purification weight α Mn .
11 . A sound signal decoding method comprising the sound signal purification method according to claim 1 as a sound signal purification step, the sound signal decoding method further comprising:
a stereo decoding step of decoding the stereo code CS to obtain the n-th channel decoded sound signal {circumflex over ( )}X n of the each channel n without using either information obtained by decoding the monaural code CM or the monaural code CM; and
a monaural decoding step of decoding the monaural code CM to obtain the monaural decoded sound signal {circumflex over ( )}X M .
12 . A sound signal purification device for obtaining, for each frame, an n-th channel purified decoded sound signal ˜ X n that is a sound signal of each channel of stereo by using at least an n-th channel decoded sound signal {circumflex over ( )}X n (n is each integer of 1 or more and 2 or less) that is a decoded sound signal of the each channel of the stereo obtained by decoding a stereo code CS and a monaural decoded sound signal {circumflex over ( )}X M that is a monaural decoded sound signal obtained by decoding a monaural code CM that is a code different from the stereo code CS, wherein
the n-th channel decoded sound signal {circumflex over ( )}X n is obtained by decoding the stereo code CS without using either information obtained by decoding the monaural code CM or the monaural code CM, and
the sound signal purification device comprises
a decoded sound common signal estimation circuitry configured to obtain, for the each frame, a decoded sound common signal {circumflex over ( )}Y M that is a signal common to all channels of the stereo by using at least all of one or more and two or less n-th channel decoded sound signals {circumflex over ( )}X n ,
a decoded sound common signal upmixing circuitry configured to obtain, for the each frame, an n-th channel upmixed common signal {circumflex over ( )}Y Mn that is a signal obtained by upmixing the decoded sound common signal {circumflex over ( )}Y M for the each channel by an upmixing process using the decoded sound common signal {circumflex over ( )}Y M and inter-channel relationship information that is information indicating a relationship between the channels of the stereo,
a monaural decoded sound upmixing circuitry configured to obtain, for the each frame, an n-th channel upmixed monaural decoded sound signal {circumflex over ( )}X Mn that is a signal obtained by upmixing the monaural decoded sound signal {circumflex over ( )}X M for the each channel by an upmixing process using the monaural decoded sound signal {circumflex over ( )}X M and information indicating a relationship between the channels of the stereo,
an n-th channel signal purification circuitry configured to obtain, for the each frame and for each corresponding sample t with respect to the each channel n, a sequence based on a value ˜ y Mn (t)=(1−α Mn )×{circumflex over ( )}y Mn (t)+α Mn ×{circumflex over ( )}x Mn (t) obtained by adding a value α Mn ×{circumflex over ( )}x Mn (t) obtained by multiplying an n-th channel purification weight α Mn by a sample value {circumflex over ( )}x Mn (t) of the n-th channel upmixed monaural decoded sound signal {circumflex over ( )}X Mn and a value (1−α Mn )×{circumflex over ( )}y Mn (t) obtained by multiplying a value (1−α Mn ) obtained by subtracting the n-th channel purification weight α Mn from 1 by a sample value {circumflex over ( )}y Mn (t) of the n-th channel upmixed common signal {circumflex over ( )}Y Mn , as an n-th channel purified upmixed signal ˜ Y Mn ,
an n-th channel separation combination weight estimation circuitry configured to obtain, for the each frame with respect to the each channel n, a normalized inner product value for the n-th channel upmixed common signal {circumflex over ( )}Y Mn of the n-th channel decoded sound signal {circumflex over ( )}X n as an n-th channel separation combination weight β n , and
an n-th channel separation combination circuitry configured to obtain, for the each frame and for each corresponding sample t with respect to the each channel n, a sequence based on a value ˜ x n (t)={circumflex over ( )}x n (t)−β n ×{circumflex over ( )}y Mn (t)+β n × ˜ y Mn (t) obtained by subtracting a value β n ×{circumflex over ( )}y Mn (t) obtained by multiplying the n-th channel separation combination weight β n by the sample value {circumflex over ( )}y Mn (t) of the n-th channel upmixed common signal {circumflex over ( )}Y Mn from a sample value {circumflex over ( )}x n (t) of the n-th channel decoded sound signal {circumflex over ( )}X n and adding a value β n × ˜ y Mn (t) obtained by multiplying the n-th channel separation combination weight β n by a sample value ˜ y Mn (t) of the n-th channel purified upmixed signal ˜ Y Mn , as the n-th channel purified decoded sound signal ˜ X n , and
the decoded sound common signal estimation circuitry
uses a sample value {circumflex over ( )}x 1 (t) of the first channel decoded sound signal {circumflex over ( )}X 1 , a sample value {circumflex over ( )}x 2 (t) of the second channel decoded sound signal {circumflex over ( )}X 2 , and an inter-channel correlation coefficient γ of a first channel and a second channel to obtain,
on a basis of information indicating which of the first channel and the second channel is preceding,
a sequence based on {circumflex over ( )}y M (t)=((1+γ)/2)×{circumflex over ( )}x 1 (t)+((1−γ)/2)×{circumflex over ( )}x 2 (t) as the decoded sound common signal {circumflex over ( )}Y M in a case where the first channel is preceding,
a sequence based on {circumflex over ( )}y M (t)=((1−γ)/2)×{circumflex over ( )}x 1 (t)+((1+γ)/2)×{circumflex over ( )}x 2 (t) as the decoded sound common signal {circumflex over ( )}Y M in a case where the second channel is preceding, or
a sequence based on {circumflex over ( )}y M (t)=({circumflex over ( )}x 1 (t)+{circumflex over ( )}x 2 (t))/2 as the decoded sound common signal {circumflex over ( )}Y M in a case where none of the channels is preceding.
13 . A sound signal decoding device comprising the sound signal purification device according to claim 12 as a sound signal purification circuitry, the sound signal decoding device further comprising:
a stereo decoding circuitry configured to decode the stereo code CS to obtain the n-th channel decoded sound signal {circumflex over ( )}X n of the each channel n without using either information obtained by decoding the monaural code CM or the monaural code CM; and
a monaural decoding circuitry configured to decode the monaural code CM to obtain the monaural decoded sound signal {circumflex over ( )}X M .
14 . A non-transitory recording medium recording a program for causing a computer to execute the sound signal purification method according to claim 1 .