Sound signal refinement method, sound signal decode method, apparatus thereof, program, and storage medium
View Patent ↗There is provided a technology that improves, in a case where there is a sound signal obtained from a different code that is different from a code from which a decoded sound signal is obtained and that is derived from the same sound signal, the decoded sound signal by using the sound signal obtained from the different code. A signal (hereinafter, referred to as an upmixed common signal) obtained by upmixing a decoded sound common signal obtained by downmixing a decoded sound signal of each channel is subjected to signal purification using a signal (hereinafter, referred to as an upmixed monaural decoded sound signal) obtained by upmixing a monaural decoded sound signal to thereby generate a purified upmixed signal, and in each channel, the upmixed common signal is subtracted from the decoded sound signal and the purified upmixed signal is added thereto, to thereby generate a purified decoded sound signal.
1. A sound signal purification method for obtaining, for each frame, an n-th channel purified decoded sound signal ˜ X n that is a sound signal of each channel of stereo by using at least an n-th channel decoded sound signal {circumflex over ( )}X n (n is each integer of 1 or more and 2 or less) that is a decoded sound signal of the each channel of the stereo obtained by decoding a stereo code CS and a monaural decoded sound signal {circumflex over ( )}X M that is a monaural decoded sound signal obtained by decoding a monaural code CM that is a code different from the stereo code CS, wherein
the n-th channel decoded sound signal {circumflex over ( )}X n is obtained by decoding the stereo code CS without using either information obtained by decoding the monaural code CM or the monaural code CM, and
the sound signal purification method comprises
a decoded sound common signal estimation step of obtaining, for the each frame, a decoded sound common signal {circumflex over ( )}Y M that is a signal common to all channels of the stereo by using at least all of one or more and two or less n-th channel decoded sound signals {circumflex over ( )}X n ,
a decoded sound common signal upmixing step of obtaining, for the each frame, an n-th channel upmixed common signal {circumflex over ( )}Y Mn that is a signal obtained by upmixing the decoded sound common signal {circumflex over ( )}Y M for the each channel by an upmixing process using the decoded sound common signal {circumflex over ( )}Y M and inter-channel relationship information that is information indicating a relationship between the channels of the stereo,
a monaural decoded sound upmixing step of obtaining, for the each frame, an n-th channel upmixed monaural decoded sound signal {circumflex over ( )}X Mn that is a signal obtained by upmixing the monaural decoded sound signal {circumflex over ( )}X M for the each channel by an upmixing process using the monaural decoded sound signal {circumflex over ( )}X M and information indicating a relationship between the channels of the stereo,
an n-th channel signal purification step of obtaining, for the each frame and for each corresponding sample t with respect to the each channel n, a sequence based on a value ˜ y Mn (t)=(1−α Mn )×{circumflex over ( )}y Mn (t)+α Mn ×{circumflex over ( )}x Mn (t) obtained by adding a value α Mn ×{circumflex over ( )}x Mn (t) obtained by multiplying an n-th channel purification weight α Mn by a sample value {circumflex over ( )}x Mn (t) of the n-th channel upmixed monaural decoded sound signal {circumflex over ( )}X Mn and a value (1−α Mn )×{circumflex over ( )}y Mn (t) obtained by multiplying a value (1−α Mn ) obtained by subtracting the n-th channel purification weight α Mn from 1 by a sample value {circumflex over ( )}y Mn (t) of the n-th channel upmixed common signal {circumflex over ( )}Y Mn , as an n-th channel purified upmixed signal ˜ Y Mn ,
an n-th channel separation combination weight estimation step of obtaining, for the each frame with respect to the each channel n, a normalized inner product value for the n-th channel upmixed common signal {circumflex over ( )}Y Mn of the n-th channel decoded sound signal {circumflex over ( )}X n as an n-th channel separation combination weight β n , and
an n-th channel separation combination step of obtaining, for the each frame and for each corresponding sample t with respect to the each channel n, a sequence based on a value ˜ x n (t)={circumflex over ( )}x n (t)−β n ×{circumflex over ( )}y Mn (t)+β n × ˜ y Mn (t) obtained by subtracting a value β n ×{circumflex over ( )}y Mn (t) obtained by multiplying the n-th channel separation combination weight β n by the sample value {circumflex over ( )}y Mn (t) of the n-th channel upmixed common signal {circumflex over ( )}Y Mn from a sample value {circumflex over ( )}x n (t) of the n-th channel decoded sound signal {circumflex over ( )}X n and adding a value β n ×{circumflex over ( )}y Mn (t) obtained by multiplying the n-th channel separation combination weight β n by a sample value {circumflex over ( )}y Mn (t) of the n-th channel purified upmixed signal {circumflex over ( )}Y Mn , as the n-th channel purified decoded sound signal ˜ X n ,
the inter-channel relationship information includes information indicating a number of samples |τ| corresponding to a time difference between channels of a first channel and a second channel, information indicating which of the first channel and the second channel is preceding, and an inter-channel correlation coefficient γ that is a correlation coefficient between a first channel decoded sound signal and a second channel decoded sound signal, and
the decoded sound common signal upmixing step
uses the decoded sound common signal without change as a temporary first channel upmixed common signal Y′ M1 and uses a signal obtained by delaying the decoded sound common signal by |τ| samples as a temporary second channel upmixed common signal Y′ M2 in a case where the first channel is preceding,
uses a signal obtained by delaying the decoded sound common signal by |τ| samples as a temporary first channel upmixed common signal Y′ M1 and uses the decoded sound common signal without change as a temporary second channel upmixed common signal Y′ M2 in a case where the second channel is preceding, and
obtains, with respect to the each channel n, a sequence based on {circumflex over ( )}y MN (t)=(1−γ)×{circumflex over ( )}x n (t)+γ×y′ Mn (t) based on a sample value y′ Mn (t) of the temporary n-th channel upmixed common signal Y′ Mn , a sample value {circumflex over ( )}x n (t) of the n-th channel decoded sound signal {circumflex over ( )}X n , and the inter-channel correlation coefficient γ as the n-th channel upmixed common signal {circumflex over ( )}Y Mn .
2. The sound signal purification method according to claim 1 , wherein
the decoded sound common signal estimation step
uses a number of samples per frame as T,
obtains w cand having a minimum value obtained by
[
Math
.
50
]
∑
t
=
1
T
❘
"\[LeftBracketingBar]"
(
1
+
w
cand
2
x
^
1
(
t
)
+
1
-
w
cand
2
x
^
2
(
t
)
)
-
x
^
M
(
t
)
❘
"\[RightBracketingBar]"
2
among w cand of −1 or more and 1 or less as a weighting coefficient w, and
obtains a sequence based on {circumflex over ( )}y M (t) obtained by
[
Math
.
51
]
y
^
M
(
t
)
=
1
+
w
2
x
^
1
(
t
)
+
1
-
w
2
x
^
2
(
t
)
for each sample number t as the decoded sound common signal {circumflex over ( )}Y M .
3. The sound signal purification method according to claim 1 , further comprising
an n-th channel purification weight estimation step of obtaining,
for the each frame with respect to the each channel n,
the n-th channel purification weight α Mn by
[
Math
.
52
]
α
Mn
=
2
-
2
b
m
T
2
-
2
b
m
T
+
2
-
2
b
M
T
using a number of samples T per frame, a number of bits b m corresponding to a common signal in a number of bits of the stereo code CS, and a number of bits b M of the monaural code CM.
4. The sound signal purification method according to claim 1 , further comprising
an n-th channel purification weight estimation step of obtaining,
for the each frame with respect to the each channel n,
a value that is larger than 0 and smaller than 1, 0.5 when b m and bare equal, closer to 0 than 0.5 as b m is larger than b M , and closer to 1 than 0.5 as b M is larger than b m by using at least a number of bits b m corresponding to a common signal in a number of bits of the stereo code CS, and a number of bits b M of the monaural code CM, as the n-th channel purification weight α Mn .
5. The sound signal purification method according to claim 1 , further comprising
an n-th channel purification weight estimation step of obtaining,
for the each frame with respect to the each channel n,
a value c n ×r n obtained by multiplying
a normalized inner product value r n for the n-th channel upmixed monaural decoded sound signal {circumflex over ( )}X Mn of the n-th channel upmixed common signal {circumflex over ( )}Y Mn by
a correction coefficient c n obtained by
[
Math
.
53
]
c
n
=
2
-
2
b
m
T
2
-
2
b
m
T
+
2
-
2
b
M
T
using a number of samples T per frame, a number of bits b m corresponding to a common signal in a number of bits of the stereo code CS, and a number of bits b M of the monaural code CM, as the n-th channel purification weight α Mn .
6. The sound signal purification method according to claim 1 , further comprising
an n-th channel purification weight estimation step of obtaining,
for the each frame with respect to the each channel n,
with a number of bits corresponding to a common signal in a number of bits of the stereo code CS as b m and a number of bits of the monaural code CM as b M ,
a value c n ×r n obtained by multiplying
r n that is a value closer to 1 as a correlation between the n-th channel upmixed common signal {circumflex over ( )}Y Mn and the n-th channel upmixed monaural decoded sound signal {circumflex over ( )}X Mn is higher, and closer to 0 as the correlation is lower by
a correction coefficient c n that is a value larger than 0 and smaller than 1, 0.5 when b m and b M are equal, closer to 0 than 0.5 as b m is larger than b M , and closer to 1 than 0.5 as b m is smaller than b M , as the n-th channel purification weight α Mn .
7. The sound signal purification method according to claim 1 , wherein
T is a number of samples per frame and each of εn and εMn is a value larger than 0 and smaller than 1, and
the sound signal purification method further comprises
an n-th channel purification weight estimation step of obtaining,
for the each frame with respect to the each channel n,
a value cn×rn obtained by multiplying
a normalized inner product value rn obtained by
r n =E n (0)/ E Mn (0) [Math. 56]
using an inner product value En(0) obtained by
[
Math
.
54
]
E
n
(
0
)
=
ϵ
n
E
n
(
-
1
)
+
(
1
-
ϵ
n
)
T
∑
t
=
1
T
y
^
Mn
(
t
)
x
^
Mn
(
t
)
using each sample value {circumflex over ( )}yMn(t) of the n-th channel upmixed common signal {circumflex over ( )}YMn, each sample value {circumflex over ( )}xMn(t) of the n-th channel upmixed monaural decoded sound signal {circumflex over ( )}XMn, and an inner product value En(−1) of a previous frame, and
energy EMn(0) of the n-th channel upmixed monaural decoded sound signal obtained by
[
Math
.
55
]
E
Mn
(
0
)
=
ϵ
Mn
E
Mn
(
-
1
)
+
(
1
-
ϵ
Mn
)
T
∑
t
=
1
T
x
^
Mn
(
t
)
x
^
Mn
(
t
)
using the each sample value {circumflex over ( )}xMn(t) of the n-th channel upmixed monaural decoded sound signal {circumflex over ( )}XMn and energy EMn(−1) of the n-th channel upmixed monaural decoded sound signal of the previous frame, by
a correction coefficient cn obtained by
[
Math
.
57
]
c
n
=
2
-
2
b
m
T
2
-
2
b
m
T
+
2
-
2
b
M
T
using a number of samples T per frame, a number of bits bm corresponding to a common signal in a number of bits of the stereo code CS, and a number of bits b M of the monaural code CM, as the n-th channel purification weight αMn.
8. The sound signal purification method according to claim 5 , wherein
the n-th channel purification weight estimation step
obtains a value λ×cn×rn obtained by multiplying the normalized inner product value rn, the correction coefficient cn, and λ that is a predetermined value larger than 0 and smaller than 1 as the n-th channel purification weight αMn.
9. The sound signal purification method according to claim 5 , wherein
the n-th channel purification weight estimation step
obtains a value γ×cn×rn obtained by multiplying the normalized inner product value rn, the correction coefficient cn, and an inter-channel correlation coefficient γ that is a correlation coefficient between the first channel decoded sound signal and the second channel decoded sound signal as the n-th channel purification weight αMn.
10. A sound signal decoding method comprising the sound signal purification method according to claim 1 as a sound signal purification step, the sound signal decoding method further comprising:
a stereo decoding step of decoding the stereo code CS to obtain the n-th channel decoded sound signal {circumflex over ( )}Xn of the each channel n without using either information obtained by decoding the monaural code CM or the monaural code CM; and
a monaural decoding step of decoding the monaural code CM to obtain the monaural decoded sound signal {circumflex over ( )}XM.
11. A non-transitory recording medium recording a program for causing a computer to execute the sound signal purification method according to claim 1 .
12. A sound signal purification device for obtaining, for each frame, an n-th channel purified decoded sound signal ˜Xn that is a sound signal of each channel of stereo by using at least an n-th channel decoded sound signal {circumflex over ( )}Xn (n is each integer of 1 or more and 2 or less) that is a decoded sound signal of the each channel of the stereo obtained by decoding a stereo code CS and a monaural decoded sound signal {circumflex over ( )}XM that is a monaural decoded sound signal obtained by decoding a monaural code CM that is a code different from the stereo code CS, wherein
the n-th channel decoded sound signal {circumflex over ( )}Xn is obtained by decoding the stereo code CS without using either information obtained by decoding the monaural code CM or the monaural code CM, and
the sound signal purification device comprises
a decoded sound common signal estimation circuitry configured to obtain, for the each frame, a decoded sound common signal {circumflex over ( )}YM that is a signal common to all channels of the stereo by using at least all of one or more and two or less n-th channel decoded sound signals {circumflex over ( )}Xn,
a decoded sound common signal upmixing circuitry configured to obtain, for the each frame, an n-th channel upmixed common signal {circumflex over ( )}YMn that is a signal obtained by upmixing the decoded sound common signal {circumflex over ( )}YM for the each channel by an upmixing process using the decoded sound common signal {circumflex over ( )}YM and inter-channel relationship information that is information indicating a relationship between the channels of the stereo,
a monaural decoded sound upmixing circuitry configured to obtain, for the each frame, an n-th channel upmixed monaural decoded sound signal {circumflex over ( )}XMn that is a signal obtained by upmixing the monaural decoded sound signal {circumflex over ( )}XM for the each channel by an upmixing process using the monaural decoded sound signal {circumflex over ( )}XM and information indicating a relationship between the channels of the stereo,
an n-th channel signal purification circuitry configured to obtain, for the each frame and for each corresponding sample t with respect to the each channel n, a sequence based on a value ˜yMn(t)=(1−αMn)×{circumflex over ( )}yMn(t)+αMn×{circumflex over ( )}xMn(t) obtained by adding a value αMn×{circumflex over ( )}xMn(t) obtained by multiplying an n-th channel purification weight αMn by a sample value {circumflex over ( )}xMn(t) of the n-th channel upmixed monaural decoded sound signal {circumflex over ( )}XMn and a value (1−αMn)×{circumflex over ( )}yMn(t) obtained by multiplying a value (1−αMn) obtained by subtracting the n-th channel purification weight αMn from 1 by a sample value {circumflex over ( )}yMn(t) of the n-th channel upmixed common signal {circumflex over ( )}YMn, as an n-th channel purified upmixed signal ˜YMn,
an n-th channel separation combination weight estimation circuitry configured to obtain, for the each frame with respect to the each channel n, a normalized inner product value for the n-th channel upmixed common signal {circumflex over ( )}YMn of the n-th channel decoded sound signal {circumflex over ( )}Xn as an n-th channel separation combination weight βn, and
an n-th channel separation combination circuitry configured to obtain, for the each frame and for each corresponding sample t with respect to the each channel n, a sequence based on a value ˜xn(t)={circumflex over ( )}xn(t)−βn×{circumflex over ( )}yMn(t)+βnטyMn(t) obtained by subtracting a value βn×{circumflex over ( )}yMn(t) obtained by multiplying the n-th channel separation combination weight βn by the sample value {circumflex over ( )}yMn(t) of the n-th channel upmixed common signal {circumflex over ( )}YMn from a sample value {circumflex over ( )}xn(t) of the n-th channel decoded sound signal {circumflex over ( )}Xn and adding a value βnטyMn(t) obtained by multiplying the n-th channel separation combination weight βn by a sample value ˜yMn(t) of the n-th channel purified upmixed signal ˜YMn, as the n-th channel purified decoded sound signal ˜Xn,
the inter-channel relationship information includes information indicating a number of samples |τ| corresponding to a time difference between channels of a first channel and a second channel, information indicating which of the first channel and the second channel is preceding, and an inter-channel correlation coefficient γ that is a correlation coefficient between a first channel decoded sound signal and a second channel decoded sound signal, and
the decoded sound common signal upmixing circuitry
uses the decoded sound common signal without change as a temporary first channel upmixed common signal Y′M1 and uses a signal obtained by delaying the decoded sound common signal by |τ| samples as a temporary second channel upmixed common signal Y′M2 in a case where the first channel is preceding,
uses a signal obtained by delaying the decoded sound common signal by |τ| samples as a temporary first channel upmixed common signal Y′M1 and uses the decoded sound common signal without change as a temporary second channel upmixed common signal Y′M2 in a case where the second channel is preceding, and
obtains, with respect to the each channel n, a sequence based on {circumflex over ( )}yMN(t)=(1−γ)×{circumflex over ( )}xn(t)+γ×y′Mn(t) based on a sample value y′Mn(t) of the temporary n-th channel upmixed common signal Y′Mn, a sample value {circumflex over ( )}xn(t) of the n-th channel decoded sound signal {circumflex over ( )}Xn, and the inter-channel correlation coefficient γ as the n-th channel upmixed common signal {circumflex over ( )}YMn.
13. A sound signal decoding device comprising the sound signal purification device according to claim 12 as a sound signal purification circuitry, the sound signal decoding device further comprising:
a stereo decoding circuitry configured to decode the stereo code CS to obtain the n-th channel decoded sound signal {circumflex over ( )}Xn of the each channel n without using either information obtained by decoding the monaural code CM or the monaural code CM; and
a monaural decoding circuitry configured to decode the monaural code CM to obtain the monaural decoded sound signal {circumflex over ( )}XM.