Sound event early detection
Systems and methods for Evidence-based Sound Event Early Detection is provided. The system/method includes parsing collected labeled audio corpus data and real time audio streaming data utilizing mel-spectrogram, encoding features of the parsed mel-spectrograms using a trained neural network, and generating a final predicted result for a sound event based on the belief, disbelief and uncertainty outputs from the encoded mel-spectrograms.
1 . A computer implemented method for Evidence-based Sound Event Early Detection, comprising:
parsing overlapping polyphony sound from collected labeled audio corpus data and real-time audio streaming data into spectrogram features utilizing a mel-spectrogram;
extracting parsed mel-spectrograms with zero-padded frames from the collected labeled audio corpus data and real-time audio streaming data and the spectrogram features;
encoding features of the parsed mel-spectrograms using a trained neural network to obtain encoded mel-spectrograms; and
detecting sound events from the overlapping polyphony sound by generating a final predicted result for the sound events with a trained neural network that reduces sound event detection delay by utilizing subjective logic with a vacuity threshold and an uncertainty calibration loss based on sums of evidential uncertainty determined based on prediction samples with corresponding strengths of binomial opinion based on belief, disbelief and uncertainty outputs from the encoded mel-spectrograms.
2 . The method as recited in claim 1 , further comprising applying ground truth labels to the real-time audio streaming data being parsed.
3 . The method as recited in claim 2 , wherein the neural network is a convolutional neural network with a gated recurrent unit (GRU) recurrent neural network.
4 . The method as recited in claim 3 , wherein the real-time audio data is recorded as audio frames padded with additional zeros to a predefined hop size.
5 . The method as recited in claim 4 , wherein the audio frames have a frame size of 64 milliseconds.
6 . The method as recited in claim 5 , wherein the belief and disbelief are utilized to calculate an expected probability, {circumflex over (p)} ik t as
p
ˆ
i
k
t
=
b
i
k
t
b
ik
t
+
d
i
k
t
+
2
,
where
b
ik
t
and
d
ik
t
represents the belief and the disbelief for a k th class at a time, t.
7 . The method as recited in claim 6 , wherein a cross entropy is based on the expected probability,
p
^
ik
t
,
as follows:
l
B
e
t
a
(
α
i
t
,
y
i
t
)
=
∑
k
=
1
K
y
ik
t
log
p
ˆ
i
k
t
+
(
1
-
y
i
k
t
)
log
(
1
-
p
ˆ
i
k
t
)
.
8 . The method as recited in claim 7 , wherein an uncertainty estimation is calculated by the uncertainty calibration loss as:
ℒ
u
n
=
∑
i
=
1
N
∑
m
=
0
M
∑
t
=
m
+
1
T
∑
k
=
1
K
𝕀
(
x
ik
[
t
,
t
-
m
]
)
·
U
(
α
ik
[
t
,
t
-
m
]
)
,
where
𝕀
(
x
ik
[
t
,
t
-
m
]
)
=
{
1
,
correct
prediction
-
1
,
incorrect
prediction
.
9 . The method as recited in claim 8 , wherein the real-time audio streaming data is collected as audio segments with time stamps.
10 . The method as recited in claim 1 , further comprising notifying entities of an undesirable sound event related to a security issue for a building based on the final predicted result.
11 . The method as recited in claim 1 , further comprising training a neural network with the uncertainty calibration loss to obtain the trained neural network.
12 . A computer system for Evidence-based Sound Event Early Detection, comprising:
one or more processors;
a display screen coupled to the one or more processors through a bus; and
memory coupled to the one or more processors through the bus, wherein the memory includes:
an audio processing tool that parses overlapping polyphony sound from collected labeled audio corpus data and real-time audio streaming data into spectrogram features by utilizing a mel-spectrogram and extracts parsed mel-spectrograms with zero-padded frames from the collected labeled audio corpus data and real time audio streaming data and the spectrogram features,
an audio encoding tool that encodes features of the parsed mel-spectrograms using a trained neural network to obtain encoded mel-spectrograms, and
a sound detection tool that detects sound events from the overlapping polyphony sound and generates a final predicted result for the sound events with the trained neural network that reduces sound event detection delay by utilizing subjective logic with a vacuity threshold and an uncertainty calibration loss based on sums of evidential uncertainty determined based on prediction samples with corresponding strengths of binomial opinion based on the belief, disbelief and uncertainty outputs from the encoded mel-spectrograms.
13 . The computer system as recited in claim 12 , wherein the audio processing tool is further configured to apply ground truth labels to the real-time audio streaming data being parsed.
14 . The computer system as recited in claim 13 , wherein the neural network is a convolutional neural network with a gated recurrent unit (GRU) recurrent neural network.
15 . The computer system as recited in claim 14 , wherein the real-time audio data is recorded as audio frames padded with additional zeros to a predefined hop size.
16 . The computer system as recited in claim 15 , wherein the audio frames have a frame size of 64 milliseconds.
17 . The computer system as recited in claim 16 , wherein the belief and disbelief are utilized to calculate an expected probability, {circumflex over (p)} ik t as
p
ˆ
i
k
t
=
b
i
k
t
b
ik
t
+
d
i
k
t
+
2
,
where
b
ik
t
and
d
ik
t
represents the belief and the disbelief for a k th class at a time, t.
18 . The computer system as recited in claim 17 , wherein the cross entropy is based on a expected probability,
p
^
ik
t
,
as follows:
l
B
e
t
a
(
α
i
t
,
y
i
t
)
=
∑
k
=
1
K
y
ik
t
log
p
ˆ
i
k
t
+
(
1
-
y
i
k
t
)
log
(
1
-
p
ˆ
i
k
t
)
.
19 . The computer system as recited in claim 18 , wherein an uncertainty estimation is calculated by the uncertainty calibration loss as:
ℒ
u
n
=
∑
i
=
1
N
∑
m
=
0
M
∑
t
=
m
+
1
T
∑
k
=
1
K
𝕀
(
x
ik
[
t
,
t
-
m
]
)
·
U
(
α
ik
[
t
,
t
-
m
]
)
,
where
𝕀
(
x
ik
[
t
,
t
-
m
]
)
=
{
1
,
correct
prediction
-
1
,
incorrect
prediction
.
20 . The computer system as recited in claim 19 , wherein the real-time audio streaming data is collected as audio segments with time stamps.