Method and system for multi-sensor fusion in the presence of missing and noisy labels
This disclosure relates to a method and system for multi-sensor fusion in the presence of missing and noisy labels. Prior methods for multi-sensor fusion do not estimate and correct labels for learning effective models in semi-supervised learning methods. Embodiments of the present disclosure provides a method for learning robust sensor-specific autoencoder based fusion model by utilizing a graph structure to perform label propagation and correction. In the disclosed Graph regularized AutoFuse (GAF) method latent representation for each sensor is learnt using the sensor-specific autoencoders. Further these latent representations are combined and fed to a classifier for multi-class classification. The disclosure presents a joint optimization formulation for multi-sensor fusion where label propagation and correction, sensor-specific learning and classification are executed together.
1 . A processor implemented method comprising the steps of:
receiving, via one or more hardware processors, (i) a set of sensor data (X 1 , X 2 , . . . , X n ) from a set of sensors and (ii) class labels (Y) corresponding to the set of sensor data for learning a set of sensor-specific autoencoders for multi-sensor fusion wherein learning the set of sensor-specific autoencoders involves a joint optimization of a set of parameters including (i) a set of sensor-specific encoder weights (W e1 , . . . , W en ) (ii) a set of sensor-specific decoder weights (W d1 , . . . , W dn ) (iii) a set of sensor-specific latent representations (H 1 , . . . , H n ) corresponding to the set of sensor data (iv) a set of new class labels (F) and (v) a set of classifier weights (M), wherein the class labels are one or more of partially labelled and noisy labelled;
generating, via the one or more hardware processors, a set of graphs corresponding to the set of sensor data wherein a graph amongst the set of graphs corresponds to at most one sensor data from the set of sensors;
generating, via the one or more hardware processors, a fused graph (L) from the set of graphs;
iteratively learning, via the one or more hardware processors, the set of sensor-specific autoencoders by jointly learning the set of parameters based on the joint optimization until a termination criteria is met, the learning comprises,
updating, via the one or more hardware processors, the set of sensor-specific encoder weights;
updating, via the one or more hardware processors, the set of sensor-specific decoder weights;
updating, via the one or more hardware processors, the set of sensor-specific latent representations;
combining, via the one or more hardware processors, the set of sensor-specific latent representations to obtain the fused latent representation;
estimating, via the one or more hardware processors, the set of new class labels (F) using the fused graph, the set of new class labels from a previous iteration, the class labels, the fused latent representation and the classifier weights; and
updating, via the one or more hardware processors, the set of classifier weights using the set of new class labels and the fused latent representation;
providing, via the one or more hardware processors, a set of test sensor data
(
X
1
test
,
X
2
test
,
…
,
X
n
test
)
for predicting a set of test labels (Ŷ test );
computing, via the one or more hardware processors, a set of sensor-specific test latent representations
(
H
1
test
,
…
,
H
n
test
)
corresponding to the set of test sensor data using the updated set of sensor-specific encoder weights;
combining, via the one or more hardware processors, the set of sensor-specific test latent representations
(
H
1
test
,
…
,
H
n
test
)
to obtain a fused test latent representation (H test ); and
predicting, via the one or more hardware processors, the set of test labels (Ŷ test ) using the fused test latent representations and the learnt set of classifier weights (M);
wherein the joint optimization is represented as,
min
W
d
1
,
…
,
W
dn
,
W
e
1
,
…
,
W
en
,
H
1
,
…
,
H
n
,
M
,
F
X
1
-
W
d
1
H
1
F
2
+
W
e
1
X
1
-
H
1
F
2
+
…
+
X
n
-
W
dn
H
n
F
2
+
W
e
n
X
n
-
H
n
F
2
+
α
Tr
(
FL
F
T
)
+
β
Tr
(
(
F
-
Y
)
U
(
F
-
Y
)
T
)
+
λ
F
-
MH
F
2
such that H 1 ≥0, . . . , H n ≥0, where α, β and λ are real positive number parameters, U is a diagonal matrix with U jj =1 when the j th measurement of the set of sensor data is labelled else U jj =0.
2 . The method as claimed in claim 1 , wherein estimating the set of new class labels in a first iteration utilizes the fused graph and the class labels.
3 . The method as claimed in claim 1 , wherein updating the set of sensor-specific latent representations is represented as,
H
i
←
min
H
i
X
i
-
W
di
H
i
F
2
+
W
e
i
X
i
-
H
i
F
2
+
λ
F
-
(
M
i
H
i
+
∑
z
=
1
,
z
≠
i
n
M
z
H
z
)
F
2
such that H i ≥0.
4 . The method as claimed in claim 1 , wherein obtaining the fused latent representation (H) is based on simple concatenation of the set of sensor-specific latent representations.
5 . The method as claimed in claim 1 , wherein estimating the new class labels (F) is represented as,
F
←
min
F
α
Tr
(
FLF
T
)
+
β
Tr
(
(
F
-
Y
)
U
(
F
-
Y
)
T
+
λ
F
-
MH
F
2
.
6 . The method as claimed in claim 1 , wherein updating the set of classifier weights (M) is represented as,
M
←
min
M
λ
F
-
MH
F
2
.
7 . The method as claimed in claim 1 , wherein the termination criteria is convergence of a loss function associated with the joint optimization within an empirically determined threshold value.
8 . A system comprising:
a memory storing instructions;
one or more communication interfaces; and
one or more hardware processors coupled to the memory via the one or more communication interfaces, wherein the one or more hardware processors are configured by the instructions to:
receive (i) a set of sensor data (X 1 , X 2 , . . . , X n ) from a set of sensors and (ii) class labels (Y) corresponding to the set of sensor data for learning a set of sensor-specific autoencoders for multi-sensor fusion wherein learning the set of sensor-specific autoencoders involves a joint optimization of a set of parameters including (i) a set of sensor-specific encoder weights (W e1 , . . . , W en ) (ii) a set of sensor-specific decoder weights (W d1 , . . . , W dn ) (iii) a set of sensor-specific latent representations (H 1 , . . . , H n ) corresponding to the set of sensor data (iv) a set of new class labels (F) and (v) a set of classifier weights (M), wherein the class labels are one or more of partially labelled and noisy labelled;
generate a set of graphs corresponding to the set of sensor data wherein a graph amongst the set of graphs corresponds to at most one sensor data from the set of sensors;
generate a fused graph (L) from the set of graphs;
iteratively learn the set of sensor-specific autoencoders by jointly learning the set of parameters based on the joint optimization until a termination criteria is met, the learning comprises,
updating the set of sensor-specific encoder weights;
updating the set of sensor-specific decoder weights;
updating the set of sensor-specific latent representations;
combining the set of sensor-specific latent representations to obtain the fused latent representation;
estimating the set of new class labels (F) using the fused graph, the set of new class labels from a previous iteration, the class labels, the fused latent representation and the classifier weights; and
updating the set of classifier weights using the set of new class labels and the fused latent representation;
provide a set of test sensor data
(
X
1
test
,
X
2
test
,
…
,
X
n
test
)
for predicting a set of test provide a set of test sensor data labels (Ŷ test );
compute a set of sensor-specific test latent representations
(
H
1
test
,
…
,
H
n
test
)
corresponding to the set of test sensor data using the updated set of sensor-specific encoder weights;
combine the set of sensor-specific test latent representations
(
H
1
test
,
…
,
H
n
test
)
to obtain a fused test latent representation (H test ); and
predict the set of test labels (Ŷ test ) using the fused test latent representations and the learnt set of classifier weights (M);
wherein the joint optimization is represented as,
min
W
d
1
,
…
,
W
dn
,
W
e
1
,
…
,
W
en
,
H
1
,
…
,
H
n
,
M
,
F
X
1
-
W
d
1
H
1
F
2
+
W
e
1
X
1
-
H
1
F
2
+
…
+
X
n
-
W
dn
H
n
F
2
+
W
e
n
X
n
-
H
n
F
2
+
α
Tr
(
FL
F
T
)
+
β
Tr
(
(
F
-
Y
)
U
(
F
-
Y
)
T
)
+
λ
F
-
MH
F
2
such that H 1 ≥0, . . . , H n ≥0, where α, β and λ are real positive number parameters, U is a diagonal matrix with U jj =0 when the i th measurement of the set of sensor data is labelled else U jj =0.
9 . The system as claimed in claim 8 , wherein the one or more hardware processors are configured to estimate the set of new class labels in the first iteration utilizes the fused graph and the class labels.
10 . The system as claimed in claim 8 , wherein the one or more hardware processors are configured to update the set of sensor-specific latent representations is represented as,
H
i
←
min
H
i
X
i
-
W
di
H
i
F
2
+
W
e
i
X
i
-
H
i
F
2
+
λ
F
-
(
M
i
H
i
+
∑
z
=
1
,
z
≠
i
n
M
z
H
z
)
F
2
such that H i ≥0.
11 . The system as claimed in claim 8 , wherein the one or more hardware processors are configured to obtain the fused latent representation (H) is based on simple concatenation of the set of sensor-specific latent representations.
12 . The system as claimed in claim 8 , wherein the one or more hardware processors are configured to estimate the new class labels (F) is represented as,
F
←
min
F
α
Tr
(
FLF
T
)
+
β
Tr
(
(
F
-
Y
)
U
(
F
-
Y
)
T
+
λ
F
-
MH
F
2
.
13 . The system as claimed in claim 8 , wherein the one or more hardware processors are configured to update the set of classifier weights (M) is represented as,
M
←
min
M
λ
F
-
MH
F
2
.
14 . The system as claimed in claim 8 , wherein the termination criteria is convergence of a loss function associated with the joint optimization within an empirically determined threshold value.
15 . One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause:
receiving (i) a set of sensor data (X 1 ,X 2 , . . . , X n ) from a set of sensors and (ii) class labels (Y) corresponding to the set of sensor data for learning a set of sensor-specific autoencoders for multi-sensor fusion wherein learning the set of sensor-specific autoencoders involves a joint optimization of a set of parameters including (i) a set of sensor-specific encoder weights (W e1 , . . . , W en ) (ii) a set of sensor-specific decoder weights (W d1 , . . . , W dn ) (iii) a set of sensor-specific latent representations (H 1 , . . . , H n ) corresponding to the set of sensor data (iv) a set of new class labels (F) and (v) a set of classifier weights (M), wherein the class labels are one or more of partially labelled and noisy labelled;
generating a set of graphs corresponding to the set of sensor data wherein a graph amongst the set of graphs corresponds to at most one sensor data from the set of sensors;
generating a fused graph (L) from the set of graphs;
iteratively learning the set of sensor-specific autoencoders by jointly learning the set of parameters based on the joint optimization until a termination criteria is met, the learning comprises,
updating the set of sensor-specific encoder weights;
updating the set of sensor-specific decoder weights;
updating the set of sensor-specific latent representations;
combining the set of sensor-specific latent representations to obtain the fused latent representation;
estimating the set of new class labels (F) using the fused graph, the set of new class labels from a previous iteration, the class labels, the fused latent representation and the classifier weights; and
updating the set of classifier weights using the set of new class labels and the fused latent representation;
providing a set of test sensor data
(
X
1
test
,
X
2
test
,
…
,
X
n
test
)
for predicting a set of test labels (Ŷ test );
computing, via the one or more hardware processors, a set of sensor-specific test latent representations
(
H
1
test
,
…
,
H
n
test
)
corresponding to the set of test sensor data using latent representations the updated set of sensor-specific encoder weights;
combining the set of sensor-specific test latent representations
(
H
1
test
,
…
,
H
n
test
)
to obtain a fused test latent representation (H test ); and
predicting the set of test labels (Ŷ test ) using the fused test latent representations and the learnt set of classifier weights (M);
wherein the joint optimization is represented as,
min
W
d
1
,
…
,
W
dn
,
W
e
1
,
…
,
W
en
,
H
1
,
…
,
H
n
,
M
,
F
X
1
-
W
d
1
H
1
F
2
+
W
e
1
X
1
-
H
1
F
2
+
…
+
X
n
-
W
dn
H
n
F
2
+
W
e
n
X
n
-
H
n
F
2
+
α
Tr
(
FL
F
T
)
+
β
Tr
(
(
F
-
Y
)
U
(
F
-
Y
)
T
)
+
λ
F
-
MH
F
2
such that H 1 ≥0, . . . , H n ≥0, where α, β and λ are real positive number parameters, U is a diagonal matrix with U jj =1 when the i th measurement of the set of sensor data is labelled else U jj =0.