System and method for training a model to perform semantic segmentation on low visibility images using high visibility images having a close camera view
A system and a method for training a semantic segmentation model includes obtaining a plurality of sets of images each having an index z for visibility, iteratively training the model. Iteratively training the model includes (a) for each z above 1, obtaining preliminary semantic segmentation labels for each image of the set of images of index z−1 by applying the model to each image of the set of images of index z−1, (b) processing each preliminary semantic segmentation labels using semantic segmentation labels obtained using the model on a selected image of index 1, and obtaining processed semantic segmentation labels, (c) training the model using the set of images of index z−1 and the associated processed semantic segmentation labels, and (d) performing steps (a) to (c) for z+1.
1. A method for training a model to be used for semantic segmentation of images taken under low visibility conditions, comprising:
obtaining a plurality of sets of images, each set of images being associated with an index z comprised between 1 and Z, the index z indicating a level of visibility of the images of the set of images, 1 corresponding to the highest level of visibility and Z corresponding to the lowest level of visibility,
wherein the model is initially trained to perform semantic segmentation using an annotated set of images having a level of visibility of 1 and the associated semantic segmentation labels, and
for z being greater than or equal to 2, iteratively training the model comprising:
a—obtaining, for at least a first image of the set of images of index z, a preliminary semantic segmentation label by applying the model trained on the set of images of index z−1 to the first image,
b—obtaining, for the at least the first image, a processed preliminary semantic segmentation label using the preliminary semantic segmentation label and a semantic segmentation label obtained by applying the initially trained model on a selected image of the set of images of index 1, the selected image being selected as the image having a camera view which is closest to a camera view of the first image by:
using a cross bilateral filter between the semantic segmentation label associated with the selected image of the set of images of index 1 and the semantic segmentation label associated with the first image, and
performing a fusion of an output of the cross bilateral filter with the semantic segmentation label associated with the first image,
wherein the cross bilateral filter performs:
S
~
1
(
p
)
=
∑
q
∈
𝒩
(
p
)
G
σ
s
(
q
-
p
)
G
σ
r
(
I
z
(
q
)
-
I
z
(
p
)
)
S
1
(
q
)
∑
q
∈
𝒩
(
p
)
G
σ
s
(
q
-
p
)
G
σ
r
(
I
z
(
q
)
-
I
z
(
p
)
)
in which:
p and q are pixel positions,
Ŝ 1 (p) is the output of the cross bilateral filter for a pixel p,
(p) is a neighborhood of p,
G σ s is a spatial-domain Gaussian kernel,
G σ r is a color-domain kernel,
I z (q) and I z (p) respectively designate a color value at pixel q and pixel p in the first image from the set of index z, and
S 1 (q) is a semantic segmentation label at pixel q for the selected image of the set of images of index 1,
c—training the model using:
the set of images of index z and the associated processed semantic segmentation labels, and
a synthetic set of images of index z and the associated semantic segmentation labels both generated from the annotated set of images having a level of visibility of 1 and the associated semantic segmentation labels; and
d—performing steps a to c for z+1.
2. The method of claim 1 , comprising a preliminary step of obtaining the semantic segmentation labels for all the images of the set of images of index 1 using the model.
3. The method of claim 1 , wherein performing the fusion of an output of the cross bilateral filter with the semantic segmentation label obtained for the first image comprises using:
S
^
z
=
F
z
F
z
+
α
F
1
S
z
+
α
F
1
F
z
+
α
F
1
S
~
1
in which:
Ŝ z is an output of a fusion function,
F 1 is equal to
max
c
∈
S
~
c
1
(
p
)
and is a maximum output value of a semantic segmentation label at pixel p for the selected image of the set of images of index 1 for all possible classes , c designating a class having a highest output,
F z is equal to
max
c
∈
S
~
c
z
(
p
)
and is a maximum output value of a semantic segmentation label at pixel p for the first image, and
α a coefficient chosen between 0 and 1.
4. The method of claim 3 , wherein a is a function of a pixel p:
α
(
p
)
=
{
α
l
,
if
c
1
=
arg
max
c
∈
𝒞
S
~
c
1
(
p
)
∈
d
and
S
c
1
z
(
p
)
≤
η
or
c
2
=
arg
max
c
∈
𝒞
S
c
z
(
p
)
∈
d
and
S
~
c
2
1
(
p
)
≤
η
,
α
h
otherwise
.
5. A system for training a model to be used for semantic segmentation of images taken under low visibility conditions, comprising at least one processor configured for:
obtaining a plurality of sets of images, each set of images being associated with an index comprised between 1 and Z, the index z indicating a level of visibility of the images of the set of images, 1 corresponding to the highest level of visibility and Z corresponding to the lowest level of visibility,
wherein the model is initially trained to perform semantic segmentation using an annotated set of images having a level of visibility of 1 and the associated semantic segmentation labels, the at least one processor being further configured for:
for z being greater than or equal to 2, iteratively training the model by:
a—obtaining, for at least a first image of the set of images of index z, a preliminary semantic segmentation label by applying the model trained on the set of images of index z−1 to the first image,
b—obtaining, for the at least the first image, a processed preliminary semantic segmentation label using the preliminary semantic segmentation label and a semantic segmentation label obtained by applying the initially trained model on a selected image of the set of images of index 1, the selected image being selected as the image having a camera view which is closest to a camera view of the first image by:
using a cross bilateral filter between the semantic segmentation label associated with the selected image of the set of images of index 1 and the semantic segmentation label associated with the first image, and
performing a fusion of an output of the cross bilateral filter with the semantic segmentation label associated with the first image,
wherein the cross bilateral filter performs:
S
~
1
(
p
)
=
∑
q
∈
𝒩
(
p
)
G
σ
s
(
q
-
p
)
G
σ
r
(
I
z
(
q
)
-
I
z
(
p
)
)
S
1
(
q
)
∑
q
∈
𝒩
(
p
)
G
σ
s
(
q
-
p
)
G
σ
r
(
I
z
(
q
)
-
I
z
(
p
)
)
in which:
p and q are pixel positions,
Ŝ 1 (p) is the output of the cross bilateral filter for a pixel p,
(p) is a neighborhood of p,
G σ s is a spatial-domain Gaussian kernel,
G σ r is a color-domain kernel,
I z (q) and I z (p) respectively designate a color value at pixel q and pixel p in the first image from the set of index z, and
S 1 (q) is a semantic segmentation label at pixel q for the selected image of the set of images of index 1,
c—training the model using:
the set of images of index z and the associated processed semantic segmentation labels, and
a synthetic set of images of index z and the associated semantic segmentation labels both generated from the annotated set of images having a level of visibility of 1 and the associated semantic segmentation labels; and
d—performing steps a to c for z+1.
6. A model to be used for semantic segmentation trained by the method of claim 1 .
7. A vehicle comprising an image acquisition module and a system comprising the model trained by the method of claim 1 to perform semantic segmentation on the images acquired by the image acquisition module.