Multi-domain semantic segmentation with label shifts
Methods and systems for training a model include combining data from multiple datasets, the datasets having different respective label spaces. Relationships between labels in the different label spaces are identified. A unified neural network model is trained, using the combined data and the identified relationships to generate a unified model, with a class relational binary cross-entropy loss.
1. A computer-implemented method for training a model, comprising:
combining data from multiple datasets, the datasets having different respective label spaces;
identifying relationships between labels in the different label spaces, including determining a cosine similarity between an image feature and a weight of a label class; and
training a unified neural network model, using the combined data and the identified relationships to generate a unified model, with a class relational binary cross-entropy loss.
2. The computer-implemented method of claim 1 , wherein the unified neural network model is an image segmentation model.
3. The computer-implemented method of claim 2 , wherein the multiple datasets include images, with the different respective label spaces relating to different classes of objects within the images.
4. The computer-implemented method of claim 3 , wherein the different label spaces include at least two labels that overlap in objects that they apply to.
5. The computer-implemented method of claim 1 , further comprising:
capturing a new image of a scene;
performing image segmentation using the unified neural network model to identify objects within the scene; and
navigating through the scene to avoid collision with the identified objects.
6. The computer-implemented method of claim 1 , wherein the relationships include synonym relationships and hierarchical relationships.
7. The computer-implemented method of claim 1 , wherein the class relational binary cross-entropy loss is determined as:
ℒ
s
e
g
c
l
-
b
c
e
=
-
∑
i
=
1
N
∑
k
=
1
K
i
c
∑
h
,
w
Y
~
i
,
c
h
,
w
,
k
log
(
Q
i
(
h
,
w
,
k
)
)
+
(
1
-
Y
~
i
,
c
h
,
w
,
k
)
log
(
1
-
Q
i
(
h
,
w
,
k
)
)
where N is a number of the multiple datasets, K i c is a number of labels having identified relationships to a label c, h and w are coordinates within an image, {tilde over (Y)} i,c h,w,k is amulti-class label for a pixel, and Q i (h,w,k) represents an independent probability of each class.
8. The computer-implemented method of claim 1 , wherein identifying the relationships between labels includes determining that a first label having a similarity score to a second label that exceeds a threshold is related to the second label.
9. The computer-implemented method of claim 1 , combining data from multiple datasets includes concatenating the multiple datasets in a single unified dataset.
10. A computer-implemented method for training a model, comprising:
combining data from multiple datasets of images, the datasets having different respective label spaces that relate to different classes of objects within the images;
identifying relationships between labels in the different label spaces, including hierarchical and synonym relationships, based on a cosine similarity between an image feature and a weight of a label class; and
training a unified image segmentation neural network model, using the combined data and the identified relationships to generate a unified model, with a class relational binary cross-entropy loss.
11. A system for training a model, comprising:
a hardware processor; and
a memory that stores a computer program product, which, when executed by the hardware processor, causes the hardware processor to:
combine data from multiple datasets, the datasets having different respective label spaces;
identify relationships between labels in the different label spaces, including a determination of a cosine similarity between an image feature and a weight of a label class; and
train a unified neural network model, using the combined data and the identified relationships to generate a unified model, with a class relational binary cross-entropy loss.
12. The system of claim 11 , wherein the unified neural network model is an image segmentation model.
13. The system of claim 12 , wherein the multiple datasets include images, with the different respective label spaces relating to different classes of objects within the images.
14. The system of claim 13 , wherein the different label spaces include at least two labels that overlap in objects that they apply to.
15. The system of claim 11 , wherein the relationships include synonym relationships and hierarchical relationships.
16. The system of claim 11 , wherein the class relational binary cross-entropy loss is determined as:
ℒ
s
e
g
c
l
-
b
c
e
=
-
∑
i
=
1
N
∑
k
=
1
K
i
c
∑
h
,
w
Y
~
i
,
c
h
,
w
,
k
log
(
Q
i
(
h
,
w
,
k
)
)
+
(
1
-
Y
~
i
,
c
h
,
w
,
k
)
log
(
1
-
Q
i
(
h
,
w
,
k
)
)
where N is a number of the multiple datasets, K i c is a number of labels having identified relationships to a label c, h and w are coordinates within an image, {tilde over (Y)} i,c h,w,k is a multi-class label for a pixel, and Q i (h,w,k) represents an independent probability of each class.
17. The system of claim 11 , wherein the computer program product further causes the hardware processor to determine that a first label having a similarity score to a second label that exceeds a threshold is related to the second label.
18. The system of claim 11 , wherein the computer program product further causes the hardware processor to concatenate the multiple datasets in a single unified dataset.