Video domain adaptation via contrastive learning for decision making
Video methods and systems include extracting features of a first modality and a second modality from a labeled first training dataset in a first domain and an unlabeled second training dataset in a second domain. A video analysis model is trained using contrastive learning on the extracted features, including optimization of a loss function that includes a cross-domain regularization part and a cross-modality regularization part.
1. A computer-implemented video method, comprising:
extracting features of a first modality and a second modality from a labeled first training dataset in a first domain and an unlabeled second training dataset in a second domain;
training a video analysis model using contrastive learning on the extracted features, including optimization of a loss function that includes a cross-domain regularization part that compares features from a first training data from the first training dataset and a second training data from the second training dataset, the second training data having a pseudo label that matches the label of the first training data, and a cross-modality regularization part that compares features from different cue types in a same domain, with the cross-domain less regularization part being expressed as
ℒ
d
o
=
-
log
∑
k
=
l
∈
{
a
,
m
}
ϕ
+
st
(
F
s
i
k
,
F
t
i
+
l
)
∑
k
=
l
∈
{
a
,
m
}
ϕ
+
st
(
F
s
i
k
,
F
t
i
+
l
)
+
ϕ
-
st
(
F
s
i
k
,
F
t
i
-
l
)
where ϕ + st (F s i k ,F t i + l ) measures similarity between features having a same modality and different domains for positive samples and ϕ − st (F s i k ,F t i − l ) measures similarity between features having a same modality and different domains for negative samples, and further including generating pseudo-labels for the unlabeled dataset.
2. The computer-implemented method of claim 1 , wherein the pseudo-labels are generated by the video analysis model.
3. The computer-implemented method of claim 1 , wherein the different cue types include appearance features and motion features.
4. The computer-implemented method of claim 1 , wherein the first domain relates to video taken from a first perspective and the second domain relates to video taken from a second, different perspective.
5. The computer-implemented method of claim 1 , wherein the loss function is represented as:
all = src ( V s ,Y s )+λ( mo s ( V s )+ mo t ( V t )+ do ( V s ,V t ,Ŷ t )
where V s is a set of videos in a source domain, V t is a set of videos in a target domain, Y s are labels for the source videos, Ŷ t are pseudo-labels for the target videos, src is a cross-entropy loss for the source videos, mo s is a cross-modality loss term for the source videos, mo t is a cross-modality loss term for the target videos, and A is a balancing parameter.
6. The computer-implemented method of claim 5 , wherein the cross-modality loss term for the source videos is expressed as:
ℒ
m
o
s
=
-
log
∑
k
≠
l
∈
{
a
,
m
}
ϕ
+
s
(
F
s
i
k
,
F
s
i
l
)
∑
k
≠
l
∈
{
a
,
m
}
ϕ
+
s
(
F
s
i
k
,
F
s
i
+
l
)
+
ϕ
-
s
(
F
s
i
k
,
F
s
j
-
l
)
where ϕ + s (F s i k ,F t i + l ) measures similarity between features having a different modality and same domain for positive samples and ϕ − st (F s i k ,F t i − l ) measures similarity between features having a different modality and same domain for negative samples.
7. The method of claim 1 , further comprising performing action recognition on an input video using the trained video analysis model.
8. A computer-implemented video method, comprising:
extracting features of a first modality and a second modality from a labeled first training dataset in a first domain, relating to video taken from a first perspective, and an unlabeled second training dataset in a second domain, relating to video taken from a second, different perspective;
training a video analysis model using contrastive learning on the extracted features, including:
generating pseudo-labels for the unlabeled training dataset using the video analysis model;
optimization of a loss function that includes a cross-domain regularization part, that compares features from a first training data from the first training dataset and a second training data from the second training dataset, the second training data having a pseudo label that matches the label of the first training data, and a cross-modality regularization part, that compares features from different cue types in a same domain, with the cross-domain regularization part being expressed as
ℒ
d
o
=
-
log
∑
k
=
l
∈
{
a
,
m
}
ϕ
+
st
(
F
s
i
k
,
F
t
i
+
l
)
∑
k
=
l
∈
{
a
,
m
}
ϕ
+
st
(
F
s
i
k
,
F
t
i
+
l
)
+
ϕ
-
st
(
F
s
i
k
,
F
t
i
-
l
)
where ϕ + st (F s i k ,F t i + l ) measures similarity between features having a same modality and different domains for positive samples and ϕ − st (F s i k ,F t i − l ) measures similarity between features having a same modality and different domains for negative samples.
9. A video system, comprising:
a hardware processor; and
a memory that stores a computer program that, when executed by the hardware processor, causes the hardware processor to:
extract features of a first modality and a second modality from a labeled first training dataset in a first domain and an unlabeled second training dataset in a second domain;
training a video analysis model using contrastive learning on the extracted features, including optimization of a loss function that includes a cross-domain regularization part that compares features from a first training data from the first training dataset and a second training data from the second training dataset, the second training data having a pseudo label that matches the label of the first training data, and a cross-modality regularization part that compares features from different cue types in a same domain, with the cross-domain regularization part being expressed as
ℒ
d
o
=
-
log
∑
k
=
l
∈
{
a
,
m
}
ϕ
+
s
t
(
F
s
i
k
,
F
t
i
+
l
)
∑
k
=
l
∈
{
a
,
m
}
ϕ
+
s
t
(
F
s
i
k
,
F
t
i
+
l
)
+
ϕ
-
st
(
F
s
i
k
,
F
t
i
-
l
)
where ϕ + st (F s i k ,F t i + l ) measures similarity between features having a same modality and different domains for positive samples and ϕ − st (F s i k ,F t i − l ) measures similarity between features having a same modality and different domains for negative samples; and
generating pseudo-labels for the unlabeled training dataset.
10. The system of claim 9 , wherein the pseudo-labels are generated by the video analysis model.
11. The system of claim 9 , wherein the different cue types include appearance features and motion features.
12. The system of claim 9 , wherein the loss function is represented as:
all = src ( V s ,Y s )+λ( mo s ( V s )+ mo t ( V t )+ do ( V s ,V t ,Ŷ t )
where V s is a set of videos in a source domain, V t is a set of videos in a target domain, Y s are labels for the source videos, Ŷ t are pseudo-labels for the target videos, src is a cross-entropy loss for the source videos, mo s is a cross-modality loss term for the source videos, mo t is a cross-modality loss term for the target videos, and λ is a balancing parameter.
13. The system of claim 12 , wherein the cross-modality loss term for the source videos is expressed as:
ℒ
m
o
s
=
-
log
∑
k
≠
l
∈
{
a
,
m
}
ϕ
+
s
(
F
s
i
k
,
F
s
i
l
)
∑
k
≠
l
∈
{
a
,
m
}
ϕ
+
s
(
F
s
i
k
,
F
s
i
+
l
)
+
ϕ
-
s
(
F
s
i
k
,
F
s
j
-
l
)
where ϕ + st (F s i k ,F t i + l ) measures similarity between features having a different modality and same domain for positive samples and ϕ − st (F s i k ,F t i − l ) measures similarity between features having a different modality and same domain for negative samples.