Video domain adaptation via contrastive learning for decision making
Video methods and systems include extracting features of a first modality and a second modality from a labeled first training dataset in a first domain and an unlabeled second training dataset in a second domain. A video analysis model is trained using contrastive learning on the extracted features, including optimization of a loss function that includes a cross-domain regularization part and a cross-modality regularization part. The video methods and systems can support real-time decision making in various applications, such as security surveillance where immediate responses to detected actions are required.
1. A computer-implemented machine learning method, comprising:
extracting features of a first modality and a second modality from a labeled first training dataset in a first domain and an unlabeled second training dataset in a second domain, the labeled first training dataset including source videos and action labels, the source videos being received from a camera, the action labels indicating player's actions in sports;
training a video analysis model using contrastive learning on the extracted features, including optimization of a loss function that includes a cross-domain regularization part that compares features from a first training data from the first training dataset and a second training data from the second training dataset, the second training data having a pseudo label that matches the label of the first training data, and a cross-modality regularization part that compares features from different cue types in a same domain, with the cross-domain regularization part being expressed as
ℒ
do
=
-
log
∑
k
=
l
∈
{
a
,
m
}
ϕ
+
st
(
F
s
i
k
,
F
t
i
+
l
)
∑
k
=
l
∈
{
a
,
m
}
ϕ
+
st
(
F
s
i
k
,
F
t
i
+
l
)
+
ϕ
-
st
(
F
s
i
k
,
F
t
i
-
l
)
where ϕ + st (F s i k ,F t i+ l ) measures similarity between features having a same modality and different domains for positive samples and ϕ − st (F s i k ,F t j− l ) measures similarity between features having a same modality and different domains for negative samples, and further including generating pseudo-labels for the unlabeled dataset.
2. The computer-implemented machine learning method of claim 1 , wherein the pseudo-labels are generated by the video analysis model.
3. The computer-implemented machine learning method of claim 1 , wherein the different cue types include appearance features and motion features.
4. The computer-implemented machine learning method of claim 1 , wherein the first domain relates to video taken from a first perspective and the second domain relates to video taken from a second, different perspective.
5. The computer-implemented machine learning method of claim 1 ,
wherein the loss function is represented as:
L all =L src ( V s ,Y s )+λ( L mo s ( V s )+ L do ( V s ,V t ,Ŷt ))
where V is a set of videos in a source domain, V t is a set of videos in a target domain, Y s are labels for the source videos, Ŷ t are pseudo-labels for the target videos, L src is a cross-entropy loss for the source videos, L mo s in is a cross-modality loss term for the source videos, L mo t is a cross-modality loss term for the target videos, and λ is a balancing parameter.
6. The computer-implemented machine learning method of claim 5 , wherein the cross-modality loss term for the source videos is expressed as:
ℒ
mo
s
=
-
log
∑
k
≠
l
∈
{
a
,
m
}
ϕ
+
s
(
F
s
i
k
,
F
s
i
l
)
∑
k
≠
l
∈
{
a
,
m
}
ϕ
+
s
(
F
s
i
k
,
F
s
i
+
l
)
+
ϕ
-
s
(
F
s
i
k
,
F
s
j
-
l
)
where ϕ + s (F s i k ,F s i l ) measures similarity between features having a different modality and same domain for positive samples and ϕ − s (F s i k ,F s j− l ) measures similarity between features having a different modality and same domain for negative samples.
7. A computer-implemented video domain adaptation method, comprising:
extracting features of a first modality and a second modality from a labeled first training dataset in a first domain, relating to video taken from a first perspective, and an unlabeled second training dataset in a second domain, relating to video taken from a second, different perspective, the labeled first training dataset including source videos and action labels, the source videos being received from a camera, the action labels indicating player's actions in sports;
training a video analysis model using contrastive learning on the extracted features, including:
generating pseudo-labels for the unlabeled training dataset using the video analysis model;
optimization of a loss function that includes a cross-domain regularization part, that compares features from a first training data from the first training dataset and a second training data from the second training dataset, the second training data having a pseudo label that matches the label of the first training data, and a cross-modality regularization part, that compares features from different cue types in a same domain, with the cross-domain less regularization part being expressed as
ℒ
do
=
-
log
∑
k
=
l
∈
{
a
,
m
}
ϕ
+
st
(
F
s
i
k
,
F
t
i
+
l
)
∑
k
=
l
∈
{
a
,
m
}
ϕ
+
st
(
F
s
i
k
,
F
t
i
+
l
)
+
ϕ
-
st
(
F
s
i
k
,
F
t
i
-
l
)
where ϕ + st (F s i k ,F t i+ l ) measures similarity between features having a same modality and different domains for positive samples and ϕ − st (F s i k ,F t i− l ) measures similarity between features having a same modality and different domains for negative samples; and
generating pseudo-labels for the unlabeled dataset.
8. A machine learning system, comprising:
a hardware processor; and
a memory that stores a computer program that, when executed by the hardware processor, causes the hardware processor to:
extract features of a first modality and a second modality from a labeled first training dataset in a first domain and an unlabeled second training dataset in a second domain, the labeled first training dataset including source videos and action labels, the source videos being received from a camera, the action labels indicating player's actions in sports;
training a video analysis model using contrastive learning on the extracted features, including optimization of a loss function that includes a cross-domain regularization part that compares features from a first training data from the first training dataset and a second training data from the second training dataset, the second training data having a pseudo label that matches the label of the first training data, and a cross-modality regularization part that compares features from different cue types in a same domain, with the cross-domain regularization part being expressed as
ℒ
do
=
-
log
∑
k
=
l
∈
{
a
,
m
}
ϕ
+
st
(
F
s
i
k
,
F
t
i
+
l
)
∑
k
=
l
∈
{
a
,
m
}
ϕ
+
st
(
F
s
i
k
,
F
t
i
+
l
)
+
ϕ
-
st
(
F
s
i
k
,
F
t
i
-
l
)
where ϕ + st (F s i k ,F t i+ l ) measures similarity between features having a same modality and different domains for positive samples and ϕ − st (F s i k ,F t i− l ) measures similarity between features having a same modality and different domains for negative samples, and further generating pseudo-labels for the unlabeled training dataset.