Shuffle, attend, and adapt: video domain adaptation by clip order prediction and clip attention alignment
View Patent ↗A method for performing video domain adaptation for human action recognition is presented. The method includes using annotated source data from a source video and unannotated target data from a target video in an unsupervised domain adaptation setting, identifying and aligning discriminative clips in the source and target videos via an attention mechanism, and learning spatial-background invariant human action representations by employing a self-supervised clip order prediction loss for both the annotated source data and the unannotated target data.
1. A computer-implemented method executed on a processor for performing video domain adaptation for human action recognition, the method comprising:
using annotated source data from a source video and unannotated target data from a target video in an unsupervised domain adaptation setting;
identifying and aligning discriminative clips in the source and target videos via an attention mechanism; and
learning spatial-background invariant human action representations by employing a self-supervised clip order prediction loss for both the annotated source data and the unannotated target data.
2. The method of claim 1 , wherein the attention mechanism suppresses temporal background clips.
3. The method of claim 2 , wherein the attention mechanism is learned jointly for video-level adaptation and classification.
4. The method of claim 3 , wherein the discriminative clips are selected by employing an auxiliary network such that video features are derived as a weighted combination of the identified discriminative clips.
5. The method of claim 4 , wherein the attention mechanism is pre-trained with a binary cross-entropy loss to obtain a ground truth attention vector.
6. The method of claim 1 , wherein the clip order prediction (COP) loss is given as:
L
COP
=
-
𝔼
(
x
,
y
)
∼
(
X
,
Y
)
∑
k
=
1
M
!
[
y
k
log
f
O
(
ϕ
)
]
where f O is a linear classification function for COP, ϕ=Ω(Φ(x 1 ), . . . , Φ(x M )) is a ReLU activation of multi-layer perceptron (MLP), which takes M clip features as input, and (x,y) are data.
7. The method of claim 1 , wherein a domain adversarial loss helps a shuffle and attend video domain adaptation (SAVA) model to align video-level features between the source and target videos.
8. A non-transitory computer-readable storage medium comprising a computer-readable program for performing video domain adaptation for human action recognition, wherein the computer-readable program when executed on a computer causes the computer to perform the steps of:
using annotated source data from a source video and unannotated target data from a target video in an unsupervised domain adaptation setting;
identifying and aligning discriminative clips in the source and target videos via an attention mechanism; and
learning spatial-background invariant human action representations by employing a self-supervised clip order prediction loss for both the annotated source data and the unannotated target data.
9. The non-transitory computer-readable storage medium of claim 8 , wherein the attention mechanism suppresses temporal background clips.
10. The non-transitory computer-readable storage medium of claim 9 , wherein the attention mechanism is learned jointly for video-level adaptation and classification.
11. The non-transitory computer-readable storage medium of claim 10 , wherein the discriminative clips are selected by employing an auxiliary network such that video features are derived as a weighted combination of the identified discriminative clips.
12. The non-transitory computer-readable storage medium of claim 11 , wherein the attention mechanism is pre-trained with a binary cross-entropy loss to obtain a ground truth attention vector.
13. The non-transitory computer-readable storage medium of claim 8 , wherein the clip order prediction (COP) loss is given as:
L
COP
=
-
𝔼
(
x
,
y
)
∼
(
X
,
Y
)
∑
k
=
1
M
!
[
y
k
log
f
O
(
ϕ
)
]
where f O is a linear classification function for COP, ϕ=Φ(Ψ(x 1 ), . . . , Φ(x M )) is a ReLU activation of multi-layer perceptron (MLP), which takes M clip features as input, and (x,y) are data.
14. The non-transitory computer-readable storage medium of claim 8 , wherein a domain adversarial loss helps a shuffle and attend video domain adaptation (SAVA) model to align video-level features between the source and target videos.
15. A system for performing video domain adaptation for human action recognition, the system comprising:
a memory; and
one or more processors in communication with the memory configured to:
use annotated source data from a source video and unannotated target data from a target video in an unsupervised domain adaptation setting;
identify and align discriminative clips in the source and target videos via an attention mechanism; and
learn spatial-background invariant human action representations by employing a self-supervised clip order prediction loss for both the annotated source data and the unannotated target data.
16. The system of claim 15 , wherein the attention mechanism suppresses temporal background clips.
17. The system of claim 16 , wherein the attention mechanism is learned jointly for video-level adaptation and classification.
18. The system of claim 17 , wherein the discriminative clips are selected by employing an auxiliary network such that video features are derived as a weighted combination of the identified discriminative clips.
19. The system of claim 18 , wherein the attention mechanism is pre-trained with a binary cross-entropy loss to obtain a ground truth attention vector.
20. The system of claim 15 , wherein the clip order prediction (COP) loss is given as:
L
COP
=
-
𝔼
(
x
,
y
)
∼
(
X
,
Y
)
∑
k
=
1
M
!
[
y
k
log
f
O
(
ϕ
)
]
where f O is a linear classification function for COP, ϕ=χ(Φ(x 1 ), . . . , Φ(x M )) is a ReLU activation of multi-layer perceptron (MLP), which takes M clip features as input, and (x,y) are data.