Self-learning object detectors for unlabeled videos using multi-task learning
A system and method enable generating a specific object detector for a category of interest. The method includes identifying seed objects in frames of a video sequence with a pre-trained generic detector for the category. An appearance model is iteratively learned for each of the seed objects using other frames in which the seed object is identified. The appearance models are learned jointly to optimize a loss function which accounts for the loss of incorrectly labeling sub-images and a regularization term which measures a distance between the appearance models. The loss of incorrectly labeling sub-images is determined using a motion model which predicts the location of the seed object in the subsequent frames so that sub-images outside the location that the current appearance model contribute to the loss. The specific object detector is then generated by aggregating the optimized appearance models.
1. A method for generating a specific object detector comprising:
with a generic detection model trained to identify objects in an object category, identifying a plurality of seed objects, each seed object being identified in a frame of a video sequence;
for a plurality of iterations,
for each of the plurality of seed objects:
with a current motion model, identifying an object region in a different frame of the video sequence where the seed object is predicted to be located, and
with a current appearance model, labeling sub-images of the different frame as positive or negative, and identifying as hard negatives, sub-images outside the object region that the current appearance model labels as positive for the object category; and
with a processor, jointly updating the current appearance models for the plurality of seed objects to optimize a loss function which includes a regularization term that is a function of a distance between the current appearance models; and
forming the specific object detector, comprising aggregating the appearance models of at least one of the iterations.
2. The method of claim 1 , wherein the regularization term is a function of a sum over all pairs of appearance models, of the distance between the current appearance models.
3. The method of claim 2 , wherein the distance between the current appearance models is measured as the l 2 norm between each pair of appearance models.
4. The method of claim 1 , wherein the loss function includes a loss term which aggregates, over the appearance models, the loss incurred by classifying a set of sub-images erroneously with the current appearance model, based on the identified hard negatives.
5. The method of claim 1 , wherein the loss function is of the general form:
W
*
=
arg
min
W
L
(
X
,
y
,
W
)
+
λΩ
(
W
)
,
(
3
)
where L(X,y,W) is the loss, over all the appearance models, incurred by classifying a sub-image as y based on current appearance model parameters and features of the sub-image, Ω(W) is the regularization term, and λ is a weight.
6. The method of claim 5 , wherein the loss L(X,y,W) is a sum of the loss for each of the appearance models.
7. The method of claim 6 , wherein the loss for each of the appearance models is computed as the logistic loss.
8. The method of claim 5 , wherein the regularization term is a computed as a function of the sum over all pairs of appearance models of the distance between the appearance models in the pair.
9. The method of claim 5 , wherein the regularization term is a computed as:
Ω
(
W
)
=
1
2
N
2
∑
i
=
1
N
∑
j
=
1
N
w
i
-
w
j
2
2
(
4
)
where
w
i
-
w
j
2
denotes the l 2 norm of a pair of appearance models w i and w j and N represents the number of the seed objects.
10. The method of claim 5 , wherein the loss function is optimized by stochastic gradient descent.
11. The method of claim 5 , wherein the aggregating the appearance models includes averaging the appearance models.
12. The method of claim 1 , wherein each appearance model comprises a vector of parameters and the specific object detector is a linear classifier which classifies a sub-image as a function of the vector of parameters and a feature vector for the sub-image.
13. The method of claim 12 , wherein the feature vector comprises a Fisher Vector.
14. The method of claim 1 , wherein for each of the plurality of iterations the method further comprises updating each motion model by identifying a most probable location of the seed object with the respective updated appearance model.
15. The method of claim 1 , wherein the method further comprises traversing at least one frame of the video sequence with a window of fixed size to generate the sub-images.
16. The method of claim 1 , wherein the object category is selected from people and cars.
17. The method of claim 1 , further comprising at least one of:
outputting the specific object detector; and
applying the specific object detector to the same or a new video sequence to detect objects in the object category.
18. A computer program product comprising a non-transitory recording medium storing instructions, which when executed on a computer, causes the computer to perform the method of claim 1 .
19. A system comprising memory which stores instructions for performing the method of claim 1 and a processor in communication with the memory which executes the instructions.
20. A system for generating a specific object detector comprising:
a generic detection model trained to identify objects in an object category, the generic detection model identifying a plurality of seed objects, each seed object being identified in a frame of a video sequence;
a tracking component which instantiates an ensemble of trackers one for each of a plurality of seed objects detected with the generic detection model, each tracker comprising a motion model and a detection model;
a learning component which learns a specific object detector, the learning comprising, for a plurality of iterations,
for each of the plurality of seed objects:
with a current motion model, identifying an object region in a different frame of the video sequence where the seed object is predicted to be located, and
with a current appearance model, labeling sub-images of the different frame as positive or negative, and identifying as hard negatives, sub-images outside the object region that the current appearance model labels as positive for the object category; and
jointly updating the current appearance models for the plurality of seed objects by optimizing a loss function which includes a loss term which accounts for the hard negatives and a regularization term that is a function of a distance between the current appearance models; and
aggregating the appearance models of at least one of the iterations to generate the specific object detector; and
a processor which implements the generic detection model, tracking component, and learning component.
21. A method for generating a specific object detector comprising:
with a generic detection model trained to identify objects in an object category, identifying a plurality of seed objects, each seed object being identified in a frame of a video sequence;
instantiating a tracker for each of the seed objects for tracking the seed object in subsequent frames of the video sequence, each tracker comprising a motion model and an appearance model, the motion model configured for identifying a region of a subsequent frame where the seed object is expected to be located, the appearance model configured for assigning a label that is selected from positive and negative for the seed object to each of a set of sub-images of the subsequent frame, hard negatives being identified as the sub-images in the set of sub images which are assigned a positive label by the appearance model and which are outside the identified region; and
learning a specific detector for the category of objects comprising, jointly learning the appearance models by sampling the sub-images and iteratively updating the appearance models to optimize a loss function which includes a regularization term, which is a function of a distance between the current appearance models, and a loss term which aggregates, over the appearance models, the loss incurred by classifying a set of sub-images erroneously with the current appearance model, based on the identified hard negatives, and aggregating the updated appearance models,
wherein at least one of the identifying of the plurality of seed objects, instantiating the trackers, and learning the specific detector is performed with a computer processor.