Method and apparatus with object tracking
A method and apparatus with object tracking is provided. The method includes generating a mixed filter by fusing a short-term filter with a long-term filter; and performing object tracking on a current frame image based on the mixed filter. The short-term filter is dependent on a prediction of the current frame image in a video sequence, and the long-term filter is a previously generated long-term filter or is generated by optimizing the previously generated long-term filter based on an object template feature pool.
1. A processor-implemented method, comprising:
generating a mixed filter by fusing a short-term filter with a long-term filter; and
performing object tracking on a current frame image based on the mixed filter,
wherein the short-term filter is dependent on a prediction of the current frame image in a video sequence, and
the long-term filter is a previously generated long-term filter or is generated by optimizing the previously generated long-term filter based on an object template feature pool.
2. The method of claim 1 , further comprising, prior to the generating of the mixed filter, predicting the short-term filter based on a first frame image of the video sequence, the current frame image and an auxiliary frame image of the video sequence,
wherein the auxiliary frame image is an image frame that has a determined greater tracking success confidence than a first threshold value and is closest to the current frame image in time sequence.
3. The method of claim 2 , wherein the predicting of the short-term filter comprises:
extracting features, through a feature extraction network, for a first search region from the first frame image, an auxiliary search region from the auxiliary frame image, and a current search region from the current frame image, and extracting a first deep feature of the first search region, an auxiliary deep feature of the auxiliary search region, and a current deep feature of the current search region;
generating an object state encoding vector by performing object state encoding on the first deep feature, a first bounding box of the first frame image with respect to the object, the auxiliary deep feature, and an auxiliary bounding box of the auxiliary frame image with respect to the object;
obtaining a current frame encoding vector by performing encoding on the current deep feature;
generate a hidden feature using a trained transformer model provided an input based on the object state encoding vector and the current frame encoding vector to thus; and
generating the short-term filter by linearly transforming the hidden feature,
wherein the first search region is determined according to the first bounding box,
the auxiliary search region is determined according to the auxiliary bounding box, and
the current search region is determined according to a predicted bounding box of a predicted object based on N number of frame images prior to the current frame image, wherein N is an integer greater than or equal to 1.
4. The method of claim 1 , further comprising, prior to the generating of the mixed filter,
in response to the current frame image being determined to be an image frame at a predetermined position in the video sequence, generating the long-term filter by optimizing the previously obtained long-term filter based on the object template feature pool; or
in response to the current frame image being determined to not be an image frame at the predetermined position in the video sequence, generating the previously obtained long-term filter as the long-term filter.
5. The method of claim 1 , wherein the optimizing of the previously obtained long-term filter comprises:
extracting a predetermined number of deep features and bounding boxes of the object corresponding to respective ones of accumulated deep features from the object template feature pool and determining the extracted deep features and bounding boxes to be a filter training set; and
training and/or optimizing, based on the filter training set, the previously obtained long-term filter through a filter optimization algorithm.
6. The method of claim 1 , wherein the generating of the mixed filter by fusing the short-term filter with the long-term filter comprises:
generating a short-term object positioning response map and a long-term object positioning response map by respectively performing correlation processing on the current frame image using the short-term filter and the long-term filter; and
generating the mixed filter by fusing the short-term filter with the long-term filter according to the short-term object positioning response map and the long-term object positioning response map.
7. The method of claim 6 , wherein the generating of the mixed filter further comprises:
evaluating short-term map quality of the short-term object positioning response map, and long-term map quality of the long-term object positioning response map;
determining a mixture weight of the short-term filter and a mixture weight of the long-term filter according to a result of comparing a second predetermined threshold value to the short-term map quality and the long-term map quality; and
generating the mixed filter by fusing the short-term filter with the long-term filter according to the mixture weight of the short-term filter and the mixture weight of the long-term filter.
8. The method of claim 7 , wherein the determining of the mixture weight of the short-term filter and the mixture weight of the long-term filter comprises:
in response to the short-term map quality being determined greater than or equal to the second predetermined threshold value and the long-term map quality is less than the second predetermined threshold value, setting the mixture weight of the short-term filter as 1 and the mixture weight of the long-term filter as 0;
in response to the short-term map quality being determined less than the second predetermined threshold value and the long-term map quality is greater than or equal to the second predetermined threshold value, setting the mixture weight of the short-term filter as 0 and the mixture weight of the long-term filter as 1;
in response to both the mixture weights of the short-term filter and the long-term map being determined to have respective qualities that are less than the second predetermined threshold value, setting each of the mixture weights as a weight value corresponding to a previously obtained mixed filter; or
in response to both the mixture weights of the short-term filter and the long-term map being determined to have respective qualities that are greater than or equal to the second predetermined threshold value, setting each of the mixture weights as a mixture weight of a normalized output of a Softmax activation function of the short-term map quality and the long-term map quality.
9. The method of claim 6 , wherein the generating of the mixed filter further comprises:
generating a mixture weight of the short-term filter and a mixture weight of the long-term filter by using a convolutional neural network and a normalization function, according to the short-term object positioning response map and the long-term object positioning response map; and
generating the mixed filter by fusing the short-term filter with the long-term filter according to the mixture weight of the short-term filter and the mixture weight of the long-term filter.
10. The method of claim 9 , wherein the generating of the mixture weight of the short-term filter and the mixture weight of the long-term filter further comprises:
generating a mixed response map by mixing and processing the short-term object positioning response map and the long-term object positioning response map;
extracting a feature from the mixed response map using the convolutional neural network, and generating a mixture weight vector by linearly transforming the extracted feature using a linear transformation layer; and
generating the mixture weight of the short-term filter and the mixture weight of the long-term filter by normalizing the mixture weight vector according to a Softmax activation function.
11. The method of claim 1 , wherein the performing of the object tracking further comprises:
generating, using the mixed filter, an object positioning response map by performing correlation processing on a current deep feature of an object search region of the current frame image;
predicting an initial target object bounding box of the current frame image based on the object positioning response map; and
segmenting a target object by using a segmentation network based on the initial target object bounding box and generating a target object segmentation map of the target object and a bounding box of a final target object,
wherein the object search region of the current frame image is determined according to a bounding box of a predicted target object based on N number (N is an integer greater than or equal to 1) of frame images prior to the current frame image.
12. The method of claim 1 , wherein the object template feature pool comprises accumulated deep features including at least:
a deep feature of an object search region of the current frame image;
a plurality of deep features obtained by performing multi-augmentation processing on an object search region of a first frame image of the video sequence to obtain a plurality of augmented images and extracting features with respect to the plurality of augmented images;
and/or a deep feature of an image frame having a determined greater tracking success confidence than a first predetermined threshold value when object tracking is performed on an image frame prior to the current frame image in the video sequence.
13. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method of claim 1 .
14. A computing apparatus comprising:
one or more processors configured to execute instructions; and
one or more memories storing the instructions,
wherein the execution of the instructions by the one or more processors configures the one or more processors to control:
a mixed filter generator to generate a mixed filter by fusing a short-term filter with a long-term filter; and
an object tracker to perform object tracking on a current frame image based on the mixed filter,
wherein
the short-term filter is dependent on a prediction of the current frame image in a video sequence, and
the long-term filter is a previously obtained long-term filter or is obtained by optimizing the previously obtained long-term filter based on an object template feature pool.
15. The computing apparatus of claim 14 , wherein the one or more processors are configured to further control a short-term filter generator to predict the short-term filter based on a first frame image of the video sequence, the current frame image and an auxiliary frame image of the video sequence,
wherein the auxiliary frame image is an image frame that has a determined greater tracking success confidence than a first threshold value and is closest to the current frame image in time sequence.
16. The computing apparatus of claim 14 , wherein the one or more processors are configured to further control a long-term filter generator, in response to the current frame image being determined to be an image frame at a predetermined position in the video sequence to generate the long-term filter by optimizing the previously obtained long-term filter based on the object template feature pool, or in response to the current frame image being determined to not be an image frame at the predetermined position in the video sequence, to generate the previously obtained long-term filter as the long-term filter.
17. The computing apparatus of claim 14 , wherein the mixed filter generator is configured to:
generate a short-term object positioning response map and a long-term object positioning response map by respectively performing correlation processing on the current frame image, using the short-term filter and the long-term filter; and
generate the mixed filter by fusing the short-term filter with the long-term filter according to the short-term object positioning response map and the long-term object positioning response map.
18. The computing apparatus of claim 17 , wherein, the mixed filter generator is further configured to:
evaluate short-term map quality of the short-term object positioning response map, and long-term map quality of the long-term object positioning response map;
determine a mixture weight of the short-term filter and a mixture weight of the long-term filter according to a result of comparing a second predetermined threshold value to the short-term map quality and the long-term map quality; and
generate the mixed filter by fusing the short-term filter with the long-term filter according to the mixture weight of the short-term filter and the mixture weight of the long-term filter.
19. The computing apparatus of claim 17 , wherein, the mixed filter generator is further configured to:
generate a mixture weight of the short-term filter and a mixture weight of the long-term filter by using a convolutional neural network and a normalization function, according to the short-term object positioning response map and the long-term object positioning response map; and
generate the mixed filter by fusing the short-term filter with the long-term filter according to the mixture weight of the short-term filter and the mixture weight of the long-term filter.
20. The computing apparatus of claim 14 , wherein the object tracker comprises:
an object positioning response map generator configured to obtain, using the mixed filter, an object positioning response map by performing correlation processing on a current deep feature of an object search region of the current frame image;
an initial target object bounding box predictor configured to predict an initial target object bounding box of the current frame image based on the object positioning response map; and
a final target object bounding box predictor configured to segment a target object by using a segmentation network based on the initial target object bounding box and generating a target object segmentation map of the target object and a bounding box of a final target object,
wherein the object search region of the current frame image is determined according to a bounding box of a predicted target object based on N number (N is an integer greater than or equal to 1) of frame images prior to the current frame image.