Robust tracking of objects in videos

1. In a digital environment for tracking objects in videos, a method of identifying objects in videos comprising:

receiving a video;

extracting a plurality of video frames from the video;

generating an image search index from the plurality of video frames;

receiving an indication of a query object within one or more key frames of the plurality of video frames; and

for each of the plurality of video frames in the image search index:

determining a similarity score between a key frame and the video frame based on a search area in the key frame, wherein a size of the search area is determined based on a distance between the key frame and the video frame, and

generating a voting map that utilizes the determined similarity score to localize the query object in the video frame.

2. The method as recited in claim 1 , further comprising identifying one or more auxiliary key frames.

3. The method as recited in claim 2 , wherein identifying one or more auxiliary key frames comprises:

selecting a candidate video frame from the image search index;

determining, based on a spatially-constrained area within the key frame, a similarity between the candidate video frame and each of the one or more key frames; and

determining that the similarity between the candidate video frame and a key frame of the one or more key frames is greater than a predetermined threshold;

re-categorizing, based on the similarity being greater than the predetermined threshold, the candidate video frame as an auxiliary key frame.

4. The method as recited in claim 3 , further comprising:

determining a first candidate query object for the video frame based on the key frame;

determining a second candidate query object for the video frame based on the auxiliary key frame;

weighting a similarity score for the first candidate query object using a time decay function;

weighting a similarity score for the second candidate query object using the time decay function; and

selecting as the query object one of the first candidate query object or the second candidate query object that has the maximum weighted similarity score.

5. The method as recited in claim 1 , wherein generating the image search index comprises:

identifying one or more video frames in the received video;

extracting one or more features from each of the one or more video frames.

6. The method as recited in claim 1 , further comprising redacting the query object from the video frames in which the query object is identified.

7. The method as recited in claim 6 , wherein redacting the query object from the video frames in which the query object is identified comprises:

identifying, within each of the video frames in which the query object is identified, an area around the localized query object;

changing a color of pixels within the area around the localized query object.

8. The method as recited in claim 1 , further comprising determining a location of the search area based on a location of the query object in a keyframe.

9. The method as recited in claim 1 , further comprising:

sequentially determining similarity scores working backward and forward from the key frame,

adjusting the determined similarity scores using penalty variables, wherein a penalty variable for a given similarity score is based on the given similarity score, a penalty variable for a previous similarity score, and a lower threshold.

10. A system for tracking objects in videos comprising:

a memory comprising a video;

a computing device, storing instructions thereon that, when executed by the computing device, cause the system to:

extract a plurality of video frames from the video;

generate an image search index from the plurality of video frames by extracting one or more features from each of the video frames;

receive an indication of a query object within one or more key frames of the plurality of video frames and a location of the query object with the one or more key frames; and

for each of the plurality of video frames:

determine a similarity score between a key frame and a video frame based on a search area in the key frame by comparing features of the query object in the key frame to features of the video frame within the search area, wherein a size of the search area is determined based on a distance between the key frame and the video frame, and

generate a voting map that utilizes the determined similarity score to localize the query object in the video frame.

11. The system as recited in claim 10 , wherein the instructions, when executed by the computing device, further cause the system to:

track backward from the key frame to identify the query object by performing acts comprising:

identifying a bounding box around the query object in the key frame,

identifying a preceding candidate video frame that has an earlier time stamp than the key frame,

identifying a search area for the preceding candidate video frame based on a location of the bounding box and a distance from the key frame to the preceding candidate video frame, and

track forward from the key frame to identify the query object by performing acts comprising:

identifying a subsequent candidate video frame that has a later time stamp than the key frame,

identifying a search area for the subsequent candidate video frame based on a location of the bounding box and a distance from the key frame to the subsequent candidate video frame.

12. The system as recited in claim 11 , wherein the instructions, when executed by the computing device, further cause the system to

redact the query object from the video frames in which the query object is identified; and

generate a redacted video by merging the video frames in which the query object has been redacted with a remainder of the plurality of video frames based on time stamps associated with each video frame.

13. A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause a computer system to:

extract a plurality of video frames from the video;

generate an image search index from the plurality of video frames by extracting one or more features from each of the video frames;

receive an indication of a query object within one or more key frames of the plurality of video frames and a location of the query object with the one or more key frames; and

for each of the plurality of video frames:

determine a similarity score between a key frame and a video frame based on a search area in the key frame, wherein a size of the search area is determined based on a distance between the key frame and the video frame, and

generate a voting map that utilizes the determined similarity score to localize the query object in the video frame.

14. The non-transitory computer-readable medium as recited in claim 13 , further storing instructions thereon that, when executed by the at least one processor, cause the system to identify one or more auxiliary key frames.

15. The non-transitory computer-readable medium as recited in claim 14 , wherein identifying one or more auxiliary key frames comprises:

selecting a candidate video frame from the image search index;

determining, based on a spatially-constrained area within the key frame, a similarity between the candidate video frame and each of the one or more key frames; and

determining that the similarity between the candidate video frame and a key frame of the one or more key frames is greater than a predetermined threshold; and

re-categorizing, based on the similarity being greater than the predetermined threshold, the candidate video frame as an auxiliary key frame.

16. The non-transitory computer-readable medium as recited in claim 15 , further storing instructions thereon that, when executed by the at least one processor, cause the system to:

determine a first candidate query object for the video frame based on the key frame;

determine a second candidate query object for the video frame based on the auxiliary key frame;

weight a similarity score for the first candidate query object using a time decay function;

weight a similarity score for the second candidate query object using the time decay function; and

select as the query object one of the first candidate query object or the second candidate query object that has the maximum weighted similarity score.

17. The non-transitory computer-readable medium as recited in claim 16 , wherein generating the image search index comprises:

identifying one or more video frames in the received video;

extracting one or more features from each of the one or more video frames.

18. The non-transitory computer-readable medium as recited in claim 17 , further storing instructions thereon that, when executed by the at least one processor, cause the system to redact the query object from the video frames in which the query object is identified.

19. The non-transitory computer-readable medium as recited in claim 18 , wherein redacting the query object from the video frames in which the query object is identified comprises:

identifying, which each of the video frames in which the query object is identified, an area around the localized query object; and

changing a color of pixels within the area around the localized query object.

20. The non-transitory computer-readable medium as recited in claim 19 , further storing instructions thereon that, when executed by the at least one processor, cause the system to determine a location of the search area based on a location of the query object in a key frame.