Video object tracking
A technique is disclosed for automating tracking of annotated objects and improves the throughput and efficiency of existing methods while maintaining a degree of accuracy comparable to a human annotator. In particular, the disclosed technique provides an automated annotated object tracking tool that allows machine-learning teams to annotate an object within a frame and have that annotation persist across frames as the annotated object is tracked within a series of frames, still ensuring that every frame is accurately reviewed by a human where high quality annotation is required. This technique incorporates human feedback via a user adjustment that allows the tool to adapt and improve its accuracy in tracking an annotated object across a sequence of frames.
1. A system for object-level tracking comprising:
a processor and a memory coupled with the processor, wherein the memory is configured to provide the processor with instructions which when executed cause the processor to:
present a first frame in a series of pre-recorded frames;
receive a first annotation of a first bounding region of an object in the first frame;
assign an object identifier associated with the first bounding region;
obtain a second bounding region of the object in a second frame in the series of pre-recorded frames based at least in part on a prediction of a location of the object in the second frame, wherein the second bounding region is associated with the object identifier;
output the second bounding region;
maintain a prediction of the location of the object in subsequent frames using the object identifier;
predict the location of the object in the second frame;
receive a user adjustment of the prediction;
determine a difference between the prediction and the user adjustment; and
adjust a subsequent prediction of a location of the object in a subsequent frame in response to the difference.
2. The system of claim 1 wherein the first annotation is input by a user.
3. The system of claim 1 wherein the processor is further configured to determine the first annotation.
4. The system of claim 1 wherein the object's location in the first frame is different from the object's location in the second frame.
5. The system of claim 1 wherein the processor is further configured to predict the location of the object in the second frame.
6. The system of claim 1 wherein the processor is further configured to predict the location of the object in the second frame including by applying a tracker.
7. The system of claim 1 wherein the processor is further configured to:
predict the location of the object in the second frame including by applying a tracker; and
provide a confidence score of the prediction.
8. The system of claim 1 wherein the processor is further configured to predict the location of the object in the second frame including by applying a plurality of trackers.
9. The system of claim 1 wherein the processor is further configured to predict the location of the object in the second frame including by applying a plurality of trackers, wherein the prediction is based at least in part by selecting an output from one of the plurality of trackers.
10. The system of claim 1 wherein the processor is further configured to predict the location of the object in the second frame including by comparing a cropped region in the first frame to a cropped region in the second frame.
11. The system of claim 1 wherein the processor is further configured to receive an indication that the object has moved off screen or out of the second frame.
12. The system of claim 1 wherein the processor is further configured to:
detect that the object has disappeared from view in the second frame;
provide a flag or indicator in response to detecting that the object has disappeared from view in the second frame; and
prompt a user to confirm whether the object is visible in the second frame.
13. The system of claim 1 , wherein:
to predict the location of the object in the second frame includes to apply a tracker; and
the processor is further configured to adapt the tracker based at least in part on the user adjustment.
14. The system of claim 1 wherein:
to predict the location of the object in the second frame includes to apply a tracker; and
the processor is further configured to adapt the tracker based at least in part on the user adjustment, including by adjusting a model online as subsequent predictions are made.
15. The system of claim 1 wherein:
to predict the location of the object in the second frame includes to apply a tracker;
the user adjustment comprises a plurality of user adjustments from multiple users;
the difference comprises a plurality of differences between the prediction and each one of the plurality of user adjustments from multiple users; and
the processor is further configured to:
adjust a subsequent prediction of a location of the object in a subsequent frame in response to the difference; and
adapt the tracker based at least in part on the user adjustment.
16. The system of claim 1 wherein:
to predict the location of the object in the second frame includes to apply a tracker;
the user adjustment comprises a plurality of individual user adjustments from multiple users; and
the processor is further configured to:
determine an aggregate adjustment based on the user adjustment;
determine a difference between the prediction and the aggregate adjustment;
adjust a subsequent prediction of a location of the object in a subsequent frame in response to the difference; and
adapt the tracker based on at least one of the user adjustment or the aggregate adjustment.
17. The system of claim 1 wherein the processor is further configured to:
predict the location of the object in the second frame including by applying a tracker, wherein the tracker comprises a model configured to adapt based at least in part on transfer learning.
18. A method for object-level tracking comprising:
presenting a first frame in a series of pre-recorded frames;
receiving a first annotation of a first bounding region of an object in the first frame;
assigning an object identifier associated with the first bounding region;
obtaining a second bounding region of the object in a second frame in the series of pre-recorded frames based at least in part on a prediction of a location of the object in the second frame, wherein the second bounding region is associated with the object identifier;
outputting the second bounding region;
maintaining a prediction of the location of the object in subsequent frames using the object identifier;
predicting the location of the object in the second frame;
receiving a user adjustment of the prediction;
determining a difference between the prediction and the user adjustment; and
adjusting a prediction of a location of the object in a subsequent frame in response to the difference.
19. A computer program product for annotation, the computer program product being embodied in a tangible computer readable storage medium and comprising computer instructions for:
presenting a first frame in a series of pre-recorded frames;
receiving a first annotation of a first bounding region of an object in the first frame;
assigning an object identifier associated with the first bounding region;
obtaining a second bounding region of the object in a second frame in the series of pre-recorded frames based at least in part on a prediction of a location of the object in the second frame, wherein the second bounding region is associated with the object identifier;
outputting the second bounding region;
maintaining a prediction of the location of the object in subsequent frames using the object identifier;
predicting the location of the object in the second frame;
receiving a user adjustment of the prediction;
determining a difference between the prediction and the user adjustment; and
adjusting a prediction of a location of the object in a subsequent frame in response to the difference.