Multiple object tracking in a video stream
Multiple Object Tracking (MOT) procedures are used to analyze a video stream to identify and track objects and events of interest across frames in the video stream. According to various embodiments, two or more different models may be separately applied to track an object across multiple video frames. A model may be dynamically evaluated for a frame or group of frames by determining a performance metric for the model, for instance on the level of a frame or group of frames. Then, two or more models may be fused together using a weighting scheme based at least in part on performance metrics for the different models. The fused model may be used to track objects across the frames.
1 . A method comprising:
determining a bounding box around an object within a designated frame of a plurality of frames within a video stream;
determining a first object track and a second object track for the object from a plurality of object tracks by applying an appearance-based tracking model and a motion-based object tracking model to the video stream, each of the first and second object tracks identifying a correspondence between bounding boxes for the object across different ones of the plurality of frames;
determining a plurality of raw performance metrics based at least in part on one or more visual features within the designated frame, the plurality of raw performance metrics including a first model fitness coefficient corresponding to the first object tracking model and measuring appearance similarity for the designated frame, the plurality of raw performance metrics including a second model fitness coefficient corresponding to the second object tracking model and measuring crowdedness of the designated frame;
selecting an object tracking model for the object based at least in part on the plurality of raw performance metrics;
selecting a designated object track of the first and second object tracks based on the object tracking model; and
storing the designated object track on a storage device.
2 . The method recited in claim 1 , the method further comprising:
determining whether the two or more object tracks are different, wherein the plurality of raw performance metrics are determined when it is determined that the two or more object tracks are different.
3 . The method recited in claim 1 , the method further comprising:
determining a plurality of normalized performance metrics based on the raw performance metrics, the normalized performance metrics being located on a designated performance metric scale.
4 . The method recited in claim 1 , the method further comprising:
determining a plurality of raw cost matrices for the designated frame, each of the raw cost matrices corresponding to a respective one of the object tracking models and identifying a respective distance between one or more bounding boxes included in a current frame and one or more bounding boxes of existing tracks.
5 . The method recited in claim 4 , wherein a designated one of the raw cost matrices includes a plurality of values, each of the values corresponding with a respective one of the plurality of object tracks and a respective bounding box within the designated frame.
6 . The method recited in claim 4 , the method further comprising:
determining a plurality of weighted cost matrices for the designated frame, each of the weighted cost matrices corresponding with a respective raw cost matrix, each of the weighted cost matrices being weighted based on a respective weighting factor, the weighting factors determined based on the plurality of raw performance metrics.
7 . The method recited in claim 1 , the method further comprising:
determining an object identifier for the bounding box by applying an object recognition algorithm to all or a portion of the designated frame.
8 . The method recited in claim 1 , wherein a designated one of the performance metrics is determined for an individual frame.
9 . The method recited in claim 1 , wherein a designated one of the performance metrics is determined for a group of temporally proximate frames.
10 . A system comprising a processor and a storage device, the processor configured to perform a method comprising:
determining a bounding box around an object within a designated frame of a plurality of frames within a video stream;
determining a first object track and a second object track for the object from a plurality of object tracks by applying an appearance-based tracking model and a motion-based object tracking model to the video stream, each of the first and second object tracks identifying a correspondence between bounding boxes for the object across different ones of the plurality of frames;
determining a plurality of raw performance metrics based at least in part on one or more visual features within the designated frame, the plurality of raw performance metrics including a first model fitness coefficient corresponding to the first object tracking model and measuring appearance similarity for the designated frame, the plurality of raw performance metrics including a second model fitness coefficient corresponding to the second object tracking model and measuring crowdedness of the designated frame;
selecting an object tracking model for the object based at least in part on the plurality of raw performance metrics;
selecting a designated object track of the first and second object tracks based on the object tracking model; and
storing the designated object track on a storage device.
11 . The system recited in claim 10 , the method further comprising:
determining whether the two or more object tracks are different, wherein the plurality of raw performance metrics are determined when it is determined that the two or more object tracks are different.
12 . The system recited in claim 10 , the method further comprising:
determining a plurality of normalized performance metrics based on the raw performance metrics, the normalized performance metrics being located on a designated performance metric scale.
13 . The system recited in claim 10 , the method further comprising:
determining a plurality of raw cost matrices for the designated frame, each of the raw cost matrices corresponding to a respective one of the object tracking models and identifying a respective distance between one or more bounding boxes included in a current frame and one or more bounding boxes of existing tracks.
14 . The system recited in claim 13 , wherein a designated one of the raw cost matrices includes a plurality of values, each of the values corresponding with a respective one of the plurality of object tracks and a respective bounding box within the designated frame.
15 . The system recited in claim 13 , the method further comprising:
determining a plurality of weighted cost matrices for the designated frame, each of the weighted cost matrices corresponding with a respective raw cost matrix, each of the weighted cost matrices being weighted based on a respective weighting factor, the weighting factors determined based on the plurality of raw performance metrics.
16 . The system recited in claim 10 , the method further comprising:
determining an object identifier for the bounding box by applying an object recognition algorithm to all or a portion of the designated frame.
17 . One or more non-transitory computer readable media having instructions stored thereon for performing a method, the method comprising:
determining a bounding box around an object within a designated frame of a plurality of frames within a video stream;
determining a first object track and a second object track for the object from a plurality of object tracks by applying an appearance-based tracking model and a motion-based object tracking model to the video stream, each of the first and second object tracks identifying a correspondence between bounding boxes for the object across different ones of the plurality of frames;
determining a plurality of raw performance metrics based at least in part on one or more visual features within the designated frame, the plurality of raw performance metrics including a first model fitness coefficient corresponding to the first object tracking model and measuring appearance similarity for the designated frame, the plurality of raw performance metrics including a second model fitness coefficient corresponding to the second object tracking model and measuring crowdedness of the designated frame;
selecting an object tracking model for the object based at least in part on the plurality of raw performance metrics;
selecting a designated object track of the first and second object tracks based on the object tracking model; and
storing the designated object track on a storage device.