Image stabilization techniques for video surveillance systems
View Patent ↗A behavioral recognition system may include both a computer vision engine and a machine learning engine configured to observe and learn patterns of behavior in video data. Certain embodiments may provide image stabilization of a video stream obtained from a camera. An image stabilization module in the behavioral recognition system obtains a reference image from the video stream. The image stabilization module identifies alignment regions within the reference image based on the regions of the image that are dense with features. Upon determining that the tracked features of a current image is out of alignment with the reference image, the image stabilization module uses the most feature dense alignment region to estimate an affine transformation matrix to apply to the entire current image to warp the image into proper alignment.
1. A method for stabilizing camera images captured by a video recording device and evaluated by a video analytics application, the method comprising:
identifying candidate regions within a reference frame to use in aligning a stream of frames captured by a video camera, wherein the candidate regions are sorted in descending order by a density of tracked features within each candidate region;
designating a candidate region having a highest density of tracked features among the candidate regions as an alignment region for each captured frame; and
upon determining that the stream is unstable based on the tracked features in a current frame being misaligned relative to the corresponding features of the reference frame:
determining an affine transformation matrix for the current frame based on the alignment region,
warping the current frame using the affine transformation matrix such that the features in the alignment region of the current frame match the corresponding features of the reference frame,
identifying a registration error between the warped current frame and the reference frame, and
upon determining that the registration error is within a tolerance threshold, designating the warped current frame as a stabilized frame.
2. The method of claim 1 , wherein the affine transformation matrix is determined over a specified number of iterations based on a vertical translation, a horizontal translation, and a rotation of the features within the alignment region of the reference frame and the current frame using a gradient descent analysis.
3. The method of claim 2 , wherein the number of iterations is increased and thresholds of the vertical translation and of the horizontal translation are decreased based on a previous affine transformation matrix being determined under a specified length of time.
4. The method of claim 1 , wherein the reference frame is an initial raw frame captured by the camera on start-up.
5. The method of claim 1 , further comprising, updating the reference frame to the stabilized frame.
6. The method of claim 1 , further comprising, assigning a statistical weight to the alignment region based on registration error.
7. The method of claim 6 , further comprising, upon determining that the registration error is not within the tolerance threshold:
reducing a statistical weight of the alignment region; and
designating a next statistically significant candidate region as the alignment region.
8. A non-transitory computer-readable storage medium storing instructions, which, when executed on a processor, performs an operation for stabilizing camera images captured by a video recording device and evaluated by a video analytics application, the operation comprising:
identifying candidate regions within a reference frame to use in aligning a stream of frames captured by a video camera, wherein the candidate regions are sorted in descending order by a density of tracked features within each candidate region;
designating a candidate region having a highest density of tracked features among the candidate regions as an alignment region for each captured frame; and
upon determining that the stream is unstable based on the tracked features in a current frame being misaligned relative to the corresponding features of the reference frame:
determining an affine transformation matrix for the current frame based on the alignment region,
warping the current frame using the affine transformation matrix such that the features in the alignment region of the current frame match the corresponding features of the reference frame,
identifying a registration error between the warped current frame and the reference frame, and
upon determining that the registration error is within a tolerance threshold, designating the warped current frame as a stabilized frame.
9. The non-transitory computer-readable storage medium of claim 8 , wherein the affine transformation matrix is determined over a specified number of iterations based on a vertical translation, a horizontal translation, and a rotation of the features within the alignment region of the reference frame and the current frame using a gradient descent analysis.
10. The non-transitory computer-readable storage medium of claim 9 , wherein the number of iterations is increased and thresholds of the vertical translation and of the horizontal translation are decreased based on a previous affine transformation matrix being determined under a specified length of time.
11. The non-transitory computer-readable storage medium of claim 8 , wherein the reference frame is an initial raw frame captured by the camera on start-up.
12. The non-transitory computer-readable storage medium of claim 8 , further comprising, updating the reference frame to the stabilized frame.
13. The non-transitory computer-readable storage medium of claim 8 , further comprising, assigning a statistical weight to the alignment region based on registration error.
14. The non-transitory computer-readable storage medium of claim 13 , further comprising, upon determining that the registration error is not within the tolerance threshold:
reducing a statistical weight of the alignment region; and
designating a next statistically significant candidate region as the alignment region.
15. A system comprising:
a processor; and
a memory hosting an application, which, when executed on the processor, performs an operation for stabilizing camera images captured by a video recording device and evaluated by a video analytics application, the operation comprising:
identifying candidate regions within a reference frame to use in aligning a stream of frames captured by a video camera, wherein the candidate regions are sorted in descending order by a density of tracked features within each candidate region;
designating a candidate region having a highest density of tracked features among the candidate regions as an alignment region for each captured frame; and
upon determining that the stream is unstable based on the tracked features in a current frame being misaligned relative to the corresponding features of the reference frame:
determining an affine transformation matrix for the current frame based on the alignment region,
warping the current frame using the affine transformation matrix such that the features in the alignment region of the current frame match the corresponding features of the reference frame,
identifying a registration error between the warped current frame and the reference frame, and
upon determining that the registration error is within a tolerance threshold, designating the warped current frame as a stabilized frame.
16. The system of claim 15 , wherein the affine transformation matrix is determined over a specified number of iterations based on a vertical translation, a horizontal translation, and a rotation of the features within the alignment region of the reference frame and the current frame using a gradient descent analysis.
17. The system of claim 16 , wherein the number of iterations is increased and thresholds of the vertical translation and of the horizontal translation are decreased based on a previous affine transformation matrix being determined under a specified length of time.
18. The system of claim 15 , wherein the reference frame is an initial raw frame captured by the camera on start-up.
19. The system of claim 15 , further comprising, updating the reference frame to the stabilized frame.
20. The system of claim 15 , further comprising, assigning a statistical weight to the alignment region based on registration error.
21. The system of claim 20 , further comprising, upon determining that the registration error is not within the tolerance threshold:
reducing a statistical weight of the alignment region; and
designating a next statistically significant candidate region as the alignment region.
22. A method for stabilizing camera images captured by a video recording device, the method comprising:
identifying candidate regions within a reference frame to use in aligning a frame captured by a video camera, wherein the candidate regions are sorted in descending order by a density of tracked features within each candidate region;
designating a candidate region having a highest density of tracked features among the candidate regions as an alignment region for each captured frame; and
upon determining that the stream is unstable based on the tracked features in a current frame being misaligned relative to the corresponding features of the reference frame, aligning the current frame.
23. The method of claim 22 , wherein the current frame is aligned using perturbation theory methods.