Systems and methods for digital video stabilization via constraint-based rotation smoothing
View Patent ↗Systems and methods for digital video stabilization via constraint-based rotation smoothing are provided. Digital video data including a set of image frames having associated time stamps and a set of camera orientation data having associated time stamps may be provided. A smoothed set of camera orientation data may be generated by minimizing a rate of rotation between successive image frames while minimizing an amount of empty regions in a resulting set of smoothed image frames reoriented based on the smoothed set of camera orientation data.
1. A computer implemented method comprising:
providing, by a computer system, digital video data comprising a set of image frames having associated time stamps and a set of camera orientation data having associated time stamps, wherein the set of camera orientation data is based on an orientation of a camera during a capture of the digital video data by the camera; and
generating, by the computer system, a smoothed set of camera orientation data by minimizing a rate of rotation between successive image frames while minimizing an amount of empty regions in a resulting set of smoothed image frames reoriented based on the smoothed set of camera orientation data.
2. The method of claim 1 , wherein the amount of empty regions in the resulting set of smoothed image frames is minimized to zero.
3. The method of claim 1 , wherein the amount of empty regions in the resulting set of smoothed image frames is minimized below a threshold value.
4. The method of claim 1 , further comprising warping the set of image frames based on the associated time stamps for the set of image frames and the smoothed set of camera orientation data to form a set of corrected image frames.
5. The method of claim 4 , wherein the warping the set of image frames based on the associated time stamps for the set of image frames and the smoothed set of camera orientation data to form a set of corrected image frames comprises:
dividing an individual image frame into a plurality of subsections, each subsection having an associated time stamp and camera orientation; and
realigning each subsection based on the associated time stamp and camera orientation to form an individual corrected image frame.
6. The method of claim 4 , further comprising displaying the set of corrected image frames as a video.
7. The method of claim 6 , wherein the amount of empty regions in the resulting set of smoothed image frames is minimized below a threshold value.
8. The method of claim 7 , further comprising inpainting the amount of empty regions below the threshold value.
9. The method of claim 1 , wherein the set of camera orientation data having associated time stamps is provided from a gyroscope of a handheld device.
10. The method of claim 1 , wherein the set of image frames is provided from a camera of a handheld device.
11. The method of claim 1 , wherein the set of camera orientation data having associated time stamps is provided from a gyroscope of a handheld device comprising a mobile phone and a digital camera, and wherein the set of image frames is provided from the digital camera.
12. The method of claim 1 , wherein the generating the smoothed set of camera orientation data comprises an iterative optimization based on gradient descent.
13. The method of claim 1 , wherein the generating the smoothed set of camera orientation data comprises filtering based on a Gaussian filter.
14. The method of claim 1 , wherein the generating the smoothed set of camera orientation data comprises filtering based on a temporal derivative.
15. The method of claim 1 , wherein the set of camera orientation data includes rotations without any translations.
16. The method of claim 1 , wherein the set of camera orientation data includes vectors having both rotations and translations.
17. The method of claim 1 , wherein the generating the smoothed set of camera orientation data is performed by a social networking system.
18. The method of claim 17 , wherein the set of image frames having associated time stamps and the set of camera orientation data having associated time stamps are uploaded to the social networking system by a user of the social networking system.
19. A system comprising:
at least one processor, and
a memory storing instructions configured to instruct the at least one processor to perform:
providing digital video data comprising a set of image frames having associated time stamps and a set of camera orientation data having associated time stamps; and
generating a smoothed set of camera orientation data by minimizing a rate of rotation between successive image frames while minimizing an amount of empty regions in a resulting set of smoothed image frames reoriented based on the smoothed set of camera orientation data, wherein minimizing the rate of rotation between successive image frames is based on at least a portion of the set of camera orientation data.
20. A non-transitory computer storage medium storing computer-executable instructions that, when executed, cause a computer system to perform computer-implemented method comprising:
providing digital video data comprising a set of image frames having associated time stamps and a set of camera orientation data having associated time stamps, wherein the set of camera orientation data is based on an orientation of a camera during a capture of the digital video data by the camera; and
generating a smoothed set of camera orientation data by minimizing a rate of rotation between successive image frames while minimizing an amount of empty regions in a resulting set of smoothed image frames reoriented based on the smoothed set of camera orientation data, wherein minimizing the rate of rotation between successive image frames is based on at least a portion of the set of camera orientation data.