Edge-based electronic image stabilization
View Patent ↗An image stabilization system applies a “pinned-edge” or “soft pinned edge” image stabilization technique to digital video to compensate for unwanted camera motion in a captured video. In these stabilization techniques, a warping function is applied to an image frame to achieve a non-uniform shifting of depicted points in the image frame such that a reference point is stabilized with respect to a reference frame. In pinned-edge image stabilization, the final stabilized output video has the same dimensions as the pre-stabilized input video captured by the image sensor. In soft pinned-edge image stabilization, the pre-stabilized input video has slightly larger dimensions than the stabilized output video but these larger dimensions are still reduced compared to traditional electronic image stabilization.
1. A method for stabilizing a digital video, the method comprising:
receiving a reference video frame depicting a scene, the reference video frame depicting a reference point represented by one or more pixels at a reference pixel location;
receiving a second frame depicting at least a portion of the scene, wherein movement of the reference point between the first and second frame results in the reference point being depicted at a different pixel location in the second frame;
non-uniformly shifting non-edge points depicted in the second frame by an amount based on distances of the non-edge points from an edge of the second frame and applying zero shift to edge points depicted in the second frame; and
generating a stabilized video comprising a sequence of frames including the reference frame and the stabilized second frame, wherein depicted motion of the reference point between the first frame and the stabilized second frame is reduced relative to depicted motion of the reference point between the first frame and the second frame.
2. The method of claim 1 , wherein non-uniformly shifting non-edge points comprises applying a shift as a linear function of the distance.
3. The method of claim 1 , wherein non-uniformly shifting non-edge points comprises applying a shift as a non-linear function of the distance.
4. A method for stabilizing a digital video, the method comprising:
receiving the digital video comprising a sequence of frames depicting a scene, the sequence of frames including a reference frame and a second frame each depicting a common plurality of depicted points in the scene;
determining a reference point from the common plurality of points, wherein the reference point corresponds to different pixel locations in the reference frame and the second frame; and
applying a warping function to the second frame of the digital video to generate a warped second frame, wherein depicted points of the depicted scene in the warped second frame are shifted non-uniformly relative to corresponding depicted points in the second frame, and wherein applying the warping function decreases depicted movement of the reference point between the reference frame and the second frame, wherein applying the warping function to generate the warped second frame comprises:
applying a first shift to a non-edge point, the first shift having a component in a first direction; and
applying a second shift to an edge point on an edge of the second frame, wherein the non-edge point is shifted by a greater amount in the first direction than the edge point.
5. The method of claim 4 , wherein an amount of shift applied to depicted points along a line between an edge of the second frame and the reference point varies linearly with distance from the edge of the second frame.
6. The method of claim 4 , wherein an amount of shift applied to depicted points along a line between an edge of the second frame and the reference point varies non-linearly with distance from the edge of the second frame.
7. The method of claim 4 , wherein the first direction comprises a direction perpendicular to the edge of the second frame.
8. A method for stabilizing a digital video, the method comprising:
receiving the digital video comprising a sequence of frames depicting a scene, the sequence of frames including a reference frame and a second frame each depicting a common plurality of depicted points in the scene;
determining a reference point from the common plurality of points, wherein the reference point corresponds to different pixel locations in the reference frame and the second frame; and
applying a warping function to the second frame of the digital video to generate a warped second frame, wherein depicted points of the depicted scene in the warped second frame are shifted non-uniformly relative to corresponding depicted points in the second frame, and wherein applying the warping function decreases depicted movement of the reference point between the reference frame and the second frame, wherein applying the warping function comprises:
applying a first shift to a non-edge point, the first shift having a component in a first direction; and
applying no shift in the first direction to an edge point on an edge of the second frame.
9. The method of claim 8 , wherein the first direction comprises a direction perpendicular to the edge of the second frame.
10. The method of claim 8 , wherein an amount of shift applied to depicted points along a line between an edge of the second frame and the reference point varies linearly with distance from the edge of the second frame.
11. The method of claim 8 , wherein an amount of shift applied to depicted points along a line between an edge of the second frame and the reference point varies non-linearly with distance from the edge of the second frame.
12. A non-transitory computer-readable storage medium storing computer-executable instructions for stabilizing a digital video, the computer-executable instructions when executed by one more processors cause the one or more processors to performs steps including:
receiving the digital video comprising a sequence of frames depicting a scene, the sequence of frames including a reference frame and a second frame each depicting a common plurality of depicted points in the scene;
determining a reference point from the common plurality of points, wherein the reference point corresponds to different pixel locations in the reference frame and the second frame; and
applying a warping function to the second frame of the digital video to generate a warped second frame, wherein depicted points of the depicted scene in the warped second frame are shifted non-uniformly relative to corresponding depicted points in the second frame, and wherein applying the warping function decreases depicted movement of the reference point between the reference frame and the second frame, wherein applying the warping function to generate the warped second frame comprises:
applying a first shift to a non-edge point, the first shift having a component in a first direction; and
applying a second shift to an edge point on an edge of the second frame, wherein the non-edge point is shifted by a greater amount in the first direction than the edge point.
13. The computer-readable storage medium of claim 12 , wherein an amount of shift applied to depicted points along a line between an edge of the second frame and the reference point varies linearly with distance from the edge of the second frame.
14. The computer-readable storage medium of claim 12 , wherein an amount of shift applied to depicted points along a line between an edge of the second frame and the reference point varies non-linearly with distance from the edge of the second frame.
15. The computer-readable storage medium of claim 12 , wherein the first direction comprises a direction perpendicular to the edge of the second frame.
16. A non-transitory computer-readable storage medium storing computer-executable instructions for stabilizing a digital video, the computer-executable instructions when executed by one more processors cause the one or more processors to performs steps including:
receiving the digital video comprising a sequence of frames depicting a scene, the sequence of frames including a reference frame and a second frame each depicting a common plurality of depicted points in the scene;
determining a reference point from the common plurality of points, wherein the reference point corresponds to different pixel locations in the reference frame and the second frame; and
applying a warping function to the second frame of the digital video to generate a warped second frame, wherein depicted points of the depicted scene in the warped second frame are shifted non-uniformly relative to corresponding depicted points in the second frame, and wherein applying the warping function decreases depicted movement of the reference point between the reference frame and the second frame, wherein applying the warping function comprises:
applying a first shift to a non-edge point, the first shift having a component in a first direction; and
applying no shift in the first direction to an edge point on an edge of the second frame.
17. The computer-readable storage medium of claim 16 , wherein the first direction comprises a direction perpendicular to the edge of the second frame.
18. The computer-readable storage medium of claim 16 , wherein an amount of shift applied to depicted points along a line between an edge of the second frame and the reference point varies linearly with distance from the edge of the second frame.
19. The computer-readable storage medium of claim 16 , wherein an amount of shift applied to depicted points along a line between an edge of the second frame and the reference point varies non-linearly with distance from the edge of the second frame.
20. An image stabilization system for capturing and stabilizing digital video, the image stabilization system comprising:
a camera system for capturing digital video;
one or more processors; and
a computer-readable storage medium storing computer-executable instructions for stabilizing the digital video, the computer-executable instructions when executed by the one more processors cause the one or more processors to performs steps including:
receiving the digital video comprising a sequence of frames depicting a scene, the sequence of frames including a reference frame and a second frame each depicting a common plurality of depicted points in the scene;
determining a reference point from the common plurality of points, wherein the reference point corresponds to different pixel locations in the reference frame and the second frame; and
applying a warping function to the second frame of the digital video to generate a warped second frame, wherein depicted points of the depicted scene in the warped second frame are shifted non-uniformly relative to corresponding depicted points in the second frame, and wherein applying the warping function decreases depicted movement of the reference point between the reference frame and the second frame, wherein applying the warping function to generate the warped second frame comprises:
applying a first shift to a non-edge point, the first shift having a component in a first direction; and
applying a second shift to an edge point on an edge of the second frame, wherein the non-edge point is shifted by a greater amount in the first direction than the edge point.
21. The image stabilization system of claim 20 , wherein an amount of shift applied to depicted points along a line between an edge of the second frame and the reference point varies linearly with distance from the edge of the second frame.
22. The image stabilization system of claim 20 , wherein an amount of shift applied to depicted points along a line between an edge of the second frame and the reference point varies non-linearly with distance from the edge of the second frame.
23. The image stabilization system of claim 20 , wherein the first direction comprises a direction perpendicular to the edge of the second frame.
24. An image stabilization system for capturing and stabilizing digital video, the image stabilization system comprising:
a camera system for capturing digital video;
one or more processors; and
a computer-readable storage medium storing computer-executable instructions for stabilizing the digital video, the computer-executable instructions when executed by the one more processors cause the one or more processors to performs steps including:
receiving the digital video comprising a sequence of frames depicting a scene, the sequence of frames including a reference frame and a second frame each depicting a common plurality of depicted points in the scene;
determining a reference point from the common plurality of points, wherein the reference point corresponds to different pixel locations in the reference frame and the second frame; and
applying a warping function to the second frame of the digital video to generate a warped second frame, wherein depicted points of the depicted scene in the warped second frame are shifted non-uniformly relative to corresponding depicted points in the second frame, and wherein applying the warping function decreases depicted movement of the reference point between the reference frame and the second frame, wherein applying the warping function comprises:
applying a first shift to a non-edge point, the first shift having a component in a first direction; and
applying no shift in the first direction to an edge point on an edge of the second frame.
25. The image stabilization system of claim 24 , wherein the first direction comprises a direction perpendicular to the edge of the second frame.
26. The image stabilization system of claim 24 , wherein an amount of shift applied to depicted points along a line between an edge of the second frame and the reference point varies linearly with distance from the edge of the second frame.
27. The image stabilization system of claim 24 , wherein an amount of shift applied to depicted points along a line between an edge of the second frame and the reference point varies non-linearly with distance from the edge of the second frame.