Methods and apparatus for electronic image stabilization based on a lens polynomial
Methods and apparatus for stabilizing image data based on a lens polynomial. Non-rectilinear footage can be captured and rectified in-camera; the rectified images may be stabilized to provide rectified stable video. In one exemplary embodiment, the footage is rectified and stabilized based on a lens polynomial and the camera's own movement. In some variants, the rectified stable video may be stored along with its margin track. In-camera rectified stable video provides several benefits over traditional techniques (e.g., the ability to share rectilinear content from the camera without additional post-processing, as well as reduced file sizes of the shared videos). Lens-aware post-processing can reuse portions of the in-camera rectified stable videos while providing additional benefits (e.g., the ability to re-frame the video in post-production).
1. A post-processing apparatus, comprising:
a processor subsystem; and
a non-transitory computer-readable medium that stores instructions which when executed by the processor subsystem, causes the post-processing apparatus to:
obtain a video comprising previously stabilized rectilinear image data according to an originally captured view;
obtain a margin track that corresponds to the previously stabilized rectilinear image data;
obtain telemetry data and a lens characteristic;
determine a new view based on the previously stabilized rectilinear image data and at least one of the margin track, the telemetry data, and the lens characteristic;
correct a first portion of the previously stabilized rectilinear image data and a second portion of the margin track based on the lens characteristic;
stabilize the first portion of the previously stabilized rectilinear image data and the second portion of the margin track based on the telemetry data; and
stitch the first portion and the second portion into a video frame.
2. The post-processing apparatus of claim 1 , further comprising a codec that is configured to approximate image motion based on straight-line motion vectors and address image data based on row and column addressing; and
where the instructions are further configured to cause the post-processing apparatus to encode the video frame into a re-framed rectilinear stabilized video.
3. The post-processing apparatus of claim 2 , further comprising a display, and where the instructions, when executed by the processor subsystem, further causes the post-processing apparatus to display the re-framed rectilinear stabilized video.
4. The post-processing apparatus of claim 2 , further comprising a network interface, and where the instructions, when executed by the processor subsystem, further causes the post-processing apparatus to transfer the re-framed rectilinear stabilized video to another device for display.
5. The post-processing apparatus of claim 1 , further comprising a network interface, and where the instructions, when executed by the processor subsystem, further causes the post-processing apparatus to obtain the previously stabilized rectilinear image data from another device that is streaming the video.
6. The post-processing apparatus of claim 1 , further comprising a user interface that enables a user to identify a subject-of-interest within a frame of the video; and
where the instructions, when executed by the processor subsystem, further causes the post-processing apparatus to determine the new view based on the subject-of-interest.
7. A method, comprising:
obtaining a video comprising previously stabilized rectilinear image data according to an originally captured view;
obtaining a margin track that corresponds to the previously stabilized rectilinear image data;
obtaining telemetry data and a lens characteristic;
determining a new view based on the previously stabilized rectilinear image data and at least one of the margin track, the telemetry data, and the lens characteristic;
correcting a first portion of the previously stabilized rectilinear image data and a second portion of the margin track based on the lens characteristic;
stabilizing the first portion of the previously stabilized rectilinear image data and the second portion of the margin track based on the telemetry data; and
stitching the first portion and the second portion into a video frame.
8. The method of claim 7 , further encoding the video frame into a re-framed rectilinear stabilized video.
9. The method of claim 8 , further comprising displaying the re-framed rectilinear stabilized video.
10. The method of claim 8 , further comprising transferring the re-framed rectilinear stabilized video to another device for display.
11. The method of claim 7 , further comprising obtaining the previously stabilized rectilinear image data from another device that is streaming the video.
12. The method of claim 7 , further comprising identifying a subject-of-interest within a frame of the video and determining the new view based on the subject-of-interest.
13. A method, comprising:
obtaining a video, a margin track, a telemetry track, and a lens characteristic that corresponds to the video;
determining a view based on the video and at least one of the margin track, the telemetry track, and the lens characteristic;
correcting a first frame of the video and a margin portion of the margin track based on the lens characteristic;
stabilizing the first frame of the video and the margin portion of the margin track based on the telemetry track; and
stitching the first frame and the margin portion into a new video frame.
14. The method of claim 13 , where the video comprises a previously stabilized video.
15. The method of claim 13 , where the video, the margin track, and the telemetry track, are obtained from a shared data structure.
16. The method of claim 13 , where at least one of the video, the margin track, and the telemetry track, are obtained from different data structures.
17. The method of claim 13 , where the video, the margin track, and the telemetry track, are obtained from a ready-to-share data stream.
18. The method of claim 13 , where the view is based on user input.
19. The method of claim 13 , where the view is based on the telemetry track.
20. The method of claim 19 , where the telemetry track comprises a flag that identifies an error in stabilization.