METHODS AND APPARATUS FOR ELECTRONIC IMAGE STABILIZATION BASED ON A LENS POLYNOMIAL
Methods and apparatus for stabilizing image data based on a lens polynomial. Non-rectilinear footage can be captured and rectified in-camera; the rectified images may be stabilized to provide rectified stable video. In one exemplary embodiment, the footage is rectified and stabilized based on a lens polynomial and the camera's own movement. In some variants, the rectified stable video may be stored along with its margin track. In-camera rectified stable video provides several benefits over traditional techniques (e.g., the ability to share rectilinear content from the camera without additional post-processing, as well as reduced file sizes of the shared videos). Lens-aware post-processing can reuse portions of the in-camera rectified stable videos while providing additional benefits (e.g., the ability to re-frame the video in post-production).
1 . A camera apparatus, comprising:
a lens that focuses light onto a camera sensor;
an accelerometer and a gyroscope;
a processor subsystem; and
a non-transitory computer-readable medium that stores instructions which when executed by the processor subsystem, causes the camera apparatus to:
obtain telemetry data based on the accelerometer and the gyroscope;
obtain image data based on the camera sensor;
rectify the image data based on a characteristic of the lens to generate rectified image data;
stabilize the rectified image data based on the telemetry data to generate stabilized rectilinear image data; and
encode the stabilized rectilinear image data into a video.
2 . The camera apparatus of claim 1 , where the lens is characterized by a field-of-view that is greater than 120°; and
where the image data is non-rectilinear.
3 . The camera apparatus of claim 1 , further comprising:
a second lens that focuses second light onto a second camera sensor;
where the lens and the second lens are characterized by a field-of-view that is greater than 180°; and
where the image data comprises spherical content.
4 . The camera apparatus of claim 1 , where the characteristic of the lens is a polynomial that describes a distortion of the light according to a sensor radius and a corresponding angle relative to the camera sensor.
5 . The camera apparatus of claim 1 , where the processor subsystem further comprises a codec that is configured to approximate image motion based on straight-line motion vectors and address image data based on row and column addressing.
6 . The camera apparatus of claim 1 , where the instructions are further configured to cause the camera apparatus to store cropped portions of the rectified image data into a stabilization margin data structure.
7 . The camera apparatus of claim 6 , where the video and the stabilization margin data structure are encoded within separate tracks of a single data structure.
8 . A method for electronic image stabilization based on a lens polynomial of a lens attachment, comprising:
obtaining previously stabilized image data and orientation data, the previously stabilized image data stabilized according to a different lens polynomial than the lens polynomial of the lens attachment;
determining a designated view of the previously stabilized image data according to the lens polynomial of the lens attachment;
rectifying the previously stabilized image data based on the lens polynomial and the designated view; and
encoding a rectified stable video based on the previously stabilized image data.
9 . The method of claim 8 , where the previously stabilized image data is obtained by capturing multiple exposures at different exposure settings and compositing the multiple exposures together.
10 . The method of claim 8 , where rectifying the previously stabilized image data comprises stretching or shrinking based on a pixel mapping of the previously stabilized image data to a rectilinear image.
11 . The method of claim 10 , where the previously stabilized image data is obtained by capturing multiple exposures at spatially different orientations and stitching the multiple exposures together.
12 . The method of claim 11 , where the pixel mapping is extrapolated beyond a field-of-view of at least one exposure of the multiple exposures.
13 .- 20 . (canceled)
21 . A post-processing apparatus, comprising:
a processor subsystem; and
a non-transitory computer-readable medium that stores instructions which when executed by the processor subsystem, causes the post-processing apparatus to:
obtain a video comprising previously stabilized rectilinear image data according to an originally smoothed camera motion;
obtain a margin track that corresponds to the previously stabilized rectilinear image data;
obtain orientation data and a lens characteristic;
determine a new smoothed camera motion based on the previously stabilized rectilinear image data and at least one of the margin track, the orientation data, and the lens characteristic;
correct a first portion of the previously stabilized rectilinear image data and a second portion of the margin track based on the lens characteristic;
stabilize the first portion of the previously stabilized rectilinear image data and the second portion of the margin track based on the orientation data; and
stitch the first portion and the second portion into a video frame.
22 . The post-processing apparatus of claim 21 , further comprising instructions that cause the post-processing apparatus to obtain an in-camera stabilization error flag that corresponds to the first portion of the previously stabilized rectilinear image data or the second portion of the margin track.
23 . The post-processing apparatus of claim 22 , where the orientation data comprises a partial record of device movement, and the instructions further cause the post-processing apparatus to determine device orientation or image orientation with image analysis.
24 . The post-processing apparatus of claim 21 , further comprising instructions that cause the post-processing apparatus to re-rectify at least one of the first portion of the previously stabilized rectilinear image data or the second portion of the margin track for a different lens angle based on the lens characteristic.
25 . The post-processing apparatus of claim 24 , further comprising instructions that cause the post-processing apparatus to obtain a user identified subject-of-interest, and where re-rectification is based on the subject-of-interest.
26 . The post-processing apparatus of claim 21 , further comprising a codec that is configured to approximate image motion based on straight-line motion vectors and address image data based on row and column addressing; and where the instructions are further configured to cause the post-processing apparatus to encode the video frame into a re-framed rectilinear stabilized video.
27 . The post-processing apparatus of claim 26 , further comprising a display, and where the instructions, when executed by the processor subsystem, further causes the post-processing apparatus to display the re-framed rectilinear stabilized video.
28 . The post-processing apparatus of claim 21 , further comprising a network interface, and where the instructions, when executed by the processor subsystem, further causes the post-processing apparatus to obtain the previously stabilized rectilinear image data from another device that is streaming the video.