Image stitching with electronic rolling shutter correction
Systems and methods are disclosed for image signal processing. For example, methods may include receiving a first image from a first image sensor, receiving a second image from a second image sensor; obtaining corrected images based on the first image and the second image; obtaining stabilized images based on the corrected images; applying a parallax correction to the stabilized images to obtain a composite image; obtaining a transformed image from the composite image; and encoding an output image based on the transformed image.
1 . A system comprising:
a first image sensor configured to detect a first image;
a second image sensor configured to detect a second image;
an angular rate sensor; and
a processing apparatus that is configured to:
obtain angular rate measurements from the angular rate sensor for a device including the first image sensor and the second image sensor for times during capture of the first image and the second image;
determine an electronic rolling shutter correction based on the angular rate measurements and times when portions of the first image and the second image were captured using an electronic rolling shutter;
obtain corrected images based on the first image and the second image, wherein the processing apparatus is configured to determine a near point for an initial epipolar line, and a compensated near point, wherein the compensated near point is determined based on the near point and electronic rolling shutter data associated with the near point;
obtain stabilized images based on the corrected images;
apply a parallax correction to the stabilized images to obtain a composite image;
obtain a transformed image from the composite image; and
encode an output image based on the transformed image.
2 . The system of claim 1 , wherein the processing apparatus is further configured to:
determine compensated epipolar lines based on the electronic rolling shutter data; and
determine one of the compensated epipolar lines based on the compensated near point.
3 . The system of claim 2 , in which an electronic rolling shutter correction is determined at a lower resolution than the parallax correction.
4 . The system of claim 3 , wherein the initial epipolar line is determined geometrically based on a relative position of the first image sensor and the second image sensor.
5 . The system of claim 1 , wherein the parallax correction is based on a pre-compensated electronic rolling shutter distortion.
6 . The system of claim 1 , in which the processing apparatus is configured to:
determine rotations for respective portions of the first image and the second image based on the angular rate measurements corresponding to times when respective portions were captured;
interpolate the rotations to determine interpolated rotations for portions of the first image and the second image; and
determine an electronic rolling shutter correction based on the interpolated rotations.
7 . The system of claim 6 , wherein the parallax correction is based on a one-dimensional search.
8 . The system of claim 6 , comprising:
a first fish-eye lens attached to the first image sensor; and
a second fish-eye lens attached to the second image sensor.
9 . The system of claim 1 , wherein the parallax correction is based on a pre-compensated electronic rolling shutter distortion.
10 . The system of claim 1 , further comprising:
a storage device configured to store the transformed image; and
a touchscreen display configured to display the transformed images.
11 . A method comprising:
detecting a first image;
detecting a second image;
obtaining angular rate measurements from an angular rate sensor for a device for times during capture of the first image and the second image;
determining an electronic rolling shutter correction based on the angular rate measurements and times when portions of the first image and the second image were captured using an electronic rolling shutter;
determining a near point for an initial epipolar line, and a compensated near point, wherein the compensated near point is determined based on the near point and electronic rolling shutter data associated with the near point, to obtain corrected images based on the first image and the second image;
obtaining stabilized images based on the corrected images;
applying a parallax correction to the stabilized images to obtain a composite image;
obtaining a transformed image from the composite image; and
encoding an output image based on the transformed image.
12 . The method of claim 11 , further comprising:
determining compensated epipolar lines based on the electronic rolling shutter data.
13 . The method of claim 12 , wherein determining the compensated epipolar lines comprises:
determining a far point for the initial epipolar line; and
determining one of the compensated epipolar lines based on the far point and the compensated near point.
14 . The method of claim 13 , wherein the parallax correction is based on a pre-compensated electronic rolling shutter distortion.
15 . The method of claim 14 , wherein the parallax correction includes a stitching profile.
16 . The method of claim 11 , comprising:
obtaining angular rate measurements from an angular rate sensor for a device including a first image sensor and a second image sensor for times during capture of the first image and the second image; and
determining the electronic rolling shutter correction based on the angular rate measurements and times when portions of the first image and the second image were captured using an electronic rolling shutter.
17 . A non-transitory computer-readable medium comprising instructions, that when executed by a processor, cause the processor to perform operations comprising:
detecting a first image;
detecting a second image;
obtaining angular rate measurements from an angular rate sensor for a device for times during capture of the first image and the second image;
determining an electronic rolling shutter correction based on the angular rate measurements and times when portions of the first image and the second image were captured using an electronic rolling shutter;
determining a near point for an initial epipolar line, and a compensated near point, wherein the compensated near point is determined based on the near point and electronic rolling shutter data associated with the near point, to obtain corrected images based on the first image and the second image;
obtaining stabilized images based on the corrected images;
applying a parallax correction to the stabilized images to obtain a composite image;
obtaining a transformed image from the composite image; and
encoding an output image based on the transformed image.
18 . The non-transitory computer-readable medium of claim 17 , wherein the operations further comprise:
determining compensated epipolar lines based on the electronic rolling shutter data.
19 . The non-transitory computer-readable medium of claim 18 , wherein determining the compensated epipolar lines comprises:
determining a far point for the initial epipolar line; and
determining one of the compensated epipolar lines based on the far point and the compensated near point.
20 . The non-transitory computer-readable medium of claim 17 , wherein the parallax correction is based on a pre-compensated electronic rolling shutter distortion.