Joint rolling shutter image stitching and rectification
View Patent ↗A computer-implemented method executed by at least one processor for applying rolling shutter (RS)-aware spatially varying differential homography fields for simultaneous RS distortion removal and image stitching is presented. The method includes inputting two consecutive frames including RS distortions from a video stream, performing keypoint detection and matching to extract correspondences between the two consecutive frames, feeding the correspondences between the two consecutive frames into an RS-aware differential homography estimation component to filter out outlier correspondences, sending inlier correspondences to an RS-aware spatially varying differential homography field estimation component to compute an RS-aware spatially varying differential homography field, and using the RS-aware spatially varying differential homography field in an RS stitching and correction component to produce stitched images with removal of the RS distortions.
1. A computer-implemented method executed by at least one processor for applying rolling shutter (RS)-aware spatially varying differential homography fields for simultaneous RS distortion removal and image stitching, the method comprising:
inputting two consecutive frames including RS distortions from a video stream;
performing keypoint detection and matching to extract correspondences between the two consecutive frames;
feeding the correspondences between the two consecutive frames into an RS-aware differential homography estimation component to filter out outlier correspondences;
sending inlier correspondences to an RS-aware spatially varying differential homography field estimation component to compute an RS-aware spatially varying differential homography field; and
using the RS-aware spatially varying differential homography field in an RS stitching and correction component to produce stitched images with removal of the RS distortions.
2. The method of claim 1 , wherein the outlier correspondences are filtered out by employing robust estimation with the RS-aware differential homography estimation component and remaining inlier correspondences are sent to the RS-aware spatially varying differential homography estimation component.
3. The method of claim 1 , wherein the RS-aware spatially varying differential homography field is derived under a constant acceleration motion assumption regarding an RS camera capturing the video stream.
4. The method of claim 1 , wherein a minimal solver is applied to account for scanline varying camera poses of an RS camera capturing the video stream.
5. The method of claim 1 , wherein a relative motion between two first scanlines of the two consecutive frames are parameterized.
6. The method of claim 5 , wherein poses corresponding to all other scanlines are obtained by interpolation.
7. The method of claim 1 , wherein the RS-aware spatially varying differential homography field is given as:
h
*
(
x
)
=
arg
min
h
∑
i
∈
I
w
i
(
x
)
(
β
(
k
,
y
1
,
y
2
)
b
i
h
-
u
i
)
2
,
where w i (x) is defined as
w
i
(
x
)
=
max
(
exp
(
-
x
-
x
i
2
σ
2
)
,
τ
)
,
σ and τ are pre-defined scale and regularization parameters, k is a pure motion parameter independent of a scene, b i and u i are constraints, x is an image point, y 1 , y 2 are scanlines, h is a total number of scanlines, and β(k, y 1 , y 2 ) is a scale describing a constant acceleration motion.
8. A system for applying rolling shutter (RS)-aware spatially varying differential homography fields for simultaneous RS distortion removal and image stitching, the system comprising:
a memory; and
a processor in communication with the memory, wherein the processor runs program code to:
input two consecutive frames including RS distortions from a video stream;
perform keypoint detection and matching to extract correspondences between the two consecutive frames;
feed the correspondences between the two consecutive frames into an RS-aware differential homography estimation component to filter out outlier correspondences;
send inlier correspondences to an RS-aware spatially varying differential homography field estimation component to compute an RS-aware spatially varying differential homography field; and
use the RS-aware spatially varying differential homography field in an RS stitching and correction component to produce stitched images with removal of the RS distortions.
9. The system of claim 8 , wherein the outlier correspondences are filtered out by employing robust estimation with the RS-aware differential homography estimation component and remaining inlier correspondences are sent to the RS-aware spatially varying differential homography estimation component.
10. The system of claim 8 , wherein the RS-aware spatially varying differential homography field is derived under a constant acceleration motion assumption regarding an RS camera capturing the video stream.
11. The system of claim 8 , wherein a minimal solver is applied to account for scanline varying camera poses of an RS camera capturing the video stream.
12. The system of claim 8 , wherein a relative motion between two first scanlines of the two consecutive frames are parameterized.
13. The system of claim 12 , wherein poses corresponding to all other scanlines are obtained by interpolation.
14. The system of claim 1 , wherein the RS-aware spatially varying differential homography field is given as:
h
*
(
x
)
=
arg
min
h
∑
i
∈
I
w
i
(
x
)
(
β
(
k
,
y
1
,
y
2
)
b
i
h
-
u
i
)
2
,
where w i (x) is defined as
w
i
(
x
)
=
max
(
exp
(
-
x
-
x
i
2
σ
2
)
,
τ
)
,
σ and τ are pre-defined scale and regularization parameters, k is a pure motion parameter independent of a scene, b i and u i are constraints, x is an image point, y 1 , y 2 are scanlines, h is a total number of scanlines, and β(k, y 1 , y 2 ) is a scale describing the constant acceleration motion.
15. A non-transitory computer-readable storage medium comprising a computer-readable program for applying rolling shutter (RS)-aware spatially varying differential homography fields for simultaneous RS distortion removal and image stitching, wherein the computer-readable program when executed on a computer causes the computer to perform the steps of:
inputting two consecutive frames including RS distortions from a video stream;
performing keypoint detection and matching to extract correspondences between the two consecutive frames;
feeding the correspondences between the two consecutive frames into an RS-aware differential homography estimation component to filter out outlier correspondences;
sending inlier correspondences to an RS-aware spatially varying differential homography field estimation component to compute an RS-aware spatially varying differential homography field; and
using the RS-aware spatially varying differential homography field in an RS stitching and correction component to produce stitched images with removal of the RS distortions.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the outlier correspondences are filtered out by employing robust estimation with the RS-aware differential homography estimation component and remaining inlier correspondences are sent to the RS-aware spatially varying differential homography estimation component.
17. The non-transitory computer-readable storage medium of claim 15 , wherein the RS-aware spatially varying differential homography field is derived under a constant acceleration motion assumption regarding an RS camera capturing the video stream.
18. The non-transitory computer-readable storage medium of claim 15 , wherein a minimal solver is applied to account for scanline varying camera poses of an RS camera capturing the video stream.
19. The non-transitory computer-readable storage medium of claim 15 , wherein a relative motion between two first scanlines of the two consecutive frames are parameterized.
20. The non-transitory computer-readable storage medium of claim 19 , wherein poses corresponding to all other scanlines are obtained by interpolation.