Automatic alignment of video frames for image processing
View Patent ↗The provided is a method that can automatically align image frames in recorded video clips. Individual frames in video may shift and rotate due to shaking or vibration of a video camera. Unaligned frames make some imaging processing techniques difficult or infeasible. One example of image processing techniques is to isolate, recognize, and/or quantitatively analyze vapor plume images captured by an Infrared (IR) camera. Such techniques have a great potential to be used to automatically detect volatile organic compounds (VOC) leaked from process equipment at refineries and chemical plants. This invention is a technique for various subsequent image processing techniques. The invention uses spatially based Fast Fourier Transforms (FFT) to determine amount of shift, rotation, and scaling to align image frames, and uses a digital filtering technique to improve the alignment.
1. A method for aligning image frames, comprising steps of:
selecting a reference image;
selecting a sample image to be aligned to the reference image;
Fourier-transforming the reference image to obtain a Fourier transform of the reference image;
Fourier-transforming the sample image to obtain a Fourier transform of the sample image;
coordinate-transforming an absolute value of the Fourier transform of the reference image to obtain a coordinate-transformed Fourier transform of the reference image;
coordinate-transforming an absolute value of the Fourier transform of the sample image to obtain a coordinate-transformed Fourier transform of the sample image;
obtaining a first phase shift from the coordinate-transformed Fourier transform of the reference image and the coordinate-transformed Fourier transform of the sample image;
inverse-Fourier-transforming the first phase shift to obtain an inverse Fourier transform of the first phase shift;
finding a first transformation factor from the inverse Fourier transform of the first phase shift;
transforming the sample image by the first transformation factor to obtain a first-transformed sample image;
Fourier-transforming the first-transformed sample image to obtain a Fourier transform of the first-transformed sample image;
obtaining a second phase shift of the Fourier transform of the reference image and the Fourier transform of the first-transformed sample image;
inverse-Fourier-transforming the second phase shift to obtain an inverse Fourier transform of the second phase shift;
finding a second transformation factor from the inverse Fourier transform of the second phase shift; and
transforming the first-transformed sample image by the second transformation factor.
2. The method of claim 1 , comprised of each of the reference image and the sample image being represented in Cartesian coordinates.
3. The method of claim 2 , comprised of the step of coordinate-transforming the absolute value of the Fourier transform of the reference image including a step of transforming an absolute value of the Fourier transform of the reference image from Cartesian coordinates to log-polar coordinates; and
comprised of the step of coordinate-transforming the absolute value of the Fourier transform of the sample image including a step of transforming an absolute value of the Fourier transform of the sample image from Cartesian coordinates to log-polar coordinates.
4. The method of claim 3 , comprised of the first transformation factor including a rotational shift, by which the sample image is rotated to be aligned to the reference image.
5. The method of claim 4 , comprised of the first transformation factor including a scaling factor, by which the sample image is rescaled to be aligned to the reference image.
6. The method of claim 4 , comprised of the second transformation factor including a translational shift, by which the sample image is translated to be aligned to the reference image.
7. The method of claim 6 , comprised of the step of finding the second transformation factor comprising steps of:
finding a first location (x 1 , y 1 ) at which the inverse Fourier transform of the second phase shift is maximized;
finding a second location (x 2 , y 2 ) among four points (x 1 ±1, y 1 ±1) at which the inverse Fourier transform of the second phase shift is the largest; and
finding the translational shift from formulas
x
T
=
w
x
1
x
1
+
w
x
2
x
2
w
x
1
+
w
x
2
y
T
=
w
y
1
y
1
+
w
y
2
y
2
w
y
1
+
w
y
2
where (x T , y T ) is the translational shift, and w xi and w yi are defined by formulas
w xi =ƒ(| F ( x i ,y 1 )|)+ƒ(| F ( x i ,y 2 )|)
w yi =ƒ(| F ( x 1 ,y i )|)+ƒ(| F ( x 2 ,y i )|)
where i stands for 1 or 2, F stands for Fourier transform, and ƒ is an empirical function.
8. The method of claim 7 , comprised of the empirical function including a function of ƒ(z)=z α , where α is a parameter.
9. The method of claim 1 , comprised of the step of finding a first transformation factor comprising steps of:
finding a first location (x 1 , y 1 ) at which the inverse Fourier transform of the first phase shift is maximized; and
finding a second location (x 2 , y 2 ) among four points (x 1 ±1, y 1 ±1) at which the inverse Fourier transform of the first phase shift is the largest.
10. The method of claim 1 , comprised of the step of obtaining the first phase shift including a step of calculating the first phase shift from a formula
R
1
=
F
1
(
ξ
,
η
)
·
conj
(
F
2
(
ξ
,
η
)
)
abs
(
F
1
(
ξ
,
η
)
)
·
abs
(
F
2
(
ξ
,
η
)
)
;
and
comprised of the step of obtaining the second phase shift including a step of calculating the second phase shift from a formula
R
2
=
F
1
(
ξ
,
η
)
·
conj
(
F
2
′
(
ξ
,
η
)
)
abs
(
F
1
(
ξ
,
η
)
)
·
abs
(
F
2
′
(
ξ
,
η
)
)
where R 1 is the first phase shift, R 2 is the second phase shift, F 1 (ξ, η) is the Fourier transform of the reference image, F 2 (ξ, η) is the coordinate-transformed Fourier transform of the sample image, F 2 ′(ξ, η) is the Fourier transform of the first-transformed sample image, ξ and η are a vertical and a horizontal frequencies, respectively, conj is a complex conjugate, and abs is an absolute value.
11. The method of claim 1 , further comprising:
applying an erosion-dilation filter to process a difference image of the reference image and the transformed first-transformed sample image; and
reconstructing the transformed first-transformed sample image by adding the difference image to the reference image.
12. A non-transitory machine readable storage medium providing instructions that cause the machine to perform operations to align image frames, the operations comprising:
selecting a reference image that is represented in Cartesian coordinates;
selecting a sample image to be aligned to the reference image, the sample image being represented in Cartesian coordinates;
Fourier-transforming the reference image to obtain a Fourier transform of the reference image;
Fourier-transforming the sample image to obtain a Fourier transform of the sample image;
coordinate-transforming an absolute value of the Fourier transform of the reference image from Cartesian coordinates to log-polar coordinates to obtain a coordinate-transformed Fourier transform of the reference image;
coordinate-transforming an absolute value of the Fourier transform of the sample image from Cartesian coordinates to log-polar coordinates to obtain a coordinate-transformed Fourier transform of the sample image;
obtaining a rotational phase shift from the coordinate-transformed Fourier transform of the reference image and the coordinate-transformed Fourier transform of the sample image;
inverse-Fourier-transforming the rotational phase shift to obtain an inverse Fourier transform of the rotational phase shift;
finding a rotational shift from the inverse Fourier transform of the rotational phase shift;
rotating the sample image by the rotational shift to obtain a rotated sample image;
Fourier-transforming the rotated sample image to obtain a Fourier transform of the rotated sample image;
obtaining a translational phase shift of the Fourier transform of the reference image and the Fourier transform of the rotated sample image;
inverse-Fourier-transforming the translational phase shift to obtain an inverse Fourier transform of the translational phase shift;
finding a translational shift from the inverse Fourier transform of the translational phase shift; and
translating the rotated sample image by the translational shift.
13. The non-transitory machine readable storage medium of claim 12 , the operations further comprising:
finding a scaling factor from the inverse Fourier transform of the rotational phase shift; and
rescaling the sample image by the scaling factor.
14. The non-transitory machine readable storage medium of claim 12 , wherein the step of obtaining the rotational phase shift includes a step of calculating the rotational phase shift from a formula
R
=
ⅇ
-
j
2
π
(
ξ
·
dx
+
η
·
dy
)
=
Flp
1
(
ξ
,
η
)
·
conj
(
Flp
2
(
ξ
,
η
)
)
abs
(
Flp
1
(
ξ
,
η
)
)
·
abs
(
Flp
2
(
ξ
,
η
)
)
where R is the rotational phase shift, Flp 1 (ξ, η) is the coordinate-transformed Fourier transform of the reference image, Flp 2 (ξ, η) is the coordinate-transformed Fourier transform of the sample image, ξ and η are a vertical and a horizontal frequencies, respectively, conj is a complex conjugate, and abs is an absolute value.
15. The non-transitory machine readable storage medium of claim 12 , wherein the step of obtaining the translational phase shift includes a step of calculating the translational phase shift from a formula
R
=
ⅇ
-
j
2
π
(
ξ
·
dx
+
η
·
dy
)
=
F
1
(
ξ
,
η
)
·
conj
(
F
2
(
ξ
,
η
)
)
abs
(
F
1
(
ξ
,
η
)
)
·
abs
(
F
2
(
ξ
,
η
)
)
where R is the translational phase shift, F 1 (ξ, η) is the Fourier transform of the reference image, F 2 (ξ, η) is the Fourier transform of the rotated sample image, ξ and η are a vertical and a horizontal frequencies, respectively, conj is a complex conjugate, and abs is an absolute value.
16. The non-transitory machine readable storage medium of claim 12 , wherein the step of finding a rotational shift includes steps of:
finding a first location (x 1 , y 1 ) at which the inverse Fourier transform of the rotational shift is maximized; and
finding a second location (x 2 , y 2 ) among four points (x 1 ±1, y 1 ±1) at which the inverse Fourier transform of the rotational shift is the largest.
17. The non-transitory machine readable storage medium of claim 12 , wherein the step of finding the translational shift comprises steps of:
finding a first location (x 1 , y 1 ) at which the inverse Fourier transform of the translational shift is maximized;
finding a second location (x 2 , y 2 ) among four points (x 1 ±1, y 1 ±1) at which the inverse Fourier transform of the translational shift is the largest; and
finding the translational shift from formulas
x
T
=
w
x
1
x
1
+
w
x
2
x
2
w
x
1
+
w
x
2
y
T
=
w
y
1
y
1
+
w
y
2
y
2
w
y
1
+
w
y
2
where (x T , y T ) is the translational shift, and w xi and w yi are defined by formulas
w xi =ƒ(| F ( x i ,y 1 )|)+ƒ(| F ( x i ,y 2 )|)
w yi =ƒ(| F ( x 1 ,y i )|)+ƒ(| F ( x 2 ,y i )|)
where i stands for 1 or 2, F stands for Fourier transform, and ƒ is an empirical function.
18. The non-transitory machine readable storage medium of claim 17 , wherein the empirical function includes a function of ƒ(z)=z α , where α is a parameter.
19. The non-transitory machine readable storage medium of claim 12 , the operations further comprising:
applying an erosion-dilation filter to process a difference image of the reference image and the transformed first-transformed sample image; and
reconstructing the transformed first-transformed sample image by adding the difference image to the reference image.