Image merging method using viewpoint transformation and system therefor
View Patent ↗An image merging method using viewpoint transformation and a system therefor are provided. The method includes obtaining images captured by a plurality of cameras included in the camera system, performing viewpoint transformation for each of the images using a depth map for the images, and merging the images, the viewpoint transformation of which is performed.
1. An image merging method using viewpoint transformation executed by a computer included in a camera system including a plurality of cameras, the method comprising:
performing, by the camera system, viewpoint transformation for each image obtained by the plurality of cameras, using a depth map; and
merging images, the viewpoint transformation of which is performed;
wherein said performing the viewpoint transformation for each of the images comprises:
setting a viewpoint transformation relationship including a backward warping relationship;
determining a movement parameter in the viewpoint transformation relationship, the movement parameter being a value associated with a distance to move when a location at which each of the images is captured moves to a virtual specific location; and
transforming viewpoints of the images to be matched using the viewpoint transformation relationship;
wherein the backward warping is performed in accordance with the following relationship:
I
(
x
,
y
)
=
I
′
(
x
′
(
fB
+
m
z
D
)
+
f
m
x
D
fB
-
c
x
′
y
′
(
fB
+
m
y
D
)
+
f
m
z
D
fB
-
c
y
)
in which I(x, y) denotes the images captured by the plurality of cameras included in the camera system, x, y denote pixel locations in each of the images captured by the plurality of cameras included in the camera system, I′(x′, y′) denotes transformed images resulting from the viewpoint relationship which is performed, x′, y′ denote pixel locations in each of the transformed images resulting from the viewpoint transformation which is performed, f denotes a focal length of each of the plurality of cameras, B denotes a length of a baseline of the camera system, D denotes a parallax between the plurality of cameras, m x , m y , m denote the movement parameter, and c x , c y denote coordinates of an image center passing through an optical axes.
2. The method of claim 1 , wherein the determining of the movement parameter in the viewpoint transformation formula comprises:
determining the movement parameter based on a distance to move when an optical center of each of the plurality of cameras moves to a virtual reference location of the camera system.
3. The method of claim 2 , wherein said determining of the movement parameter in the viewpoint transformation relationship comprises:
determining the movement parameter based on a length of a baseline of the camera system.
4. The method of claim 1 , wherein said transforming the viewpoints of the images to be matched comprises:
transforming the viewpoint of each of the images into a viewpoint of an image captured at a reference location of the camera system.
5. The method of claim 1 , wherein said performing the viewpoint transformation for each of the images further comprises:
filling a hole of the depth map before performing the viewpoint transformation for each of the images.
6. The method of claim 1 , wherein said performing the viewpoint transformation for each of the images further comprises:
filling a hole of each of the images, the viewpoint transformation of which is performed.
7. The method of claim 1 , wherein the merging of the images, the viewpoint transformation of which is performed, further comprises:
transforming a viewpoint of the merged image to fill a hole of the merged image.
8. The method of claim 1 , wherein obtaining the images by the plurality of cameras comprises:
obtaining the images with different exposure values from the plurality of cameras; and
wherein said merging the images, the viewpoint transformation of which is performed, comprises:
merging the images, with the different exposure values, the viewpoint transformation of which is performed, to generate a hyper dynamic range (HDR) image.
9. The method of claim 1 , wherein each image captured by the plurality of cameras is obtained by:
obtaining images with different focus positions from the plurality of cameras; and
wherein said merging the images, the viewpoint transformation of which is performed, comprises:
merging the images, with the different focus positions, the viewpoint transformation of which is performed, to generate an all-in-focus image.
10. An image merging system using viewpoint transformation, implemented with a computer included in a camera system, the system comprising:
at least one processor configured to execute computer-readable instructions,
wherein the at least one processor comprises:
an image obtaining unit configured to obtain images captured by a plurality of cameras included in the camera system;
a viewpoint transformation performing unit configured to perform viewpoint transformation for each of the images using a depth map for the images; and
a merging unit configured to merge the images, the viewpoint transformation of which is performed;
wherein said viewpoint transformation for each of the images comprises:
setting a viewpoint transformation relationship including a backward warping relationship;
determining a movement parameter in the viewpoint transformation relationship, the movement parameter being a value associated with a distance to move when a location at which each of the images is captured moves to a virtual specific location; and
transforming viewpoints of the images to be matched using the viewpoint transformation relationship;
wherein the backward warping is performed in accordance with the following relationship:
I
(
x
,
y
)
=
I
′
(
x
′
(
fB
+
m
z
D
)
+
f
m
x
D
fB
-
c
x
′
y
′
(
fB
+
m
y
D
)
+
f
m
z
D
fB
-
c
y
)
in which I(x, y) denotes the images captured by the plurality of cameras included in the camera system, x, y denote pixel locations in each of the images captured by the plurality of cameras included in the camera system, I′(x′, y′) denotes transformed images resulting from the viewpoint relationship which is performed, x′, y′ denote pixel locations in each of the transformed images resulting from the viewpoint transformation which is performed, f denotes a focal length of each of the plurality of cameras, B denotes a length of a baseline of the camera system, D denotes a parallax between the plurality of cameras, m x , m x , m z denote the movement parameter, and c x , c y denote coordinates of an image center passing through an optical axes.
11. An image registration method executed by a computer included in a camera system, the method comprising:
obtaining images captured by a plurality of cameras included in the camera system;
setting a viewpoint transformation relationship based on a focal length of each of the plurality of cameras, a length of a baseline of the camera system, and parallax between the plurality of cameras, based on a depth map for the images;
determining a movement parameter in the viewpoint transformation formula, the movement parameter being a value associated with a distance to move when a location where each of the images is captured moves to a virtual specific location; and
registering images by transforming viewpoints of the images to be matched using the viewpoint transformation relationship;
wherein said setting the viewpoint transformation for each of the images further comprises:
setting a viewpoint transformation relationship including a backward warping relationship;
determining a movement parameter in the viewpoint transformation relationship, the movement parameter being a value associated with a distance to move when a location at which each of the images is captured moves to a virtual specific location; and
transforming viewpoints of the images to be matched using the viewpoint transformation relationship,
wherein the backward warping is performed in accordance with the following relationship:
I
(
x
,
y
)
=
I
′
(
x
′
(
fB
+
m
z
D
)
+
f
m
x
D
fB
-
c
x
′
y
′
(
fB
+
m
y
D
)
+
f
m
z
D
fB
-
c
y
)
in which I(x, y) denotes the images captured by the plurality of cameras included in the camera system, x, y denote pixel locations in each of the images captured by the plurality of cameras included in the camera system, I′(x′, y′) denotes transformed images resulting from the viewpoint relationship which is performed, x′, y′ denote pixel locations in each of the transformed images resulting from the viewpoint transformation which is performed, f denotes a focal length of each of the plurality of cameras, B denotes a length of a baseline of the camera system, D denotes a parallax between the plurality of cameras, m x , m x , m z denote the movement parameter, and c x , c y , denote coordinates of an image center passing through an optical axes.
12. The method of claim 11 , wherein said determining the movement parameter in the viewpoint transformation formula comprises:
determining the movement parameter based on a distance to move when an optical center of each of the plurality of cameras moves to a virtual reference location of the camera system.
13. An image registration system implemented with a computer included in a camera system, the system comprising:
at least one processor configured to execute computer-readable instructions,
wherein the at least one processor comprises:
an image obtaining unit configured to obtain images captured by a plurality of cameras included in the camera system;
a formula setting unit configured to set a viewpoint transformation relationship based on a focal length of each of the plurality of cameras, a length of a baseline of the camera system, and parallax between the plurality of cameras, based on a depth map for the images;
a movement parameter determining unit configured to determine a movement parameter in the viewpoint transformation relationship, the movement parameter being a value associated with a distance to move when a location at which each of the images is captured moves to a virtual specific location; and
an image registration unit configured to register the images by transforming viewpoints of the images to be matched using the viewpoint transformation relationship;
wherein said performing the viewpoint transformation for each of the images further comprises:
setting a viewpoint transformation relationship including a backward warping relationship;
determining a movement parameter in the viewpoint transformation relationship, the movement parameter being a value associated with a distance to move when a location at which each of the images is captured moves to a virtual specific location; and
transforming viewpoints of the images to be matched using the viewpoint transformation formula;
wherein the backward warping is performed in accordance with the following relationship:
I
(
x
,
y
)
=
I
′
(
x
′
(
fB
+
m
z
D
)
+
f
m
x
D
fB
-
c
x
′
y
′
(
fB
+
m
y
D
)
+
f
m
z
D
fB
-
c
y
)
in which I(x, y) denotes the images captured by the plurality of cameras included in the camera system, x, y denote pixel locations in each of the images captured by the plurality of cameras included in the camera system, I′(x′, y′) denotes transformed images resulting from the viewpoint relationship which is performed, x′, y′ denote pixel locations in each of the transformed images resulting from the viewpoint transformation which is performed, f denotes a focal length of each of the plurality of cameras, B denotes a length of a baseline of the camera system, D denotes a parallax between the plurality of cameras, m x , m x , m z denote the movement parameter, and c x , c y , denote coordinates of an image center passing through an optical axes.