Method and device for image stitching
The present application discloses a method and a device for image stitching. The method includes: obtaining a first image acquired by a first camera and a second image acquired by a second camera, wherein the first image is an image with visible light information, the second image is an image including depth information and visible light information, the first camera and the second camera are disposed adjacent to each other, and there is an overlapping area between the acquired first image and second image; mapping pixels in the second image to the overlapping area and/or an extended area, wherein the extended area is an image area formed by pixels in the second image which are mapped to the outside of the first image; stitching the mapped pixels in the second image to the overlapping area and/or the extended area based on depth information and visible light information of the pixels in the second image to obtain a stitched image. The present application solves the technical problem of a low stitching efficiency in the image stitching process in the prior art.
1. A method for image stitching, comprising
obtaining a first image acquired by a first camera and a second image acquired by a second camera, wherein the first image is an image with visible light information, the second image is an image comprising depth information and visible light information, the first camera and the second camera are disposed adjacent to each other, and there is an overlapping area between the acquired first image and second image;
mapping pixels in the second image to the overlapping area and/or an extended area, wherein the extended area is an image area formed by pixels in the second image which are mapped to the outside of the first image;
stitching the mapped pixels in the second image to the overlapping area and/or the extended area based on depth information and visible light information of the pixels in the second image to obtain a stitched image.
2. The method according to claim 1 , wherein the step of mapping pixels in the second image to the overlapping area and/or an extended area, comprises:
reading coordinate information of a pixel in the second image in a second image coordinate system;
mapping the coordinate information of the pixel in the second image in the second image coordinate system to a first image coordinate system by using coordinate transformation to obtain coordinate information of the pixel in the second image in the first image coordinate system;
determining a position of the pixel in the second image in the overlapping area and/or the extended area based on the coordinate information of the pixel in the second image in the first image coordinate system.
3. The method according to claim 2 , wherein the step of mapping the coordinate information of the pixel in the second image in the second image coordinate system to a first image coordinate system by using coordinate transformation to obtain coordinate information of the pixel in the second image in the first image coordinate system, comprises:
mapping the coordinate information of the pixel in the second image in the second image coordinate system to a second camera coordinate system to obtain coordinate information of the pixel in the second image in the second camera coordinate system;
mapping the coordinate information of the pixel in the second image in the second camera coordinate system to a first camera coordinate system to obtain coordinate information of the pixel in the second image in the first camera coordinate system;
mapping the coordinate information of the pixel in the second image in the first camera coordinate system to the first image coordinate system to obtain coordinate information of the pixel in the second image in the first image coordinate system.
4. The method according to claim 3 , wherein the coordinate information m 2 (u 2 ,v 2 ) of the pixel in the second image in the second image coordinate system is mapped to the second camera coordinate system by using a first formula as follows to obtain coordinate information m 2 (X 2 ,Y 2 ,Z 2 ) of the pixel in the second image in the second camera coordinate system;
[
X
2
Y
2
Z
2
1
]
=
D
2
*
A
2
-
1
*
[
u
2
v
2
1
]
,
wherein A 2 is an intrinsic parameter of the second camera; and D 2 is a scale factor of the second image.
5. The method according to claim 4 , wherein the coordinate information m 2 (X 2 ,Y 2 ,Z 2 ) of the pixel in the second image in the second camera coordinate system is mapped to the first camera coordinate system by using a second formula as follows to obtain coordinate information m 1 (X 1 ,Y 1 ,Z 1 ) of the pixel in the second image in the first camera coordinate system;
[
X
1
Y
1
Z
1
1
]
=
[
R
,
t
0
T
,
1
]
[
X
2
Y
2
Z
2
1
]
,
wherein R is a rotation matrix for the first camera coordinate system relative to the second camera coordinate system, t is a translation vector for the first camera coordinate system relative to the second camera coordinate system.
6. The method according to claim 5 , wherein the coordinate information m 1 (X 1 ,Y 1 ,Z 1 ) of the pixel in the second image in the first camera coordinate system is mapped to the first image coordinate system by using a third formula as follows to obtain coordinate information m 1 (u 1 ,v 1 ) of the pixel in the second image in the first image coordinate system;
[
u
1
v
1
1
]
=
A
1
D
1
[
X
1
Y
1
Z
1
1
]
wherein A 1 is an intrinsic parameter of the first camera; and D 1 is a scale factor of the first mage.
7. The method according to claim 1 , wherein the step of stitching the mapped pixels in the second image to the overlapping area and/or the extended area based on depth information and visible light information of the pixels in the second image to obtain a stitched image, comprises:
when a pixel in the second image is mapped to the overlapping area, performing a weighting operation on the visible light information of the pixel in the first image and the visible light information of the pixel in the second image, and assigning the weighted visible light information to the visible light information of the pixel in the overlapping area in the stitched image;
when a pixel in the second image is mapped to the extended area, assigning the visible light information of the pixel in the second image to the visible light information of the pixel in the extended area in the stitched image.
8. The method according to claim 2 , wherein the step of stitching the mapped pixels in the second image to the overlapping area and/or the extended area based on depth information and visible light information of the pixels in the second image to obtain a stitched image, comprises:
determining whether a plurality of pixels in the second image are mapped to a same pixel in the overlapping area and/or the extended area, based on the coordinate information of the mapped pixel in the second image in the first image coordinate system;
when the plurality of pixels in the second image are mapped to the same pixel in the overlapping area and/or the extended area, determining visible light information of the same pixel, in the overlapping area and/or the extended area in the stitched image based on a plurality of pieces of depth information of the plurality of pixels in the second image.
9. The method according to claim 8 , wherein the step of determining the visible light information of the same pixel, in the overlapping area and/or the extended area in the stitched image based on a plurality of pieces of depth information of the plurality of pixels in the second image, comprises:
comparing the plurality of pieces of depth information of the plurality of pixels in the second image, and assigning visible light information of a pixel with the smallest depth information among the plurality of pieces of depth information to visible light information of the same pixel in the overlapping area and/or the extended area in the stitched image; or
performing a weighting operation on visible light information of the plurality of pixels in the second image, and assigning the weighted visible light information to visible light information of the same pixel in the overlapping area and/or the extended area in the stitched image.
10. A non-transitory computer readable storage medium for storing executable program codes, the executable program codes are executed to perform the method for image stitching of claim 1 .
11. An electronic device, which comprises: one or more processors, one or more memories, wherein the one or more memories are configured to store executable program codes; the executable program codes stored in the one or more memories are executed by the one or more processors to perform:
obtaining a first image acquired by a first camera and a second image acquired by a second camera, wherein the first image is an image with visible light information, the second image is an image comprising, depth information and visible light information, the first camera and the second camera are disposed adjacent to each other, and there is an overlapping area between the acquired first image and second image;
mapping pixels in the second image to the overlapping area and/or an extended area, wherein the extended area is an image area formed by pixels in the second image which are mapped to the outside of the first image;
stitching the mapped pixels in the second image to the overlapping area and/or the extended area based on depth information and visible light information of the pixels in the second image to obtain a stitched image.
12. The device according to claim 11 , wherein the executable program codes stored in the one or more memories are executed by the one or more processors to perform:
reading coordinate information of a pixel in the second image in a second image coordinate system;
mapping the coordinate information of the pixel in the second image in the second image coordinate system to a first image coordinate system by using coordinate transformation to obtain coordinate information of the pixel in the second image in the first image coordinate system;
determining a position of the pixel in the second image in the overlapping area and/or the extended area based on the coordinate information of the pixel in the second image in the first image coordinate system.
13. The device according to claim 12 wherein the executable program codes stored in the one or more memories are executed by the one or more processors to perform:
mapping the coordinate information of the pixel in the second image in the second image coordinate system to a second camera coordinate system to obtain coordinate information of the pixel in the second image in the second camera coordinate system;
mapping the coordinate information of the pixel in the second image in the second camera coordinate system to a first camera coordinate system to obtain coordinate information of the pixel in the second image in the first camera coordinate system;
mapping the coordinate information of the pixel in the second image in the first camera coordinate system to the first image coordinate system to obtain coordinate information of the pixel in the second image in the first image coordinate system.
14. The device according to claim 13 , wherein the executable program codes stored in the one or more memories are executed by the one or more processors to perform: obtaining coordinate information m 2 (X 2 ,Y 2 ,Z 2 ) of the pixel in the second image in the second camera coordinate system through calculation according to a first formula as follows:
[
X
2
Y
2
Z
2
1
]
=
D
2
*
A
2
-
1
*
[
u
2
v
2
1
]
,
wherein A 2 is an intrinsic parameter of the second camera; and D 2 is a scale factor of the second image.
15. The device according to claim 14 , wherein the executable program codes stored in the one or more memories are executed by the one or more processors to perform: obtaining coordinate information m 1 (X 1 ,Y 1 ,Z 3 ) of the pixel in the second image in the first camera coordinate system through calculation according to a second formula as follows:
[
X
1
Y
1
Z
1
1
]
=
[
R
,
t
0
T
,
1
]
[
X
2
Y
2
Z
2
1
]
,
wherein R is a rotation matrix for the first camera coordinate system relative to the second camera coordinate system, t is a translation vector for the first camera coordinate system relative to the second camera coordinate system.
16. The device according to claim 15 , wherein the executable program codes stored in the one or more memories are executed by the one or more processors to perform: obtaining coordinate information m 1 (u 1 ,v 1 ) of the pixel in the second image in the first image coordinate system through calculation according to a third formula as follows:
[
u
1
v
1
1
]
=
A
1
D
1
[
X
1
Y
1
Z
1
1
]
wherein A 1 is an intrinsic parameter of the first camera; and D 1 is a scale factor of the first image.
17. The device according to claim 12 , wherein the executable program codes stored in the one or more memories are executed by the one or more processors to perform:
determining whether a plurality of pixels in the second image are mapped to a same pixel in the overlapping area and/or the extended area, based on the coordinate information of the mapped pixel in the second image in the first image coordinate system;
when the plurality of pixels in the second image are mapped to the same pixel in the overlapping area and/or the extended area, determining visible light information of the same pixel, in the overlapping area and/or the extended area in the stitched image based on a plurality of pieces of depth information of the plurality of pixels in the second image.
18. The device according to claim 17 , wherein the executable program codes stored in the one or more memories are executed by the one or more processors to perform:
comparing the plurality of pieces of depth information of the plurality of pixels in the second image, and assigning visible light information of a pixel with the smallest depth information among the plurality of pieces of depth information to visible light information of the same pixel in the overlapping area and/or the extended area in the stitched image;
performing a weighting operation on visible light information of the plurality of pixels in the second image, and assigning the weighted visible light information to visible light information of the same pixel in the overlapping area and/or the extended area in the stitched image.
19. The device according to claim 11 , wherein the executable program codes stored in the one or more memories are executed by the one or more processors to perform:
when a pixel in the second image is mapped to the overlapping area, performing a weighting operation on the visible light information of the pixel in the first image and the visible light information of the pixel in the second image, and assigning the weighted visible light information to the visible light information of the pixel in the overlapping area in the stitched image;
when a pixel in the second image is mapped to the extended area, assigning the visible light information of the pixel in the second image to the visible light information of the pixel in the extended area in the stitched image.