Real-time high-resolution binocular camera distortion correction implementation method based on FPGA
Provided in the present disclosure is a real-time high-resolution binocular camera distortion correction implementation method based on FPGA. The method includes acquiring input video information in a YUV format, and performing YUV channel separation on the input video information; compressing U and V video signals inputted; performing distortion correction on a Y video signal, and performing distortion correction on the compressed U and V video signals, including calculating binocular camera distortion parameters, calculating a coordinate mapping relationship of Y, U, and V channels, and calculating and generating target image data by using an interpolation algorithm; amplifying U and V target image data after distortion correction to fit an input image; and outputting separated Y, U, and V channel target image data according to a raw image format. According to the present disclosure, high resolution and high frame rate videos can be processed in real time.
1 . A real-time high-resolution binocular camera distortion correction implementation method based on FPGA, comprising the following steps:
acquiring input video information in a YUV format, and performing YUV channel separation on the input video information to respectively obtain Y, U, and V video signals;
compressing the U and V video signals inputted, and not compressing the Y video signal inputted;
performing distortion correction on the Y video signal, and performing distortion correction on the compressed U and V video signals, comprising calculating binocular camera distortion parameters, calculating a coordinate mapping relationship of Y, U, and V channels, and calculating and generating target image data by using an interpolation algorithm;
amplifying U and V target image data that has been subjected to distortion correction to fit an input image; and
outputting separated Y, U, and V channel target image data according to a raw image format.
2 . The method as claimed in claim 1 , wherein
an FPGA end is used to perform distortion correction; the FPGA end is a xilinx ultrascale FPGA chip; and the chip is provided with ultra_ram to replace BRAM for data caching.
3 . The method as claimed in claim 1 , wherein
when the input video information in the YUV format is acquired, the input video information in RGB or other formats is converted into the YUV format for display.
4 . The method as claimed in claim 1 , wherein
when the coordinate mapping relationship of the Y, U, and V channels is calculated, pixel points on an image plane are re-arranged according to a distortion model; and gray values of the pixel points after spatial transformation are re-assigned.
5 . The method as claimed in claim 4 , wherein
mapping coordinates are calculated by using an internal parameter matrix H of a camera, and distortion coefficients ki(i=1, 2, 3) and pi(i=1, 2); and assuming that (xp, yp) is a pixel point on a target image, and (xs, ys) is a pixel mapping coordinate corresponding to the pixel point (xp, yp), a calculation formula of the pixel mapping coordinate (xs, ys) is expressed as (1) and (2):
[
x
d
y
d
1
]
=
H
-
1
×
[
x
p
y
d
1
]
=
[
1
dx
0
u
0
0
1
dy
v
o
0
0
1
]
×
[
x
p
y
p
1
]
(
1
)
[
x
s
y
s
]
=
[
1
dx
0
0
1
dx
]
×
[
x
d
×
(
1
+
k
1
r
2
+
k
2
r
4
+
k
3
r
6
)
+
[
2
×
p
1
×
y
d
+
p
2
×
(
r
2
+
2
x
d
2
)
]
y
d
×
(
1
+
k
1
r
2
+
k
2
r
4
+
k
3
r
6
)
+
[
2
×
p
2
×
x
d
+
p
1
×
(
r
2
+
2
y
d
2
)
]
]
+
[
u
0
v
0
]
.
(
2
)
6 . The method as claimed in claim 1 , wherein
when the target image data is calculated and generated by using the interpolation algorithm, gray values of pixel points after spatial transformation are re-assigned by using bilinear interpolation, and after linear interpolation is performed once respectively in x and y directions, a gray value of a target pixel is obtained.
7 . The method as claimed in claim 6 , wherein
assuming that the value of a function f at a point P=(x, y) is unknown, and assuming that the values of the function f at four points of Q11=(x1, y1), Q12=(x1, y2), Q21=(x2, y1), and Q22=(x2, y2) are known, linear interpolation is performed in the x direction to obtain formulas (3) and (4):
f
(
R
1
)
≈
x
2
-
x
x
2
-
x
1
f
(
Q
1
1
)
+
x
-
x
1
x
2
-
x
1
f
(
Q
2
1
)
,
R
1
=
(
x
,
y
1
)
(
3
)
f
(
R
2
)
≈
x
2
-
x
x
2
-
x
1
f
(
Q
1
2
)
+
x
-
x
1
x
2
-
x
1
f
(
Q
2
2
)
,
R
2
=
(
x
,
y
2
)
(
4
)
then linear interpolation is performed in the y direction to obtain a formula (5):
f
(
P
)
≈
y
2
-
y
y
2
-
y
1
f
(
R
1
)
+
y
-
y
1
y
2
-
y
1
f
(
R
2
)
(
5
)
therefore, a final result of bilinear interpolation is expressed as a formula (6):
f
(
P
)
≈
(
x
2
-
x
)
×
(
y
2
-
y
)
(
x
2
-
x
1
)
×
(
y
2
-
y
1
)
f
(
Q
1
1
)
+
(
x
-
x
1
)
×
(
y
2
-
y
)
(
x
2
-
x
1
)
×
(
y
2
-
y
1
)
f
(
Q
2
1
)
+
(
x
2
-
x
)
×
(
y
-
y
1
)
(
x
2
-
x
1
)
×
(
y
2
-
y
1
)
f
(
Q
1
2
)
+
(
x
-
x
1
)
×
(
y
-
y
1
)
(
x
2
-
x
1
)
×
(
y
2
-
y
1
)
f
(
Q
2
2
)
.
(
6
)
8 . The method as claimed in claim 1 , wherein
when distortion correction is performed on the video signals of the Y, U, and V channels, respectively, the video signals are transmitted to an FPGA end according to the format of a pixel flow, received video data is cached into ULTRA_RAM, and since the ULTRA_RAM of a selected signal is 72 bits, 8-bit YUV422 data is first subjected to data extension and aligned to 72 bits, and then pixel data is stored into the ULTRA_RAM; and
according to the binocular camera distortion parameters provided by a host computer, by means of assembly line arrangement, a pixel mapping coordinate in a raw image corresponding to one pixel is guaranteed to be calculated within each clock cycle, and the pixel mapping coordinate is stored according to the format of Q12.20.
9 . The method as claimed in claim 1 , wherein
according to a correspondence relationship between integer parts of the pixel mapping coordinate, pixel data of four adjacent pixels are read from ULTRA_RAM for row caching, and when a coordinate after a mapping coordinate is calculated as (m, n), pixel values of four points of (m, n), (m+1, n), (m, n+1), and (m+1, n+1) are read;
a final pixel value is calculated according to a bilinear interpolation calculation formula and a coordinate mapping fractional part; and
video streaming data of a single channel that has been subjected to distortion correction is outputted.
10 . The method as claimed in claim 1 , wherein
the Y target image data that has been subjected to distortion correction is sent to data reorganization FIFO;
the U target image data that has been subjected to distortion correction is amplified, and then sent to the data reorganization FIFO; and
the V target image data that has been subjected to distortion correction is amplified, further compressed, and then sent to the data reorganization FIFO.