Granted Patent B2

US 12,736,804 · App. 18/826,466 · Granted Sep 15, 2026

Inventors: Fan Dong (Beijing, CN); Cho-Han Wu (New Taipei City, TW)

Assignee: VIA TECHNOLOGIES, INC.

G02B27/0012G01M11/0257G06T7/80G06T2207/20221G06T2207/30256

Abstract

Claims (54)

1 . A method of a simulated platform for online optimizing extrinsic parameters of fisheye-lens cameras in stitching surround-view images, the method comprising:

establishing a simulated vehicle in the simulation platform according to vehicular characteristics associated with an actual vehicle;

generating a plurality of simulated fisheye-lens cameras according to intrinsic parameters of simulated fisheye lenses of the plurality of the simulated fisheye-lens cameras;

disposing the plurality of simulated fisheye-lens cameras on the simulated vehicle;

placing a plurality of simulated calibration plates surrounding the simulated vehicle;

obtaining initial extrinsic parameters of each of the simulated fisheye-lens cameras, comprising the steps of:

generating a first fisheye image by each of the simulated fisheye-lens cameras, wherein the first fisheye image comprises an image of one of the plurality of the simulated calibration plates; and

generating the initial extrinsic parameters of each of the simulated fisheye-lens cameras by using a first characteristic point within the image of one of the plurality of the simulated calibration plates;

optimizing the extrinsic parameters of each of the simulated fisheye-lens cameras, comprising the steps of:

generating at least one second fisheye image according to the initial extrinsic parameters of each of the simulated fisheye-lens cameras;

generating a second stitching image by using the second fisheye images of two adjacently simulated fisheye-lens cameras according to a predetermined direction; and

determining if the extrinsic parameters of each of the simulated fisheye-lens cameras is optimized according to errors of overlap regions of the second stitching image.

2 . The method of claim 1 , further comprising the simulated platform automatically places a plurality of predetermined simulated calibration plates surrounding the simulated vehicle when a first shortcut key is activated.

3 . The method of claim 1 , further comprising the simulated platform automatically disposes the plurality of simulated fisheye-lens cameras on the simulated vehicle when a second shortcut key is activated.

4 . The method of claim 3 , further comprising the simulated platform automatically stores the first fisheye images after the first fisheye images are generated.

5 . The method of claim 3 , wherein the simulated fisheye-lens cameras are indicated by a user over the simulated platform.

6 . The method of claim 1 , wherein the simulated platform automatically aligns camera coordinate systems of the simulated fisheye-lens cameras with a world coordinate system after the initial extrinsic parameters of the simulated fisheye-lens cameras are generated.

7 . The method of claim 6 , wherein the simulated platform adjusts a transform matrix and a rotation matrix associated with the initial extrinsic parameters to align the camera coordinate systems with the world coordinate system.

8 . The method of claim 6 , wherein the step of optimizing the extrinsic parameters comprises:

transforming a second characteristic point to a third characteristic point after the camera coordinate systems align with the world coordinate system; and

optimizing the extrinsic parameters of each of the simulated fisheye-lens cameras by determining if the errors of overlap regions of the second stitching image is larger than a stitching-error threshold by referring to the third characteristic point.

9 . The method of claim 8 , further comprising:

adjusting the initial extrinsic parameters of the two adjacently simulated fisheye-lens cameras to generate an updated second stitching image if the errors of overlap regions of the second stitching image is determined larger than the stitching-error threshold; and

determining errors of overlap regions of the updated second stitching image is determined larger than the stitching-error threshold.

10 . The method of claim 8 , wherein the second characteristic point is an edge of the simulated calibration plates, interior vertex point, a traffic line, a texture of the overlap regions recognizable by the simulated platform.

11 . The method of claim 1 , wherein the first characteristic point is an interior vertex point of one of the simulated checkerboard calibration plates.

12 . The method of claim 11 , further comprising:

obtaining a position of the interior vertex point in the world coordinate system by using a calibration-plate-corner detection algorithm; and

generating the initial extrinsic parameters according to the position of the interior vertex point in the world coordinate system.

13 . The method of claim 1 , wherein the first fisheye image is identical to the second fisheye image.

14 . The method of claim 1 , further comprising optimizing the second stitching images after all of the initial extrinsic parameters of the simulated fisheye-lens camera are optimized.

15 . The method of claim 14 , wherein the step of optimizing the second stitching images comprises re-projection-error minimizations, color/brightness adjustments, and smooth transitions to remove jagged edges or inconsistences of the seam regions.

16 . The method of claim 1 , further comprising:

generating a reference extrinsic parameter for one of the adjacently simulated fisheye-lens cameras;

generating a third fisheye image by one of the adjacently simulated fisheye-lens cameras being introduced the reference extrinsic parameters;

generating a third stitching image by using the third fisheye image and the fisheye image generated by the other one of the adjacently simulated fisheye-lens cameras; and

outputting an offset of the reference extrinsic parameters as a tolerance if errors of overlap regions of the third stitching image is determined larger than a stitching-error threshold.

17 . The method of claim 16 , wherein the reference extrinsic parameters are generated by introducing the offset into the initial extrinsic parameters.

18 . The method of claim 16 , further comprising the step of increasing the offset an increment and introducing to the initial extrinsic parameters of the one of the adjacently simulated fisheye-lens cameras if errors of overlap regions of the third stitching image is determined smaller than a stitching-error threshold.

19 . The method of claim 1 , wherein the simulation platform comprises Carla, PerScan, CarSim, VIRES VTD, PTV Vissim, or TESS NG.

20 . A method of a simulated platform for online optimizing extrinsic parameters of fisheye-lens cameras in stitching surround-view images, the method comprising:

generating at least one fisheye images by each of the simulated fisheye-lens cameras;

generating a stitching image by using the fisheye images of two adjacently simulated fisheye-lens cameras according to a predetermined direction;

determining if the extrinsic parameters of each of the simulated fisheye-lens cameras is optimized according to errors of overlap regions of the stitching image;

transforming a first characteristic point to a second characteristic point after camera coordinate systems of each of the simulated fisheye-lens cameras align with a world coordinate system; and

optimizing the extrinsic parameters of each of the simulated fisheye-lens cameras by determining if the errors of overlap regions of the stitching image is larger than a stitching-error threshold by referring to the third characteristic point.

21 . The method of claim 20 , further comprising:

adjusting current extrinsic parameters of the two adjacently simulated fisheye-lens cameras to generate an updated stitching image if the errors of overlap regions of the stitching image is determined larger than a stitching-error threshold; and

determining errors of overlap regions of the updated stitching image is determined larger than the stitching-error threshold.

22 . The method of claim 21 , wherein the steps of generating the stitching image, determining if the extrinsic parameters of each of the simulated fisheye-lens cameras is optimized, transforming the first characteristic point to the third characteristic point, adjusting the current extrinsic parameters of the two adjacently simulated fisheye-lens cameras is larger than the stitching-error threshold, generating the updated stitching image, and determining errors of overlap regions of the updated stitching image is determined larger than the stitching-error threshold are operated by the simulated platform after the fisheye images are generated.

23 . The method of claim 20 , wherein the first characteristic point is an edge of edges or interior vertex points of a calibration plate, a traffic line, or any identifiable textures of the overlap regions of the stitching image.

24 . The method of claim 20 , further comprising optimizing a surround-view stitching image made by all of the simulated fisheye-lens cameras after all the extrinsic parameters of the simulated fisheye-lens cameras are optimized.

25 . The method of claim 24 , wherein the optimizations of the surround-view stitching image comprise re-projection-error minimizations, color/brightness adjustments, and smooth transitions to remove jagged edges or inconsistences of the seam regions.

26 . The method of claim 20 , wherein each of the images is generated by a fisheye-lens camera installed on a vehicle.

Assignments (1)

Priority Claims (2)

Continuity (1)

References Cited (77)