Granted Patent B1

US 10,726,579 · App. 16/683,062 · Granted Jul 28, 2020

Inventor: Jiawei Huang (San Jose, CA)

Assignee: Honda Motor Co., Ltd.

G06T7/80G01S7/4972G01S17/42G01S17/86H04N17/002G05D1/0231G06T2207/10028G06T2207/20101G06T2207/30244G06T2207/30252

Abstract

Claims (76)

1. A calibration system, comprising:

control circuitry communicatively coupled to a sensor system comprising a Light Detection and Ranging (LiDAR) sensor and a camera,

wherein the control circuitry is configured to:

receive a plurality of image frames comprising a calibration pattern;

receive a plurality of point cloud data (PCD) frames comprising the calibration pattern;

extract a first normal to a first plane of the calibration pattern in a first PCD frame of the received plurality of PCD frames;

extract a second normal to a second plane of the calibration pattern in a first image frame of the received plurality of image frames;

compute a transform between the extracted first normal and the extracted second normal,

wherein the computed transform comprises final values of extrinsic calibration parameters for the sensor system; and

calibrate the LiDAR sensor with the camera based on the computed transform.

2. The calibration system according to claim 1 , further comprising the sensor system, wherein

the camera is configured to capture the plurality of image frames from a plurality of viewpoints, and

the LiDAR sensor is configured to:

scan the calibration pattern from the plurality of viewpoints; and

generate the plurality of PCD frames based on the scan of the calibration pattern from the plurality of viewpoints.

3. The calibration system according to claim 1 , wherein the control circuitry is further configured to:

estimate final values of intrinsic calibration parameters of the camera based on the plurality of image frames of the calibration pattern and a canonical image of the calibration pattern; and

calibrate the camera based on the estimated final values of the intrinsic calibration parameters.

4. The calibration system according to claim 3 , wherein the control circuitry is further configured to calibrate the LiDAR sensor with the calibrated camera based on the computed transform.

5. The calibration system according to claim 1 , wherein the control circuitry is further configured to determine a distance between the calibration pattern and a position of the camera.

6. The calibration system according to claim 5 , wherein the control circuitry is further configured to:

initialize values of the extrinsic calibration parameters with seed values;

compute a dot product between the extracted first normal and the extracted second normal based on the initialized values of the extrinsic calibration parameters;

compute a value of an objective function based on the computed dot product and a square of the determined distance; and

update the initialized values of the extrinsic calibration parameters until the computed value of the objective function is below a threshold,

wherein when the computed value of the objective function is below the threshold, the updated values of the extrinsic calibration parameters are the final values of the extrinsic calibration parameters.

7. The calibration system according to claim 6 , wherein the objective function is based on one of Trust Region Reflective (TRR) method, Levenberg-Marquardt method, or dogleg method.

8. The calibration system according to claim 1 , wherein the extrinsic calibration parameters comprise a rotation matrix and a translation vector for the sensor system.

9. The calibration system according to claim 1 , wherein the calibration pattern is a checkerboard pattern.

10. The calibration system according to claim 1 , wherein a position of the LiDAR sensor is fixed with respect to the camera.

11. The calibration system according to claim 1 , wherein

each PCD frame of the plurality of PCD frames includes a plurality of points that represents the calibration pattern, and

the control circuitry is further configured to:

select at least three points from the first PCD frame, wherein a Field-of-View (FOV) of the first PCD frame is same as that of the first image frame;

determine the first plane based on the selected at least three points; and

extract the first normal to the determined first plane.

12. The calibration system according to claim 11 , wherein the control circuitry is further configured to:

receive a user input; and

select at least three points from the first PCD frame based on the received user input.

13. The calibration system according to claim 1 , wherein

the control circuitry is further configured to transmit the extrinsic calibration parameters to an Electronic Control Unit (ECU) of a vehicle, and

wherein, while the sensor system in mounted on the vehicle, the ECU is configured to calibrate the LiDAR sensor with the camera based on the extrinsic calibration parameters.

14. A calibration system, comprising:

a sensor system that comprises a Light Detection and Ranging (LiDAR) sensor and a camera; and

control circuitry communicatively coupled to the sensor system, wherein the control circuitry is configured to:

receive a plurality of image frames comprising a calibration pattern;

receive a plurality of point cloud data (PCD) frames comprising the calibration pattern;

extract a first normal to a first plane of the calibration pattern in a first PCD frame of the received plurality of PCD frames;

extract a second normal to a second plane of the calibration pattern in a first image frame of the received plurality of image frames;

compute a transform between the extracted first normal and the extracted second normal, wherein the computed transform comprises final values of extrinsic calibration parameters for the sensor system; and

transmit the extrinsic calibration parameters to an Electronic Control Unit (ECU) of a vehicle,

wherein the ECU is configured to calibrate the LiDAR sensor with the camera based on the final values of the extrinsic calibration parameters, and

wherein the LiDAR sensor is calibrated while the sensor system in mounted on the vehicle and the vehicle is in an operational state.

15. A method, comprising:

in a calibration system communicatively coupled to a sensor system comprising a Light Detection and Ranging (LiDAR) sensor and a camera:

receiving a plurality of image frames comprising a calibration pattern;

receiving a plurality of point cloud data (PCD) frames comprising the calibration pattern;

extracting a first normal to a first plane of the calibration pattern in a first PCD frame of the received plurality of PCD frames;

extracting a second normal to a second plane of the calibration pattern in a first image frame of the received plurality of image frames;

computing a transform between the extracted first normal and the extracted second normal, wherein the computed transform comprises final values of extrinsic calibration parameters for the sensor system; and

calibrating the LiDAR sensor with the camera based on the computed transform.

16. The method according to claim 15 , further comprising:

estimating final values of intrinsic calibration parameters of the camera based on the plurality of image frames of the calibration pattern and a canonical image of the calibration pattern; and

calibrating the camera based on the estimated final values of the intrinsic calibration parameters.

17. The method according to claim 16 , further comprising calibrating the LiDAR sensor with the calibrated camera based on the computed transform.

18. The method according to claim 15 , further comprising determining a distance between the calibration pattern and a position of the camera.

19. The method according to claim 18 , further comprising:

initializing values of the extrinsic calibration parameters with seed values;

computing a dot product between the extracted first normal and the extracted second normal based on the initialized values of the extrinsic calibration parameters;

computing a value of an objective function based on the computed dot product and a square of the determined distance; and

updating the initialized values of the extrinsic calibration parameters until the computed value of the objective function is below a threshold,

wherein when the computed value of the objective function is below the threshold, the updated values of the extrinsic calibration parameters are the final values of the extrinsic calibration parameters.

20. The method according to claim 15 , further comprising:

selecting at least three points from a plurality of points representing the calibration pattern in the first PCD frame, wherein a Field-of-View (FOV) of the first PCD frame is same as that of the first image frame;

determining the first plane based on the selected at least three points; and

extracting the first normal to the determined first plane.

Assignments (1)

Cited By (10)

US 12,189,036 US 12,277,800 US 12,323,574 US 12,399,278 US 12,399,279 US 12,400,365 US 12,561,942 US 12,682,488 US 12,694,536 US 12,730,202