3D vision system with automatically calibrated stereo vision sensors and LiDAR sensor

1. An automatically calibrated vision system for an autonomous movable machine, the vision system comprising:

a vision calibration system;

a first sensor system configured to process stereo image data of a scene captured by first and second camera sensors on the movable machine and to output a first map based on the stereo image data; and

a second sensor system configured to process lidar data of the scene captured by a lidar sensor on the movable machine and to output a second map based on the lidar data,

wherein the first and second camera sensors and the lidar sensor are synchronized such that the stereo image data and the lidar data are captured simultaneously,

wherein the vision calibration system is configured to:

compute calibration data by performing a lidar/stereo calibration based on the first and second maps, the lidar/stereo calibration simultaneously minimizing errors based on the first map and errors based on the second map,

supply the calibration data to the first sensor system and the second sensor system for automatic calibration of the first sensor system and automatic calibration of the second sensor system, and

output in real time or nearly real time, to a controller of the movable machine, a calibrated depth map comprising depth measurements calibrated using the calibration data, and

wherein the first sensor system and the second sensor system automatically perform self-calibration using the calibration data, the self-calibration comprising:

calibrating a rotation parameter R of the lidar sensor relative to the first camera sensor,

calibrating a translation parameter T of the lidar sensor relative to the first camera sensor, and

calibrating a stereo yaw angle of the first and second camera sensors.

2. The vision system of claim 1 , wherein:

the first map is a depth map,

the second map is a point cloud, and

the vision calibration system computes the calibration data by:

generating an enhanced image based assigning distance measurements of pixels of the depth map to pixels of a rectified image of the stereo image data,

projecting the point cloud onto the enhanced image, and

determining differences between the point cloud and the enhanced image.

3. The vision system of claim 2 , wherein the calibration data is computed based on an average of the differences between the point cloud and the enhanced image.

4. The vision system of claim 2 , wherein:

the vision calibration system computes the calibration data by converting the differences between the point cloud and the enhanced image into disparities, and

the calibration data is computed based on an average of the disparities.

5. The vision system of claim 1 , wherein the self-calibration comprises calibrating a rotation parameter R of the first and second camera sensors and calibrating a translation parameter T of the first and second camera sensors.

6. The vision system of claim 1 , wherein the self-calibration comprises calibrating a rotation parameter R of the first camera sensor relative to the second stereo camera sensor and/or calibrating a translation parameter T of the first camera sensor relative to the second stereo camera sensor.

7. The vision system of claim 2 , wherein the vision calibration system is configured to:

accumulate a plurality of sets of differences for a plurality of enhanced images and a plurality of point clouds determined from a plurality of sets of lidar data and a plurality of sets of stereo image data, and

compute the calibration data based on an average of the plurality of sets of differences.

8. A non-transitory computer-readable storage medium storing computer-executable code that, when executed by a processing system comprising at least one computer processor, causes the processing system to automatically perform a calibration method to calibrate a vision system for an autonomous movable machine, the calibration method is-comprising:

computing calibration data by performing a lidar/stereo calibration based on a first map obtained from a stereo vision system and a second map obtained from a lidar system, wherein:

the first map corresponds to stereo image data captured by first and second camera sensors of the stereo vision system,

the second map corresponds to lidar data captured by a lidar sensor of the lidar system,

the stereo image data and the lidar data are captured simultaneously, and

the lidar/stereo calibration simultaneously minimizes errors based on the first map and errors based on the second map;

outputting the calibration data to to the stereo vision system and the lidar system for automatic self-calibration of the stereo vision system and the lidar system in real time or nearly real time using the calibration data, the self-calibration comprising:

calibrating a rotation parameter R of the lidar sensor relative to the first camera sensor,

calibrating a translation parameter T of the lidar sensor relative to the first camera sensor, and

calibrating a stereo yaw angle of the first and second camera sensors of the stereo vision system; and

outputting in real time or nearly real time, to a controller of the movable machine, a calibrated depth map comprising depth measurements calibrated using the calibration data.

9. The computer-readable storage medium of claim 8 ,

wherein:

the first map is a depth map,

the second map is a point cloud, and

wherein the calibration data is computed by:

generating an enhanced image based assigning distance measurements of pixels of the depth map to pixels of a rectified image of the stereo image data,

projecting the point cloud onto the enhanced image, and

determining differences between the point cloud and the enhanced image.

10. The computer-readable storage medium of claim 9 , wherein the calibration data is computed based on an average of the differences between the point cloud and the enhanced image.

11. The computer-readable storage medium of claim 9 , wherein:

the calibration data is computed by converting the differences between the point cloud and the enhanced image into disparities, and

the calibration data is based on an average of the disparities.

12. The computer-readable storage medium of claim 9 , wherein the calibration method further comprises:

accumulating a plurality of sets of differences for a plurality of enhanced images and a plurality of point clouds determined from a plurality of sets of lidar data and a plurality of sets of stereo image data, and

computing the calibration data based on an average of the plurality of sets of differences.