Lidar to camera calibration for generating high definition maps

1. A non-transitory computer readable storage medium storing instructions for performing calibration of sensors of a vehicle, wherein the instructions when executed by a processor, cause the processor to perform the steps including:

receiving a close-up sensor data of an object displaying a calibration pattern, the close-up sensor data comprising a camera image and a lidar scan of the object positioned less that than a threshold distance from the vehicle;

determining an approximate lidar-to-camera transform based on the close-up sensor data;

receiving a distant sensor data of the object, the distant sensor data comprising a camera image and a lidar scan of the object positioned greater than the threshold distance from the vehicle;

determining an accurate lidar-to-camera transform based on the distant sensor data and the approximate lidar-to-camera transform; and

using the accurate lidar-to-camera transform for calibrating sensor data captured by the vehicle.

2. The non-transitory computer readable storage medium of claim 1 , wherein instructions for determining the accurate lidar-to-camera transform comprises instructions for:

determining points on the object in the lidar scan of the distant sensor data based on:

the approximate lidar-to-camera transform, and

the points on the object in the camera image of the distant sensor data.

3. The non-transitory computer readable storage medium of claim 1 , wherein the instructions when executed by the processor, further cause the processor to perform steps including:

sending signals to the controls of the vehicle based on a high definition map.

4. The non-transitory computer readable storage medium of claim 1 , wherein the camera is a left camera, and the camera image is a left camera image, wherein the instructions when executed by the processor, further cause the processor to perform steps including:

receiving a right camera image of the object, the right camera image captured by a right camera mounted on the vehicle; and

for each point from a plurality of points on the object:

obtaining a 3D location of the point by triangulating a location of the point obtained from the left camera image and a location of the point obtained from the right camera image; and

determining 3D coordinates of the point by applying the approximate lidar-to-camera transform to the 3D location of the point.

5. The non-transitory computer readable storage medium of claim 4 , wherein the instructions further cause the processor to perform steps including:

fitting a dominant plane within the plurality of points; and

adjusting 3D coordinates of one or more points by projecting the 3D coordinates of one or more points to the dominant plane.

6. The non-transitory computer readable storage medium of claim 1 , wherein the instructions for determining the accurate lidar-to-camera transform cause the processor to perform steps including:

determining a bounding polygon of the calibration pattern of the object using the camera image of the distant sensor data;

projecting a set of 3D points of lidar scan of the distant sensor data onto the camera image of the distant sensor data; and

identifying a subset of 3D points of the lidar scan of the distant sensor data such that a projected point corresponding to each of the subset of 3D points is within a threshold of the bounding polygon.

7. The non-transitory computer readable storage medium of claim 6 , wherein the instructions for determining the accurate lidar-to-camera transform further cause the processor to perform steps including:

fitting a dominant plane within the subset of 3D points; and

adjusting locations of one or more 3D points by projecting the one or more 3D points to the dominant plane.

8. The non-transitory computer readable storage medium of claim 1 , wherein the instructions when executed by the processor, further cause the processor to perform steps including:

monitoring portions of a viewing area of at least one of the camera or lidar that are covered by the object in the distant sensor data captured; and

determining a portion of the viewing area that is not covered by the object in the distant sensor data captured; and

determining the position of the object for subsequently capturing distant sensor data based on the portion of the viewing area that is not covered by the object.

9. The non-transitory computer readable storage medium of claim 8 , wherein the instructions when executed by the processor, further cause the processor to perform steps including:

transmitting information describing the location for placing the object corresponding to the determined position.

10. The non-transitory computer readable storage medium of claim 1 , wherein the instructions when executed by the processor, further cause the processor to perform steps including:

determining positions of points on the lidar scan based on intensity data in the lidar scan.

11. A computer system comprising:

a processor; and

a non-transitory computer readable storage medium storing instructions for

performing calibration of sensors of a vehicle, wherein the instructions when executed by a processor, cause the processor to perform the steps including: receiving a close-up sensor data of an object displaying a calibration pattern,

the close-up sensor data comprising a camera image and a lidar scan of

the object positioned less that than a threshold distance from the vehicle;

determining an approximate lidar-to-camera transform based on the close-up sensor data;

receiving a distant sensor data of the object, the distant sensor data comprising a camera image and a lidar scan of the object positioned greater than the threshold distance from the vehicle;

determining an accurate lidar-to-camera transform based on the distant sensor data and the approximate lidar-to-camera transform; and

using the accurate lidar-to-camera transform for calibrating sensor data captured by the vehicle.

12. The computer system of claim 11 , wherein instructions for determining the accurate lidar-to-camera transform comprises instructions for:

determining points on the object in the lidar scan of the distant sensor data based on: the approximate lidar-to-camera transform, and the points on the object in the camera image of the distant sensor data.

13. The computer system of claim 11 , wherein the instructions when executed by the processor, further cause the processor to perform steps including:

sending signals to the controls of the vehicle based on a high definition map.

14. The computer system of claim 11 , wherein the camera is a left camera, and the camera image is a left camera image, wherein the instructions when executed by the processor, further cause the processor to perform steps including:

receiving a right camera image of the object, the right camera image captured by a right camera mounted on the vehicle; and

for each point from a plurality of points on the object:

obtaining a 3D location of the point by triangulating a location of the point obtained from the left camera image and a location of the point obtained from the right camera image; and

determining 3D coordinates of the point by applying the approximate lidar-to-camera transform to the 3D location of the point.

15. The computer system of claim 14 , wherein the instructions further cause the processor to perform steps including:

fitting a dominant plane within the plurality of points; and

adjusting 3D coordinates of one or more points by projecting the 3D coordinates of one or more points to the dominant plane.

16. The computer system of claim 11 , wherein the instructions for determining the accurate lidar-to-camera transform cause the processor to perform steps including:

determining a bounding polygon of the calibration pattern of the object using the camera image of the distant sensor data;

projecting a set of 3D points of lidar scan of the distant sensor data onto the camera image of the distant sensor data; and

identifying a subset of 3D points of the lidar scan of the distant sensor data such that a projected point corresponding to each of the subset of 3D points is within a threshold of the bounding polygon.

17. The computer system of claim 16 , wherein the instructions the instructions for determining the accurate lidar-to-camera transform cause the processor to perform steps including:

fitting a dominant plane within the subset of 3D points; and

adjusting locations of one or more 3D points by projecting the one or more 3D points to the dominant plane.

18. The computer system of claim 11 , wherein the instructions when executed by the processor, further cause the processor to perform steps including:

monitoring portions of a viewing area of at least one of the camera or lidar that are covered by the object in the distant sensor data captured; and

determining a portion of the viewing area that is not covered by the object in the distant sensor data captured; and

determining the position of the object for subsequently capturing distant sensor data based on the portion of the viewing area that is not covered by the object.

19. The computer system of claim 18 , wherein the instructions when executed by the processor, further cause the processor to perform steps including:

transmitting information describing the location for placing the object corresponding to the determined position.

20. The computer system of claim 11 , wherein the instructions when executed by the processor, further cause the processor to perform steps including:

determining positions of points on the lidar scan based on intensity data in the lidar scan.

21. A method for performing calibration of sensors of a vehicle, the method comprising:

receiving a close-up sensor data of an object displaying a calibration pattern, the close-up sensor data comprising a camera image and a lidar scan of the object positioned less that than a threshold distance from the vehicle;

determining an approximate lidar-to-camera transform based on the close-up sensor data;

receiving a distant sensor data of the object, the distant sensor data comprising a camera image and a lidar scan of the object positioned greater than the threshold distance from the vehicle;

determining an accurate lidar-to-camera transform based on the distant sensor data and the approximate lidar-to-camera transform; and

using the accurate lidar-to-camera transform for calibrating sensor data captured by the vehicle.

22. The method of claim 21 , further comprising:

determining points on the object in the lidar scan of the distant sensor data based on:

the approximate lidar-to-camera transform, and the points on the object in the camera image of the distant sensor data.

23. The method of claim 21 , further comprising:

sending signals to the controls of the vehicle based on a high definition map.

24. The method of claim 21 , wherein the camera is a left camera, and the camera image is a left camera image, the method further comprising:

receiving a right camera image of the object, the right camera image captured by a right camera mounted on the vehicle; and

for each point from a plurality of points on the object:

obtaining a 3D location of the point by triangulating a location of the point obtained from the left camera image and a location of the point obtained from the right camera image; and

determining 3D coordinates of the point by applying the approximate lidar-to-camera transform to the 3D location of the point.

25. The method of claim 24 , further comprising:

fitting a dominant plane within the plurality of points; and

adjusting 3D coordinates of one or more points by projecting the 3D coordinates of one or more points to the dominant plane.

26. The method of claim 21 , wherein determining the accurate lidar-to-camera transform comprises:

determining a bounding polygon of the calibration pattern of the object using the camera image of the distant sensor data;

projecting a set of 3D points of lidar scan of the distant sensor data onto the camera image of the distant sensor data; and

identifying a subset of 3D points of the lidar scan of the distant sensor data such that a projected point corresponding to each of the subset of 3D points is within a threshold of the bounding polygon.

27. The method of claim 26 , wherein determining the accurate lidar-to-camera transform further comprises:

fitting a dominant plane within the subset of 3D points; and

adjusting locations of one or more 3D points by projecting the one or more 3D points to the dominant plane.

28. The method of claim 21 , further comprising:

monitoring portions of a viewing area of at least one of the camera or lidar that are covered by the object in the distant sensor data captured; and

determining a portion of the viewing area that is not covered by the object in the distant sensor data captured; and

determining the position of the object for subsequently capturing distant sensor data based on the portion of the viewing area that is not covered by the object.

29. The method of claim 28 , further comprising:

transmitting information describing the location for placing the object corresponding to the determined position.

30. The method of claim 21 , further comprising:

determining positions of points on the lidar scan based on intensity data in the lidar scan.