Lidar to camera calibration for generating high definition maps

1. A non-transitory computer readable storage medium storing instructions for performing calibration of sensors of a vehicle, wherein the instructions when executed by a processor, cause the processor to perform the steps including:

receiving a first lidar scan of a first view comprising a pattern, the first lidar scan captured by a lidar mounted on an autonomous vehicle, wherein the pattern is positioned less that a first threshold distance from the autonomous vehicle;

receiving a first camera image of the first view, the first camera image captured by a camera mounted on the autonomous vehicle;

determining an approximate lidar-to-camera transform based on the first lidar scan of the first view and the first camera image of the first view;

receiving a second lidar scan of a second view comprising the pattern, the second lidar scan captured by the lidar mounted on the autonomous vehicle, wherein the pattern is positioned greater than a second threshold distance from the autonomous vehicle;

receiving, by a camera mounted on the autonomous vehicle, a second camera image of the second view;

determining an accurate lidar-to-camera transform based on the location of the pattern in the second lidar scan and the location of the pattern in the camera image of the second view;

receiving sensor data comprising images received from the camera and lidar scans from the lidar;

generating a high definition map based on the sensor data using the accurate lidar-to-camera transform; and

storing the high definition map in a computer readable storage medium for use in navigating the autonomous vehicle.

2. The non-transitory computer readable storage medium of claim 1 , wherein the instructions when executed by the processor, further cause the processor to perform steps including:

sending signals to the controls of the autonomous vehicle based on the high definition map.

3. The non-transitory computer readable storage medium of claim 1 , wherein instructions for determining the accurate lidar-to-camera transform comprises instructions for:

detecting location of the pattern in the camera image of the second view; and

determining points on the pattern in the second lidar scan based on the approximate lidar-to-camera transform and the points on the pattern in the image of the second view.

4. The non-transitory computer readable storage medium of claim 1 , wherein the camera is a left camera, and the camera image is a left camera image, wherein the instructions when executed by the processor, further cause the processor to perform steps including:

receiving, by a right camera mounted on the autonomous vehicle, a right camera image of the second view;

for a plurality of points on the pattern:

detecting a first location of the point from the left camera image;

detecting a second location of point from the right camera image;

triangulating the first location and the second location to obtain a 3D location of the point in camera coordinates; and

determining 3D coordinates of the point by applying the approximate lidar-to-camera transform to the 3D location of the point;

fitting a dominant plane within the plurality of points; and

adjusting locations of one or more 3D points by projecting the one or more 3D points to the dominant plane.

5. The non-transitory computer readable storage medium of claim 1 , wherein the instructions when executed by the processor, further cause the processor to perform steps including:

determining a bounding polygon of the pattern using the camera image;

projecting a set of 3D points of lidar scan onto the camera image;

identifying a subset of 3D points of the lidar scan such that a projected point corresponding to each of the subset of 3D points is within a threshold of the bounding polygon;

fitting a dominant plane within the subset of 3D points; and

adjusting locations of one or more 3D points by projecting the one or more 3D points to the dominant plane.

6. The non-transitory computer readable storage medium of claim 1 , wherein the instructions when executed by the processor, further cause the processor to perform steps including:

monitoring portions of viewing area of at least one of the camera or lidar that are covered by the pattern in sensor data captured; and

determining a portion of the viewing area that is not covered by the pattern in sensor data captured; and

determining the position of the pattern for subsequently capturing sensor data based on the portion of the viewing area that is not covered by the pattern.

7. The non-transitory computer readable storage medium of claim 6 , wherein the instructions when executed by the processor, further cause the processor to perform steps including:

transmitting information describing the location for placing the pattern corresponding to the determined position.

8. The non-transitory computer readable storage medium of claim 1 , wherein the instructions when executed by the processor, further cause the processor to perform steps including:

determining position of points on the lidar scan based on intensity data in the lidar scan.

9. A method for performing calibration of sensors of a vehicle, the method comprising:

receiving a first lidar scan of a first view comprising a pattern, the first lidar scan captured by a lidar mounted on an autonomous vehicle, wherein the pattern is positioned less that a first threshold distance from the autonomous vehicle;

receiving a first camera image of the first view, the first camera image captured by a camera mounted on the autonomous vehicle;

determining an approximate lidar-to-camera transform based on the first lidar scan of the first view and the first camera image of the first view;

receiving a second lidar scan of a second view comprising the pattern, the second lidar scan captured by the lidar mounted on the autonomous vehicle, wherein the pattern is positioned greater than a second threshold distance from the autonomous vehicle;

receiving, by a camera mounted on the autonomous vehicle, a second camera image of the second view;

determining an accurate lidar-to-camera transform based on the location of the pattern in the second lidar scan and the location of the pattern in the camera image of the second view;

receiving sensor data comprising images received from the camera and lidar scans from the lidar;

generating a high definition map based on the sensor data using the accurate lidar-to-camera transform; and

storing the high definition map in a computer readable storage medium for use in navigating the autonomous vehicle.

10. A computer system comprising:

one or more processors; and

a non-transitory computer readable storage medium storing instructions for performing calibration of sensors of a vehicle, wherein the instructions when executed by a processor, cause the processor to perform the steps including:

receiving a first lidar scan of a first view comprising a pattern, the first lidar scan captured by a lidar mounted on an autonomous vehicle, wherein the pattern is positioned less that a first threshold distance from the autonomous vehicle;

receiving a first camera image of the first view, the first camera image captured by a camera mounted on the autonomous vehicle;

determining an approximate lidar-to-camera transform based on the first lidar scan of the first view and the first camera image of the first view;

receiving a second lidar scan of a second view comprising the pattern, the second lidar scan captured by the lidar mounted on the autonomous vehicle, wherein the pattern is positioned greater than a second threshold distance from the autonomous vehicle;

receiving, by a camera mounted on the autonomous vehicle, a second camera image of the second view;

determining an accurate lidar-to-camera transform based on the location of the pattern in the second lidar scan and the location of the pattern in the camera image of the second view;

receiving sensor data comprising images received from the camera and lidar scans from the lidar;

generating a high definition map based on the sensor data using the accurate lidar-to-camera transform; and

storing the high definition map in a computer readable storage medium for use in navigating the autonomous vehicle.