Camera to camera calibration

1. A computer-implemented method comprising:

obtaining first image data captured by a first camera of a vehicle during a given period of operation of the vehicle;

obtaining second image data captured by a second camera of the vehicle during the given period of operation of the vehicle; and

applying optimization to determine a combination of an extrinsics transformation and a time offset that jointly minimizes an aggregated reprojection error determined based on the first and second image data, wherein the optimization involves iteratively:

determining a combination of a respective candidate extrinsics transformation and a respective candidate time offset for which to determine an aggregated reprojection error;

determining a respective reprojection error for each respective first image in a given set of first images from the first image data by (i) identifying a corresponding pair of second images from the second image data that bound the respective first image, (ii) determining a candidate pose for the first camera based on the respective candidate extrinsics transformation, the respective candidate time offset, and estimated poses of the second camera associated with the corresponding pair of second images, and (iii) determining the respective reprojection error for the respective first image based on at least one landmark that is reprojected from at least one of the corresponding pair of second images into the respective first image using the candidate pose for the first camera; and

based on the respective reprojection errors that are determined for the respective first images in the given set of first images, determining the aggregated reprojection error produced by the combination of the respective candidate extrinsics transformation and the respective time offset;

wherein the determined combination of the extrinsics transformation and the time offset is utilized to calibrate a first sensor system comprising the first camera with a second sensor system comprising the second camera.

2. The computer-implemented method of claim 1 , wherein the first sensor system comprises a LiDAR-based sensor system and the second sensor system comprises a camera-based sensor system.

3. The computer-implemented method of claim 1 , wherein:

each image in the first image data has an associated timestamp represented according to a first clock; and

each image in the second image data has an associated timestamp represented according to a second clock that differs from the first clock.

4. The computer-implemented method of claim 1 , wherein, during an initial iteration of the optimization, determining the respective candidate extrinsics transformation comprises:

comparing respective poses of the first and second cameras in relation to a common landmark.

5. The computer-implemented method of claim 1 , wherein, during an initial iteration of the optimization, determining the respective candidate time offset comprises:

determining the respective candidate time offset using the respective candidate extrinsics transformation.

6. The computer-implemented method of claim 1 , wherein determining the candidate pose for the first camera based on the respective candidate extrinsics transformation, the respective candidate time offset, and the estimated poses of the second camera associated with the corresponding pair of second images comprises:

determining the estimated poses of the second camera associated with the corresponding pair of second images;

using the respective candidate time offset to determine an intermediate pose between the estimated poses of the second camera associated with the corresponding pair of second images; and

applying the respective candidate extrinsics transformation to the intermediate pose.

7. The computer-implemented method of claim 6 , using the respective candidate time offset to determine the intermediate pose between the estimated poses of the second camera associated with the corresponding pair of second images comprises:

assuming linear motion of the second camera between the estimated poses.

8. The computer-implemented method of claim 1 , wherein the candidate extrinsics transformation is fixed between iterations of the optimization.

9. A non-transitory computer-readable medium comprising program instructions stored thereon that are executable to cause a computing system to:

obtain first image data captured by a first camera of a vehicle during a given period of operation of the vehicle;

obtain second image data captured by a second camera of the vehicle during the given period of operation of the vehicle; and

apply optimization to determine a combination of an extrinsics transformation and a time offset that jointly minimizes an aggregated reprojection error determined based on the first and second image data, wherein the optimization involves iteratively:

determining a combination of a respective candidate extrinsics transformation and a respective candidate time offset for which to determine an aggregated reprojection error;

determining a respective reprojection error for each respective first image in a given set of first images from the first image data by (i) identifying a corresponding pair of second images from the second image data that bound the respective first image, (ii) determining a candidate pose for the first camera based on the respective candidate extrinsics transformation, the respective candidate time offset, and estimated poses of the second camera associated with the corresponding pair of second images, and (iii) determining the respective reprojection error for the respective first image based on at least one landmark that is reprojected from at least one of the corresponding pair of second images into the respective first image using the candidate pose for the first camera; and

based on the respective reprojection errors that are determined for the respective first images in the given set of first images, determining the aggregated reprojection error produced by the combination of the respective candidate extrinsics transformation and the respective time offset;

wherein the determined combination of the extrinsics transformation and the time offset is utilized to calibrate a first sensor system comprising the first camera with a second sensor system comprising the second camera.

10. The computer-readable medium of claim 9 , wherein the first sensor system comprises a LiDAR-based sensor system and the second sensor system comprises a camera-based sensor system.

11. The computer-readable medium of claim 9 , wherein:

each image in the first image data has an associated timestamp represented according to a first clock; and

each image in the second image data has an associated timestamp represented according to a second clock that differs from the first clock.

12. The computer-readable medium of claim 9 , wherein, during an initial iteration of the optimization, determining the respective candidate extrinsics transformation comprises:

comparing respective poses of the first and second cameras in relation to a common landmark.

13. The computer-readable medium of claim 9 , wherein, during an initial iteration of the optimization, determining the respective candidate time offset comprises:

determining the respective candidate time offset using the respective candidate extrinsics transformation.

14. The computer-readable medium of claim 9 , wherein determining the candidate pose for the first camera based on the respective candidate extrinsics transformation, the respective candidate time offset, and the estimated poses of the second camera associated with the corresponding pair of second images comprises:

determining the estimated poses of the second camera associated with the corresponding pair of second images;

using the respective candidate time offset to determine an intermediate pose between the estimated poses of the second camera associated with the corresponding pair of second images; and

applying the respective candidate extrinsics transformation to the intermediate pose.

15. The computer-readable medium of claim 14 , wherein using the respective candidate time offset to determine the intermediate pose between the estimated poses of the second camera associated with the corresponding pair of second images comprises:

assuming linear motion of the second camera between the estimated poses.

16. The computer-readable medium of claim 9 , wherein the candidate extrinsics transformation is fixed between iterations of the optimization.

17. A computing system comprising:

at least one processor;

at least one non-transitory computer-readable medium; and

program instructions stored on the at least one non-transitory computer-readable medium that are executable by the at least one processor such that the computing system is capable of:

obtaining first image data captured by a first camera of a vehicle during a given period of operation of the vehicle;

obtaining second image data captured by a second camera of the vehicle during the given period of operation of the vehicle; and

applying optimization to determine a combination of an extrinsics transformation and a time offset that jointly minimizes an aggregated reprojection error determined based on the first and second image data, wherein the optimization involves iteratively:

determining a combination of a respective candidate extrinsics transformation and a respective candidate time offset for which to determine an aggregated reprojection error;

determining a respective reprojection error for each respective first image in a given set of first images from the first image data by (i) identifying a corresponding pair of second images from the second image data that bound the respective first image, (ii) determining a candidate pose for the first camera based on the respective candidate extrinsics transformation, the respective candidate time offset, and estimated poses of the second camera associated with the corresponding pair of second images, and (iii) determining the respective reprojection error for the respective first image based on at least one landmark that is reprojected from at least one of the corresponding pair of second images into the respective first image using the candidate pose for the first camera; and

based on the respective reprojection errors that are determined for the respective first images in the given set of first images, determining the aggregated reprojection error produced by the combination of the respective candidate extrinsics transformation and the respective time offset;

wherein the determined combination of the extrinsics transformation and the time offset is utilized to calibrate a first sensor system comprising the first camera with a second sensor system comprising the second camera.

18. The computing system of claim 17 , wherein the first sensor system comprises a LiDAR-based sensor system and the second sensor system comprises a camera-based sensor system.

19. The computing system of claim 17 , wherein determining the candidate pose for the first camera based on the respective candidate extrinsics transformation, the respective candidate time offset, and the estimated poses of the second camera associated with the corresponding pair of second images comprises:

determining the estimated poses of the second camera associated with the corresponding pair of second images;

using the respective candidate time offset to determine an intermediate pose between the estimated poses of the second camera associated with the corresponding pair of second images; and

applying the respective candidate extrinsics transformation to the intermediate pose.

20. The computing system of claim 19 , wherein using the respective candidate time offset to determine the intermediate pose between the estimated poses of the second camera associated with the corresponding pair of second images comprises:

assuming linear motion of the second camera between the estimated poses.