Sensor calibration

1. A system comprising:

one or more processors; and

computer-readable media storing instructions executable by the one or more processors, wherein the instructions cause the system to perform operations comprising:

determining, as a point pair, a first point in a first image captured by a first camera and a second point in a second image captured by a second camera, the first point and the second point associated with a particular image feature;

determining a projection error associated with the point pair;

determining a subset of a plurality of point pairs based at least in part on the projection error, the subset of the plurality of point pairs including the point pair;

determining, from the subset of the plurality of point pairs, a first correction function representative of a misalignment of an estimated relative positioning of the first and second cameras;

in parallel with at least one of the determining the point pair, the determining the projection error, the determining the subset of the plurality of point pairs, or the determining the first correction function, determining a second correction function representative of a misalignment of at least one of the first camera or the second camera relative to a vehicle; and

calibrating the first camera and the second camera based at least in part on the first correction function and the second correction function.

2. The system of claim 1 , wherein the first image and the second image are captured at substantially a first time, the system further comprising:

a vehicle configured to traverse through an environment and comprising a body upon which the first and second cameras are disposed;

at least one sensor disposed on the body of the vehicle and configured to generate point cloud data for portions of the environment in a field of view of the first and second cameras,

wherein the operations further comprise:

comparing point cloud data captured at the first time to edges determined in at least one image of the first image or second image; and wherein

the determining the second correction function is based on the comparing.

3. The system of claim 1 , wherein the determining the projection error comprises:

determining a fundamental matrix describing a relationship between the first and second images;

determining, for the point pair and based at least in part on the fundamental matrix, an epipolar line corresponding to the second point as represented in the first image; and

comparing the first point to the epipolar line in the first image.

4. The system of claim 1 , wherein

the vehicle is an autonomous vehicle configured to traverse through an environment and comprising a body upon which the first and second cameras are disposed, and

the operations further comprise:

calibrating, based at least in part on the first correction function, additional image data received from the first and second cameras as calibrated data;

generating, based at least in part on the calibrated data, a trajectory to control the autonomous vehicle; and

controlling the autonomous vehicle based at least in part on the trajectory.

5. A method comprising:

determining a point pair based on a first image captured by a first camera and a second image captured by a second camera, the point pair comprising a first point in the first image and a second point in the second image corresponding to the first point;

determining a projection error associated with the point pair;

determining a subset of a plurality of point pairs based at least in part on the projection error, the subset of the plurality of point pairs including the point pair; and

determining, from the subset of the plurality of point pairs, a first calibration function representative of an estimated relative positioning of the first camera and the second camera;

in parallel with at least one of the determining the point pair, the determining the projection error, the determining the subset of the plurality of point pairs, or the determining the first calibration function, determining a second calibration function representative of a misalignment of at least one of the first camera or the second camera relative to a vehicle, and

calibrating the first camera and the second camera based at least in part on the first calibration function and the second calibration function.

6. The method of claim 5 , wherein the determining the projection error comprises:

determining a fundamental matrix;

determining, for point pair and based at least in part on the fundamental matrix, an epipolar line corresponding to the second point;

projecting the epipolar line in the first image; and

determining a distance between the first point and the epipolar line in the first image.

7. The method of claim 5 , wherein the determining the point pair comprises:

extracting a plurality of first features from the first image;

extracting a plurality of second features from the second image;

matching, as matched features, a first feature from the first plurality of features and a second feature of the second plurality of features; and

determining, as point pair, a plurality of pixels associated with the matched features,

wherein the plurality of first features and the plurality of second features comprise one or more of FREAK, AKAZE, FAST, SURF, SIFT, ORB, or BRIEF features.

8. The method of claim 5 , wherein the determining the subset of the plurality of point pairs comprises:

determining point pairs of the plurality of point pairs having projection errors equal to or below a threshold error.

9. The method of claim 5 , wherein the determining the first calibration function comprises:

dividing the first image into a plurality of regions,

associating first points of the plurality of point pairs with the plurality of regions; and

applying, based on the association between the first points of the plurality of point pairs and the plurality of regions, a weighting factor to individual ones of the plurality of point pairs.

10. The method of claim 5 , further comprising:

receiving lidar data comprising a point cloud of an environment captured by a lidar sensor, the point cloud comprising data about a portion of a second field of view of the second image overlapping a first field of view of the first image;

comparing the point cloud to the first image or the second image; and

determining, based at least in part on the comparing, a correction function representative of a misalignment of at least one of the first camera or the second camera and the lidar sensor.

11. The method of claim 10 , further comprising:

detecting edges in the first image or the second image;

determining a plurality of depth discontinuities in the lidar data;

comparing the plurality of depth discontinuities to the edges by projecting the plurality of depth discontinuities into the first image or the second image as a plurality of projected points; and

determining distances between the edges and the plurality of projected points.

12. A non-transitory computer-readable medium storing instructions that, when executed, cause one or more processors to perform operations comprising:

determining a point pair based on a first image captured by a first camera and a second image captured by a second camera, the point pair comprising a first point in the first image and a second point in the second image corresponding to the first point;

determining a projection error for the point pair by:

determining, for a first point in the first image, an epipolar line corresponding to the first point as represented in the second image, and

comparing the second point to the epipolar line in the second image;

determining a subset of a plurality of point pairs based at least in part on the projection error, the subset of the plurality of point pairs including the point pair; and

determining, from the subset of the plurality of point pairs, a calibration function representative of a misalignment of the first camera relative to the second camera;

in parallel with at least one of the determining the point pair, the determining the projection error, determining the subset of the plurality of point pairs, or the determining the first calibration function, determining a second calibration function representative of a misalignment of at least one of the first camera or the second camera relative to a vehicle; and

calibrating the first camera and the second camera based at least in part on the first calibration function and the second calibration function.

13. The non-transitory computer-readable medium of claim 12 , wherein the determining the point pair comprises using computer vision techniques to match a first feature extracted from the first image with a second feature extracted from the second image.

14. The non-transitory computer-readable medium of claim 12 , wherein the determining the subset of the plurality of point pairs comprises:

determining point pairs of the plurality of point pairs having projection errors equal to or below a threshold error.

15. The non-transitory computer-readable medium of claim 12 , wherein the determining the subset of the plurality of point pairs comprises:

dividing the first image into a plurality of regions;

determining positions of the plurality of point pairs relative to the plurality of regions; and

determining, based at least in part on the positions and the projection error, to remove a subset of the plurality of point pairs.

16. The non-transitory computer-readable medium of claim 15 , wherein determining to remove the subset comprises retaining a similar number of points in each region.

17. The non-transitory computer-readable medium of claim 12 , storing further instructions that, when executed, cause the one or more processors to perform operations further comprising:

receiving lidar data comprising a point cloud of an environment captured by a lidar sensor having a third field of view overlapping with a first field of view of the first image and a second field of view of the second image;

comparing the point cloud to the first image or the second image; and

determining, based at least in part on the comparing, a correction function representative of a misalignment of at least one of the first camera or the second camera and the lidar sensor.

18. The non-transitory computer-readable medium of claim 17 , storing further instructions that, when executed, cause the one or more processors to perform operations further comprising:

detecting edges in the first image or the second image;

detecting a plurality of depth discontinuities in the point cloud;

projecting, as a plurality of projected points, the plurality of depth discontinuities into either the first image or the second image;

comparing the plurality of projected points to the edges; and

determining distances between the edges and the plurality of projected points.

19. The non-transitory computer-readable medium of claim 18 , wherein the detecting the edges comprises applying a Canny edge detector to the first image or the second image.

20. The non-transitory computer-readable medium of claim 12 , wherein: the first image and the second image are captured at substantially a first time; the vehicle is configured to traverse through an environment and comprising a body upon which the first and second cameras are disposed; at least one sensor disposed on the body of the vehicle and configured to generate point cloud data for portions of the environment in a field of view of the first and second cameras; and wherein the operations further comprise: comparing point cloud data captured at the first time to edges determined in at least one image of the first image or second image and wherein the determining the second calibration function is based on the comparing.