Multi-sensor calibration system
Techniques for performing multi-sensor calibration on a vehicle are described. A method includes obtaining, from each of at least two sensors located on a vehicle, sensor data item of a road comprising a lane marker, extracting, from each sensor data item, a location information of the lane marker, and calculating extrinsic parameters of the at least two sensors based on determining a difference between the location information of the lane marker from each sensor data item and a previously stored location information of the lane marker.
1. A method of estimating parameters for vehicle operation, comprising:
obtaining, from each of at least two sensors located on a vehicle, a sensor data item of a segment of a road comprising a lane marker, wherein point cloud data (PCD) of the segment of the road that comprises the lane marker is obtained from a Light Detection and Ranging (LiDAR) sensor located on the vehicle, wherein the at least two sensors include the LiDAR sensor, and wherein the sensor data item is obtained from each of the at least two sensors as the vehicle is driven on the road;
determining, from each sensor data item, a location information of the lane marker comprising pixel locations of corners of the lane marker;
deriving, for each corner of the lane marker, a first distance between a pixel location of a corner of the lane marker and previously stored 3D world coordinates of the corner of the lane marker;
deriving, for each corner of the lane marker, a second distance between 3D world coordinates of the corner of the lane marker calculated based on a coordinate system of the LiDAR sensor and the previously stored 3D world coordinates of the corner of the lane marker; and
estimating extrinsic parameters for the at least two sensors based on the first distance and the second distance, respectively.
2. The method of claim 1 , wherein the previously stored 3D world coordinates are obtained based on a location of the vehicle.
3. The method of claim 1 , wherein the parameters for the vehicle operation are estimated while the vehicle is operated on the road.
4. The method of claim 1 , wherein the 3D world coordinates of the corner of the lane marker are calculated based on a coordinate system of the LiDAR sensor by:
obtaining a transformed point cloud by correlating the PCD received from the LiDAR sensor to one or more points of a point cloud map reference;
extracting a submap from a previously obtained PCD map;
correlating the transformed point cloud to the submap to obtain a first transformation matrix that describes transformation between the transformed point cloud and the submap;
determining additional transformation matrices that describe transformation between a position of the LiDAR sensor, the one or more points of the point cloud map reference, and a reference point associated with an inertial measurement unit-global navigation satellite system (IMU-GNSS) sensor located in the vehicle; and
calculating 3D world coordinates of the corners of the lane marker in a frame based on the first transformation matrix, the additional transformation matrices, and the previously stored 3D world coordinates of the corners of the lane marker.
5. The method of claim 4 , wherein:
the LiDAR sensor and the IMU-GNSS sensor are located at different positions at the vehicle and/or have different orientations.
6. The method of claim 4 ,
wherein the PCD is correlated to the point cloud map reference based on a measurement provided by the IMU-GNSS sensor and previously stored extrinsic parameters associated with the LiDAR and IMU, and
wherein the previously stored extrinsic parameters associated with the LiDAR and IMU describe a spatial relationship between positions and orientations of the LiDAR sensor and the IMU-GNSS sensor.
7. The method of claim 4 , wherein the submap comprises 3D world coordinates of points within a pre-determined distance of the location of the vehicle when the PCD is received.
8. A non-transitory computer readable storage medium having code stored thereon, the code, when executed by a processor, causing the processor to implement a method of estimating parameters for vehicle operation, the method comprising:
obtaining, from each of at least two sensors located on a vehicle, a sensor data item of a segment of a road comprising a lane marker, wherein point cloud data (PCD) of the segment of the road that comprises the lane marker is obtained from a Light Detection and Ranging (LiDAR) sensor located on the vehicle, wherein the at least two sensors include the LiDAR sensor, and wherein the sensor data item is obtained from each of the at least two sensors as the vehicle is driven on the road;
determining, from each sensor data item, a location information of the lane marker comprising pixel locations of corners of the lane marker;
deriving, for each corner of the lane marker, a first distance between a pixel location of a corner of the lane marker and previously stored 3D world coordinates of the corner of the lane marker;
deriving, for each corner of the lane marker, a second distance between 3D world coordinates of the corner of the lane marker calculated based on a coordinate system of the LiDAR sensor and the previously stored 3D world coordinates of the corner of the lane marker; and
estimating extrinsic parameters for the at least two sensors based on the first distance and the second distance, respectively.
9. The non-transitory computer readable storage medium of claim 8 , wherein the previously stored 3D world coordinates are obtained based on a location of the vehicle.
10. The non-transitory computer readable storage medium of claim 8 , wherein the parameters for the vehicle operation are estimated while the vehicle is operated on the road.
11. The non-transitory computer readable storage medium of claim 8 , wherein the 3D world coordinates of the corner of the lane marker are calculated based on a coordinate system of the LiDAR sensor by:
obtaining a transformed point cloud by correlating the PCD received from the LiDAR sensor to one or more points of a point cloud map reference;
extracting a submap from a previously obtained PCD map;
correlating the transformed point cloud to the submap to obtain a first transformation matrix that describes transformation between the transformed point cloud and the submap;
determining additional transformation matrices that describe transformation between a position of the LiDAR sensor, the one or more points of the point cloud map reference, and a reference point associated with an inertial measurement unit-global navigation satellite system (IMU-GNSS) sensor located in the vehicle; and
calculating 3D world coordinates of the corners of the lane marker in a frame based on the first transformation matrix, the additional transformation matrices, and the previously stored 3D world coordinates of the corners of the lane marker.
12. The non-transitory computer readable storage medium of claim 11 , wherein:
the LiDAR sensor and the IMU-GNSS sensor are located at different positions at the vehicle and/or have different orientations.
13. The non-transitory computer readable storage medium of claim 11 ,
wherein the PCD is correlated to the point cloud map reference based on a measurement provided by the IMU-GNSS sensor and previously stored extrinsic parameters associated with the LiDAR and IMU, and
wherein the previously stored extrinsic parameters associated with the LiDAR and IMU describe a spatial relationship between positions and orientations of the LiDAR sensor and the IMU-GNSS sensor.
14. The non-transitory computer readable storage medium of claim 11 , wherein the submap comprises 3D world coordinates of points within a pre-determined distance of the location of the vehicle when the PCD is received.
15. A system for estimating parameters for vehicle operation, the system comprising a processor configured to:
obtain, from each of at least two sensors located on a vehicle, a sensor data item of a segment of a road comprising a lane marker, wherein point cloud data (PCD) of the segment of the road that comprises the lane marker is obtained from a Light Detection and Ranging (LiDAR) sensor located on the vehicle, wherein the at least two sensors include the LiDAR sensor, and wherein the sensor data item is obtained from each of the at least two sensors as the vehicle is driven on the road;
determine, from each sensor data item, a location information of the lane marker comprising pixel locations of corners of the lane marker;
derive, for each corner of the lane marker, a first distance between a pixel location of a corner of the lane marker and previously stored 3D world coordinates of the corner of the lane marker;
derive, for each corner of the lane marker, a second distance between 3D world coordinates of the corner of the lane marker calculated based on a coordinate system of the LiDAR sensor and the previously stored 3D world coordinates of the corner of the lane marker; and
estimate extrinsic parameters for the at least two sensors based on the first distance and the second distance, respectively.
16. The system of claim 15 , wherein the previously stored 3D world coordinates are obtained based on a location of the vehicle.
17. The system of claim 15 , wherein the parameters for the vehicle operation are estimated while the vehicle is operated on the road.
18. The system of claim 15 , wherein the 3D world coordinates of the corner of the lane marker are calculated based on a coordinate system of the LiDAR sensor by the processor configured to:
obtain a transformed point cloud by correlating the PCD received from the LiDAR sensor to one or more points of a point cloud map reference;
extract a submap from a previously obtained PCD map;
correlate the transformed point cloud to the submap to obtain a first transformation matrix that describes transformation between the transformed point cloud and the submap;
determine additional transformation matrices that describe transformation between a position of the LiDAR sensor, the one or more points of the point cloud map reference, and a reference point associated with an inertial measurement unit-global navigation satellite system (IMU-GNSS) sensor located in the vehicle; and
calculate 3D world coordinates of the corners of the lane marker in a frame based on the first transformation matrix, the additional transformation matrixes matrices, and the previously stored 3D world coordinates of the corners of the lane marker.
19. The system of claim 18 , wherein:
the LiDAR sensor and the IMU-GNSS sensor are located at different positions at the vehicle and/or have different orientations.
20. The system of claim 18 ,
wherein the PCD is correlated to the point cloud map reference based on a measurement provided by the IMU-GNSS sensor and previously stored extrinsic parameters associated with the LiDAR and IMU, and
wherein the previously stored extrinsic parameters associated with the LiDAR and IMU describe a spatial relationship between positions and orientations of the LiDAR sensor and the IMU-GNSS sensor.