Vehicular sensor system calibration
The present disclosure provides devices, systems and methods for a vehicular sensor system that detects and classifies objects, and further determining a vanishing point and aspect ratios, which enable detecting a misalignment of a lens focus and determining quality metrics; and calibrating the sensor system.
1 . A method, comprising:
receiving, at a device within a vehicle, visual data captured by a camera that is affixed to the vehicle;
detecting, by the device and based on the visual data, a plurality of instances of a same object and a position of the object in the field of view of the camera for an environment outside of the vehicle;
for each instance of the plurality of instances, determining a ray angle from a vehicle position to the object;
identifying, using a distribution of a plurality of ray angles, a central ray angle and a deviation from the central ray angle;
determining an object class to which the detected object belongs and represents a recognition of a type of the detected object;
generating a plurality of calibration data points corresponding to the ray angle based on the position of the object in the field of view to the vehicle position, wherein a ray angle exhibiting greater deviation from the central ray angle contributes more to calibration of a vehicle sensor system than a ray angle closer to the central ray angle; and
determining at least one calibration parameter of the vehicular sensor system based at least in part on a plurality of calibration data points in a calibration database, wherein the calibration database is configured to adjust the at least one calibration parameter for a sensor of the vehicle sensor system.
2 . The method of claim 1 , further comprising determining whether to store the calibration data points based on a determination that an object class to which the detected object belongs is one of a predetermined set of object classes having an attribute within the environment.
3 . The method of claim 2 , wherein the predetermined set of object classes comprises object classes that tend to be stationary and on or near a road.
4 . The method of claim 2 , wherein the predetermined set of object classes comprises traffic lights, traffic signs, and road markings.
5 . The method of claim 1 , further comprising:
receiving, at the device, the vehicle position from a positional sensor affixed to the vehicle, wherein the vehicular sensor system comprises the camera and the positional sensor;
determining a plurality of vehicle locations, wherein each vehicle location of the plurality of vehicle locations corresponds to a location of the vehicle at a time of a plurality of times when a visual data frame was captured by the camera; and
determining a plurality of object detections, wherein each object detection comprises a position of the object in the field of view of the camera at a time of the plurality of times.
6 . The method of claim 5 , wherein determining whether to store the data point is further based on a property of the plurality of object detections.
7 . The method of claim 6 ,
wherein the property of the plurality of object detections is a property of the plurality of ray angles.
8 . The method of claim 7 , wherein the property of the plurality of ray angles is a range of ray angles to the object, and wherein a calibration data point corresponding to the position of the object is stored in response to a determination that the range of ray angles is above a threshold.
9 . The method of claim 7 ,
wherein the property of the plurality of ray angles is the central ray angle.
10 . The method of claim 9 , wherein a calibration data point corresponding to the position of the object is stored in response to a determination that the corresponding ray angle deviates from the central ray angle.
11 . The method of claim 10 , further comprising:
determining a plurality of deviation values, wherein each deviation value corresponds to a difference between a ray angle of the plurality of ray angles and the central ray angle.
12 . The method of claim 11 ,
wherein each ray angle of the plurality of ray angles is stored along with a weighting parameter, wherein the weighting parameter is based on a corresponding deviation value, and wherein the weighting parameter is increased for ray angles of the plurality of ray angles that have an increased corresponding deviation value.
13 . The method of claim 11 , further comprising:
selecting a subset of ray angles from the plurality of ray angles; wherein a probability of selecting a ray angle is increased for ray angles that have an increased corresponding deviation value; and wherein a data point corresponding to the position of the object is stored in response to a determination that the ray angle belongs to the selected subset.
14 . The method of claim 6 , further comprising:
determining a vanishing point;
determining a distance from each object detection of the plurality of object detections to the vanishing point;
storing a data point corresponding to a position of each object detection in the calibration database, wherein each data point is stored along with a weighting parameter, and wherein the weighting parameter is increased for object detections that are more distant from the vanishing point.
15 . The method of claim 6 , further comprising:
determining a plurality of aspect ratios, each aspect ratio based on a bounding box associated with an object detection of the plurality of object detections; and
determining a variance in the plurality of aspect ratios; wherein the property of the plurality of object detections is the variance in the plurality of aspect ratios, and wherein a data point corresponding to the position of the object is stored in response to a determination that the variance is below a threshold.
16 . The method of claim 6 , wherein the property of the plurality of object detections is a hit rate of object detections, wherein the hit rate is a ratio of a number of object detections to a number of times in the plurality of times when a visual data frame was captured by the camera, and wherein the calibration data point is stored in response to a determination that the hit rate is above a threshold.
17 . The method of claim 5 , wherein determining whether to store the calibration data point is further based on a property of the plurality of vehicle locations.
18 . The method of claim 17 :
wherein the property of the plurality of vehicle locations corresponds to a distance travelled by the vehicle, and wherein the calibration data point corresponding to the position of the object is stored in response to a determination that the distance travelled is above a threshold.
19 . The method of claim 17 :
wherein the property of the plurality of vehicle locations corresponds to a curvature of a trajectory of the vehicle, and wherein the calibration data point corresponding to the position of the object is stored in response to a determination that the curvature is above a threshold.
20 . The method of claim 1 , further comprising:
filtering the calibration database; and wherein the plurality of stored calibration data points on which the determination of the at least one calibration parameter of the vehicular sensor system is based is a plurality of data points in the filtered calibration database.
21 . The method of claim 20 , further comprising:
determining a plurality of uncertainty values; wherein each uncertainty value corresponds to an extent of locations in an extra-vehicular reference frame that are consistent with a plurality of ray angles and vehicular positions that are stored in the calibration database and that correspond to a detected object; and wherein filtering the calibration database comprises ignoring calibration data points associated with uncertainty values that are atypically large with respect to the plurality of uncertainty values.
22 . The method of claim 1 , further comprising:
determining one or more keypoints within the detected object; and wherein the calibration data point corresponds to a keypoint of the one or more determined keypoints.
23 . The method of claim 1 , wherein a calibration parameter of the at least one calibration parameters is an offset between the camera and a positioning sensor, wherein the positioning sensor is affixed to the vehicle.
24 . The method of claim 1 , wherein the at least one calibration parameter is at least one of:
a distance and a direction between a camera and a positional sensor;
a height of the positional sensor when the vehicle is at rest;
a height of the camera when the vehicle is at rest;
a pitch of the camera when the vehicle is at rest;
a roll of the camera when the vehicle is at rest;
a yaw of the camera when the vehicle is at rest; or
an intrinsic parameter of camera.
25 . A computer program product, the computer program product comprising:
a non-transitory computer-readable medium having program code recorded thereon, the program code, when executed by one or more processors, causes the one or more processors to:
detect, by the first processor and based on the visual data, a plurality of instances of a same object and a position of the object in the field of view of the camera for an environment outside of the vehicle;
for each instance of the plurality of instances, determining a ray angle from a vehicle position to the object;
identifying, using a distribution of a plurality of ray angles, a central ray angle and a deviation from the central ray angle;
determining an object class to which the detected object belongs and represents a recognition of a type of the detected object;
generating a plurality of calibration data points corresponding to the ray angle based on the position of the object in the field of view to the vehicle position, wherein a ray angle exhibiting greater deviation from the central ray angle contributes more to calibration of a vehicle sensor system than a ray angle closer to the central ray angle; and
determine at least one calibration parameter of the vehicular sensor system based at least in part on a plurality of calibration data points in a calibration database, wherein the calibration database is configured to adjust the at least one calibration parameter for a sensor of the vehicle sensor system.
26 . The computer program product of claim 25 , wherein the object is an object belonging to a class of objects that tends to be stationary and situated on or near a road.