Dynamic calibration of multi-camera systems using multiple multi-view image frames
System, apparatus, method, and computer readable media for on-the-fly dynamic calibration of multi-camera platforms using images of multiple different scenes. Image frame sets previously captured by the platform are scored as potential candidates from which new calibration parameters may be computed. The candidate frames are ranked according to their score and iteratively added to the calibration frame set according to an objective function. The selected frame set may be selected from the candidates based on a reference frame, which may be a most recently captured frame, for example. A device platform including a CM and comporting with the exemplary architecture may enhance multi-camera functionality in the field by keeping calibration parameters current. Various computer vision algorithms may then rely upon these parameters, for example.
1. A camera calibration method, comprising:
collecting a plurality of image frame sets, wherein each image frame set comprises two or more image frames of a single scene captured with two or more cameras;
defining one of the plurality as a reference frame set and others of the plurality as candidate frame sets;
scoring the plurality of candidate frame sets;
determining a frame set selection comprising the reference frame set and one or more of the candidate frame sets, the determining based on the scoring and an objective function of error associated with camera calibration parameter values determined from the frame set selection; and
updating a value of one or more camera calibration parameters based, at least in part, on the frame set selection, the updating further comprising executing, on the frame set selection, a calibration routine with all of the frame sets in the frame set selection sharing an extrinsic camera calibration parameter value determined from the reference frame set, and with an intrinsic camera calibration parameter value uniquely determined from each of the frame sets in the frame set selection.
2. The method of claim 1 , wherein determining the frame set selection further comprises:
determining a baseline rectification error on the reference frame set associated with one or more current calibration parameter values; and
iteratively adding to the reference frame set the candidate frame sets sequentially based on the scoring until the camera calibration parameter values determined from the reference frame set and those of the candidate frame sets thus far added result in a rectification error on the reference frame set that is not less than the baseline rectification error.
3. The method of claim 2 , wherein the reference frame set comprises a set of the plurality of image frame sets that has been captured by the cameras most recently in time.
4. The method of claim 2 , wherein the scoring further comprises:
executing a single-frame calibration routine on each of the candidate frame sets independently; and
determining from the single-frame calibration routine at least one of:
a count of feature points in each of the candidate frame sets;
a 3D distribution of the feature points in each of the candidate frame sets for the captured scene; or
a measure of confidence in each of the candidate frame sets.
5. The method of claim 4 , wherein the scoring is based on the count of feature points, and the 3D distribution of the feature points, and the measure of confidence.
6. The method of claim 4 , wherein the scoring is based on at least the 3D distribution of the feature points, and the 3D distribution of the feature points for each of the candidate frame sets is based on a joint histogram of at least one of the candidate frame sets and the reference frame set.
7. The method of claim 6 , wherein the method further comprises determining the joint histogram by binning the feature points according to their location within a 3D scene space divided into cubes or frustums.
8. The method of claim 7 , wherein the scoring is based on the measure of confidence in the candidate frame set; and
the method further comprises determining the measure of confidence for the candidate frame sets by computing a mean rectification error for individual ones of the candidate frame sets.
9. The method of claim 1 , wherein updating the camera calibration parameters further comprises updating intrinsic and extrinsic parameter matrices for at least one of the cameras, and the method further comprises:
executing a computer vision algorithm based on the parameter matrices following the updating; and
outputting to a display one or more attributes computed based on the computer vision algorithm.
10. The method of claim 1 , wherein:
the plurality of frame sets is associated with a capture rate; and
the updating of one or more camera calibration parameters is at a rate at least equal to the capture rate.
11. A camera calibration system, comprising:
one or more camera hardware modules (CM) including two or more cameras to collect a plurality of image frame sets, wherein each image frame set comprises two or more image frames of a single scene captured with the cameras; and
one or more processors coupled to the CM, the processors to:
define one of the plurality as a reference frame set and others of the plurality as candidate frame sets;
determine scores for each of the plurality of candidate frame sets;
determine a frame set selection comprising the reference frame set and one or more of the candidate frame sets based on the scores and an objective function of error associated with camera calibration parameter values determined from the frame set selection;
execute, on the frame set selection, a calibration routine with all of the frame sets in the frame set selection sharing an extrinsic camera calibration parameter value determined from the reference frame set, and with an intrinsic camera calibration parameter value uniquely determined from each of the frame sets in the frame set selection; and
update one or more camera calibration parameter values based, at least in part, on an output of the calibration routine.
12. The system of claim 11 , wherein the processor is to:
determine a baseline rectification error on the reference frame set associated with one or more current calibration parameter values; and
iteratively add, to the reference frame set, the candidate frame sets sequentially based on the scores until the camera calibration parameter values determined from the reference frame set and the candidate frame sets result in a rectification error on the reference frame set that is not less than the baseline rectification error.
13. The system of claim 12 , wherein the reference frame set comprises a set of the plurality of image frames that has been captured by the cameras most recently in time.
14. The system of claim 11 , wherein the processor is to:
execute a single-frame calibration routine on each of the candidate frame sets independently; and
determine from the single-frame calibration routine at least one of:
a count of feature points in each of the candidate frame sets;
a 3D distribution of the feature points in each of the candidate frame sets for the captured scene; or
a measure of confidence in each of the candidate frame sets.
15. The system of claim 14 , wherein the processor is to score the candidate frames sets based on the count of feature points, and the 3D distribution of the feature points, and the measure of confidence.
16. The system of claim 15 , wherein the processor is to determine the 3D distribution of the feature points for each of the candidate frame sets based on a joint histogram of at least one of the candidate frame sets and the reference frame set.
17. The system of claim 16 , wherein the processor is to determine the joint histogram by binning the feature points according to their location within a 3D scene space divided into cubes or frustums.
18. One or more non-transitory computer-readable storage media, with instructions stored thereon, which when executed by a processor, cause the processor to perform a method, comprising:
collecting a plurality of image frame sets, wherein each image frame set comprises two or more image frames of a single scene captured with two or more cameras;
defining one of the plurality as a reference frame set and others of the plurality as candidate frame sets;
scoring the plurality of candidate frame sets;
determining a frame set selection comprising the reference frame set and one or more of the candidate frame sets based on the scoring and an objective function of error associated with camera calibration parameter values determined from the frame set selection; and
updating one or more camera calibration parameter values based, at least in part, on the frame set selection, the updating further comprising executing, on the frame set selection, a calibration routine with all of the frame sets in the frame set selection sharing an extrinsic camera calibration parameter value determined from the reference frame set, and with an intrinsic camera calibration parameter value uniquely determined from each of the frame sets in the frame set selection.