Camera calibration for augmented reality
Approaches herein provide for multi-camera calibration and subsequent application in augmented reality applications. The approach obtains image data of a three-dimensional (3D) calibration object from different directions using the at least one camera. A determination is made for extrinsic parameters associated with a relative position of the 3D calibration object to the at least one camera. A further determination is made for intrinsic parameters associated with the at least one camera relative to the 3D calibration object facing in the different positions. Revised intrinsic parameters are generated from the intrinsic parameters. The revised intrinsic parameters provide a calibration output that may be applied to generate views of an item in an augmented reality application.
1. A computing system, comprising:
at least one processor;
a turntable;
at least one camera; and
a memory device including instructions that, when executed by the at least one processor, enable the computing system to:
obtain, using the at least one camera, image data of a calibration object that comprises planar markers and that is located on the turntable to allow the at least one camera to obtain the image data from different positions of the turntable;
determine a set of parallel planes in 3D space based at least in part on images capture by the at least one camera;
use plane fitting to fit the set of parallel planes to trajectories of optical camera centers of the plurality of cameras;
determine an orientation of the fitted set of parallel planes, the orientation corresponding to an axis of rotation of the turntable;
determine extrinsic parameters comprising a rotation and a translation associated with the planar markers of the calibration object being in the different positions on the turntable, and further comprising the axis of rotation;
determine intrinsic parameters comprising focal length, principal point, and skew for the at least one camera relative to the planar markers of the calibration object being in the different positions;
determine that at least one of the intrinsic parameters exceeds a predetermined threshold;
generate revised intrinsic parameters from one or more of the focal length, the principal point, and the skew of the intrinsic parameters, wherein the revised intrinsic parameters do not exceed the predetermined threshold;
determine calibration output including the revised intrinsic parameters in combination with the extrinsic parameters; and
utilize the calibration output to provide a rendering of a virtual item in an augmented reality view of a live environment.
2. The computing system of claim 1 , wherein the instructions, when executed by the at least one processor, further enable the computing system to:
determine a projection matrix comprising the extrinsic parameters and the intrinsic parameters;
invert a portion of the projection matrix to provide an inverted representation;
perform QR decomposition of the inverted representation to provide parameters, which are modified to generate the revised intrinsic parameters.
3. The computing system of claim 1 , wherein the instructions, when executed by the at least one processor, further enable the computing system to:
modify the principal point so that a modified principal point is located at a center of a calibration image provided in the image data, the modified principal paint used to determine the calibration output.
4. The computing system of claim 1 , wherein the instructions, when executed by the at least one processor, further enable the computing system to:
modify the skew so that a modified skew of zero (0) is used to determine the calibration output.
5. The computing system of claim 1 , wherein the instructions, when executed by the at least one processor, further enable the computing system to:
obtain the image data using at least one RGBD sensor and at least one RGB sensor.
6. The computing system of claim 5 , wherein the instructions, when executed by the at least one processor, further enable the computing system to:
apply the calibration output to calibrate a portion of the augmented reality view associated with the at least one RGBD sensor without affecting portions of the augmented realty view associated with the at least one RGB sensor.
7. A system, comprising:
at least one processor;
at least one camera;
a memory device including instructions that, when executed by the at least one processor, enable the system to:
obtain, using the at least one camera, image data of a three-dimensional (3D) calibration object from different directions;
determine a set of parallel planes in 3D space based at least in part on images capture by a plurality of cameras;
use plane fitting to fit the set of parallel planes to trajectories of optical camera centers of the plurality of cameras;
determine an orientation of the fitted set of parallel planes, the orientation corresponding to an axis of rotation of a turntable providing a resting surface for the 3D calibration object;
determine extrinsic parameters associated with a relative position of features of the 3D calibration object to the at least one camera, and further including the axis of rotation;
determine intrinsic parameters associated with the at least one camera relative to the 3D calibration object facing in the different directions;
generate revised intrinsic parameters from the intrinsic parameters, the revised intrinsic parameters in combination with the extrinsic parameters providing combined parameters; and
provide a calibration output using the combined parameters.
8. The system of claim 7 , wherein the instructions, when executed further enable the system to:
detect the features associated with planar markers on the 3D calibration object;
obtain information for a 3D coordinate system from the features; and
associate two-dimension coordinates for the at least one camera to the 3D coordinate system.
9. The system of claim 7 , wherein the intrinsic parameters comprise at least one of a focal length, a principal point, and a skew, and the extrinsic parameters comprise at least one of position and camera orientation.
10. The system of claim 7 , wherein the different positions are obtained from the 3D calibration object positioned on a turntable, and wherein the image data includes a plurality of images captured from a plurality of cameras positioned with respect to the turntable.
11. The system of claim 7 , wherein the features of the 3D calibration object comprises at least one of a checkerboard style pattern, data matrices, regions of different intensities, color patterns, text, markings, or salient points.
12. The system of claim 7 , wherein the image data includes a plurality of images, and wherein the instructions, when executed further enable the system to:
for individual images of the plurality of images, use a feature detection technique to determine the features of the 3D calibration object, the features associated with position of the 3D calibration object relative to a camera coordinate system.
13. The system of claim 7 , wherein the image data includes a plurality of images, and wherein the instructions, when executed further enable the system to:
analyze the image data to detect the features of the 3D calibration object represented in the image data, the features associated with position of the 3D calibration object relative to a camera coordinate system;
associate the features to known 3D positions of corresponding features of the 3D calibration object in a reference coordinate system; and
use plane fitting to determine a set of parallel 3D planes in the reference coordinate system based at least in part on the known 3D positions.
14. The system of claim 7 , wherein the instructions, when executed further enable the system to:
use shape fitting to fit a sphere to optical camera centers of at least two cameras; and
determine a center of the sphere, the center corresponding to a center of a turntable providing a resting surface for the 3D calibration object, the center providing part of the extrinsic parameters.
15. The system of claim 7 , wherein the instructions, when executed further enable the system to:
use shape fitting to fit a set of concentric circles to optical camera centers of at least two cameras; and
determine a center of the concentric circles, the center corresponding to a center of a turntable providing a resting surface for the 3D calibration object, the center providing part of the extrinsic parameters.
16. The system of claim 4 , wherein the instructions, when executed further enable the system to:
compare the extrinsic parameters and the extrinsic parameters to new intrinsic parameters and new extrinsic parameters to determine an amount of change;
compare the amount of change to a threshold; and
determine whether to use the new intrinsic parameters and the new extrinsic parameters based at least in part on the amount of change.
17. A method, comprising:
obtaining, using the at least one camera, image data of a three-dimensional (3D) calibration object from different directions;
determining a set of parallel planes in 3D space based at least in part on images capture by a plurality of cameras;
using plane fitting to fit the set of parallel planes to trajectories of optical camera centers of the plurality of cameras;
determining an orientation of the fitted set of parallel planes, the orientation corresponding to an axis of rotation of a turntable providing a resting surface for the 3D calibration object;
determining extrinsic parameters associated with a relative position of features of the 3D calibration object to the at least one camera, and further including the axis of rotation;
determining intrinsic parameters associated with the at least one camera relative to the features of the 3D calibration object facing in the different directions;
generating revised intrinsic parameters from the intrinsic parameters, the revised intrinsic parameters in combination with the extrinsic parameters providing combined parameters; and
provide a calibration output using the combined parameters.
18. The method of claim 17 , further comprising:
detecting the features associated with planar markers on the 3D calibration object;
obtaining information for a 3D coordinate system from the features; and
associating two-dimension coordinates for the at least one camera to the 3D coordinate system.
19. The method of claim 17 , wherein the intrinsic parameters comprise at least one of a focal length, a principal point, and a skew, and the extrinsic parameters comprise at least one of position and camera orientation.