Calibration for virtual or augmented reality systems
Techniques for addressing deformations in a virtual or augmented headset described. In some implementations, cameras in a headset can obtain image data at different times as the headset moves through a series of poses of the headset. One or more miscalibration conditions for the headset that have occurred as the headset moved through the series of poses can be detected. The series of poses can be divided into groups of poses based on the one or more miscalibration conditions, and bundle adjustment for the groups of poses can be performed using a separate set of camera calibration data. The bundle adjustment for the poses in each group is performed using a same set of calibration data for the group. The camera calibration data for each group is estimated jointly with bundle adjustment estimation for the poses in the group.
1 . A method comprising:
obtaining image data from a plurality of cameras of a headset, including obtaining image data at different times as the headset moves through a series of poses of the headset;
detecting that one or more miscalibration conditions for the headset have occurred as the headset moved through the series of poses;
dividing the series of poses into groups of poses based on the one or more miscalibration conditions;
performing bundle adjustment for the groups of poses using a separate set of camera calibration data for each group of poses of the groups of poses, wherein the bundle adjustment for a set of poses in each group of poses is performed using a same set of calibration data for each group of poses, and wherein the separate set of camera calibration data for each group of poses is estimated jointly with bundle adjustment estimation for the set of poses in each group of poses, and comprising:
optimizing the separate set of camera calibration data for each group of poses to align the headset to a set of mapping reference points in an augmented reality image, comprising:
applying a first constraint that limits a difference between rotation parameters of the separate set of camera calibration data for two groups of poses of the groups of poses to be less than a rotation threshold; and
applying a second constraint that limits a difference between translation parameters of the separate set of camera calibration data for two groups of poses of the groups of poses to be less than a translation threshold; and
changing one or more of an alignment and an orientation of one or more of the plurality of cameras based on the bundle adjustment to calibrate the headset.
2 . The method of claim 1 , wherein obtaining the image data at different times as the headset moves through the series of poses of the headset comprises:
determining position and orientation values for each camera of the headset relative to a map reference point for each pose of the headset.
3 . The method of claim 1 , wherein obtaining the image data at different times as the headset moves through the series of poses of the headset comprises:
determining a pose of the headset for each selected frame of image data; and
selecting a subset of poses as the series of poses.
4 . The method of claim 1 , wherein detecting that the one or more miscalibration conditions for the headset have occurred comprises one or more of:
determining that the headset has been placed on or taken off of a user's head;
determining that a location of the headset has changed;
detecting a fluctuation in an ambient temperature surrounding the headset;
detecting a fluctuation in a temperature of the headset; and
detecting a mechanical displacement of a portion of the headset.
5 . The method of claim 1 , wherein dividing the series of poses into the groups of poses comprises:
determining a first time that a first miscalibration condition occurred;
identifying all poses of the headset prior to the first time when the first miscalibration condition occurred; and
grouping the poses of the headset prior to the first time into a first pose group.
6 . The method of claim 5 , wherein dividing the series of poses into the groups of poses comprises:
determining a second time that a second miscalibration condition occurred;
identifying all poses of the headset after the first time and prior to the second time when the second miscalibration condition occurred; and
grouping the poses of the headset between the first time and the second time into a second pose group.
7 . The method of claim 6 , wherein a number of poses in the second pose group is different from a number of poses in the first pose group.
8 . The method of claim 1 , wherein performing the bundle adjustment for the groups of poses using the separate set of camera calibration data for each group of poses comprises:
determining a separate extrinsic parameter for each group of poses, the separate extrinsic parameter comprising at least two matrices of rotation and translation parameter values associated with at least two cameras of the headset.
9 . The method of claim 1 , wherein:
the rotation threshold is 2 arc minutes; and
the translation threshold is 3 mm.
10 . The method of claim 1 , comprising:
displaying the augmented reality image at a display of the headset in response to calibration of the headset.
11 . A virtual or augmented reality system, comprising:
a headset comprising a display device configured to display an augmented reality image and cameras configured to obtain image data for rendering the augmented reality image; and
a processor coupled to the headset, the processor configured to:
obtain the image data from the cameras, including image data obtained at different times as the headset moves through a series of poses of the headset;
detect that one or more miscalibration conditions for the headset have occurred as the headset moved through the series of poses;
divide the series of poses into groups of poses based on the one or more miscalibration conditions;
perform bundle adjustment for the groups of poses using a separate set of camera calibration data for each group of poses of the groups of poses, wherein the bundle adjustment for a set of poses in each group of poses is performed using a same set of calibration data for each group of poses, and wherein the separate set of camera calibration data for each group of poses is estimated jointly with bundle adjustment estimation for the set of poses in each group of poses, and comprising:
optimize the separate set of camera calibration data for each group of poses to align the headset to a set of mapping reference points in the augmented reality image, comprising:
apply a first constraint that limits a difference between rotation parameters of the separate set of camera calibration data for two groups of poses of the groups of poses to be less than a rotation threshold; and
apply a second constraint that limits a difference between translation parameters of the separate set of camera calibration data for two groups of poses of the groups of poses to be less than a translation threshold; and
change one or more of an alignment and an orientation of one or more of the cameras based on the bundle adjustment to calibrate the headset.
12 . The virtual or augmented reality system of claim 11 , wherein to obtain the image data at different times as the headset moves through the series of poses of the headset, the processor is configured to:
determine a pose of the headset for each selected frame of image data; and
select a subset of poses as the series of poses.
13 . The virtual or augmented reality system of claim 11 , wherein to divide the series of poses into the groups of poses, the processor is configured to:
determine a first time that a first miscalibration condition occurred;
identify all poses of the headset prior to the first time when the first miscalibration condition occurred;
group the poses of the headset prior to the first time into a first pose group;
determine a second time that a second miscalibration condition occurred;
identify all poses of the headset after the first time and prior to the second time when the second miscalibration condition occurred; and
group the poses of the headset between the first time and the second time into a second pose group.
14 . The virtual or augmented reality system of claim 11 , wherein to perform the bundle adjustment for the groups of poses using the separate set of camera calibration data for each group of poses, the processor is configured to:
determine a separate extrinsic parameter for each group of poses, the separate extrinsic parameter comprising at least two matrices of rotation and translation parameter values associated with at least two cameras of the headset.
15 . A non-transitory computer-readable storage medium comprising instructions, which, when executed by one or more processors, cause the one or more processors to perform operations comprising:
obtaining image data from a plurality of cameras of a headset, including obtaining image data at different times as the headset moves through a series of poses of the headset;
detecting that one or more miscalibration conditions for the headset have occurred as the headset moved through the series of poses;
dividing the series of poses into groups of poses based on the one or more miscalibration conditions;
performing bundle adjustment for the groups of poses using a separate set of camera calibration data for each group of poses of the groups of poses, wherein the bundle adjustment for a set of poses in each group of poses is performed using a same set of calibration data for each group of poses, and wherein the separate set of camera calibration data for each group of poses is estimated jointly with bundle adjustment estimation for the set of poses in each group of poses, and comprising:
optimizing the separate set of camera calibration data for each group of poses to align the headset to a set of mapping reference points in an augmented reality image, comprising:
applying a first constraint that limits a difference between rotation parameters of the separate set of camera calibration data for two groups of poses of the groups of poses to be less than a rotation threshold; and
applying a second constraint that limits a difference between translation parameters of the separate set of camera calibration data for two groups of poses of the groups of poses to be less than a translation threshold; and
changing one or more of an alignment and an orientation of one or more of the plurality of cameras based on the bundle adjustment to calibrate the headset.
16 . The non-transitory computer-readable storage medium of claim 15 , obtaining the image data at different times as the headset moves through the series of poses of the headset comprises:
determining a pose of the headset for each selected frame of image data; and
selecting a subset of poses as the series of poses.
17 . The non-transitory computer-readable storage medium of claim 15 , wherein dividing the series of poses into the groups of poses comprises:
determining a first time that a first miscalibration condition occurred;
identifying all poses of the headset prior to the first time when the first miscalibration condition occurred;
grouping the poses of the headset prior to the first time into a first pose group;
determining a second time that a second miscalibration condition occurred;
identifying all poses of the headset after the first time and prior to the second time when the second miscalibration condition occurred; and
grouping the poses of the headset between the first time and the second time into a second pose group.
18 . The non-transitory computer-readable storage medium of claim 15 , wherein performing the bundle adjustment for the groups of poses using the separate set of camera calibration data for each group of poses comprises:
determining a separate extrinsic parameter for each group of poses, the separate extrinsic parameter comprising at least two matrices of rotation and translation parameter values associated with at least two cameras of the headset.