Efficient augmented reality display calibration
Techniques of augmented reality device calibration are disclosed. In some example embodiments, a system calibrates a visual inertial navigation (VIN) camera of a head mounted display (HMD) device with at least one eye camera of the HMD device to generate multi-camera calibration parameters, calibrating the at eye camera(s) with a display module of the HMD to generate display calibration parameters, calibrating the IMU with the VIN camera to generate VIN calibration parameters, and calibrating the IMU to the display module using the multi-camera calibration parameters, the display calibration parameters, and the VIN calibration parameters.
1. A computer-implemented method comprising:
calibrating multiple cameras of a head mounted display (HMD) device with respect to one another, the multiple cameras comprising at least one eye camera, the at least one eye camera simulating the positioning of at least one human eye within an interior space of the HMD device looking through a display screen of the HMD device to an exterior space of the HMD, the multiple cameras further comprising a visual inertial navigation (VIN) camera, the VIN camera being configured to be used with an inertial measurement unit (IMU) of the HMD device to determine a VIN state, the calibrating of the multiple cameras comprising calculating multi-camera calibration parameters;
calibrating the at least one eye camera with a display module of the HMD, the display module being configured to display AR content on the display screen, the calibrating of the at least one eye camera with the display module comprising calculating display calibration parameters;
calibrating the IMU to the VIN camera, the calibrating of the IMU to the VIN camera comprising calculating VIN calibration parameters;
calibrating, by a machine having a memory and at least one processor, one of a plurality of components of the HMD device with another one of the plurality of components of the HMD device using the multi-camera calibration parameters and the display calibration parameters, the plurality of components comprising the display module and at least one of the multiple cameras, the calibrating of the one of a plurality of components of the HMD device with another one of the plurality of components of the HMD device comprising calibrating the IMU to the display module using the multi-camera calibration parameters, the display calibration parameters, and the VIN calibration parameters;
performing a loop closure process using the HMD device using the VIN calibration parameters; and
validating the VIN calibration parameters based on the loop closure process and ground-truth data provided by a trajectory of a calibration robot.
2. The computer-implemented method of claim 1 , wherein the multiple cameras comprise at least one additional camera other than the VIN camera and the at least one eye camera.
3. The computer-implemented method of claim 1 , wherein the calibrating of the multiple cameras comprises receiving VIN data and eye camera data, the VIN data having been captured by the VIN camera and the eye camera data having been captured by the at least one eye camera while the VIN camera, the at least one eye camera, and the HMD device were physically coupled to a calibration robot, and the calculating of the multi-camera calibration parameters comprises calculating the multi-camera calibration parameters using the VIN data and the eye camera data while the HMD device is not physically coupled to the calibration robot.
4. The computer-implemented method of claim 1 , wherein the calibrating of the at least one eye camera to the display module comprises receiving eye camera data, the eye camera data having been captured by the at least one eye camera while the VIN camera, the at least one eye camera, and the HMD device were physically coupled to a calibration robot, and the calculating of the display calibration parameters comprises calculating the display calibration parameters using the eye camera data while the HMD device is not physically coupled to the calibration robot.
5. The computer-implemented method of claim 1 , wherein the calibrating of the IMU to the VIN camera comprises receiving VIN data, the VIN data having been captured by the VIN camera while the VIN camera and the HMD device were physically coupled to a calibration robot, the calculating of the VIN calibration parameters comprising calculating the VIN calibration parameters using the VIN data while the HMD device is not coupled to the calibration robot.
6. The computer-implemented method of claim 1 , wherein the HMD device comprises a helmet.
7. The computer-implemented method of claim 1 , wherein the display screen is transparent.
8. The computer-implemented method of claim 1 , wherein the display module comprises at least one projector configured to project the AR content onto the display screen.
9. A system comprising:
at least one processor; and
a non-transitory computer-readable medium storing executable instructions that, when executed, cause the at least one processor to perform operations comprising:
calibrating a visual inertial navigation (VIN) camera of a head mounted display (HMD) device to at least one eye camera of the HMD device, the at least one eye camera simulating the positioning of at least one human eye within an interior space of the HMD device looking through a display screen of the HMD device to an exterior space of the HMD, the VIN camera being configured to be used with an inertial measurement unit (IMU) of the HMD device to determine a VIN state, the calibrating of the VIN camera to the at least one eye camera comprising calculating multi-camera calibration parameters;
calibrating the at least one eye camera to a display module of the HMD, the display module being configured to display AR content on the display screen, the calibrating of the at least one eye camera to the display module comprising calculating display calibration parameters;
calibrating the IMU to the VIN camera, the calibrating of the IMU to the VIN camera comprising calculating VIN calibration parameters;
calibrating the IMU to the display module using the multi-camera calibration parameters, the display calibration parameters, and the VIN calibration parameters;
performing a loop closure process using the HMD device using the VIN calibration parameters; and
validating the VIN calibration parameters based on the loop closure process and ground-truth data provided by a trajectory of a calibration robot.
10. The system of claim 9 , wherein the HMD device comprises a helmet.
11. The system of claim 9 , wherein the calibrating of the VIN camera to the at least one eye camera comprises receiving VIN data and eye camera data, the VIN data having been captured by the VIN camera and the eye camera data having been captured by the at least one eye camera while the VIN camera, the at least one eye camera, and the HMD device were physically coupled to a calibration robot, and the calculating of the multi-camera calibration parameters comprises calculating the multi-camera calibration parameters using the VIN data and the eye camera data while the HMD device is not physically coupled to the calibration robot.
12. The system of claim 9 , wherein the calibrating of the at least one eye camera to the display module comprises receiving eye camera data, the eye camera data having been captured by the at least one eye camera while the VIN camera, the at least one eye camera, and the HMD device were physically coupled to a calibration robot, and the calculating of the display calibration parameters comprises calculating the display calibration parameters using the eye camera data while the HMD device is not physically coupled to the calibration robot.
13. The system of claim 9 , wherein the calibrating of the IMU to the VIN camera comprises receiving VIN data, the VIN data having been captured by the VIN camera while the VIN camera and the HMD device were physically coupled to a calibration robot, the calculating of the VIN calibration parameters comprising calculating the VIN calibration parameters using the VIN data while the HMD device is not coupled to the calibration robot.
14. The system of claim 9 , wherein the display screen is transparent.
15. The system of claim 9 , wherein the display module comprises at least one projector configured to project the AR content onto the display screen.
16. A non-transitory machine-readable storage medium storing a set of instructions that, when executed by at least one processor, causes the at least one processor to perform operations comprising:
calibrating a visual inertial navigation (VIN) camera of a head mounted display (HMD) device to at least one eye camera of the HMD device, the at least one eye camera simulating the positioning of at least one human eye within an interior space of the HMD device looking through a display screen of the HMD device to an exterior space of the HMD, the VIN camera being configured to be used with an inertial measurement unit (IMU) of the HMD device to determine a VIN state, the calibrating of the VIN camera to the at least one eye camera comprising calculating multi-camera calibration parameters;
calibrating the at least one eye camera to a display module of the HMD, the display module being configured to display AR content on the display screen, the calibrating of the at least one eye camera to the display module comprising calculating display calibration parameters;
calibrating the IMU to the VIN camera, the calibrating of the IMU to the VIN camera comprising calculating VIN calibration parameters;
calibrating the IMU to the display module using the multi-camera calibration parameters, the display calibration parameters, and the VIN calibration parameters;
performing a loop closure process using the HMD device using the VIN calibration parameters; and
validating the VIN calibration parameters based on the loop closure process and ground-truth data provided by a trajectory of a calibration robot.
17. The non-transitory machine-readable storage medium of claim 16 , wherein the calibrating of the VIN camera to the at least one eye camera comprises receiving VIN data and eye camera data, the VIN data having been captured by the VIN camera and the eye camera data having been captured by the at least one eye camera while the VIN camera, the at least one eye camera, and the HMD device were physically coupled to a calibration robot, and the calculating of the multi-camera calibration parameters comprises calculating the multi-camera calibration parameters using the VIN data and the eye camera data while the HMD device is not physically coupled to the calibration robot.