Geometry modeling of eyewear devices with flexible frames
An eyewear device with flexible frame for Augmented Reality (AR) is disclosed. At least two sensors and a display are mounted on the flexible frame. When in use, the real time geometry of the eyewear device may change from factory calibrated geometry, resulting in low quality AR rendering. A modeling module is provided to model the real time geometry of the eyewear device on the fly using sensor information of the at least two sensors. The modeled real time geometry is then provided to a rendering module to accurately display the AR to the user.
1. An eyewear device for displaying augmented reality images, comprising:
a first sensor;
a second sensor;
at least one image sensor;
at least one display for displaying the augmented reality images, wherein the eyewear device has a predetermined geometry defining spatial relations of at least two of the first sensor, the second sensor, or the at least one display; and
an augmented reality image rendering system that receives inputs from the at least one image sensor and readings from at least one of the first sensor or the second sensor to compute poses of the at least one image sensor and the at least one of the first and second sensors from the inputs and predetermined geometry during use of the eyewear device in an augmented reality application, the augmented reality system further configured to estimate an updated geometry of the eyewear device from the poses as a result of a geometry change of the eyewear device and to render the augmented reality images including virtual content and real-world objects on the at least one display in accordance with the estimated updated geometry.
2. The eyewear device of claim 1 , wherein the augmented reality image rendering system comprises a motion tracking module that implements a computer vision processing algorithm that computes poses of the at least one image sensor and at least one of the first or second sensor the from the estimated updated geometry.
3. The eyewear device of claim 2 , wherein the motion tracking module further provides the poses of the at least one image sensor and the at least one of the first or second sensor to a rendering module to render the augmented reality images.
4. The eyewear device of claim 3 , wherein the rendering module receives real-world and virtual content to be rendered as input and combines the real-world and virtual content for concurrent display on the at least one display.
5. The eyewear device of claim 4 , wherein the rendering module uses the poses to organize and arrange real-world and virtual objects in a same frame of a video sequence for display on the at least one display.
6. The eyewear device of claim 4 , wherein the rendering module receives factory calibration data and displays the real-world and virtual objects according to the calibration data.
7. The eyewear device of claim 3 , wherein the rendering module implements an algorithm that adjusts rendering of the augmented reality images on the at least one display using a bending curve model of a real-time geometry of the eyewear device based on the estimated updated geometry of the eyewear device.
8. The eyewear device of claim 7 , wherein a bending curve of the bending curve model is at least one of asymmetrical, non-smooth, or uneven.
9. The eyewear device of claim 2 , wherein the motion tracking module comprises at least one of an Extended Kalman Filter (EKF)-driven motion tracking module, an optimization-based module that coordinates spatial relation optimization, or a machine learning-driven module that provides motion tracking.
10. The eyewear device of claim 2 , wherein the motion tracking module implements an end-to-end learned approach for tracking and modeling real-time geometry of the eyewear device.
11. A method for displaying augmented reality images on an eyewear device including a first sensor, a second sensor, at least one image sensor, and at least one display for displaying the augmented reality images, the eyewear device having a predetermined geometry defining spatial relations of at least two of the first sensor, the second sensor, or the at least one display, comprising:
receiving inputs from the at least one image sensor and readings from at least one of the first sensor or the second sensor;
computing poses of the at least one image sensor and the at least one of the first and second sensors from the inputs and predetermined geometry of the at least one image sensor and at least one of the first sensor or the second sensor during use of the eyewear device in an augmented reality application;
estimating an updated geometry of the eyewear device from the poses as a result of a geometry change of the eyewear device; and
rendering the augmented reality images including virtual content and real-world objects on the at least one display in accordance with the estimated updated geometry.
12. The method of claim 11 , further comprising receiving real-world and virtual content to be rendered and combining the real-world and virtual content for concurrent display on the at least one display.
13. The method of claim 12 , further comprising organizing and arranging real-world and virtual objects in a same frame of a video sequence for display on the at least one display using the poses.
14. The method of claim 12 , further comprising receiving factory calibration data and displaying the real-world and virtual objects according to the calibration data.
15. The method of claim 11 , further comprising adjusting rendering of the augmented reality images on the at least one display using a bending curve model of a real-time geometry of the eyewear device based on the estimated updated geometry of the eyewear device.
16. The method of claim 15 , wherein a bending curve of the bending curve model is at least one of asymmetrical, non-smooth, or uneven.
17. The method of claim 11 , wherein estimating an updated geometry of the eyewear device from the poses as a result of a geometry change of the eyewear device comprises at least one of:
providing motion tracking of at least one of the first sensor or the second sensor, the at least one image sensor, and the at least one display using at least one of an Extended Kalman Filter (EKF)-driven motion tracking module, an optimization-based module that coordinates spatial relation optimization, or a machine learning-driven module that provides motion tracking; or
implementing an end-to-end learned approach for tracking and modeling a real-time geometry of the eyewear device.
18. The method of claim 11 , wherein estimating an updated geometry of the eyewear device from the poses as a result of a geometry change of the eyewear device comprises receiving and processing at least one of inertial data from at least one of the first sensor or the second sensor or image data from the at least one image sensor to track motion of the at least one of the first sensor or the second sensor, the at least one image sensor, and the at least one display with respect to each other.
19. A non-transitory computer-readable medium comprising instructions stored therein that, when executed by one or more processors, cause the one or more processors to display augmented reality images on an eyewear device including a first sensor, a second sensor, at least one image sensor, and at least one display for displaying the augmented reality images, the eyewear device having a predetermined geometry defining spatial relations of at least two of the first sensor, the second sensor, or the at least one display, by performing operations comprising:
receiving inputs from the at least one image sensor and readings from at least one of the first sensor or the second sensor;
computing poses of the at least one image sensor and the at least one of the first and second sensors from the inputs and predetermined geometry of the at least one image sensor and at least one of the first sensor or the second sensor during use of the eyewear device in an augmented reality application;
estimating an updated geometry of the eyewear device from the poses as a result of a geometry change of the eyewear device; and
rendering the augmented reality images including virtual content and real-world objects on the at least one display in accordance with the estimated updated geometry.
20. The medium of claim 19 , further comprising instructions that, when executed by the one or more processors, further cause the one or more processors to estimate an updated geometry of the eyewear device from the poses as a result of a geometry change of the eyewear device by performing operations comprising at least one of:
providing motion tracking of at least one of the first sensor or the second sensor, the at least one image sensor, and the at least one display using at least one of an Extended Kalman Filter (EKF)-driven motion tracking module, an optimization-based module that coordinates spatial relation optimization, or a machine learning-driven module that provides motion tracking;
implementing an end-to-end learned approach for tracking and modeling a real-time geometry of the eyewear device; or
receiving and processing at least one of inertial data from at least one of the first sensor or the second sensor or image data from the at least one image sensor to track motion of the at least one of the first sensor or the second sensor, the at least one image sensor, and the at least one display with respect to each other.