Authentic eye region capture through artificial reality headset
In holographic calling, it is difficult to capture the eyes of a caller due to lighting effects on an artificial reality (XR) headset. However, it can be important to capture the eyes when rendering the caller as they can show emotion, gaze, physical characteristics, etc., that aid in natural communication. Thus, implementations can capture the eyes of the caller using an external image capture device by briefly turning off the lighting effects on the XR headset. Some implementations can trigger the image capture device to capture an image of the eyes by temporal multiplexing in which timers on both the image capture device and the XR headset are synchronized. In other implementations, the image capture device can be an event-based camera that is automatically triggered to capture an image of the eyes based on a detected pixel change caused by deactivation of the lighting effects on the XR headset.
1. A method for capturing an eye region of a user viewing an artificial reality experience on an artificial reality headset, the method comprising:
displaying the artificial reality experience by applying a lighting effect to lenses of the artificial reality headset worn by the user, wherein a view of the eye region of the user is at least partially obscured from an image capture device, separate from and external to the artificial reality headset, during application of the lighting effect to the lenses of the artificial reality headset;
identifying an eye capture trigger event based on a temporally multiplexed timer synchronized between the artificial reality headset and the image capture device; and
in response to the eye capture trigger event, deactivating the lighting effect on the artificial reality headset during a time period,
wherein the image capture device, in response to a corresponding eye capture trigger event based on a corresponding temporally multiplexed timer, captures the eye region of the user through the lenses of the artificial reality headset during the time period, and
wherein the eye region of the user is visible by the image capture device through the lenses of the artificial reality headset during the time period in which the lighting effect, on the artificial reality headset, is deactivated.
2. The method of claim 1 , wherein the time period is less than or equal to 25 milliseconds.
3. The method of claim 1 , wherein the time period in which the lighting effect is deactivated is based on a refresh rate of a display of the artificial reality headset.
4. The method of claim 1 , wherein the artificial reality experience is a 3D holographic call.
5. The method of claim 4 , wherein the user is a sending call participant and the artificial reality headset is a sending-side artificial reality headset, and wherein a receiving-side artificial reality headset renders a hologram of the sending call participant including the captured eye region of the sending call participant.
6. The method of claim 1 , wherein the lighting effect includes tinting of the lenses and/or light output from a display source.
7. A non-transitory computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for capturing an eye region of a user viewing an artificial reality experience on an artificial reality headset, the process comprising:
displaying the artificial reality experience by applying at least one lighting effect to lenses of the artificial reality headset worn by the user, wherein a view of the eye region of the user is at least partially obscured from an image capture device, separate from and external to the artificial reality headset, during application of the at least one lighting effect to the lenses of the artificial reality headset;
identifying an eye capture trigger event based on A) a temporally multiplexed timer synchronized between the artificial reality headset and the image capture device or B) a periodic trigger generated for the artificial reality headset; and
in response to the eye capture trigger event, deactivating the at least one lighting effect during a time period,
wherein the image capture device, in response to a corresponding eye capture trigger event based on A) a corresponding temporally multiplexed timer or B) a detection of the deactivation of the at least one lighting effect, captures the eye region of the user through the lenses of the artificial reality headset during the time period, and
wherein the eye region of the user is visible by the image capture device through the lenses of the artificial reality headset during the time period in which the at least one lighting effect, on the artificial reality headset, is deactivated.
8. The non-transitory computer-readable storage medium of claim 7 , wherein the image capture device is an event-based camera that detects the deactivation of the at least one lighting effect based on a change in at least one detected pixel.
9. The non-transitory computer-readable storage medium of claim 7 , wherein the periodic trigger is generated at equal intervals.
10. The non-transitory computer-readable storage medium of claim 7 , wherein the periodic trigger is based on a detection of a change in the eye region of the user by an internal image capture device integral with the artificial reality headset.
11. The non-transitory computer-readable storage medium of claim 7 , wherein the time period is less than or equal to 25 milliseconds.
12. The non-transitory computer-readable storage medium of claim 7 , wherein the artificial reality experience is a 3D holographic call.
13. The non-transitory computer-readable storage medium of claim 12 , wherein the user is a sending call participant and the artificial reality headset is a sending-side artificial reality headset, and wherein a receiving-side artificial reality headset renders a hologram of the sending call participant including the captured eye region of the sending call participant.
14. The non-transitory computer-readable storage medium of claim 7 , wherein the at least one lighting effect includes at least one of tinting of the lenses and/or light output from a display source.
15. A computing system for capturing an eye region of a user viewing an artificial reality experience on an artificial reality headset by an image capture device separate from and external to the artificial reality headset, the computing system comprising:
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising:
identifying an eye capture trigger event based on A) a temporally multiplexed timer synchronized between the artificial reality headset and the image capture device separate from and external to the artificial reality headset or B) a detection of deactivation of at least one lighting effect; and
in response to the eye capture trigger event, capturing the eye region of the user through lenses of the artificial reality headset during a time period,
wherein the artificial reality headset, in response to a corresponding eye capture trigger event based on A) a corresponding temporally multiplexed timer or B) a periodic trigger generated for the artificial reality headset, deactivates the at least one lighting effect during the time period, and
wherein the eye region of the user is visible by the image capture device through the lenses of the artificial reality headset during the time period in which the at least one lighting effect, on the artificial reality headset, is deactivated.
16. The computing system of claim 15 , wherein the image capture device is an event-based camera that detects the deactivation of the at least one lighting effect based on a change in at least one detected pixel.
17. The computing system of claim 15 , wherein the time period is less than or equal to 25 milliseconds.
18. The computing system of claim 15 , wherein the artificial reality experience is a 3D holographic call.
19. The computing system of claim 18 , wherein the user is a sending call participant and the artificial reality headset is a sending-side artificial reality headset, and wherein a receiving-side artificial reality headset renders a hologram of the sending call participant including the captured eye region of the sending call participant.
20. The computing system of claim 15 , wherein the at least one lighting effect includes at least one of tinting of the lenses, light output from a display source, adjusting gratings, or any combination thereof.