Gaze-driven autofocus camera for mixed-reality passthrough
Particular embodiments are directed to passthrough image generation for a mixed-reality experience. A device may determine, using an eye-tracking system of a head-mounted device, eye-tracking data associated with a user of the head-mounted device. The device may determine, based on the eye-tracking data, a desired scene depth for the user. The device may instruct a first autofocus camera of the head-mounted device to adjust a first focus distance based on the desired scene depth and capture a first image of a real-world environment of the user. The device may generate a first passthrough image based on the first image. The device may display the first passthrough image to a first eye of the user via a first display of the head-mounted device.
1 . A method comprising, by computing system:
determining at a first time, based on eye-tracking data, whether a user is in a vergence movement or a fixating on an object;
instructing, responsive to the determining that the user is fixating on the object, a first autofocus camera of a head-mounted device to implement an autofocusing operation, the autofocusing operation including:
adjusting a first focus distance based on a desired scene depth, and
capturing a first image of a real-world environment of the user that includes the object;
determining at a second time, based on additional eye-tracking data, whether the user is in the vergence movement or fixating on the object; and
instructing, responsive to the determining that the user is in the vergence movement, the first autofocus camera to suspend the autofocusing operation.
2 . The method of claim 1 , further comprising:
instructing a second autofocus camera of the head-mounted device to adjust a second focus distance based on the desired scene depth and capture a second image of the real-world environment of the user;
generating a second passthrough image based on the second image; and
displaying the second passthrough image to a second eye of the user via a second display of the head-mounted device.
3 . The method of claim 1 , further comprising:
generating predicted gazes of the user based on the eye-tracking data;
wherein the desired scene depth is determined based on the predicted gazes.
4 . The method of claim 3 , wherein the predicted gazes are associated with a future time determined based on a predetermined duration corresponding to a latency associated with a pipeline for rendering passthrough images.
5 . The method of claim 3 , wherein the desired scene depth is determined based on a vergence of the predicted gazes of the user.
6 . The method of claim 3 , wherein the desired scene depth is determined based on an intersection between the predicted gazes of the user and a three-dimensional model representation of the real-world environment.
7 . The method of claim 3 , wherein the desired scene depth is determined based on an intersection between the predicted gazes of the user and one or more virtual objects positioned relative to the real-world environment.
8 . The method of claim 1 , wherein the desired scene depth is further determined based on contextual information or an interaction between the user and one or more virtual objects or physical objects.
9 . One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
determine at a first time, based on eye-tracking data, whether a user is in a vergence movement or fixating on an object;
instruct, responsive to the determination that the user is fixating on the object, a first autofocus camera of a head-mounted device to implement an autofocusing operation, the autofocusing operation including:
adjusting a first focus distance based on a desired scene depth, and
capturing a first image of a real-world environment of the user that including the object;
determine at a second time, based on additional eye-tracking data, whether the user is in the vergence movement or fixating on the object; and
instruct, responsive to the determining that the user is in the vergence movement, the first autofocus camera to suspend the autofocusing operation.
10 . The one or more computer-readable non-transitory storage media of claim 9 , wherein the software further operable when executed to:
instruct a second autofocus camera of the head-mounted device to adjust a second focus distance based on the desired scene depth and capture a second image of the real-world environment of the user;
generate a second passthrough image based on the second image; and
display the second passthrough image to a second eye of the user via a second display of the head-mounted device.
11 . The one or more computer-readable non-transitory storage media of claim 9 , wherein the software further operable when executed to:
generate predicted gazes of the user based on the eye-tracking data;
wherein the desired scene depth is determined based on the predicted gazes.
12 . The one or more computer-readable non-transitory storage media of claim 11 , wherein the predicted gazes are associated with a future time determined based on a predetermined duration corresponding to a latency associated with a pipeline for rendering passthrough images.
13 . The one or more computer-readable non-transitory storage media of claim 11 , wherein the desired scene depth is determined based on a vergence of the predicted gazes of the user.
14 . The one or more computer-readable non-transitory storage media of claim 11 , wherein the desired scene depth is determined based on an intersection between the predicted gazes of the user and a three-dimensional model representation of the real-world environment.
15 . A system comprising:
one or more processors; and
one or more computer-readable non-transitory storage media coupled to the one or more processors and comprising instructions operable when executed by the one or more processors to cause the system to:
determine at a first time, based on eye-tracking data, whether a user is in a vergence movement or fixating on an object;
instruct, responsive to the determining that the user is fixating on the object, a first autofocus camera of a head-mounted device to implement an autofocusing operation, the autofocusing operation including:
adjusting a first focus distance based on a desired scene depth, and
capturing a first image of a real-world environment of the user;
determine at a second time, based on additional eye-tracking data, whether the user is in the vergence movement or fixating on the object; and
instruct, responsive to the determining that the user is in the vergence movement, the first autofocus camera to suspend the autofocusing operation.
16 . The system of claim 15 , wherein the instructions are further operable when executed by the one or more processors to:
instruct a second autofocus camera of the head-mounted device to adjust a second focus distance based on the desired scene depth and capture a second image of the real-world environment of the user;
generate a second passthrough image based on the second image; and
display the second passthrough image to a second eye of the user via a second display of the head-mounted device.
17 . The system of claim 15 , wherein the instructions are further operable when executed by the one or more processors to:
generate predicted gazes of the user based on the eye-tracking data;
wherein the desired scene depth is determined based on the predicted gazes.
18 . The system of claim 17 , wherein the predicted gazes are associated with a future time determined based on a predetermined duration corresponding to a latency associated with a pipeline for rendering passthrough images.
19 . The system of claim 17 , wherein the desired scene depth is determined based on a vergence of the predicted gazes of the user.
20 . The system of claim 17 , wherein the desired scene depth is determined based on an intersection between the predicted gazes of the user and a three-dimensional model representation of the real-world environment.