Camera focusing for video passthrough systems

1 . A device, comprising:

a display configured to display virtual content to at least one of a first eye or a second eye of a pair of eyes;

a gaze tracker;

at least one camera configured to capture images of a scene; and

a controller comprising one or more processors configured to:

determine first focus distance information based on gaze information from the gaze tracker and a depth map;

determine second focus distance information based on vergence of a left gaze vector from the first eye and a right gaze vector from the second eye as determined by the gaze tracker, wherein the second focus distance information is determined based on a distance between the pair of eyes and an intersection point, from the vergence, of the first gaze vector and the second gaze vector;

determine a focus distance from the first focus distance information and the second focus distance information; and

direct the camera to focus at the focus distance.

2 . The device as recited in claim 1 , wherein, to determine a focus distance from the first focus distance information and the second focus distance information, the controller is configured to:

model the first focus distance information and the second focus distance information as probability distance functions (PDFs); and

determine the focus distance from the PDFs.

3 . The device as recited in claim 2 , wherein the PDF corresponding to the first focus distance information indicates two or more possible distances, and wherein, to determine the focus distance from the two PDFs, the controller is configured to select one of the two or more distances that most closely corresponds to a distance indicated by the PDF corresponding to the second focus distance information.

4 . The device as recited in claim 1 , wherein, to determine a focus distance from the first focus distance information and the second focus distance information, the controller is configured to:

collect first focus distance observations based on the gaze information from the gaze tracker and the depth map;

collect second focus distance observations based on the vergence of left and right gaze vectors as determined by the gaze tracker;

apply a linear regression function to the collected first and second focus distance observations to generate calibrated vergence distances; and

determine the focus distance from the calibrated vergence distances.

5 . The device as recited in claim 4 , wherein said collect first focus distance observations, said collect second focus distance observations, and said apply a linear regression function are performed during an enrollment process for the device.

6 . The device as recited in claim 4 , wherein the observations are based on real objects in the scene that are imaged by the camera and displayed by the display or virtual objects rendered by the controller and displayed by the display.

7 . The device as recited in claim 1 , wherein, to determine a focus distance from the first focus distance information and the second focus distance information, the controller is configured to:

collect first focus distance observations based on the gaze information from the gaze tracker and the depth map;

collect second focus distance observations based on the vergence of left and right gaze vectors as determined by the gaze tracker;

train a model based on the collected first and second focus distance observations; and

input at least one focus distance observation to the model, wherein the focus distance is output by the model in response to the input.

8 . The device as recited in claim 7 , wherein said collect first focus distance observations, said collect second focus distance observations, and said train a model are performed during an enrollment process for the device.

9 . The device as recited in claim 1 , wherein, to determine a focus distance from the first focus distance information and the second focus distance information, the controller is configured to:

cause display of one or more targets at known ground truth positions;

direct a user to fixate on at least one of the one or more targets;

record left and right gaze vectors for the user while fixated on the targets;

compute intersection points of the left and right gaze vectors and compare the intersection points with the ground truth positions of respective targets;

store results of the comparison as calibrated vergence distances; and

determine the focus distance from the calibrated vergence distances.

10 . The device as recited in claim 1 , wherein the device is a head-mounted device (HMD) of an extended reality (XR) system.

11 . A method, comprising:

performing, by a controller comprising one or more processors:

determining first focus distance information based on gaze information from a gaze tracker and a depth map;

determining second focus distance information based on vergence of a left gaze vector from a first eye of a pair of eyes and a right gaze vector from a second eye of the pair of eyes as determined by the gaze tracker, wherein the second focus distance information includes a distance between the pair of eyes and an intersection point, from the vergence, between the first gaze vector and the second gaze vector;

determining a focus distance from the first focus distance information and the second focus distance information including the distance between the pair of eyes and the intersection point between the first gaze vector and the second gaze vector; and

directing a camera to focus at the focus distance.

12 . The method as recited in claim 11 , wherein determining a focus distance from the first focus distance information and the second focus distance information comprises:

modeling the first focus distance information and the second focus distance information as probability distance functions (PDFs); and

determining the focus distance from the PDFs.

13 . The method as recited in claim 12 , wherein the PDF corresponding to the first focus distance information indicates two or more possible distances, and wherein determining the focus distance from the two PDFs comprises selecting one of the two or more distances that most closely corresponds to a distance indicated by the PDF corresponding to the second focus distance information.

14 . The method as recited in claim 11 , wherein determining a focus distance from the first focus distance information and the second focus distance information comprises:

collecting first focus distance observations based on the gaze information from the gaze tracker and the depth map;

collecting second focus distance observations based on the vergence of left and right gaze vectors as determined by the gaze tracker;

applying a linear regression function to the collected first and second focus distance observations to generate calibrated vergence distances; and

determining the focus distance from the calibrated vergence distances.

15 . The method as recited in claim 14 , wherein the observations are based on real objects in a scene that are imaged by the camera and displayed by a display or virtual objects rendered by the controller and displayed by the display.

16 . The method as recited in claim 11 , wherein determining a focus distance from the first focus distance information and the second focus distance information comprises:

collecting first focus distance observations based on the gaze information from the gaze tracker and the depth map;

collecting second focus distance observations based on the vergence of left and right gaze vectors as determined by the gaze tracker;

training a model based on the collected first and second focus distance observations; and

inputting at least one focus distance observation to the model, wherein the focus distance is output by the model in response to the input.

17 . The method as recited in claim 16 , wherein said collect first focus distance observations, said collect second focus distance observations, and said train a model are performed during an enrollment process for the device.

18 . The method as recited in claim 11 , wherein determining a focus distance from the first focus distance information and the second focus distance information comprises:

displaying one or more targets at known ground truth positions;

directing a user to fixate on at least one of the one or more targets;

recording left and right gaze vectors for the user while fixated on the targets;

computing intersection points of the left and right gaze vectors and compare the intersection points with the ground truth positions of respective targets;

storing results of the comparison as calibrated vergence distances; and

determining the focus distance from the calibrated vergence distances.

19 . The method as recited in claim 11 , wherein the controller, a display, the gaze tracker, and the camera are components of a head-mounted device (HMD) of an extended reality (XR) system.

20 . A system, comprising:

a head-mounted device (HMD), comprising:

a display configured to display virtual content;

a gaze tracker;

at least one camera configured to capture images of a scene; and

a controller comprising one or more processors configured to:

determine first focus distance information based on gaze information from the gaze tracker and a depth map;

determine second focus distance information based on vergence of a left gaze vector from a first eye of a pair of eyes and a right gaze vector from a second eye of the pair of eyes as determined by the gaze tracker, wherein the second focus distance information includes a distance between the pair of eyes and an intersection point, from the vergence, between the first gaze vector and the second gaze vector;

determine a focus distance from the first focus distance information and the second focus distance information including the distance between the pair of eyes and the intersection point between the first gaze vector and the second gaze vector; and

direct the camera to focus at the focus distance.