Head-mounted device with spatially aware camera adjustments
A head-mounted device may include one or more sensors for capturing images of a real-world environment and for determining where the head-mounted device is facing in the real-world environment. The sensors can be used to capture images for creating a lighting map that indicates where one or more light sources are located within the environment and for creating a physical map representing a geometry of the environment. The sensors can be used to predict a direction in which the head-mounted device will be facing in the real-world environment. One of the sensors can include an image sensor for capturing images that are displayed to a user of the head-mounted device. One or more settings of the image sensor can be adjusted based on the lighting map and based on the predicted direction of where the head-mounted device will be facing when capturing a future frame.
1 . A method of operating a head-mounted device in a real-world environment, comprising:
capturing first images of the real-world environment using a first image sensor;
capturing second live images of the real-world environment using a second image sensor;
predicting a direction in which the head-mounted device will be facing in the real-world environment; and
adjusting one or more settings of the first image sensor based on the second live images and the predicted direction of the head-mounted device in the real-world environment.
2 . The method of claim 1 , further comprising:
displaying the first images captured using the first image sensor to an eye box.
3 . The method of claim 2 , wherein the first image sensor has a first field of view and the second image sensor has a second field of view wider than the first field of view.
4 . The method of claim 2 , wherein capturing the first images comprises capturing color images using the first image sensor.
5 . The method of claim 4 , wherein capturing the second images of the real-world environment using the second image sensor comprises capturing monochrome images of the real-world environment using the second image sensor.
6 . The method of claim 1 , wherein the first image sensor is front-facing and the second image sensor is sideways-facing or downward-facing.
7 . The method of claim 1 , wherein predicting a direction in which the head-mounted device will be facing in the real-world environment comprises using one or more motion sensors to determine head pose information associated with the head-mounted device.
8 . The method of claim 1 , wherein adjusting the one or more settings of the first image sensor comprises adjusting an exposure of the first image sensor based on the second live images and the predicted direction of the head-mounted device in the real-world environment.
9 . The method of claim 8 , wherein adjusting the exposure of the first image sensor comprises adjusting an exposure duration of the first image sensor or adjusting a gain of the first image sensor.
10 . The method of claim 1 , wherein adjusting the one or more settings of the first image sensor comprises adjusting a white balance and adjusting a global or local tone mapping for images captured by the first image sensor based on the second live images and the predicted direction of the head-mounted device in the real-world environment.
11 . The method of claim 1 , further comprising:
obtaining a first flicker sensor measurement and a first intensity measurement for a first light source located within the real-world environment;
obtaining a second flicker sensor measurement and a second intensity measurement for a second light source located within the real-world environment;
determining a higher intensity measurement of the first and second intensity measurements; and
adjusting an exposure time of the first image sensor based on the flicker sensor measurement associated with the higher intensity measurement.
12 . The method of claim 1 , further comprising:
obtaining a flicker sensor measurement indicating a flicker period of a light source located within the real-world environment; and
adjusting an exposure time of the first image sensor to a multiple of the flicker period.
13 . The method of claim 1 , further comprising:
obtaining a flicker sensor measurement; and
adjusting an exposure time of the first image sensor based on the flicker sensor measurement.
14 . A head-mounted device operable in a real-world environment containing lighting from one or more light sources, comprising:
a first sensor configured to capture a first image of the real-world environment, wherein the first sensor has a first field of view;
a display configured to present the first image;
a second sensor configured to capture a second image of the lighting within the real-world environment, wherein the second sensor has a second field of view wider than the first field of view;
a third sensor configured to output motion data that is used to predict a direction in which the first sensor will be pointing within the real-world environment; and
control circuitry configured to adjust a setting for the first sensor based on real-time information from the second image captured by the second sensor and the predicted direction of the first sensor within the real-world environment.
15 . The head-mounted device of claim 14 , wherein the second sensor comprises a tracking camera.
16 . The head-mounted device of claim 14 , wherein the first sensor comprises a front-facing camera and the second sensor comprises a sideways-facing camera.
17 . The head-mounted device of claim 14 , wherein the first sensor comprises a color image sensor and wherein the second sensor comprises a monochrome image sensor.
18 . The head-mounted device of claim 14 , wherein the third sensor comprises an inertial measurement unit.
19 . The head-mounted device of claim 14 , further comprising:
an ambient light sensor configured to output an ambient light sensor measurement; and
a flicker sensor configured to output a flicker sensor measurement, wherein the control circuitry is configured to create a lighting map indicating where the light sources are located within the real-world environment based on the second image, the ambient light sensor measurement, and the flicker sensor measurement.