Adaptive exposure
Various implementations disclosed herein improve the appearance of captured video by accounting for motion-induced blur, noise, lighting, and/or other factors related to the appearance of video captured by rotating or otherwise moving wearable electronic device. Some implementations reduce or eliminate camera motion blur during video capture by dynamically adjusting the exposure used to capture the video based on user motion. For example, exposure may be reduced to reduce the amount of blur during motion. Since such a reduced exposure captures a lesser amount of light, gain may be increased such that the captured video maintains a relatively constant image brightness in spite of the changing exposure over time.
1 . A method comprising:
at a head-mounted device (HMD) having a processor, one or more cameras, and one or more displays:
detecting a current motion corresponding to a current rotation of the HMD during the capture of multiple frames by the one or more cameras in a physical environment, using a sensor wherein the current rotation is detected using an inertial measurement unit or a gyroscope;
predicting a continued rotation of the device for one or more next frames after the multiple frames based on the current rotation;
determining an exposure parameter to limit the appearance of blur that would result from the continued rotation of the HMD, wherein the exposure parameter is a maximum exposure time resulting in a predetermined level of blur on an image sensor;
adjusting an exposure of the one or more cameras based on the determined exposure parameter;
capturing video comprising the one or more next frames of the physical environment via the one or more cameras of the HMD according to the adjusted exposure; and
presenting a view on the one or more displays of the HMD based on the captured video, the view providing a viewpoint of the physical environment.
2 . The method of claim 1 , wherein the view provides a left-eye viewpoint or a right-eye viewpoint of the physical environment.
3 . The method of claim 1 , wherein the one or more displays comprises a first display to display a left-eye viewpoint and a second display to display a right-eye viewpoint.
4 . The method of claim 1 , wherein the one or more cameras comprise a first camera capturing from a left-eye viewpoint and a second camera capturing from a right-eye viewpoint.
5 . The method of claim 1 further comprising providing pass-through video via the HMD in which:
a first video captured via one or more left-eye outward facing cameras is presented on a left-eye display to provide a live view of a physical environment from the left-eye viewpoint; and
a second video captured via one or more right-eye outward facing cameras is presented on a right-eye display to provide the live view of the physical environment from the right-eye viewpoint.
6 . The method of claim 1 , wherein the view is generated by altering the captured video to provide the view from an eye-specific viewpoint different than a viewpoint from which the video is captured.
7 . He method of claim 1 , wherein the rotation of the HMD corresponds to a rotation of a head upon which the HMD is worn.
8 . The method of claim 1 , wherein the motion corresponds to a change in a six degree of freedom pose of the HMD and a head upon which the HMD is worn.
9 . The method of claim 1 , wherein the maximum exposure time is determined based on the motion and a motion curve corresponding to a constant blur amount.
10 . The method of claim 1 further comprising adjusting a gain based on the adjusting of the exposure to provide an approximately constant image brightness.
11 . The method of claim 1 , wherein the exposure is adjusted based on a minimum exposure determined to ensure noise is less than a threshold.
12 . The method of claim 11 , wherein the minimum exposure is determined based on a measure of light, a tone mapping, or an image sensor histogram.
13 . The method of claim 1 , wherein exposure is adjusted based on a minimum exposure determined based on a persistence of the display, the persistence of the display corresponding to an amount of time each frames is displayed.
14 . The method of claim 1 , wherein the exposure is adjusted based on user gaze direction.
15 . The method of claim 1 , where the exposure is adjusted based on determining whether a gaze direction is tracking a moving object.
16 . The method of claim 1 , wherein the exposure is adjusted based on a distance or contrast of a portion of a physical environment at which a user is gazing.
17 . The method of claim 1 , wherein the exposure is adjusted based on a head motion prediction.
18 . The method of claim 1 , wherein the HMD adjusts the exposure of two imaging components, each of the imaging components providing live video to one or two displays of the HMD device.
19 . The method of claim 1 , wherein the exposure parameter is determined based on determining that the physical environment includes a flickering light source.
20 . The method of claim 19 , wherein the exposure parameter is determined based on selecting an exposure time corresponding to a multiple of a flicker of the flickering light source.
21 . The method of claim 20 further comprising disabling motion-based exposure reduction based on the flickering light source.
22 . A head-mounted device (HMD) comprising:
a motion sensor;
a left-eye display;
a right-eye display;
one or more left-eye outward facing cameras associated with a left-eye viewpoint;
one or more right-eye outward facing cameras associated with a right-eye viewpoint;
a non-transitory computer-readable storage medium; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the HMD to perform operations comprising:
providing pass-through video via the HMD in which video captured via the one or more left-eye outward facing cameras is presented on the left-eye display to provide a live view of a physical environment from the left-eye viewpoint and video captured via the one or more right-eye outward facing cameras is presented on the right-eye display to provide the live view of the physical environment from the right-eye viewpoint;
detecting a current motion corresponding to a current rotation of the HMD during the capture of multiple frames by the one or more cameras in a physical environment, wherein the current rotation is detected using an inertial measurement unit or a gyroscope;
predicting a continued rotation of the device for one or more next frames after the multiple frames based on the current rotation;
determining an exposure parameter to limit the appearance of blur that would result from the continued rotation of the HMD, wherein the exposure parameter is a maximum exposure time resulting in a predetermined level of blur on an image sensor;
adjusting an exposure of the one or more left-eye outward facing cameras and the one or more right-eye outward facing cameras to provide the pass-through video comprising the one or more next frames based on the determined exposure parameter; and
presenting the video on the display of the HMD.
23 . A non-transitory computer-readable storage medium, storing program instructions computer-executable on a computer to perform operations comprising:
detecting a current motion corresponding to a current rotation of the HMD during the capture of multiple frames by the one or more cameras in a physical environment, wherein the current rotation is detected using an inertial measurement unit or a gyroscope;
predicting a continued rotation of the device for one or more next frames after the multiple frames based on the current rotation;
determining an exposure parameter to limit the appearance of blur that would result from the continued rotation of the HMD, wherein the exposure parameter is a maximum exposure time resulting in a predetermined level of blur on an image sensor;
adjusting an exposure of one or more cameras based on the determined exposure parameter;
capturing video comprising the one or more next frames of the physical environment via the one or more cameras of the HMD according to the adjusted exposure; and
presenting a view on the one or more displays of the HMD based on the captured video, the view providing a viewpoint of the physical environment.
24 . The method of claim 1 , further comprising determining a second exposure parameter to ensure noise is less than a threshold, wherein the second exposure parameter is a minimum exposure determined to correspond to a threshold maximum amount of noise approximated using light in the physical environment, wherein the exposure is adjusted based on determined exposure parameter and the determined second exposure parameter.