Warping an input image based on depth and offset information
Various implementations disclosed herein include a method performed at an electronic device including one or more processors, a non-transitory memory, an image sensor, and a display device. The method includes obtaining, via the image sensor, an input image that includes an object. The method includes obtaining depth information characterizing the object, wherein the depth information characterizes a first distance between the image sensor and a portion of the object. The method includes determining a distance warp map for the input image based on a function of the depth information and a first offset value characterizing an estimated distance between eyes of a user and the display device. The method includes setting an operational parameter for the electronic device based on the distance warp map and generating, by the electronic device set to the operational parameter, a warped image from the input image.
1 . A method comprising:
at an electronic device including one or more processors, a non-transitory memory, an image sensor, and a display device:
capturing an input image with the image sensor that is associated with a first field of view from the image sensor that includes an object;
determining, based on an estimated eye location of an eye of a user when the electronic device is worn by the user, a second field of view from the estimated eye location that includes the object and is different from the first field of view;
generating a warped image by warping the input image from the first field of view to the second field of view, wherein the warping is based on:
a first distance between the image sensor and the object;
a second distance between the estimated eye location and the display device; and
a third distance between the display device and the image sensor; and
displaying, on the display device, the warped image towards the estimated eye location.
2 . The method of claim 1 , wherein displaying the warped image towards the estimated eye location includes performing a matting operation on the warped image in order to generate a matted image and displaying the matted image.
3 . The method of claim 2 , wherein performing the matting operation comprises:
determining an image matting warp that is based on a function of the input image and depth information associated with the object; and
generating the matted image by overlaying enhanced reality (ER) content onto the warped image according to the image matting warp.
4 . The method of claim 3 , wherein the ER content includes hole filling content.
5 . The method of claim 3 , wherein the ER content is rendered by a graphics processing unit (GPU).
6 . The method of claim 1 , further comprising:
obtaining depth information associated with the object, wherein the depth information indicates the first distance between the image sensor and the object.
7 . The method of claim 1 , wherein generating the warped image comprises:
setting an operational parameter for the electronic device based on a distance warp map; and
generating the warped image after setting the operational parameter for the electronic device based on the distance warp map.
8 . The method of claim 7 , wherein the operational parameter is associated with the image sensor.
9 . The method of claim 1 , further comprising:
applying a combination of a dilation function and a depth mapping function to depth information associated with the object; and
determining a distance warp map based further on the dilation function and the depth mapping function.
10 . An electronic device comprising:
one or more processors;
a non-transitory memory;
an image sensor;
a display device; and
one or more programs, wherein the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors, the one or more programs including instructions for:
capturing an input image with the image sensor that is associated with a first field of view from the image sensor that includes an object;
determining, based on an estimated eye location of an eye of a user when the electronic device is worn by the user, a second field of view from the estimated eye location that includes the object and is different from the first field of view;
generating a warped image by warping the input image from the first field of view to the second field of view, wherein the warping is based on:
a first distance between the image sensor and the object;
a second distance between the estimated eye location and the display device; and
a third distance between the display device and the image sensor; and
displaying, on the display device, the warped image towards the estimated eye location.
11 . The electronic device of claim 10 , wherein display the warped image includes performing a matting operation on the warped image in order to generate a matted image and displaying the matted image.
12 . The electronic device of claim 11 , wherein performing the matting operation comprises:
determining an image matting warp that is based on a function of the input image and depth information associated with the object; and
generating the matted image by overlaying enhanced reality (ER) content onto the warped image according to the image matting warp.
13 . The electronic device of claim 10 , wherein the one or more programs further include instructions for:
obtaining depth information associated with the object, wherein the depth information indicates the first distance between the image sensor and the object.
14 . The electronic device of claim 10 , wherein generating the warped image comprises:
setting an operational parameter for the electronic device based on a distance warp map; and
generating the warped image after setting the operational parameter for the electronic device based on the distance warp map.
15 . The electronic device of claim 14 , wherein the operational parameter is associated with the image sensor.
16 . The electronic device of claim 10 , wherein the one or more programs further include instructions for:
applying a combination of a dilation function and a depth mapping function to depth information associated with the object; and
determining a distance warp map based further on the dilation function and the depth mapping function.
17 . A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which, when executed by an electronic device with one or more processors, an image sensor, and a display device, cause the electronic device to:
capture an input image with the image sensor that is associated with a first field of view form the image sensor that includes an object;
determine, based on an estimated eye location of an eye of a user when the electronic device is worn by the user, a second field of view from the estimated eye location that includes the object and is different from the first field of view;
generate a warped image by warping the input image from the first field of view to the second field of view, wherein the warping is based on:
a first distance between the image sensor and the object;
a second distance between the estimated eye location and the display device; and
a third distance between the display device and the image sensor;
perform a matting operation on the warped image in order to generate a matted image; and
display the matted image on the display device towards the estimated eye location.
18 . The non-transitory computer readable storage medium of claim 17 , wherein performing the matting operation comprises:
determining an image matting warp that is based on a function of the input image and depth information associated with the object; and
generating the matted image by overlaying enhanced reality (ER) content onto the warped image according to the image matting warp.
19 . The non-transitory computer readable storage medium of claim 17 , wherein the ER content includes hole filling content and the ER content is rendered by a graphics processing unit (GPU).