Automatic generation of virtual materials from real-world materials
Methods for automatically generating a texture exemplar that may be used for rendering virtual objects that appear to be made from the texture exemplar are described. In some embodiments, a head-mounted display device (HMD) may identify a real-world object within an environment, acquire a three-dimensional model of the real-world object, determine a portion of the real-world object from which a texture exemplar is to be generated, capture one or more images of the portion of the real-world object, determine an orientation of the real-world object, and generate the texture exemplar using the one or more images, the three-dimensional model, and the orientation of the real-world object. The HMD may then render and display images of a virtual object such that the virtual object appears to be made from a virtual material associated with the texture exemplar.
1. An electronic device for generating and controlling virtual objects within an augmented reality environment, comprising:
one or more processors configured to acquire one or more images of a portion of a real-world object and acquire a three-dimensional model of the real-world object, the one or more images include a color image and a depth image, the one or more processors configured to update at least a portion of the color image with interpolated pixel values derived from pixel values within the color image based on depth information from the depth image, the one or more processors configured to identify one or more specular reflections within the one or more images and generate a texture exemplar using the one or more images and the three-dimensional model of the real-world object, the one or more processors configured to generate the texture exemplar using the color image and the depth image such that the one or more specular reflections are substantially removed from the texture exemplar and such that distortions caused by a curved surface of the real-world object are corrected, the one or more processors configured to render images of a virtual object using the texture exemplar such that the virtual object appears to be at least partially covered with a virtual material corresponding with the texture exemplar, the one or more processors configured to generate the texture exemplar and render the images of the virtual object in real-time; and
a display in communication with the one or more processors, the display configured to display the rendered images of the virtual object.
2. The electronic device of claim 1 , wherein:
the one or more processors configured to identify one or more pixel values within the color image that are not part of the real-world object using the depth information from the depth image and update the one or more pixel values within the color image that are not part of the real-world object with the interpolated pixel values derived from other pixel values within the color image that are part of the real-world object.
3. The electronic device of claim 1 , wherein:
the one or more processors configured to render the images of the virtual object using texture synthesis techniques.
4. The electronic device of claim 1 , wherein:
the one or more processors configured to automatically generate the texture exemplar and render the images of the virtual object using the texture exemplar during run-time of an application;
the electronic device comprises a head-mounted display device; and
the display comprises a see-through display.
5. The electronic device of claim 4 , wherein:
the one or more processors configured to cause a virtual camera viewfinder to be displayed using the see-through display and acquire a first image of the one or more images corresponding with the virtual camera viewfinder.
6. The electronic device of claim 1 , wherein:
the one or more processors configured to determine an orientation of the real-world object and generating the texture exemplar based on the orientation of the real-world object.
7. The electronic device of claim 1 , wherein:
the texture exemplar comprises a two-dimensional texture image that depicts the virtual material.
8. A method for generating and controlling virtual objects within an augmented reality environment using a mobile device, comprising:
capturing one or more images of a portion of a real-world object using the mobile device, the one or more images include a color image and a depth image;
acquiring a three-dimensional model of the real-world object;
updating at least a portion of the color image with interpolated pixel values derived from pixel values within the color image based on depth information from the depth image;
identifying one or more specular reflections within the one or more images;
generating a texture exemplar using the one or more images and the three-dimensional model of the real-world object, the texture exemplar is generated such that the one or more specular reflections are substantially removed from the texture exemplar and such that distortions caused by a curved surface of the real-world object are corrected; and
rendering and displaying images of a virtual object using the mobile device such that the virtual object appears to be at least partially covered with a virtual material corresponding with the texture exemplar, the generating the texture exemplar and the rendering and displaying images of the virtual object are performed in real-time by the mobile device.
9. The method of claim 8 , wherein:
the generating the texture exemplar includes unwarping the one or more images using the three-dimensional model of the real-world object such that distortions caused by the curved surface of the real-world object are corrected; and
the generating the texture exemplar and the rendering and displaying images of the virtual object are performed during run-time of an application running on the mobile device.
10. The method of claim 8 , further comprising:
identifying one or more pixel values within the color image that are not part of the real-world object using the depth information from the depth image; and
updating the one or more pixel values within the color image that are not part of the real-world object with the interpolated pixel values derived from other pixel values within the color image that are part of the real-world object.
11. The method of claim 8 , wherein:
the rendering and displaying images of the virtual object includes generating the images using texture synthesis techniques.
12. The method of claim 8 , wherein:
the mobile device comprises a head-mounted display device.
13. The method of claim 12 , further comprising:
projecting a virtual camera viewfinder using the head-mounted display device, the capturing one or more images of a portion of the real-world object includes capturing a first image of the one or more images corresponding with the virtual camera viewfinder.
14. The method of claim 8 , further comprising:
determining an orientation of the real-world object, the generating a texture exemplar includes generating the texture exemplar based on the orientation of the real-world object.
15. The method of claim 8 , wherein:
the texture exemplar comprises a two-dimensional texture image that depicts the virtual material.
16. The method of claim 8 , wherein:
the identifying one or more specular reflections includes capturing two or more images of the portion of the real-world object from two or more different angles, aligning the two or more images with each other, and identifying one or more pixels within the two or more images that do not have corresponding pixel values within a threshold pixel value range.
17. The method of claim 8 , further comprising:
determining a three-dimensional shape for the real-world object using surface reconstruction techniques.
18. One or more storage devices containing processor readable code for programming one or more processors to perform a method for generating and controlling virtual objects within an augmented reality environment using a head-mounted display device, the processor readable code comprising:
processor readable code configured to capture one or more images of a portion of a real-world object using the head-mounted display device, the one or more images include a color image and a depth image;
processor readable code configured to update at least a portion of the color image with interpolated pixel values derived from pixel values within the color image based on depth information from the depth image;
processor readable code configured to identify one or more specular reflections within the one or more images;
processor readable code configured to generate a texture exemplar using the one or more images such that the one or more specular reflections are substantially removed from the texture exemplar and such that distortions caused by a curved surface of the real-world object are corrected using the three-dimensional model of the real-world object; and
processor readable code configured to render and display images of a virtual object using the head-mounted display device such that the virtual object appears to be at least partially covered with a virtual material corresponding with the texture exemplar, the generation of the texture exemplar and the display of the images of the virtual object are performed in real-time by the head-mounted display device.