Rendering three-dimensional models on mobile devices
Images of various views of objects can be captured. An object mesh structure can be created based at least in part on the object images. The object mesh structure represents the three-dimensional shape of the object. Alpha masks indicating which pixels are associated with the object can be used to refine the object mesh structure. A request can be made to view the object from an arbitrary viewpoint which differs from the viewpoints associated with the captured images. A subset of the captured images can be used to create a synthetic image. Different weights can be assigned to the captured image to render a synthetic image that represents the view from the arbitrary viewpoint selected. The input images for the synthetic image can be prefetched, or loaded into memory before the arbitrary view is requested. The images can also be cached for future use or to avoid reloading them for another synthetic image.
1. A computer-implemented method, comprising:
under the control of one or more computer systems configured with executable instructions,
receiving a plurality of images of an object;
receiving a request to view the object portrayed from a first arbitrary view;
determining a first subset of the plurality of images based at least in part on image view alignment of the images to the first arbitrary view;
creating a plurality of alpha masks based at least in part on the plurality of images;
creating a first synthetic image portraying the object from the first arbitrary view using at least the first subset of the plurality of images;
predicting that a second arbitrary view will be requested at a future time, after portrayal of the object from the first arbitrary view;
pre-fetching a second subset of the plurality of images different from the first subset, based at least in part on a prediction that the second arbitrary view will be requested; and
creating a three-dimensional object mesh structure overlaid with at least the first synthetic image and based at least in part on the plurality of images and the plurality of alpha masks; and
overlaying, based in part on manipulation of the three-dimensional object mesh structure, a second synthetic image generated using the second subset of the plurality of images corresponding to the second arbitrary view.
2. The computer-implemented method of claim 1 , further comprising:
displaying, on a mobile device, the first synthetic image overlaying the three-dimensional object mesh structure.
3. The computer-implemented method of claim 2 , the method further comprising:
receiving a second request to view the second arbitrary view of the object;
determining the second subset of the plurality of images based on the second arbitrary view, wherein the second subset of the plurality of images includes at least one image in the first subset of the plurality of images; and
loading, into memory of the mobile device, images in the second subset of the plurality of images and not in the first subset of the plurality of images.
4. The computer-implemented method of claim 1 , further comprising:
generating a weighted composite of the first subset of the plurality of images, respective weights being determined for individual images of the first subset of the plurality of images, in the weighted composite,
wherein the weights for the individual images in the first subset of the plurality of images are determined at least in part based on a cosine of the angle between respective individual captured views and the first arbitrary view.
5. The computer-implemented method of claim 1 , further comprising:
generating a weighted composite of the first subset of the plurality of images, respective weights being determined for individual images of the first subset of the plurality of images, in the weighted composite,
wherein the weights for the individual images in the first subset of the plurality of images are determined at least in part based on visibility of a reference point in the individual images.
6. The computer-implemented method of claim 1 , wherein the plurality of object images include pose information, the pose information comprising:
a camera location component including a three-dimensional description of a location associated with a capturing camera for individual object images; and
a camera orientation component including a three-dimensional description of an orientation for the capturing camera for individual object images.
7. A system, comprising:
at least one processor;
memory including instructions that, when executed by the at least one processor, cause the system to:
receive a plurality of images of an object;
receive a request to view the object portrayed from a first arbitrary view;
create a plurality of alpha masks based at least in part on the plurality of images;
determine a first subset of the plurality of images based at least in part on image view alignment of the images to the first arbitrary view;
load, into memory of a mobile device, the first subset of the plurality of images;
create a first synthetic image portraying the object from the first arbitrary view using at least the first subset of the plurality of images;
predict that a second arbitrary view will be requested at a future time, after portrayal of the object from the first arbitrary view;
load, into the memory of the mobile device, a second subset of the plurality of images different from the first subset, based at least in part on a prediction that the second arbitrary view will be requested;
create a three-dimensional object mesh structure overlaid with at least the first synthetic image and based at least in part on the plurality of images and the plurality of alpha masks; and
overlay, based in part on manipulation of the three-dimensional object mesh structure, a second synthetic image generated using the second subset of the plurality of images corresponding to the second arbitrary view.
8. The system of claim 7 , wherein the memory further includes instructions that, when executed by the at least one processor, cause the system to:
display, on the mobile device, the first synthetic image overlaying the three-dimensional object mesh structure.
9. The system of claim 8 , wherein the memory further includes instructions that, when executed by the at least one processor, cause the system to:
receive a second request to view the second arbitrary view of the object; and
determine the second subset of the plurality of images based on the second arbitrary view, wherein the second subset of the plurality of images includes at least one image in the first subset of the plurality of images; and
load, into memory of the mobile device, images in the second subset of the plurality of images and not in the first subset of the plurality of images.
10. The system of claim 8 , wherein the memory further includes instructions that, when executed by the at least one processor, cause the system to:
project the three-dimensional object mesh structure onto the first synthetic image; and
modify the first synthetic image based on the projection of the three-dimensional object mesh structure onto the first synthetic image.
11. The system of claim 8 , wherein the memory further includes instructions that, when executed by the at least one processor, cause the system to:
determine that one or more pixels are within an object border based at least in part on the plurality of alpha masks; and
modify the first synthetic image based at least in part on determining that the one or more pixels are within the object border.
12. The system of claim 7 , wherein the memory further includes instructions that, when executed by the at least one processor, cause the system to:
receive a request to view the second arbitrary view of the object; and
evict, from memory of the mobile device, at least one image based at least in part on the request to view the second arbitrary view of the object.
13. The system of claim 7 , wherein the memory further includes instructions that, when executed by the at least one processor, cause the system to:
generate a weighted composite of the first subset of the plurality of images, respective weights being determined for individual images of the first subset of the plurality of images, in the weighted composite,
wherein the weights for the individual images in the first subset of the plurality of images are determined at least in part based on a cosine of the angle between respective individual captured views and the first arbitrary view.
14. The system of claim 13 , wherein the memory further includes instructions that, when executed by the at least one processor, cause the system to:
determine a change in position or orientation of the mobile device;
modify the weights for individual images in the first subset of the plurality of images based on the change in position or orientation of the mobile device;
create a second synthetic image based at least in part on the modified weights; and
display, on the mobile device, the second synthetic image.
15. The system of claim 7 , wherein the memory further includes instructions that, when executed by the at least one processor, cause the system to:
load, into memory of the mobile device, at least one image of the first subset of the plurality of images before receiving the request to view the object from the first arbitrary view.
16. A non-transitory computer readable medium, having stored thereon instructions that, when executed by a processor, cause the processor to:
receive a plurality of captured views of an object;
receive a request to portray a first arbitrary view of the object;
determine a first subset of the plurality of captured views based at least in part on image view alignment of the captured views to the first arbitrary view;
create a plurality of alpha masks based at least in part on the plurality of captured views;
create a first rendering portraying the object from the first arbitrary view using at least the first subset of the plurality of captured views;
predict that a second arbitrary view will be requested at a future time, after portrayal of the object from the first arbitrary view;
pre-fetch a second subset of the plurality of captured views different from the first subset, based at least in part on a prediction that the second arbitrary view will be requested; and
create a three-dimensional object mesh structure overlaid with at least the first synthetic image and based at least in part on the plurality of captured views and the plurality of alpha masks; and
overlay, based in part on manipulation of the three-dimensional object mesh structure, a second synthetic image generated using the second subset of the plurality of images corresponding to the second arbitrary view.
17. The non-transitory computer readable medium of claim 16 , wherein the instructions further cause the processor to:
display, on a mobile device, the first rendering overlaying the three-dimensional object mesh structure.
18. The non-transitory computer readable medium of claim 16 , wherein the instructions further cause the processor to:
receive a second request to view the second arbitrary view of the object;
determine the second subset of the plurality of captured views based on the second arbitrary view, wherein the second subset of the plurality of captured views includes at least one view in the first subset of the plurality of captured views; and
load, into memory of the mobile device, captured views in the second subset of the plurality of captured views and not in the first subset of the plurality of captured views.
19. The non-transitory computer readable medium of claim 16 , wherein the instructions further cause the processor to:
generate a weighted composite of the first subset of the plurality of captured views, respective weights being determined for individual views of the first subset of the plurality of captured views, in the weighted composite,
wherein the weights for the individual views in the first subset of the plurality of captured views are determined at least in part based on a cosine of the angle between respective individual captured views and the first arbitrary view.
20. The non-transitory computer readable medium of claim 16 , wherein the instructions further cause the processor to:
generate a weighted composite of the first subset of the plurality of captured views, respective weights being determined for individual views of the first subset of the plurality of captured views, in the weighted composite,
wherein the weights for the individual views in the first subset of the plurality of captured views are determined at least in part based on visibility of a reference point in the individual views.