Virtual item simulation using detected surfaces
Disclosed are various embodiments for simulating one or more virtual objects (e.g., renders) based on characteristics of detected surfaces. Mask data may be generated which can be used to delineate borders depicted in a given image. The mask data can be refined based on a quantity of depicted image features (e.g., edges). The mask data can be used to generate one or more planes on which virtual items can be applied and rendered.
1. A method comprising:
generating an image using an image sensor of a client device;
storing, on the client device, an image mask generated using a neural network, the image mask assigning a portion of pixels in the image to a surface category the image mask comprising an interface between the portion of pixels in the surface category and other pixels of the image that are not in the surface category;
rendering a three-dimensional virtual item on a finite plane on the client device, the finite plane having a side terminating at the interface of the image mask; and
displaying a rendered three-dimensional virtual item overlaid on the finite plane on the image such that a terminating side of the three-dimensional virtual item coincides with the side of the finite plane.
2. The method of claim 1 , wherein the image mask is generated by the neural network by assigning pixels of the image into a plurality of categories including the surface category and another category that comprises the other pixels.
3. The method of claim 2 , wherein the image mask is a refined image mask generated from an initial image mask, the refined image mask generated by refining the interface between the pixels in the surface category and the pixels in the another category.
4. The method of claim 3 , wherein the interface is refined by replacing pixel values near the interface with values from nearby pixels.
5. The method of claim 4 , wherein a Gaussian mixture scheme is used to replace pixel values with values from the nearby pixels.
6. The method of claim 5 , wherein the refined image mask is further refined based on a determination that the portion of the pixels assigned to the surface category satisfy a uniformness threshold.
7. The method of claim 6 , wherein the uniformness threshold is evaluated using an image uniformness analysis scheme.
8. The method of claim 5 , wherein the refined image mask is not further refined using edge detection based on a determination that the portion of pixels assigned to the surface category do not satisfy a uniformness threshold.
9. The method of claim 1 , wherein the image mask is generated on the client device.
10. The method of claim 1 , wherein the three-dimensional virtual item comprises a surface texture applied to the finite plane.
11. The method of claim 1 , wherein the three-dimensional virtual item is a 3D model of a physical item placed on or proximate to the finite plane.
12. The method of claim 11 , wherein the other pixels of the image that are not in the surface category depict a wall, and the three-dimensional virtual item is aligned with a bottom-side of the wall.
13. The method of claim 1 , wherein the image is part of a video sequence generated by the image sensor of the client device, and wherein an additional image mask is generated by the neural network for each image of the video sequence.
14. A system comprising:
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
generating an image using an image sensor;
storing an image mask generated using a neural network, the image mask assigning a portion of pixels in the image to a surface category the image mask comprising an interface between the portion of pixels in the surface category and other pixels of the image that are not in the surface category;
rendering a three-dimensional virtual item on a finite plane, the finite plane having a side terminating at the interface of the image mask; and
displaying a rendered three-dimensional virtual item overlaid on the finite plane on the image such that a terminating side of the three-dimensional virtual item coincides with the side of the finite plane.
15. The method of claim 14 , wherein the image mask is generated by the neural network by assigning pixels of the image into a plurality of categories including the surface category and another category that comprises the other pixels.
16. The system of claim 15 , wherein the image mask is a refined image mask generated from an initial image mask, the refined image mask generated by refining the interface between the pixels in the surface category and the pixels in the another category.
17. The system of claim 16 , wherein the interface is refined by replacing pixel values near the interface with values from nearby pixels.
18. The system of claim 17 , wherein a Gaussian mixture scheme is used to replace pixel values with values from the nearby pixels.
19. The system of claim 18 , wherein the refined image mask is further refined based on a determination that the portion of the pixels assigned to the surface category satisfy a uniformness threshold.
20. A machine-readable storage device embodying instructions that, when executed by a device, cause the device to perform operations comprising:
generating an image using an image sensor;
storing an image mask generated using a neural network, the image mask assigning a portion of pixels in the image to a surface category the image mask comprising an interface between the portion of pixels in the surface category and other pixels of the image that are not in the surface category;
rendering a three-dimensional virtual item on a finite plane, the finite plane having a side terminating at the interface of the image mask; and
displaying a rendered three-dimensional virtual item overlaid on the finite plane on the image such that a terminating side of the three-dimensional virtual item coincides with the side of the finite plane.