Method, system, and medium for augmented reality product recommendations
Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and a method for performing operations comprising: receiving a video that includes a depiction of a real-world object in a real-world environment; determining a classification for the real-world environment by processing the real-world object depicted in the video; selecting an augmented reality (AR) item based on the classification of the real-world environment and the real-world object depicted in the video; modifying pixels corresponding to the real-world object depicted in the video to generate a modified video that excludes the depiction of the real-world object; and adding the AR item to the modified video at a display position corresponding to the modified pixels.
1. A method comprising:
receiving, by one or more processors, a video that includes a depiction of a real-world object in a real-world environment;
determining, by a trained neural network, a classification for the real-world environment by processing the real-world object depicted in the video;
selecting an augmented reality (AR) item based on the classification of the real-world environment and based on the real-world object depicted in the video;
modifying pixel data corresponding to the real-world object depicted in the video to generate a modified video that excludes the depiction of the real-world object;
adding a depiction of the AR item to the modified video at a display position corresponding to the modified pixel data, the adding the depiction of the AR item to the modified video comprising:
calculating characteristic points for a set of elements of the real-world object to generate a mesh based on the calculated characteristic points;
generating one or more areas on the mesh of the real-world object;
aligning a position of the one or more areas of the real-world object with one or more elements of the AR item; and
modifying one or more visual properties of the one or more areas to cause a user device to display the AR item within the video at an individual display position relative to the display position of the real-world object that has had pixel data modified;
detecting input that moves the AR item to a new position in the video;
determining that the AR item in the new position no longer overlaps a portion of the real-world object that has had the corresponding pixel data modified; and
in response to determining that the AR item in the new position no longer overlaps the portion of the real-world object that has had the corresponding pixel data modified, undoing modification of the pixel data to return pixel values of the real-world object to an original value in the video.
2. The method of claim 1 , further comprising generating, for display, the modified video with the depiction of the AR item that has been added.
3. The method of claim 1 , further comprising blurring a region corresponding to the modified pixel data, wherein the depiction of the AR item is added to the blurred region.
4. The method of claim 3 , further comprising blending pixel values in the blurred region with pixel values of other real-world objects that are adjacent to the blurred region.
5. The method of claim 1 , further comprising applying a machine learning technique to the video to modify the pixel data and generate the modified video.
6. The method of claim 5 , wherein the machine learning technique is trained to establish a relationship between different types of real-world objects and image blending patterns.
7. The method of claim 6 , further comprising training the machine learning technique by:
receiving training data comprising a plurality of training images and ground truth room blending patterns for each of the plurality of training images, each of the plurality of training images depicting a different real-world environment having different real-world object types;
selecting a first real-world object depicted in a first training image of the plurality of training images;
applying the neural network to the first training image and the first real-world object to estimate a blending pattern for the real-world environment depicted in the first training image;
computing a deviation between the estimated blending pattern and the ground truth room blending pattern associated with the first training image;
updating parameters of the neural network based on the computed deviation; and
repeating the applying, computing and updating steps for a set of the plurality of training images.
8. The method of claim 1 , further comprising:
obtaining a plurality of excluded objects associated with the classification;
detecting the real-world object depicted in the video using an object recognition process; and
comparing the detected object depicted in the video to the plurality of expected excluded objects.
9. The method of claim 8 , further comprising: based on the comparing, identifying a given excluded object from the plurality of excluded objects that is excluded from the detected real-world object; and searching for an AR item corresponding to the given excluded object.
10. The method of claim 9 , further comprising: generating a three-dimensional (3D) mesh representation of the real-world environment; obtaining a plurality of real-world items corresponding to the classification; detecting that the plurality of real-world items excludes the detected object depicted in the video; determining, based on the 3D mesh representation, that physical space is available for a given one of the plurality of real-world items; and selecting an AR item corresponding to the given one of the plurality of real-world items.
11. The method of claim 1 , wherein the classification corresponds to a kitchen, the method further comprising: detecting that the real-world object depicted in the video corresponds to a first type of kitchen appliance; and searching a plurality of AR items to identify an AR item corresponding to the first type of kitchen appliance, wherein the AR item corresponding to the first type of kitchen appliance represents a model of the first type of kitchen appliance that is different than the real-world object.
12. The method of claim 1 , wherein determining the classification for the real-world environment comprises:
comparing, by the trained neural network, the real-world object depicted in the video to a plurality of lists of expected objects, each list associated with a different real-world environment classification;
computing, for each list, a relevancy score based on a quantity or percentage of objects depicted in the video that match objects in the list;
identifying the list with the highest relevancy score; and
determining the classification for the real-world environment based on the real-world environment classification associated with the identified list having the highest relevancy score.
13. The method of claim 1 , further comprising: generating a three-dimensional (3D) mesh representation of the real-world environment; detecting that the real-world object depicted in the video fails to satisfy one or more fit parameters of the 3D mesh representation; identifying, based on the 3D mesh representation, a recommended item that satisfies the one or more fit parameters of the 3D mesh representation and is of a same type as the real-world object included in the video; and retrieving an AR item corresponding to the recommended item in response to identifying the recommended item.
14. The method of claim 1 , wherein determining the classification for the real-world environment further comprises:
applying, by the trained neural network, multiple classifiers to the video, wherein at least one classifier is trained to classify a room and provide an estimated age range associated with the room;
determining, based on the estimated age range, a specific room type classification;
generating, by each classifier, a classification score indicating an accuracy of the generated real-world environment classification; and
assigning the real-world environment classification based on the classification with the highest score among the multiple classifiers.
15. The method of claim 14 , wherein the specific room type classification is selected from a plurality of different types of bedrooms each associated with a different age range, further comprising modifying an orientation of the AR item based on an orientation of a surface on which the AR item is placed in the real-world environment.
16. The method of claim 1 , further comprising: determining a type of the real-world object; determining that the type of the real-world object corresponds to a type of the AR item; and selectively modifying pixels corresponding to the real-world object in response to determining that the type of the real-world object corresponds to the type of the AR item.
17. The method of claim 1 , further comprising training a neural network classifier to determine the classification by:
receiving training data comprising a plurality of training images and ground truth room classifications for each of the plurality of training images, each of the plurality of training images depicting a different real-world environment classification;
applying the neural network classifier to a first training image of the plurality of training images to estimate a real-world classification of the real-world environment depicted in the first training image;
computing a deviation between the estimated real-world environment classification and the ground truth real-world environment classification associated with the first training image;
updating parameters of the neural network classifier based on the computed deviation; and
repeating the applying, computing and updating steps for a set of the plurality of training images.
18. A system comprising:
at least one processor configured to perform operations comprising:
receiving a video that includes a depiction of a real-world object in a real-world environment;
determining, by a trained neural network, a classification for the real-world environment by processing the real-world object depicted in the video;
selecting an augmented reality (AR) item based on the classification of the real-world environment and based on the real-world object depicted in the video;
modifying pixel data corresponding to the real-world object depicted in the video to generate a modified video that excludes the depiction of the real-world object;
adding a depiction of the AR item to the modified video at a display position corresponding to the modified pixel data, the adding the depiction of the AR item to the modified video comprising:
calculating characteristic points for a set of elements of the real-world object to generate a mesh based on the calculated characteristic points;
generating one or more areas on the mesh of the real-world object;
aligning a position of the one or more areas of the real-world object with one or more elements of the AR item; and
modifying one or more visual properties of the one or more areas to cause a user device to display the AR item within the video at an individual display position relative to the display position of the real-world object that has had pixel data modified;
detecting input that moves the AR item to a new position in the video;
determining that the AR item in the new position no longer overlaps a portion of the real-world object that has had the corresponding pixel data modified; and
in response to determining that the AR item in the new position no longer overlaps the portion of the real-world object that has had the corresponding pixel data modified, undoing modification of the pixel data to return pixel values of the real-world object to an original value in the video.
19. The system of claim 18 , wherein the operations further comprise blurring a region corresponding to the modified pixel data, wherein the AR item is added to the blurred region.
20. A non-transitory machine-readable storage medium that includes instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
receiving a video that includes a depiction of a real-world object in a real-world environment;
determining, by a trained neural network, a classification for the real-world environment by processing the real-world object depicted in the video;
selecting an augmented reality (AR) item based on the classification of the real-world environment and based on the real-world object depicted in the video;
modifying pixel data corresponding to the real-world object depicted in the video to generate a modified video that excludes the depiction of the real-world object;
adding a depiction of the AR item to the modified video at a display position corresponding to the modified pixel data, the adding the depiction of the AR item to the modified video comprising:
calculating characteristic points for a set of elements of the real-world object to generate a mesh based on the calculated characteristic points;
generating one or more areas on the mesh of the real-world object;
aligning a position of the one or more areas of the real-world object with one or more elements of the AR item; and
modifying one or more visual properties of the one or more areas to cause a user device to display the AR item within the video at an individual display position relative to the display position of the real-world object that has had pixel data modified;
detecting input that moves the AR item to a new position in the video;
determining that the AR item in the new position no longer overlaps a portion of the real-world object that has had the corresponding pixel data modified; and
in response to determining that the AR item in the new position no longer overlaps the portion of the real-world object that has had the corresponding pixel data modified, undoing modification of the pixel data to return pixel values of the real-world object to an original value in the video.