Augmented reality experiences of color palettes in a messaging system
The subject technology receives image data including a representation of a physical item. The subject technology analyzes the image data to determine an object corresponding to the physical item. The subject technology identifies a set of colors corresponding to a set of regions of the determined object. The subject technology analyzes second image data to detect a second object corresponding to a representation of a particular body part of a user. The subject technology generates augmented reality content based at least in part on the identified set of colors and the detected second object. The subject technology causes display, at a client device, the augmented reality content applied to the detected second object.
1. A method, comprising:
receiving first image data captured by a camera of a client device, the first image data indicating a beauty product;
analyzing the first image data to determine an object corresponding to the beauty product, the analyzing based at least in part on a recognition process performed on the first image data wherein the recognition process further comprises:
classifying, using a first set of image classifiers, the first image data to determine that the object includes a physical identification indicator containing text, a barcode pattern, or a QR code pattern, and
classifying, using a second set of image classifiers, the first image data to perform object classification of the object based on a set of 3D models or images;
identifying a set of colors corresponding to the beauty product;
receiving second image data captured by the camera of the client device;
identifying a body part of a user based on the second image data;
generating augmented reality content based at least in part on the identified set of colors and the identified body part of the user; and
causing display, at a client device, the augmented reality content applied to the identified body part of the user.
2. The method of claim 1 , wherein the beauty product corresponds to a makeup palette and the identified body part of the user comprises a forearm.
3. The method of claim 1 , wherein the identified set of colors corresponds to a second set of colors included in a portion of the first image data including the set of regions of the determined object, and wherein each region of the set of region is associated with a particular color.
4. The method of claim 1 , wherein identifying the set of colors further comprises:
differentiating a set of colors values of pixels in a particular region from the set of regions of the beauty product; and
determining a representative color value based on an average of the set of colors.
5. The method of claim 4 , wherein generating the augmented reality content further comprises:
generating a shape based on the representative color value; and
mapping the shape to a surface of the body part.
6. The method of claim 1 , further comprising:
determining a set of textures based on pixels from the set of regions of the beauty product.
7. The method of claim 6 , wherein generating the augmented reality content further comprises:
performing a texture transfer process based on the set of textures to a surface of the body part.
8. The method of claim 1 , further comprising:
performing a body feature detection process to detect a first body feature; and
identifying a particular body part corresponding to the detected first body feature.
9. A system comprising:
a processor; and
a memory including instructions that, when executed by the processor, cause the processor to perform operations comprising:
receiving first image data captured by a camera of a client device, the first image data indicating a beauty product;
analyzing the first image data to determine an object corresponding to the beauty product, the analyzing based at least in part on a recognition process performed on the first image data, wherein the recognition process further comprises:
classifying, using a first set of image classifiers, the first image data to determine that the object includes a physical identification indicator containing text, a barcode pattern, or a QR code pattern, and
classifying, using a second set of image classifiers, the first image data to perform object classification of the object based on a set of 3D models or images;
identifying a set of colors corresponding to the beauty product;
receiving second image data captured by the camera of the client device;
identifying a body part of a user based on the second image data;
generating augmented reality content based at least in part on the identified set of colors and the identified body part of the user; and
causing display, at a client device, the augmented reality content applied to the identified body part of the user.
10. The system of claim 9 , wherein the beauty product corresponds to a makeup palette and the identified body part of the user comprises a forearm.
11. The system of claim 9 , wherein the identified set of colors corresponds to a second set of colors included in a portion of the first image data including the set of regions of the determined object, and wherein each region of the set of region is associated with a particular color.
12. The system of claim 9 , wherein identifying the set of colors further comprises:
differentiating a set of colors values of pixels in a particular region from the set of regions of the beauty product; and
determining a representative color value based on an average of the set of colors.
13. The system of claim 12 , wherein generating the augmented reality content further comprises:
generating a shape based on the representative color value; and
mapping the shape to a surface of the body part.
14. The system of claim 9 , wherein the operations further comprise:
determining a set of textures based on pixels from the set of regions of the beauty product.
15. The system of claim 14 , wherein generating the augmented reality content further comprises:
performing a texture transfer process based on the set of textures to a surface of the body part.
16. A non-transitory computer-readable medium comprising instructions, which when executed by a computing device, cause the computing device to perform operations comprising:
receiving first image data captured by a camera of a client device, the first image data indicating a beauty product;
analyzing the first image data to determine an object corresponding to the beauty product, the analyzing based at least in part on a recognition process performed on the first image data, wherein the recognition process further comprises:
classifying, using a first set of image classifiers, the first image data to determine that the object includes a physical identification indicator containing text, a barcode pattern, or a QR code pattern, and
classifying, using a second set of image classifiers, the first image data to perform object classification of the object based on a set of 3D models or images;
identifying a set of colors corresponding to the beauty product;
receiving second image data captured by the camera of the client device;
identifying a body part of a user based on the second image data;
generating augmented reality content based at least in part on the identified set of colors and the identified body part of the user; and
causing display, at a client device, the augmented reality content applied to the identified body part of the user.