3D upper garment tracking
Methods and systems are disclosed for performing operations comprising: receiving a video that includes a depiction of a person wearing a fashion item; generating a three-dimensional (3D) model of the person depicted in the video; generating a segmentation of the fashion item worn by the person depicted in the video; outlining a portion of the 3D model based on the segmentation of the fashion item; tracking movement of the portion of the 3D model in the video; and in response to tracking the movement of the portion of the 3D model in the video, modifying a display position of one or more augmented reality elements on the fashion item in the video.
1. A method comprising:
receiving, by one or more processors of a device, a video that includes a depiction of an object;
generating, by the one or more processors, a three-dimensional (3D) model of the object;
causing a plurality of augmented reality (AR) objects to appear as coming out of a first portion of the object relative to a surface normal of the first portion of the object along different directions and at different rates; and
using the 3D model of the object to adjust a score based on detected contact between an individual AR object of the plurality of AR objects and a second portion of the object.
2. The method of claim 1 , the object comprising a person, the first portion of the object comprising a fashion item, further comprising:
applying a trained machine learning model to the video that includes the depiction of the person wearing the fashion item, the trained machine learning model extracting one or more features from the video, and the trained machine learning model generating a segmentation of the fashion item worn by the person depicted in the video using the extracted one or more features;
outlining a portion of the 3D model based on the segmentation of the fashion item;
tracking movement of the portion of the 3D model in the video to determine that a portion of the fashion item has been moved towards a particular side; and
in response to tracking the movement of the portion of the 3D model in the video, modifying a display position of one or more augmented reality (AR) elements on the fashion item in the video.
3. The method of claim 1 , further comprising:
modifying a shape of the one or more AR objects in response to detecting a pose of the object.
4. The method of claim 1 , wherein the object comprises a person, the first portion comprises a fashion item worn by the person, and the second portion comprises a body part of the person.
5. The method of claim 1 , wherein the one or more AR objects are associated with a non-fungible token, wherein the non-fungible token is associated with the one or more AR objects in response to a purchase transaction.
6. The method of claim 1 , further comprising:
generating an AR object comprising the first portion combined with the one or more AR objects; and
generating a non-fungible token associated with the AR object.
7. The method of claim 1 , further comprising maintaining a location of the one or more AR objects on the first portion in the video as the object moves around the video.
8. The method of claim 7 , further comprising:
displaying the one or more AR objects on the first portion in a first frame of the video, wherein the object is positioned at a first location in the first frame;
determining that the object has moved from the first location to a second location in a second frame of the video, the first portion being moved to a new location in the second frame of the video; and
updating a display position of the one or more AR objects in the second frame to maintain the display of the one or more AR objects on the first portion.
9. The method of claim 8 , wherein the first portion comprises a shirt pocket, wherein the one or more AR objects continue to be displayed on top of the shirt pocket as the object moves from the first location to the second location in the video.
10. The method of claim 1 , further comprising:
selecting a random pose for an avatar;
causing the avatar with the random pose to be presented to the object;
presenting a timer counting towards a specified value; and
instructing the object to perform a gesture that mirrors the random pose of the avatar before the timer reaches the specified value.
11. The method of claim 10 , wherein the one or more AR objects are displayed on a portion of a fashion item, further comprising:
identifying the portion of the fashion item on the portion of the 3D model;
determining that the portion of the 3D model has been rotated by a specified amount in a second frame of the video;
in response to determining that the 3D model has been rotated by the specified amount, determining that the portion of the fashion item fails to appear in the second frame; and
removing at least a portion of the one or more AR objects in response to determining that the portion of the fashion item fails to appear in the second frame.
12. A system comprising:
at least one processor of a device; and
a memory component having instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
receiving a video that includes a depiction of an object;
generating a three-dimensional (3D) model of the object;
causing a plurality of augmented reality (AR) objects to appear as coming out of a first portion of the object relative to a surface normal of the first portion of the object along different directions and at different rates; and
using the 3D model of the object to adjust a score based on detected contact between an individual AR object of the plurality of AR objects and a second portion of the object.
13. The system of claim 12 , the operations further comprising:
generating, as the plurality of AR objects, one or more game-based AR game boards;
generating an avatar associated with the one or more game-based AR game boards; and
displaying the one or more game-based AR game boards and the avatar.
14. The system of claim 12 , the operations further comprising:
modifying a shape of the plurality of AR objects in response to detecting a pose of the object depicted in the video.
15. The system of claim 12 , the operations further comprising:
detecting overlap between a body part of a person comprising the object and a first portion of a fashion item.
16. The system of claim 12 , wherein the object comprises a person, the first portion comprises a fashion item worn by the person, and the second portion comprises a body part of the person.
17. The system of claim 16 , the operations further comprising:
adjusting a display of the plurality of AR objects based on wrinkles detected on the fashion item.
18. The system of claim 16 , the operations further comprising:
receiving, through a microphone, a voice input associated with the person wearing the fashion item in the video;
in response to receiving the voice input, applying a second trained machine learning model to a characteristic of the voice input to detect an emotion associated with the voice input;
generating, as the plurality of AR objects on the fashion item in the video, an avatar having an avatar facial expression;
searching a database of avatar facial expressions that are associated with the emotion associated with the voice input, and adjusting the avatar facial expression to correspond to the characteristic of the voice input associated with the person depicted in the video.
19. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by at least one processor of a device, cause the at least one processor to perform operations comprising:
receiving a video that includes a depiction of an object;
generating a three-dimensional (3D) model of the object;
causing a plurality of augmented reality (AR) objects to appear as coming out of a first portion of the object relative to a surface normal of the first portion of the object along different directions and at different rates; and
using the 3D model of the object to adjust a score based on detected contact between an individual AR object of the plurality of AR objects and a second portion of the object.
20. The non-transitory computer-readable storage medium of claim 19 , the operations comprising:
modifying a shape of the plurality of AR objects in response to detecting a pose of the object.