Method, system, and medium for browsing-based augmented reality try-on experience

1 . A method comprising:

displaying, by one or more processors of a device, a graphical user interface (GUI) comprising a set of fashion items;

receiving a real-time video feed from a camera of the device, the real-time video feed depicting a person;

retrieving a first augmented reality (AR) fashion item corresponding to a first fashion item in the set of fashion items included in the GUI, the retrieving comprising:

automatically selecting, without user input, the first fashion item based on a position of the first fashion item in a code segment representing the GUI; and

automatically retrieving the first AR fashion item corresponding to the automatically selected first fashion item upon displaying the GUI;

modifying the real-time video feed based on application of the first AR fashion item to the depiction of the person in the real-time video feed, the modifying comprising:

applying one or more machine learning models to the real-time video feed to detect body parts of the person and generate tracking information for the body parts;

generating a skeletal mesh of the person, wherein parameters of the skeletal mesh are adjusted based on an appearance of the person detected with the camera;

rigging the first AR fashion item onto the skeletal mesh using the tracking information such that the first AR fashion item is attached to individual joints of the skeletal mesh corresponding to the body parts;

using input from the camera to track movement of the individual joints of the skeletal mesh as the person moves; and

moving the first AR fashion item attached to the individual joints as the joints move to maintain the first AR fashion item positioned on corresponding body parts of the person; and

presenting the modified real-time video feed together with the GUI comprising the set of fashion items.

2 . The method of claim 1 , wherein the modified real-time video feed is presented in a window on top of the GUI.

3 . The method of claim 2 , wherein the window is circular, wherein accessing the code segment comprises parsing an HTML document to identify a fashion items field tag, wherein searching the code segment comprises traversing child elements within the fashion items field tag to identify individual fashion item entries, and wherein selecting the first fashion item comprises selecting a first fashion item entry within the fashion items field tag.

4 . The method of claim 1 , wherein the modified real-time video feed is presented next to the GUI without overlaying any of the set of fashion items included in the GUI.

5 . The method of claim 1 , further comprising:

continuously monitoring for updates to the GUI presenting different fashion items; and

in response to detecting a GUI update presenting a second set of fashion items, automatically;

selecting a second fashion item from the second set without receiving user input selecting the second fashion item; and

updating the modified real-time video feed to replace the first AR fashion item with a second AR fashion item corresponding to the second fashion item.

6 . The method of claim 5 , further comprising:

generating the first AR fashion item to resemble the first fashion item in the set of fashion items.

7 . The method of claim 5 , further comprising:

accessing a server associated with the GUI to retrieve the first AR fashion item corresponding to the individual fashion item in a list of the set of fashion items.

8 . The method of claim 5 , wherein the code segment comprises a hypertext markup language (HTML) document code used to generate the display of the GUI.

9 . The method of claim 1 , further comprising:

searching the code segment to identify a list of the set of fashion items; and

obtaining one or more fashion item attributes associated with each of the fashion items in the list of the set of fashion items from the code segment.

10 . The method of claim 9 , further comprising:

applying one or more machine learning models to the real-time video feed to estimate one or more attributes of the person depicted in the real-time video feed; and

matching the one or more attributes of the person depicted in the real-time video feed to the one or more fashion item attributes obtained from the code segment.

11 . The method of claim 10 , further comprising:

ranking the fashion items in the list of the set of fashion items based on a result of matching the one or more attributes of the person depicted in the real-time video feed to the one or more fashion item attributes obtained from the code segment; and

selecting, as the first fashion item, an individual fashion item that is ranked higher than all other fashion items in the list of the set of fashion items.

12 . The method of claim 1 , wherein the set of fashion items comprises a first set of fashion items, further comprising:

receiving input to update the GUI to present a second set of fashion items; and

automatically updating the real-time video feed based on the second set of fashion items.

13 . The method of claim 12 , further comprising:

retrieving a second AR fashion item corresponding to a second fashion item in the second set of fashion items included in the updated GUI; and

modifying the real-time video feed based on application of the second AR fashion item to the depiction of the person in the real-time video feed instead of the first AR fashion item.

14 . The method of claim 13 , wherein the real-time video feed is modified in response to receiving the input to update the GUI without receiving a separate request to apply the second AR fashion item to the depiction of the person.

15 . The method of claim 1 , further comprising:

automatically presenting an indicator in association with the first fashion item of the set of fashion items displayed in the GUI to identify which of the set of fashion items is being represented by the application of the first AR fashion item to the depiction of the person in the real-time video feed.

16 . The method of claim 15 , wherein the indicator comprises a highlight region or cursor.

17 . The method of claim 1 , further comprising:

determining that an AR fashion item link is absent from the code segment for the first fashion item;

in response to determining that the AR fashion item link is absent from the code segment for the first fashion item, extracting a two-dimensional image of the first fashion item from the code segment;

applying one or more machine learning models to the two-dimensional image to generate a three-dimensional representation; and

animating the three-dimensional representation to create the first AR fashion item.

18 . The method of claim 1 , further comprising:

applying one or more machine learning models to the real-time video feed to generate tracking information for the person depicted in the real-time video feed; and

continuously updating the real-time video feed to modify placement of the first AR fashion item on the depiction of the person in the real-time video feed based on the tracking information generated by the one or more machine learning models.

19 . A system comprising:

at least one processor of a device configured to perform operations comprising:

displaying a graphical user interface (GUI) comprising a set of fashion items;

receiving a real-time video feed from a camera of the device, the real-time video feed depicting a person;

retrieving a first augmented reality (AR) fashion item corresponding to a first fashion item in the set of fashion items included in the GUI, the retrieving comprising:

automatically selecting, without user input, the first fashion item based on a position of the first fashion item in a code segment representing the GUI; and

automatically retrieving the first AR fashion item corresponding to the automatically selected first fashion item upon displaying the GUI;

modifying the real-time video feed based on application of the first AR fashion item to the depiction of the person in the real-time video feed, the modifying comprising:

applying one or more machine learning models to the real-time video feed to detect body parts of the person and generate tracking information for the body parts;

generating a skeletal mesh of the person, wherein parameters of the skeletal mesh are adjusted based on an appearance of the person detected with the camera;

rigging the first AR fashion item onto the skeletal mesh using the tracking information such that the first AR fashion item is attached to individual joints of the skeletal mesh corresponding to the body parts;

using input from the camera to track movement of the individual joints of the skeletal mesh as the person moves; and

moving the first AR fashion item attached to the individual joints as the joints move to maintain the first AR fashion item positioned on corresponding body parts of the person; and

presenting the modified real-time video feed together with the GUI comprising the set of fashion items.

20 . A non-transitory machine-readable storage medium that includes instructions that, when executed by one or more processors of a device, cause the device to perform operations comprising:

displaying a graphical user interface (GUI) comprising a set of fashion items;

receiving a real-time video feed from a camera of the device, the real-time video feed depicting a person;

retrieving a first augmented reality (AR) fashion item corresponding to a first fashion item in the set of fashion items included in the GUI, the retrieving comprising:

automatically selecting, without user input, the first fashion item based on a position of the first fashion item in a code segment representing the GUI; and

automatically retrieving the first AR fashion item corresponding to the automatically selected first fashion item upon displaying the GUI;

modifying the real-time video feed based on application of the first AR fashion item to the depiction of the person in the real-time video feed, the modifying comprising:

applying one or more machine learning models to the real-time video feed to detect body parts of the person and generate tracking information for the body parts;

generating a skeletal mesh of the person, wherein parameters of the skeletal mesh are adjusted based on an appearance of the person detected with the camera;

rigging the first AR fashion item onto the skeletal mesh using the tracking information such that the first AR fashion item is attached to individual joints of the skeletal mesh corresponding to the body parts;

using input from the camera to track movement of the individual joints of the skeletal mesh as the person moves; and

moving the first AR fashion item attached to the individual joints as the joints move to maintain the first AR fashion item positioned on corresponding body parts of the person; and

presenting the modified real-time video feed together with the GUI comprising the set of fashion items.