Granted Patent B2

US 12,335,574 · App. 17/231,184 · Granted Jun 17, 2025

Inventors: Samaneh Bayat (Kanata, CA); John Jong-Suk Lee (Waterloo, CA)

Assignee: Shopify Inc.

H04N21/47815G06Q30/0643H04N21/2187H04N21/4312H04N21/44213H04N21/8146

Abstract

Claims (67)

1. A computer-implemented method, comprising:

receiving video data for a live video stream;

identifying a product that is displayed in a video frame of the live video stream;

obtaining identifying information for a viewer of the live video stream;

determining a product variant representing a preferred version of the identified product for the viewer based on:

identifying products that have been added to a virtual shopping cart associated with an e-commerce account by the viewer;

determining preferred product attributes of the products that have been added to the virtual shopping cart; and

determining a version of the identified product that has at least a threshold number of the preferred product attributes;

obtaining a graphical representation of the determined product variant; and

providing modified video data of the live video stream in real-time based on, for each of a plurality of video frames of the live stream in which the product is detected:

determining a boundary surrounding the identified product within the video frame; and

overlaying a region bounded by the boundary in the video frame with the graphical representation of the determined product variant to replace the identified product with the determined product variant.

2. The method of claim 1 , wherein obtaining the graphical representation of the determined product variant comprises obtaining three-dimensional representation data for the product variant.

3. The method of claim 1 , wherein obtaining the graphical representation of the determined product variant comprises:

obtaining an image of the product from the video frame; and

modifying one or more attributes of the image to obtain the graphical representation of the product variant.

4. The method of claim 3 , wherein modifying the one or more attributes of the image comprises modifying a color of at least one pixel of the image to obtain a recolored image.

5. The method of claim 1 , further comprising obtaining commerce data associated with the identifying information for the viewer, wherein the product variant preference for the viewer is determined based on the commerce data.

6. The method of claim 5 , wherein the commerce data is obtained via a merchant computing system storing e-commerce account data for the viewer.

7. The method of claim 1 , wherein obtaining the graphical representation of the determined product variant comprises:

determining scene lighting data for a scene depicted in the video frame,

wherein the graphical representation of the determined product variant combines image data for the product variant with the scene lighting data.

8. The method of claim 1 , wherein providing the modified video data comprises:

detecting visual markers that are affixed to the product in a scene depicted in the video frame; and

overlaying the graphical representation of the determined product variant over a region of the video frame represented by the detected visual markers.

9. The method of claim 1 , wherein at least some overlay content for the live video stream is generated at a server and wherein providing the modified video data of the live video stream comprises transmitting overlay content that is generated at the server to a device associated with a viewer.

10. A computing system, comprising:

a processor;

a memory storing computer-executable instructions that, when executed by the processor, are to cause the processor to:

receive video data for a live video stream;

identify a product that is displayed in a video frame of the live video stream;

obtain identifying information for a viewer of the live video stream;

determine a product variant representing a preferred version of the identified product for the viewer based on:

identifying products that have been added to a virtual shopping cart associated with an e-commerce account by the viewer;

determining preferred product attributes of the products that have been added to the virtual shopping cart; and

determining a version of the identified product that has at least a threshold number of the preferred product attributes;

obtain a graphical representation of the determined product variant; and

provide modified video data of the live video stream in real-time based on, for each of a plurality of video frames of the live stream in which the product is detected:

determining a boundary surrounding the identified product within the video frame; and

overlaying a region bounded by the boundary in the video frame with the graphical representation of the determined product variant to replace the identified product with the determined product variant.

11. The computing system of claim 10 , wherein obtaining the graphical representation of the determined product variant comprises obtaining three-dimensional representation data for the product variant.

12. The computing system of claim 10 , wherein obtaining the graphical representation of the determined product variant comprises:

obtaining an image of the product from the video frame; and

modifying one or more attributes of the image to obtain the graphical representation of the product variant.

13. The computing system of claim 12 , wherein modifying the one or more attributes of the image comprises modifying a color of at least one pixel of the image to obtain a recolored image.

14. The computing system of claim 10 , wherein the instructions, when executed, are to further cause the processor to obtain commerce data associated with the identifying information for the viewer, wherein the product variant preference for the viewer is determined based on the commerce data.

15. A non-transitory computer-readable medium storing computer-executable instructions that, when executed by a processor, are to cause the processor to:

receive video data for a live video stream;

identify a product that is displayed in a video frame of the live video stream;

obtain identifying information for a viewer of the live video stream;

determine a product variant representing a preferred version of the identified product for the viewer based on:

identifying products that have been added to a virtual shopping cart associated with an e-commerce account by the viewer;

determining preferred product attributes of the products that have been added to the virtual shopping cart; and

determining a version of the identified product that has at least a threshold number of the preferred product attributes;

obtain a graphical representation of the determined product variant; and

provide modified video data of the live video stream in real-time based on, for each of a plurality of video frames of the live stream in which the product is detected:

determining a boundary surrounding the identified product within the video frame; and

overlaying a region bounded by the boundary in the video frame with the graphical representation of the determined product variant to replace the identified product with the determined product variant.

16. The computing system of claim 10 , wherein obtaining the graphical representation of the determined product variant comprises:

determining scene lighting data for a scene depicted in the video frame,

wherein the graphical representation of the product variant combines image data for the product variant with the scene lighting data.

17. The computing system of claim 10 , wherein providing the modified video data comprises:

detecting a boundary associated with the product within the video frame; and

overlaying the graphical representation of the product variant over at least a portion of the detected boundary to generate a composite video frame.

18. The method of claim 1 , wherein the determining the product variant comprises identifying product attributes of products contained in the one or more virtual shopping carts, and wherein the product variant is determined based on the identified product attributes.

19. The method of claim 18 , wherein identifying the product attributes comprises identifying attributes that are most commonly found among the products contained in the one or more virtual shopping carts.

20. The method of claim 19 , wherein the product variant is a version of the product that has a defined number of the most commonly found attributes.

Assignments (1)

Continuity (1)

References Cited (24)