Granted Patent B2

US 12,412,350 · App. 17/994,752 · Granted Sep 9, 2025

Inventors: Russ Maschmeyer (Berkeley, CA); Eric Andrew Florenzano (San Francisco, CA); Brennan Letkeman (Calgary, CA); Diego Macario Bello (Montreal, CA); Daniel Beauchamp (Toronto, CA)

Assignee: Shopify Inc.

G06T19/20G06F3/04815G06F3/04845G06T11/00G06T19/006G06T2200/24G06T2210/32G06T2219/2004

Abstract

Claims (50)

1. A computer-implemented method, comprising:

receiving a user request to edit a first two-dimensional (2D) image that is displayed on a first graphical user interface, the first 2D image depicting a product;

in response to receiving the user request to edit the first 2D image:

presenting, via an augmented reality (AR) device, an AR scene that includes a virtual 3D representation of the product and a view of the first graphical user interface;

monitoring user interactions with the virtual 3D representation of the product based on detected gestures of the user;

determining modifications of the virtual 3D representation of the product based on the monitored user interactions;

presenting, in the AR scene, a modified 3D representation of the product;

converting the modified 3D representation of the product to a second 2D image for replacing the first 2D image based on obtaining an image of the modified 3D representation of the product as output by a virtual camera associated with the AR scene; and

causing the first graphical user interface to be updated in real-time responsive to the monitored user interactions based on displaying the second 2D image in place of the first 2D image on the first graphical user interface in AR.

2. The method of claim 1 , wherein the first graphical user interface is provided on a display device that is viewable in the AR scene.

3. The method of claim 1 , wherein receiving the user request to edit the first 2D image comprises receiving at least one of: a touch input on a touch-sensitive interface; a selection using an input device such as a stylus; or a gesture for interacting with the first 2D image as displayed on a virtual display device.

4. The method of claim 1 , further comprising:

comparing image data associated with the first 2D image and metadata associated with a plurality of 3D virtual scenes; and

determining a match between the image data associated with the first 2D image and metadata of a first one of the 3D virtual scenes, wherein the AR scene is generated based on the metadata of the first 3D virtual scene.

5. The method of claim 1 , further comprising obtaining sensor data of sensors for tracking gestures of the user, wherein the gestures of the user are detected based on the obtained sensor data.

6. The method of claim 5 , wherein the sensors comprise at least one of: cameras; LiDAR array; eye trackers; or hand trackers.

7. The method of claim 1 , wherein the AR scene further includes a virtual representation of a camera that is positioned adjacent to the 3D representation of the product and wherein determining the modifications of the virtual 3D representation of the product comprises identifying imaging effects on the 3D representation of the product corresponding to adjustments of the camera by the detected gestures of the user.

8. The method of claim 7 , wherein the adjustments of the camera include at least repositioning of the virtual representation of the camera.

9. The method of claim 1 , further comprising transmitting, to an AR-enabled computing device, the first 2D image and instructions for displaying the first 2D image on the first graphical user interface.

10. The method of claim 1 , further comprising detecting completion of modifications of the 3D representation of the product, wherein the modified 3D representation of the product is converted to the second 2D image responsive to detecting the completion of modifications.

11. The method of claim 1 , further comprising, in response to receiving the user request to edit the first 2D image:

presenting a virtual studio environment in the AR scene, the virtual studio environment comprising a 3D model of a studio workbench for editing images, wherein the virtual 3D representation of the product is positioned at a default location within the virtual studio environment.

12. The method of claim 11 , wherein the virtual studio environment and the virtual 3D representation of the product are viewable on a desk tabletop in AR.

13. The method of claim 11 , wherein the virtual studio environment comprises virtual representations of one or more studio equipment in AR.

14. The method of claim 13 , further comprising monitoring user interactions with the virtual representations of the studio equipment, wherein the modifications of the virtual 3D representation of the product are determined based on the monitored user interactions with the virtual representations of the product and the studio equipment.

15. A computing system, comprising:

a processor; and

a memory coupled to the processor, the memory storing processor-executable instructions that, when executed, are to cause the processor to:

receive a user request to edit a first two-dimensional (2D) image that is displayed on a first graphical user interface, the first 2D image depicting a product;

in response to receiving the user request to edit the first 2D image:

present, via an augmented reality (AR) device, an AR scene that includes a virtual 3D representation of the product and a view of the first graphical user interface;

monitor user interactions with the virtual 3D representation of the product based on detected gestures of the user;

determine modifications of the virtual 3D representation of the product based on the monitored user interactions;

present, in the AR scene, a modified 3D representation of the product;

convert the modified 3D representation of the product to a second 2D image for replacing the first 2D image based on obtaining an image of the modified 3D representation of the product as output by a virtual camera associated with the AR scene; and

cause the first graphical user interface to be updated in real-time responsive to the monitored user interactions based on displaying the second 2D image in place of the first 2D image on the first graphical user interface in AR.

16. The computing system of claim 15 , wherein receiving the user request to edit the first 2D image comprises receiving at least one of: a touch input on a touch-sensitive interface; a selection using an input device such as a stylus; or a gesture for interacting with the first 2D image as displayed on a virtual display device.

17. The computing system of claim 15 , wherein the instructions, when executed, are to further cause the processor to:

compare image data associated with the first 2D image and metadata associated with a plurality of 3D virtual scenes; and

determine a match between the image data associated with the first 2D image and metadata of a first one of the 3D virtual scenes, wherein the AR scene is generated based on the metadata of the first 3D virtual scene.

18. The computing system of claim 15 , wherein the AR scene further includes a virtual representation of a camera that is positioned adjacent to the 3D representation of the product and wherein determining the modifications of the virtual 3D representation of the product comprises identifying imaging effects on the 3D representation of the product corresponding to adjustments of the camera by the detected gestures of the user.

19. A non-transitory processor-readable storage medium storing processor-executable instructions that, when executed by a processor, are to cause the processor to:

receive a user request to edit a first two-dimensional (2D) image that is displayed on a first graphical user interface, the first 2D image depicting a product;

in response to receiving the user request to edit the first 2D image:

present, via an augmented reality (AR) device, an AR scene that includes a virtual 3D representation of a product and a view of a first graphical user interface;

monitor user interactions with the virtual 3D representation of the product based on detected gestures of the user;

determine modifications to the virtual 3D representation of the product based on the monitored user interactions;

present, in the AR scene, a modified 3D representation of the product;

convert the modified 3D representation of the product to a 2D image based on obtaining an image of the modified 3D representation of the product as output by a virtual camera associated with the AR scene; and

cause the first graphical user interface to be updated in real-time responsive to the monitored user interactions based on displaying the 2D image at a defined location of the first graphical user interface in AR.

Assignments (9)

Continuity (2)

References Cited (17)