Universal style transfer using multi-scale feature transform and user controls

Inventors: Yijun Li (Seattle, WA); Ionut Mironic{hacek over (a)} (Bucharest, RO)

Assignee: Adobe Inc.

G06T5/50 G06N3/04 G06T5/002 G06T7/50 G06T11/001 G06T2200/24 G06T2207/20081 G06T2207/20084 G06T2207/20221

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 11,803,950
App. No.: 17/447,893
Granted: Oct 31, 2023
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

Techniques for generating style-transferred images are provided. In some embodiments, a content image, a style image, and a user input indicating one or more modifications that operate on style-transferred images are received. In some embodiments, an initial style-transferred image is generated using a machine learning model. In some examples, the initial style-transferred image comprises features associated with the style image applied to content included in the content image. In some embodiments, a modified style-transferred image is generated by modifying the initial style-transferred image based at least in part on the user input indicating the one or more modifications.

Claims (31)

1. A non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:

receiving, via a user interface, a content image, a style image, and user input indicating one or more modifications that operate on style-transferred images;

generating an initial style-transferred image corresponding to an output of a feed-forward machine learning model, wherein the initial style-transferred image comprises features associated with the style image applied to content included in the content image; and

generating a modified style-transferred image by modifying the initial style-transferred image that is the output of the feed-forward machine learning model based at least in part on the user input indicating the one or more modifications.

2. The non-transitory computer-readable medium of claim 1 , wherein the user input indicating the one or more modifications comprises an intensity of the features of the style image to be applied to the content included in the content image, and wherein generating the modified style-transferred image comprises combining the content image and the initial style-transferred image using a weighted sum, wherein a weight associated with the weighted sum is based at least in part on the intensity of the features of the style image.

3. The non-transitory computer-readable medium of claim 1 , wherein the user input indicating the one or more modifications comprises a texture level associated with texture in the modified style-transferred image, and wherein generating the modified style-transferred image comprises executing the feed-forward machine learning model for a predetermined number of iterations corresponding to the texture level, wherein each iteration provides an output image used as an input content image for a next iteration.

4. The non-transitory computer-readable medium of claim 1 , wherein the user input indicating the one or more modifications comprises a brush size with which the features of the style image are to be applied to the content included in the content image, and wherein generating the modified style-transferred image comprises combining a plurality of versions of the initial style-transferred image, wherein each of the plurality of versions of the initial style-transferred image has a different resolution.

5. The non-transitory computer-readable medium of claim 1 , wherein the user input indicating the one or more modifications comprises an indication that a background of the initial style-transferred image is to be blurred, and wherein generating the modified style-transferred image comprises modifying pixels of the initial style-transferred image based on depth estimates of corresponding pixels of the content image.

6. The non-transitory computer-readable medium of claim 1 , wherein the user input indicating the one or more modifications comprises an indication that colors of the content image are to be preserved in the modified style-transferred image, and wherein generating the modified style-transferred image comprises modifying color values of pixels of the initial style-transferred image based on corresponding color values of pixels of the content image.

7. The non-transitory computer-readable medium of claim 1 , wherein the user input indicating the one or more modifications comprises an indication that objects included in the content of the content image are to be focused in the modified style-transferred image, and wherein generating the modified style-transferred image comprises combining a plurality of versions of the initial style-transferred image based at least in part on depth estimates of the content in the content image, wherein each of the plurality of versions of the initial style-transferred image has a different resolution.

8. A method, comprising:

receiving, via a user interface, a content image, a style image, and user input indicating one or more modifications that operate on style-transferred images;

9. The method of claim 8 , wherein the user input indicating the one or more modifications comprises an intensity of the features of the style image to be applied to the content included in the content image, and wherein generating the modified style-transferred image comprises combining the content image and the initial style-transferred image using a weighted sum, wherein a weight associated with the weighted sum is based at least in part on the intensity of the features of the style image.

10. The method of claim 8 , wherein the user input indicating the one or more modifications comprises a texture level associated with texture in the modified style-transferred image, and wherein generating the modified style-transferred image comprises executing the feed-forward machine learning model for a predetermined number of iterations corresponding to the texture level, wherein each iteration provides an output image used as an input content image for a next iteration.

11. The method of claim 8 , wherein the user input indicating the one or more modifications comprises a brush size with which the features of the style image are to be applied to the content included in the content image, and wherein generating the modified style-transferred image comprises combining a plurality of versions of the initial style-transferred image, wherein each of the plurality of versions of the initial style-transferred image has a different resolution.

12. The method of claim 8 , wherein the user input indicating the one or more modifications comprises an indication that a background of the initial style-transferred image is to be blurred, and wherein generating the modified style-transferred image comprises modifying pixels of the initial style-transferred image based on depth estimates of corresponding pixels of the content image.

13. The method of claim 8 , wherein the user input indicating the one or more modifications comprises an indication that colors of the content image are to be preserved in the modified style-transferred image, and wherein generating the modified style-transferred image comprises modifying color values of pixels of the initial style-transferred image based on corresponding color values of pixels of the content image.

14. The method of claim 8 , wherein the user input indicating the one or more modifications comprises an indication that objects included in the content of the content image are to be focused in the modified style-transferred image, and wherein generating the modified style-transferred image comprises combining a plurality of versions of the initial style-transferred image based at least in part on depth estimates of the content in the content image, wherein each of the plurality of versions of the initial style-transferred image has a different resolution.

15. A computing system, comprising:

one or more processors; and

a computer-readable storage medium, coupled with the one or more processors, having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

receiving, via a user interface, a content image, a style image, and user input indicating one or more modifications that operate on style-transferred images;

16. The computing system of claim 15 , wherein the user input indicating the one or more modifications comprises an intensity of the features of the style image to be applied to the content included in the content image, and wherein generating the modified style-transferred image comprises combining the content image and the initial style-transferred image using a weighted sum, wherein a weight associated with the weighted sum is based at least in part on the intensity of the features of the style image.

17. The computing system of claim 15 , wherein the user input indicating the one or more modifications comprises a texture level associated with texture in the modified style-transferred image, and wherein generating the modified style-transferred image comprises executing the feed-forward machine learning model for a predetermined number of iterations corresponding to the texture level, wherein each iteration provides an output image used as an input content image for a next iteration.

18. The computing system of claim 15 , wherein the user input indicating the one or more modifications comprises a brush size with which the features of the style image are to be applied to the content included in the content image, and wherein generating the modified style-transferred image comprises combining a plurality of versions of the initial style-transferred image, wherein each of the plurality of versions of the initial style-transferred image has a different resolution.

19. The computing system of claim 15 , wherein the user input indicating the one or more modifications comprises an indication that a background of the initial style-transferred image is to be blurred, and wherein generating the modified style-transferred image comprises modifying pixels of the initial style-transferred image based on depth estimates of corresponding pixels of the content image.

20. The computing system of claim 15 , wherein the user input indicating the one or more modifications comprises an indication that colors of the content image are to be preserved in the modified style-transferred image, and wherein generating the modified style-transferred image comprises modifying color values of pixels of the initial style-transferred image based on corresponding color values of pixels of the content image.

Assignments (1)

ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 16, 2021

From: LI, YIJUN; MIRONICA, IONUT

To: ADOBE INC.

Reel/Frame 057505/0921 →

Continuity (1)

Related Publication 20230082050A1 · Mar 16, 2023

Cited By (2)

US 12,243,260 US 12,488,428