IP Library Granted Patent US 10,699,388
Granted Patent B2
US 10,699,388 · App. 15/879,354 · Granted Jun 30, 2020

Digital image fill

Inventors: Brian Lynn Price (Pleasant Grove, UT); Yinan Zhao (Austin, TX); Scott David Cohen (Sunnyvale, CA)
Assignee: Adobe Inc.
G06T5/005G06N3/0454G06N3/0472G06N3/08G06N3/084G06T3/0006G06T5/002G06T5/50G06T7/337G06T7/344G06T11/40G06T2207/20024G06T2207/20081G06T2207/20084
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,699,388
App. No.
15/879,354
Granted
Jun 30, 2020
Kind
B2
Abstract

Fill techniques as implemented by a computing device are described to perform hole filling of a digital image. In one example, deeply learned features of a digital image using machine learning are used by a computing device as a basis to search a digital image repository to locate the guidance digital image. Once located, machine learning techniques are then used to align the guidance digital image with the hole to be filled in the digital image. Once aligned, the guidance digital image is then used to guide generation of fill for the hole in the digital image. Machine learning techniques are used to determine which parts of the guidance digital image are to be blended to fill the hole in the digital image and which parts of the hole are to receive new content that is synthesized by the computing device.

Claims (34)

1. In a digital medium environment to fill a hole in a digital image, a method implemented by a computing device, the method comprising:

generating, by the computing device, an attention map based on the digital image and a guidance digital image, the attention map describing which part of the guidance digital image is to be used as a basis to fill the hole in the digital image;

filtering, by the computing device, a second feature map, generated from the guidance digital image using machine learning, based on the attention map;

combining, by the computing device, the filtered second feature map with a first feature map generated from the digital image using machine learning; and

generating, by the computing device, a filled digital image by decoding the filtered second feature map combined with the first feature map.

2. The method as described in claim 1 , further comprising generating the first feature map.

3. The method as described in claim 2 , wherein the generating of the first feature map is performed using a convolutional neural network and further comprising generating the second feature map using a convolutional neural network having a visual geometry group (VGG) architecture.

4. The method as described in claim 1 , wherein the generating of the attention map is performed using a convolutional neural network as part of machine learning.

5. The method as described in claim 1 , wherein the combining is performed using summation or concatenation.

6. The method as described in claim 1 , further comprising calculating an amount of error using a loss function by a generative adversarial network (GAN).

7. The method as described in claim 1 , further comprising calculating an amount of error based on a discriminator implemented by a neural network or perceptual loss.

8. The method as described in claim 1 , wherein the generating of the filled digital image using machine learning is performed by a deconvolution neural network by decoding the combined filtered second feature map, the first feature map, and at least a portion of the second feature map obtained via a skip layer from a convolutional neural network used to generate the second feature map using machine learning.

9. In a digital medium environment to fill a hole in a digital image, a computing device comprising:

a processing system; and

a computer-readable storage medium having instructions stored thereon that, responsive to execution by the processing system, causes the processing system to perform operations comprising:

generating an attention map based on the digital image and a guidance digital image, the attention map describing which a part of the guidance digital image to be used as a basis to fill the hole in the digital image;

filtering a second feature map based on the attention map, the second feature map generated using machine learning from the guidance digital image;

combining the filtered second feature map with a first feature map, the first feature map generating using machine learning from the digital image having to hole to be filled; and

generating a filled digital image by decoding the combined filtered second feature map and first feature map.

10. The computing device as described in claim 9 , the operations further comprising generating a search query to locate the guidance digital image based on a search of a digital image repository using features learned from the digital image having the hole to be filed.

11. The computing device as described in claim 9 , wherein the first feature map is generated using a convolutional neural network and the second feature map is generated using a convolutional neural network having a visual geometry group (VGG) architecture.

12. The computing device as described in claim 9 , wherein the generating of the attention map is performed using a convolutional neural network as part of machine learning.

13. In a digital medium environment to fill a hole in a digital image, a system comprising:

means for generating an attention map based on the digital image and a guidance digital image, the attention map describing which part of the guidance digital image is to be used as a basis to fill the hole in the digital image;

means for filtering a second feature map, generated from the guidance digital image using machine learning, based on the attention map;

means for combining the filtered second feature map with a first feature map generated from the digital image using machine learning; and

means for generating a filled digital image by decoding the filtered second feature map combined with the first feature map.

14. The system as described in claim 13 , wherein the attention map generating means generates the first feature map.

15. The system as described in claim 13 , further comprising means for generating the first feature map using a convolutional neural network and means for generating the second feature map using a convolutional neural network having a visual geometry group (VGG) architecture.

16. The system as described in claim 13 , wherein the attention map generating means includes a convolutional neural network.

17. The system as described in claim 13 , wherein the combining means includes means for summation or means for concatenation.

18. The system as described in claim 13 , further comprising means for calculating an amount of error using a loss function by a generative adversarial network (GAN).

19. The system as described in claim 13 , further comprising means for calculating an amount of error based on a discriminator implemented by a neural network or perceptual loss.

20. The system as described in claim 13 , wherein the filled digital image generating means includes a deconvolution neural network to decode the combined filtered second feature map, the first feature map, and at least a portion of the second feature map obtained via a skip layer from a convolutional neural network used to generate the second feature map using machine learning.

Assignments (2)
CHANGE OF NAME Recorded Jan 21, 2019
From: ADOBE SYSTEMS INCORPORATED
To: ADOBE INC.
Reel/Frame 048103/0226 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 13, 2018
From: PRICE, BRIAN LYNN; ZHAO, YINAN; COHEN, SCOTT DAVID
To: ADOBE SYSTEMS INCORPORATED
Reel/Frame 045182/0590 →
Continuity (1)
Related Publication 20190228508A1 · Jul 25, 2019