Machine learning training method, system, and device
Fill techniques as implemented by a computing device are described to perform hole filling of a digital image. In one example, deeply learned features of a digital image using machine learning are used by a computing device as a basis to search a digital image repository to locate the guidance digital image. Once located, machine learning techniques are then used to align the guidance digital image with the hole to be filled in the digital image. Once aligned, the guidance digital image is then used to guide generation of fill for the hole in the digital image. Machine learning techniques are used to determine which parts of the guidance digital image are to be blended to fill the hole in the digital image and which parts of the hole are to receive new content that is synthesized by the computing device.
1. In a digital medium environment, a method implemented by a computing device, the method comprising:
adding, by the computing device, a portion from a first digital image to a second digital image;
blending, by the computing device, the added portion as part of the second digital image;
selecting, by the computing device, a training portion of the second digital image that encompasses the blended portion and a part of the second digital image in addition to the blended portion;
generating, by the computing device, a training guidance digital image by adding the training portion to the first digital image; and
training, by the computing device, a model using machine learning based on the training guidance digital image and the first digital image as a training ground truth digital image.
2. The method as described in claim 1 , further comprising determining a loss between the training ground truth digital image and the training guidance digital image.
3. The method as described in claim 2 , further comprising adjusting the model based on the loss.
4. The method as described in claim 2 , further comprising generating a second training guidance digital image based on the loss.
5. The method as described in claim 1 , wherein the generating the training guidance digital image includes blending the training portion with the first digital image.
6. The method as described in claim 1 , further comprising identifying which pixels from a subsequent guidance image best match a subsequent digital image based on the model.
7. The method as described in claim 1 , further comprising generating an attention map of a subsequent digital image based on the model.
8. In a digital medium environment to train a model with digital images, a system comprising:
a portion selection module implemented at least partially in hardware of a computing device to add a portion from a first digital image to a second digital image;
a portion blending module implemented at least partially in hardware of the computing device to blend the added portion as part of the second digital image;
a training portion selection module implemented at least partially in hardware of the computing device to select a training portion of the second digital image that encompasses the blended portion and a part of the second digital image in addition to the blended portion;
a guidance generation module implemented at least partially in hardware of the computing device to generate a training guidance digital image by adding the training portion to the first digital image; and
a machine learning system implemented at least partially in hardware of the computing device to train the model using machine learning based on the training guidance digital image and the first digital image as a training ground truth digital image.
9. The system as described in claim 8 , the machine learning system further implemented to determine a loss between the training ground truth digital image and the training guidance digital image.
10. The system as described in claim 9 , the machine learning system further implemented to adjust the model based on the loss.
11. The system as described in claim 9 , wherein the loss is a perceptual loss.
12. The system as described in claim 9 , the machine learning system further implemented to generate a second training guidance digital image based on the loss.
13. The system as described in claim 8 , the machine learning system further implemented to identify which pixels from a subsequent guidance image best match a subsequent digital image based on the model.
14. The system as described in claim 8 , the machine learning system further implemented to generate an attention map of a subsequent digital image based on the model.
15. In a digital medium environment to train a model with digital images, a computing device comprising:
a processing system; and
a computer-readable storage medium having instructions stored thereon that, responsive to execution by the processing system, causes the processing system to perform operations including:
blending a portion from a first digital image as part of a second digital image;
selecting a training portion of the second digital image that encompasses the blended portion and a part of the second digital image in addition to the blended portion;
generating a training guidance digital image by adding the training portion to the first digital image; and
training the model using machine learning based on the training guidance digital image and the first digital image as a training ground truth digital image.
16. The computing device as described in claim 15 , the operations further comprising:
determining a loss between the training ground truth digital image and the training guidance digital image; and
adjusting the model based on the loss.
17. The computing device as described in claim 16 , the operations further comprising generating a second training guidance digital image based on the loss.
18. The computing device as described in claim 15 , wherein the generating the training guidance digital image includes blending the training portion with the first digital image.
19. The computing device as described in claim 15 , the operations further comprising identifying which pixels from a subsequent guidance image best match a subsequent digital image based on the model.
20. The computing device as described in claim 15 , the operations further comprising generating an attention map of a subsequent digital image based on the model.