Generative attribute optimization
A generative attribute optimization (“GAO”) system facilitates understanding of effects of changes of attribute values of an object on a characteristic of the object and automatically identifying attribute values to achieve a desired result for the characteristic. The GAO system trains a generator (encoder and decoder) using an attribute generative adversarial network. The GAO model includes the trained generator and a separately trained predictor model. The GAO model inputs an input image and modified attribute values and employs the encoder and the decoder to generate a modified image that is the input image modified based on the modified attribute values. The GAO model then employs the predictor model to that inputs the modified image and generate a prediction of a characteristic of the modified image. The GAO system may employ an optimizer to modify the attribute values until an objective based on the desired result is achieved.
1. A method performed by one or more computing systems for generating a modified image corresponding to an input image, the method comprising:
applying a generator encoder to the input image to generate a latent vector representing the input image, the generator encoder and a corresponding generator decoder being trained using an attribute generative adversarial network based on attribute values of attributes of images;
initializing attribute values;
repeating until a termination criterion is satisfied:
applying the generator decoder to the latent vector and the attribute values to generate a modified image;
applying a predictor to the modified image to generate a prediction; and
adjusting the attribute values based on an objective for the prediction.
2. The method of claim 1 wherein the prediction is represented by a discrete variable.
3. The method of claim 1 wherein the prediction is represented by a continuous variable.
4. The method of claim 1 further comprising training the attribute generative adversarial network based on training data that include images labeled with attribute values.
5. The method of claim 1 wherein the adjusting of the attribute values applies a gradient descent technique to optimize an objective function relating to the objective.
6. The method of claim 5 wherein the objective function is a loss function.
7. A method performed by one or more computing systems for identifying attribute values of attributes of an input image so that the input image modified based on the attribute values satisfies an objective that is based on a desired prediction, the method comprising:
initializing the attribute values; and
repeating until a termination criterion is satisfied:
generating a modified image that is the input image modified to have the attribute values;
applying a predictor to the modified image to generate a prediction for the modified image; and
adjusting the attribute values based on an objective function relating to satisfaction of the objective based on relation between the generated prediction and the desired prediction
wherein when the termination criterion is satisfied, the objective is satisfied by attribute values.
8. The method of claim 7 further comprising applying a generator encoder to the input image to generate a latent vector representing the image and wherein the generating of the modified image includes applying a generator decoder to the latent vector and the attribute values to generate the modified image.
9. The method of claim 8 wherein the generator encoder and the generator decoder are generated based on training an attribute generative adversarial network.
10. The method of claim 8 wherein the predictor is a classifier and the prediction is a classification.
11. The method of claim 8 wherein the image is of a physical object and the prediction is of a characteristic of the physical object.
12. The method of claim 11 wherein the predictor is trained using training data that includes images labeled with values for the characteristic.
13. The method of claim 12 wherein the image is a scanning electron microscope image of the physical object.
14. The method of claim 13 wherein the characteristic is peak stress of the physical object.
15. The method of claim 11 wherein the object is a person.
16. One or more computing systems for identifying attribute values of attributes, the one or more computing system comprising:
one or more computer-readable storage mediums for storing computer-executable instructions for controlling the one or more computing systems to:
a generator encoder that inputs an input image and outputs a latent vector representing the input image;
a generator decoder that inputs the latent vector and attribute values and outputs a modified image that is the input image modified based on the attribute values;
a predictor that inputs the modified image and outputs a prediction relating to the image; and
an optimizer that inputs the prediction, adjusts the attribute values based on the prediction and an objective for the prediction, and outputs the attribute values after adjustment
wherein the generator decoder, the predictor, and optimizer repeatedly generate modified images, generate predictions, and adjust the attribute values until the objective for the prediction is satisfied; and
one or more processors for executing the computer-executable instructions stored in the one or more computer-readable storage mediums.
17. The one or more computing systems of claim 16 wherein the generator encoder and the generator decoder are generated based on training an attribute generative adversarial network.
18. The one or more computing systems of claim 16 wherein the predictor is a classifier and the prediction is a classification.
19. The one or more computing systems of claim 16 wherein the input image is of a physical object and the prediction is of a characteristic of the physical object.
20. The one or more computing systems of claim 19 wherein the image is a scanning electron microscope image of the physical object.
21. The one or more computing systems of claim 20 wherein the characteristic is peak stress of the physical object.
22. A method performed by one or more computing systems for generating a prediction relating to an input image based on modifying attribute values of attributes of the input image, the method comprising:
accessing a latent vector representing the image;
accessing modified attribute values;
applying a generator decoder that inputs the latent vector and the modified attribute values and outputs a modified image that is the input image modified based on the modified attribute values; and
applying a predictor that inputs the modified image and outputs the prediction relating to the modified image.
23. The method of claim 22 wherein the modified attribute values are specified by a person.
24. The method of claim 22 wherein the modified attribute values are specified by an optimizer that seeks to identify modified attribute values that result in a desired prediction.
25. The method of claim 22 further comprising applying a generator encoder to the image to generate the latent vector.
26. The method of claim 25 wherein the generator encoder and the generator decoder are generated based on training an attribute generative adversarial network.