Machine learning based image processing techniques
A machine learning based image processing architecture and associated applications are disclosed herein. In some embodiments, a machine learning framework is trained to learn low level image attributes such as object/scene types, geometries, placements, materials and textures, camera characteristics, lighting characteristics, contrast, noise statistics, etc. Thereafter, the machine learning framework may be employed to detect such attributes in other images and process the images at the attribute level.
1. A method, comprising:
detecting a set of one or more attributes of an input image using a machine learning framework, wherein the machine learning framework is trained at least in part on a set of training images comprising a prescribed scene type to which the input image belongs and wherein training images comprising the set of training images are labeled with corresponding sets of labels from which attributes associated with the prescribed scene type are learned by the machine learning framework; and
generating an output image comprising a modified version of the input image by modifying at least a subset of the detected set of attributes.
2. The method of claim 1 , wherein the prescribed scene type comprises one or more of a constrained set of objects.
3. The method of claim 1 , wherein the set of training images includes images comprising different combinations of objects and object arrangements, camera configurations, lighting types and locations, and materials and textures.
4. The method of claim 1 , wherein a training image of the set of training images is rendered using one or more three-dimensional models, is captured by an imaging or a scanning device, or is generated from one or more other existing images.
5. The method of claim 1 , wherein at least the subset of the detected set of attributes is associated with an aesthetic.
6. The method of claim 5 , wherein the output image comprises a different aesthetic than the input image.
7. The method of claim 1 , wherein at least the subset of the detected set of attributes is associated with a style.
8. The method of claim 7 , wherein the output image comprises a restyled version of the input image.
9. The method of claim 1 , wherein at least the subset of the detected set of attributes is associated with an object in the input image.
10. The method of claim 9 , wherein the object in the input image is replaced by a different object in the output image.
11. The method of claim 1 , wherein at least the subset of the detected set of attributes is associated with lighting.
12. The method of claim 11 , wherein the output image comprises a relit version of the input image.
13. The method of claim 1 , wherein at least the subset of the detected set of attributes is associated with noise.
14. The method of claim 13 , wherein the output image comprises a denoised version of the input image.
15. The method of claim 1 , further comprising labeling or tagging the output image with a corresponding set of attributes.
16. The method of claim 1 , wherein the detected set of attributes comprises one or more attributes associated with object or scene types, geometries, placements, materials, textures, camera characteristics, lighting characteristics, noise statistics, and contrast.
17. The method of claim 1 , wherein a corresponding set of labels of a training image of the set of training images comprises one or more scene-based labels, image-based labels, or both.
18. The method of claim 1 , wherein a training image of the set of training images is labeled with ground truth data associated with generating the training image.
19. The method of claim 1 , wherein the output image comprises a photograph or a photorealistic rendering.
20. The method of claim 1 , wherein the output image comprises a video frame.
21. The method of claim 1 , wherein the prescribed scene type is associated with a retailer or a brand.
22. The method of claim 1 , wherein the prescribed scene type is associated with an animation or a video sequence.
23. The method of claim 1 , wherein the machine learning framework comprises a deep neural network, a convolutional neural network, or both.
24. A system, comprising:
a processor configured to:
detect a set of one or more attributes of an input image using a machine learning framework, wherein the machine learning framework is trained at least in part on a set of training images comprising a prescribed scene type to which the input image belongs and wherein training images comprising the set of training images are labeled with corresponding sets of labels from which attributes associated with the prescribed scene type are learned by the machine learning framework; and
generate an output image comprising a modified version of the input image by modifying at least a subset of the detected set of attributes; and
a memory coupled to the processor and configured to provide the processor with instructions.
25. A computer program product embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
detecting a set of one or more attributes of an input image using a machine learning framework, wherein the machine learning framework is trained at least in part on a set of training images comprising a prescribed scene type to which the input image belongs and wherein training images comprising the set of training images are labeled with corresponding sets of labels from which attributes associated with the prescribed scene type are learned by the machine learning framework; and
generating an output image comprising a modified version of the input image by modifying at least a subset of the detected set of attributes.