Machine learning based image processing techniques
A machine learning based image processing architecture and associated applications are disclosed herein. In some embodiments, a machine learning framework is trained to learn low level image attributes such as object/scene types, geometries, placements, materials and textures, camera characteristics, lighting characteristics, contrast, noise statistics, etc. Thereafter, the machine learning framework may be employed to detect such attributes in other images and process the images at the attribute level.
1. A method, comprising:
using a machine learning framework to detect a set of one or more attributes of an input image, wherein the set of attributes comprises attributes associated with a scene comprising the input image, wherein the machine learning framework is trained on datasets comprising labeled images rendered from three-dimensional object models, and wherein the machine learning framework is trained on image datasets comprising permutations of a constrained set of objects associated with a prescribed scene type to which the input image belongs;
modifying the input image to generate an output image comprising a modified version of the input image by modifying at least a subset of the detected set of attributes; and
outputting the output image comprising the modified version of the input image.
2. The method of claim 1 , wherein the detected set of attributes is not known for the input image prior to detection by the machine learning framework.
3. The method of claim 1 , wherein the detected set of attributes is associated with a style or aesthetic.
4. The method of claim 3 , wherein the output image comprises a restyled version of the input image.
5. The method of claim 1 , wherein the detected set of attributes is associated with a first style and a modified set of attributes of the output image is associated with a second style.
6. The method of claim 5 , wherein modifying the input image comprises modifying the first style to the second style.
7. The method of claim 1 , wherein the detected set of attributes is associated with an object in the input image.
8. The method of claim 7 , wherein the object in the input image is replaced by a different object in the output image.
9. The method of claim 1 , wherein the detected set of attributes is associated with lighting.
10. The method of claim 9 , wherein the output image comprises a relit version of the input image.
11. The method of claim 1 , wherein the detected set of attributes is associated with noise.
12. The method of claim 11 , wherein the output image comprises a denoised version of the input image.
13. The method of claim 1 , further comprising labeling or tagging the output image with a modified set of attributes.
14. The method of claim 1 , wherein the detected set of attributes comprises one or more attributes associated with object/scene types, geometries, placements, materials, textures, camera characteristics, lighting characteristics, noise statistics, and contrast.
15. The method of claim 1 , wherein the input image is edited to generate the output image based on attribute detection and modification instead of pixel level editing operations.
16. The method of claim 1 , wherein the input image and the output image each comprises a photograph or a photorealistic rendering.
17. The method of claim 1 , wherein the input image and the output image each comprises a frame of an animation or a video sequence.
18. A system, comprising:
a processor configured to:
use a machine learning framework to detect a set of one or more attributes of an input image, wherein the set of attributes comprises attributes associated with a scene comprising the input image, wherein the machine learning framework is trained on datasets comprising labeled images rendered from three-dimensional object models, and wherein the machine learning framework is trained on image datasets comprising permutations of a constrained set of objects associated with a prescribed scene type to which the input image belongs;
modify the input image to generate an output image comprising a modified version of the input image by modifying at least a subset of the detected set of attributes; and
output the output image comprising the modified version of the input image; and
a memory coupled to the processor and configured to provide the processor with instructions.
19. The system of claim 18 , wherein the detected set of attributes is not known for the input image prior to detection by the machine learning framework.
20. The system of claim 18 , wherein the detected set of attributes is associated with a style or aesthetic.
21. The system of claim 20 , wherein the output image comprises a restyled version of the input image.
22. The system of claim 18 , wherein the detected set of attributes is associated with a first style and a modified set of attributes of the output image is associated with a second style.
23. The system of claim 22 , wherein to modify the input image comprises to modify the first style to the second style.
24. The system of claim 18 , wherein the detected set of attributes is associated with an object in the input image.
25. The system of claim 24 , wherein the object in the input image is replaced by a different object in the output image.
26. The system of claim 18 , wherein the detected set of attributes is associated with lighting.
27. The system of claim 26 , wherein the output image comprises a relit version of the input image.
28. The system of claim 18 , wherein the detected set of attributes is associated with noise.
29. The system of claim 28 , wherein the output image comprises a denoised version of the input image.
30. The system of claim 18 , wherein the processor is further configured to label or tag the output image with a modified set of attributes.
31. The system of claim 18 , wherein the detected set of attributes comprises one or more attributes associated with object/scene types, geometries, placements, materials, textures, camera characteristics, lighting characteristics, noise statistics, and contrast.
32. The system of claim 18 , wherein the input image is edited to generate the output image based on attribute detection and modification instead of pixel level editing operations.
33. The system of claim 18 , wherein the input image and the output image each comprises a photograph or a photorealistic rendering.
34. The system of claim 18 , wherein the input image and the output image each comprises a frame of an animation or a video sequence.
35. A computer program product embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
using a machine learning framework to detect a set of one or more attributes of an input image, wherein the set of attributes comprises attributes associated with a scene comprising the input image, wherein the machine learning framework is trained on datasets comprising labeled images rendered from three-dimensional object models, and wherein the machine learning framework is trained on image datasets comprising permutations of a constrained set of objects associated with a prescribed scene type to which the input image belongs;
modifying the input image to generate an output image comprising a modified version of the input image by modifying at least a subset of the detected set of attributes; and
outputting the output image comprising the modified version of the input image.
36. The computer program product of claim 35 , wherein the detected set of attributes is not known for the input image prior to detection by the machine learning framework.
37. The computer program product of claim 35 , wherein the detected set of attributes is associated with a style or aesthetic.
38. The computer program product of claim 37 , wherein the output image comprises a restyled version of the input image.
39. The computer program product of claim 35 , wherein the detected set of attributes is associated with a first style and a modified set of attributes of the output image is associated with a second style.
40. The computer program product of claim 39 , wherein modifying the input image comprises modifying the first style to the second style.
41. The computer program product of claim 35 , wherein the detected set of attributes is associated with an object in the input image.
42. The computer program product of claim 41 , wherein the object in the input image is replaced by a different object in the output image.
43. The computer program product of claim 35 , wherein the detected set of attributes is associated with lighting.
44. The computer program product of claim 43 , wherein the output image comprises a relit version of the input image.
45. The computer program product of claim 35 , wherein the detected set of attributes is associated with noise.
46. The computer program product of claim 45 , wherein the output image comprises a denoised version of the input image.
47. The computer program product of claim 35 , further comprising computer instructions for labeling or tagging the output image with a modified set of attributes.
48. The computer program product of claim 35 , wherein the detected set of attributes comprises one or more attributes associated with object/scene types, geometries, placements, materials, textures, camera characteristics, lighting characteristics, noise statistics, and contrast.
49. The computer program product of claim 35 , wherein the input image is edited to generate the output image based on attribute detection and modification instead of pixel level editing operations.
50. The computer program product of claim 35 , wherein the input image and the output image each comprises a photograph or a photorealistic rendering.
51. The computer program product of claim 35 , wherein the input image and the output image each comprises a frame of an animation or a video sequence.