Granted Patent B2

US 12,367,626 · App. 18/055,584 · Granted Jul 22, 2025

Inventors: Radomir Mech (Mountain View, CA); Nathan Carr (San Jose, CA); Matheus Gadelha (San Jose, CA)

Assignee: Adobe Inc.

G06T11/60G06T7/70G06T17/20G06T19/20G06T2219/2004

Abstract

Claims (71)

1. A method comprising:

determining, by at least one processor utilizing one or more neural networks, pixel depth values representing estimated depths for pixels of a two-dimensional image;

generating, by the at least one processor utilizing the one or more neural networks, a three-dimensional mesh by incorporating depth information from the pixel depth values representing the estimated depths at the pixels of the two-dimensional image into a tessellation of the two-dimensional image according to camera parameters extracted from the two-dimensional image;

detecting, by the at least one processor, a displacement input at a two-dimensional position on the two-dimensional image;

modifying, by the at least one processor in response to the displacement input at the two-dimensional position on the two-dimensional image, the three-dimensional mesh by determining a displaced portion of the three-dimensional mesh according to a mapping of the two-dimensional position of the displacement input to a three-dimensional position of the three-dimensional mesh; and

generating, by the at least one processor, a modified two-dimensional image comprising a displacement of a portion of the two-dimensional image by re-rendering the two-dimensional image according to the displaced portion of the three-dimensional mesh and the camera parameters extracted from the two-dimensional image.

2. The method of claim 1 , wherein:

detecting the displacement input comprises determining the two-dimensional position of the displacement input within the two-dimensional image comprising two-dimensional coordinates of the displacement input relative to the two-dimensional image; and

modifying the three-dimensional mesh comprises determining three-dimensional coordinates of the displacement input relative to the three-dimensional mesh based on the mapping of the two-dimensional coordinates of the displacement input to the three-dimensional position according to the camera parameters extracted from the two-dimensional image.

3. The method of claim 1 , wherein modifying the three-dimensional mesh comprises:

determining, based on an attribute of the displacement input, that the displacement input indicates a displacement direction for a portion of the three-dimensional mesh; and

displacing the displaced portion of the three-dimensional mesh in the displacement direction.

4. The method of claim 3 , wherein modifying the three-dimensional mesh comprises:

determining, based on an additional attribute of the displacement input, that the displacement input indicates an additional displacement direction of the portion of the three-dimensional mesh; and

displacing the displaced portion of the three-dimensional mesh according to the additional displacement direction.

5. The method of claim 4 , wherein modifying the three-dimensional mesh comprises:

selecting, in response to an additional input in connection with the displacement input, a new portion of the three-dimensional mesh to displace according to a mapping of an additional two-dimensional position of the additional input to an additional three-dimensional position of the three-dimensional mesh; and

displacing the new portion of the three-dimensional mesh according to the displacement input in the displacement direction.

6. The method of claim 5 , wherein modifying the three-dimensional mesh comprises displacing, based on movement of the displacement input, one or more additional portions of the three-dimensional mesh from the portion of the three-dimensional mesh to the new portion of the three-dimensional mesh in the displacement direction.

7. The method of claim 1 , wherein:

generating the three-dimensional mesh comprises generating a mapping of the pixels of the two-dimensional image to three-dimensional coordinates of the three-dimensional mesh based on the pixel depth values and the camera parameters extracted from the two-dimensional image; and

modifying the three-dimensional mesh comprises determining the mapping of the two-dimensional position of the displacement input to the three-dimensional position based on the mapping of the pixels of the two-dimensional image to the three-dimensional coordinates of the three-dimensional mesh.

8. The method of claim 1 , wherein generating the modified two-dimensional image comprises:

providing a preview two-dimensional image comprising the displacement of the portion of the two-dimensional image in response to the displacement input;

providing, for display within a graphical user interface, an option to commit the displacement of the portion of the two-dimensional image; and

generating the modified two-dimensional image comprising the displacement of the portion of the two-dimensional image in response to detecting a selection of the option to commit the displacement of the portion of the two-dimensional image.

9. The method of claim 1 , wherein modifying the three-dimensional mesh comprises:

determining a displacement filter indicating a shape associated with the displacement input; and

displacing the displaced portion of the three-dimensional mesh according to the shape of the displacement input in a direction of the displacement input.

10. A system comprising:

a memory component; and

a processing device coupled to the memory component, the processing device to perform operations comprising:

determining, utilizing one or more neural networks, pixel depth values representing estimated depths for pixels of a two-dimensional image based on objects of the two-dimensional image;

generating, utilizing the one or more neural networks, a three-dimensional mesh by incorporating depth information from the pixel depth values corresponding to the objects of the two-dimensional image into a tessellation of the two-dimensional image according to camera parameters extracted from the two-dimensional image;

determining a three-dimensional position of the three-dimensional mesh based on a corresponding position of a displacement input at a two-dimensional position on the two-dimensional image according to a mapping of the two-dimensional image to the three-dimensional mesh;

modifying, in response to the displacement input at the two-dimensional position on the two-dimensional image, the three-dimensional mesh by determining a displaced portion of the three-dimensional mesh at the three-dimensional position of the three-dimensional mesh; and

generating a modified two-dimensional image comprising a displacement of a portion of the two-dimensional image by re-rendering the two-dimensional image according to the displaced portion of the three-dimensional mesh at the three-dimensional position of the three-dimensional mesh and the camera parameters extracted from the two-dimensional image.

11. The system of claim 10 , wherein determining the three-dimensional position of the three-dimensional mesh comprises:

determining the two-dimensional position of the displacement input comprises determining two-dimensional coordinates of the displacement input relative to the two-dimensional image; and

determining the three-dimensional position of the three-dimensional mesh based on the two-dimensional coordinates of the displacement input relative to the two-dimensional image and a projection from the two-dimensional image onto the three-dimensional mesh corresponding to the mapping of the two-dimensional image to the three-dimensional mesh.

12. The system of claim 11 , wherein modifying the three-dimensional mesh comprises:

determining that the displacement input indicates a displacement of a portion of the three-dimensional mesh according to the three-dimensional position of the three-dimensional mesh; and

determining the displaced portion of the three-dimensional mesh according to a movement of the displacement input and the projection from the two-dimensional image onto the three-dimensional mesh.

13. The system of claim 12 , wherein generating the modified two-dimensional image comprises:

determining, based on the projection from the two-dimensional image onto the three-dimensional mesh, an additional two-dimensional position of the two-dimensional image corresponding to the displaced portion of the three-dimensional mesh; and

generating, based on the additional two-dimensional position of the two-dimensional image, the modified two-dimensional image comprising the displacement of the portion of the two-dimensional image according to the displaced portion of the three-dimensional mesh.

14. The system of claim 10 , wherein modifying the three-dimensional mesh comprises:

determining a direction of movement of the displacement input within a graphical user interface displaying the two-dimensional image;

determining a displacement height and a displacement radius based on the direction of movement of the displacement input; and

determining the displaced portion of the three-dimensional mesh based on the displacement height and the displacement radius.

15. The system of claim 10 , wherein modifying the three-dimensional mesh comprises:

determining one or more normal values corresponding to one or more vertices or one or more faces at the three-dimensional position of the three-dimensional mesh; and

determining, in response to the displacement input, the displaced portion of the three-dimensional mesh in one or more directions corresponding to the one or more normal values corresponding to the one or more vertices or the one or more faces.

16. The system of claim 10 , wherein modifying the three-dimensional mesh comprises:

determining a displacement direction based on a surface of a selected portion of the three-dimensional mesh; and

determining, in response to the displacement input, the displaced portion of the three-dimensional mesh in the displacement direction.

17. A non-transitory computer readable medium comprising instructions, which when executed by a processing device, cause the processing device to perform operations comprising:

determining, utilizing one or more neural networks, pixel depth values representing estimated depths for pixels of a two-dimensional image;

generating, utilizing the one or more neural networks, a three-dimensional mesh by incorporating depth information from the pixel depth values representing the estimated depths at the pixels of the two-dimensional image into a tessellation of the two-dimensional image according to camera parameters extracted from the two-dimensional image;

detecting a displacement input at a two-dimensional position on the two-dimensional image;

modifying, in response to the displacement input at the two-dimensional position on the two-dimensional image, the three-dimensional mesh by determining a displaced portion of the three-dimensional mesh according to a mapping of the two-dimensional position of the displacement input to a three-dimensional position of the three-dimensional mesh; and

generating a modified two-dimensional image comprising a displacement of a portion of the two-dimensional image by re-rendering the two-dimensional image according to the displaced portion of the three-dimensional mesh and the camera parameters extracted from the two-dimensional image.

18. The non-transitory computer readable medium of claim 17 , wherein generating the three-dimensional mesh comprises:

generating the three-dimensional mesh by determining displacement of vertices of the tessellation of the two-dimensional image based on the pixel depth values and the camera parameters extracted from the two-dimensional image; or

generating the three-dimensional mesh by generating the tessellation based on a plurality of points sampled in the two-dimensional image according to density values determined from the pixel depth values of the two-dimensional image.

19. The non-transitory computer readable medium of claim 17 , wherein modifying the three-dimensional mesh comprises:

determining a projection from the two-dimensional image onto the three-dimensional mesh; and

determining the mapping of the two-dimensional position of the displacement input to the three-dimensional position of the three-dimensional mesh, according to the projection from the two-dimensional image onto the three-dimensional mesh.

20. The non-transitory computer readable medium of claim 19 , wherein modifying the three-dimensional mesh comprises:

determining, based on the projection from the two-dimensional image onto the three-dimensional mesh, movement of the displacement input relative to the two-dimensional image and a corresponding movement of the displacement input relative to the three-dimensional mesh; and

determining the displaced portion of the three-dimensional mesh based on the corresponding movement of the displacement input relative to the three-dimensional mesh.

Assignments (1)

Continuity (1)

References Cited (130)