Systems and methods for selective image compositing
Disclosed are techniques for generating a photorealistic image by augmenting or compositing at least a portion of a physical structure (e.g., a house) depicted in a two-dimensional (2D) image with synthetic image data. Additionally, disclosed are techniques for augmenting the depicted physical structure and applying a scene effect to the synthetic image data to create a photorealistic effect.
1. A method of generating composite images, comprising:
receiving a two-dimensional (2D) image frame, the 2D image frame including a set of pixels depicting a physical structure captured by an image capture device;
segmenting the set of pixels of the 2D image into one or more subsets of pixels;
identifying, from amongst the one or more subsets of pixels, a subset of pixels to augment with synthetic image data;
receiving metadata of the 2D image frame comprising three-dimensional (3D) orientation information of a geometry associated with the identified subset of pixels, the 3D orientation comprising a surface normal prediction relative to the geometry;
providing synthetic image data based on the metadata; and
displaying the synthetic image data over a corresponding portion of the 2D image frame, according to a position in the 2D image frame of the identified subset of pixels and the metadata.
2. The method of claim 1 , wherein the metadata comprises scene effect information.
3. The method of claim 1 , wherein the surface normal prediction is provided by extracting two 3D lines from the 2D image frame and computing a cross product of the two 3D lines.
4. The method of claim 3 , wherein extracting two 3D lines comprises selecting lines according to the segmenting of the set of pixels.
5. The method of claim 4 , wherein the two 3D lines are an eave line and a rake line of the physical structure.
6. The method of claim 3 , wherein extracting the two 3D lines comprises generating a vanishing point coordinate system.
7. The method of claim 1 , further comprising warping the synthetic image data according to the 3D orientation information.
8. The method of claim 1 , wherein providing the synthetic image data further comprises orienting the synthetic image data according to a pose of the image capture device.
9. The method of claim 1 , wherein receiving the 2D image frame further comprises extracting one or more lines from the set of pixels of the 2D image frame.
10. The method of claim 9 , further comprising rectifying the one or more lines relative to a render space.
11. The method of claim 1 , wherein surface normal prediction is provided by one or more trained machine-learning models.
12. The method of claim 1 , wherein displaying the synthetic image data over the corresponding portion of the 2D image frame comprises performing a boundary fill function using the synthetic image data, wherein the boundary fill function comprises a digital swatch or collection of pixels visually sampling a texture material to fill a close boundary of the subset of pixels.
13. A system, comprising:
one or more processors; and
one or more memory devices storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving a two-dimensional (2D) image frame and metadata, the 2D image frame including a set of pixels depicting a physical structure captured by an image capture device;
segmenting the set of pixels of the 2D image into one or more subsets of pixels;
identifying, from amongst the one or more subsets of pixels, a subset of pixels to augment with synthetic image data;
receiving metadata of the 2D image frame comprising three-dimensional (3D) orientation information of a geometry associated with the identified subset of pixels, the 3D orientation comprising a surface normal prediction relative to the geometry;
providing synthetic image data based on the metadata; and
displaying the synthetic image data over a corresponding portion of the 2D image frame, according to a position in the 2D image frame of the identified subset of pixels and the metadata.
14. The system of claim 13 , wherein the metadata comprises scene effect information.
15. The system of claim 13 , wherein the surface normal prediction is provided by extracting two 3D lines from the 2D image frame and computing a cross product of the two 3D lines, and extracting two 3D lines comprises selecting lines according to the segmenting of the set of pixels.
16. The system of claim 13 , wherein providing the synthetic image data further comprises orienting the synthetic image data according to a pose of the image capture device.
17. The system of claim 13 , wherein receiving the 2D image frame further comprises extracting one or more lines from the set of pixels of the 2D image frame, and the operations further comprise rectifying the one or more lines relative to a render space.