Systems and methods for selective image compositing
Disclosed are techniques for generating a photorealistic image by augmenting or compositing at least a portion of a physical structure (e.g., a house) depicted in a two-dimensional (2D) image with synthetic image data. Additionally, disclosed are techniques for augmenting the depicted physical structure using a minimum amount of three-dimensional (3D) geometric data and applying a scene effect to the synthetic image data to create a photorealistic effect.
1. A computer-implemented method, comprising:
receiving a two-dimensional (2D) image and metadata, the 2D image including a set of pixels depicting a physical structure captured by an image capturing device, and the metadata representing one or more characteristics of the image capturing device;
identifying a portion of the 2D image to augment with synthetic image data;
segmenting the set of pixels of the 2D image into one or more subsets of pixels;
identifying, from amongst the one or more subsets of pixels, a subset of pixels corresponding to the identified portion of the 2D image;
generating a reference three-dimensional (3D) model of the physical structure from the 2D image, the reference 3D model representing the identified portion of the 2D image in a virtual space, and the generation including determining a 3D orientation of a 3D planar surface of the reference 3D model, wherein determining the 3D orientation comprises:
associating the 3D planar surface with the identified portion of the 2D image to augment with the synthetic image data;
extracting a surface normal value of the identified portion of the 2D image; and
orienting the reference 3D model to align the 3D planar surface according to the extracted surface normal;
applying the synthetic image data onto the reference 3D model;
reprojecting a select 3D geometry of the reference 3D model with the synthetic image data over the portion of the 2D image according to the identified subset of pixels; and
rendering a photorealistic image using the 2D image, the metadata, and the reprojected selected 3D geometry with the synthetic image data.
2. The computer-implemented method of claim 1 , wherein generating the reference 3D model further comprises:
inputting the 2D image into a trained machine-learning model, the trained machine-learning model having been trained to generate the reference 3D model using the 2D image; and
generating, based on an output of the trained machine-learning model, the reference 3D model to represent the identified portion of the 2D image in the virtual space, wherein the reference 3D model is an untextured block or planar representation of the physical structure.
3. The computer-implemented method of claim 1 , wherein extracting a surface normal value of the identified portion of the 2D image further comprises:
extracting a plurality of lines from the subset of pixels corresponding to the identified portion of the 2D image;
classifying, using a trained machine-learning model, each of the plurality of lines as a structural feature of the physical structure;
determining an angular difference between two classified lines is a predetermined angle, the two classified lines forming the 3D planar surface; and
computing a cross product of the two classified lines to generate a surface normal of the 3D planar surface.
4. The computer-implemented method of claim 1 , wherein generating the reference 3D model further comprises:
extracting one or more lines from the set of pixels of the 2D image;
building a virtual wire frame in the virtual space using the extracted one or more lines, the virtual wire frame representing one or more edges of the physical structure;
identifying a portion of the virtual wire frame that corresponds to the identified portion of the 2D image; and
supplementing the portion of the virtual wire frame with the synthetic image data.
5. The computer-implemented method of claim 1 , wherein the image capturing device is a Light Detecting and Ranging (LiDAR) depth camera, and wherein generating the 3D model further comprises:
generating a 3D point cloud using the LiDAR depth camera;
identifying a portion of the 3D point cloud that models the identified portion of the 2D image, the portion of the 3D point cloud being characterized by a surface orientation;
identifying three 3D points from amongst the 3D point cloud, each of the three 3D points being associated with a vector;
computing a cross product of vectors between the three 3D points, the cross product resulting in a surface normal of the identified portion of the 3D point cloud;
determining a 3D surface orientation of the identified portion of the 3D point cloud, the 3D surface orientation being determined using the surface normal;
retrieving one or more image swatches; and
warping each image swatch of the one or more image swatches according to the surface orientation of the portion of the 3D point cloud.
6. The computer-implemented method of claim 1 , wherein rendering the photorealistic image further comprises:
determining a scene effect associated with the 2D image by detecting at least one of the following from the 2D image or the metadata:
a color cast from the 2D image;
a film grain associated with the 2D image;
a chromatic aberration;
a weather condition;
a direction of a light source; or
a lens or color effect;
modifying the synthetic image data applied to the reference 3D model, wherein the modification is based on the determined scene effect; and
rendering the photorealistic image using the modified synthetic image data applied to the reference 3D model.
7. The computer-implemented method of claim 1 , wherein the reference 3D model represents the identified portion of the physical structure depicted in the 2D image only, as opposed to modeling an entirety of the physical structure.
8. A system, comprising:
one or more processors; and
a non-transitory computer-readable storage medium containing instructions which, when executed on the one or more processors, cause the one or more processors to perform operations including:
receiving a two-dimensional (2D) image and metadata, the 2D image including a set of pixels depicting a physical structure captured by an image capturing device, and the metadata representing one or more characteristics of the image capturing device;
identifying a portion of the 2D image to augment with synthetic image data;
segmenting the set of pixels of the 2D image into one or more subsets of pixels;
identifying, from amongst the one or more subsets of pixels, a subset of pixels corresponding to the identified portion of the 2D image;
generating a reference three-dimensional (3D) model of the physical structure from the 2D image, the reference 3D model representing the identified portion of the 2D image in a virtual space, and the generation including determining a 3D orientation of a 3D planar surface of the reference 3D model, wherein determining the 3D orientation comprises:
associating the 3D planar surface with the identified portion of the 2D image to augment with the synthetic image data;
extracting a surface normal value of the identified portion of the 2D image; and
orienting the reference 3D model to align the 3D planar surface according to the extracted surface normal;
applying the synthetic image data onto the reference 3D model;
reprojecting a select 3D geometry of the reference 3D model with the synthetic image data over the portion of the 2D image according to the identified subset of pixels; and
rendering a photorealistic image using the 2D image, the metadata, and the reprojected selected 3D geometry with the synthetic image data.
9. The system of claim 8 , wherein generating the reference 3D model further comprises:
inputting the 2D image into a trained machine-learning model, the trained machine-learning model having been trained to generate the reference 3D model using the 2D image; and
generating, based on an output of the trained machine-learning model, the reference 3D model to represent the identified portion of the 2D image in the virtual space, wherein the reference 3D model is an untextured block or planar representation of the physical structure.
10. The system of claim 8 , wherein determining the 3D orientation of the 3D planar surface further comprises:
extracting a plurality of lines from the identified subset of pixels;
classifying, using a trained machine-learning model, each of the plurality of lines as a structural feature of the physical structure;
determining that an angular difference between two classified lines is a predetermined angle, the two classified lines forming the 3D planar surface;
computing a cross product of the two classified lines to generate a surface normal of the 3D planar surface; and
comparing the surface normal of the 3D planar surface with a vertical vector associated with the 2D image to determine a pitch of the 3D planar surface.
11. The system of claim 8 , wherein generating the reference 3D model further comprises:
extracting one or more lines from the set of pixels of the 2D image;
building a virtual wire frame in the virtual space using the extracted one or more lines, the virtual wire frame representing one or more edges of the physical structure;
identifying a portion of the virtual wire frame that corresponds to the identified portion of the 2D image; and
supplementing the portion of the virtual wire frame with the synthetic image data.
12. The system of claim 8 , wherein the image capturing device is a Light Detecting and Ranging (LiDAR) depth camera, and wherein generating the 3D model further comprises:
generating a 3D point cloud using the LiDAR depth camera;
identifying a portion of the 3D point cloud that models the identified portion of the 2D image, the portion of the 3D point cloud being characterized by a surface orientation;
identifying three 3D points from amongst the 3D point cloud, each of the three 3D points being associated with a vector;
computing a cross product of vectors between the three 3D points, the cross product resulting in a surface normal of the identified portion of the 3D point cloud;
determining a 3D surface orientation of the identified portion of the 3D point cloud, the 3D surface orientation being determined using the surface normal;
retrieving one or more image swatches; and
warping each image swatch of the one or more image swatches according to the surface orientation of the portion of the 3D point cloud.
13. The system of claim 8 , wherein rendering the photorealistic image further comprises:
determining a scene effect associated with the 2D image by detecting at least one of the following from the 2D image or the metadata:
a color cast from the 2D image;
a film grain associated with the 2D image;
a chromatic aberration;
a weather condition;
a direction of a light source; or
a lens or color effect;
modifying the synthetic image data applied to the reference 3D model, wherein the modification is based on the determined scene effect; and
rendering the photorealistic image using the modified synthetic image data applied to the reference 3D model.
14. The system of claim 8 , wherein the reference 3D model represents the identified portion of the physical structure depicted in the 2D image only, as opposed to modeling an entirety of the physical structure.
15. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause a processing apparatus to perform operations including:
receiving a two-dimensional (2D) image and metadata, the 2D image including a set of pixels depicting a physical structure captured by an image capturing device, and the metadata representing one or more characteristics of the image capturing device;
identifying a portion of the 2D image to augment with synthetic image data;
segmenting the set of pixels of the 2D image into one or more subsets of pixels;
identifying, from amongst the one or more subsets of pixels, a subset of pixels corresponding to the identified portion of the 2D image;
generating a reference three-dimensional (3D) model of the physical structure from the 2D image, the reference 3D model representing the identified portion of the 2D image in a virtual space, and the generation including determining a 3D orientation of a 3D planar surface of the reference 3D model, wherein determining the 3D orientation comprises:
associating the 3D planar surface with the identified portion of the 2D image to augment with the synthetic image data;
extracting a surface normal value of the identified portion of the 2D image; and
orienting the reference 3D model to align the 3D planar surface according to the extracted surface normal;
applying the synthetic image data onto the reference 3D model;
reprojecting a select 3D geometry of the reference 3D model with the synthetic image data over the portion of the 2D image according to the identified subset of pixels; and
rendering a photorealistic image using the 2D image, the metadata, and the reprojected selected 3D geometry with the synthetic image data.
16. The computer-program product of claim 15 , wherein generating the reference 3D model further comprises:
inputting the 2D image into a trained machine-learning model, the trained machine-learning model having been trained to generate the reference 3D model using the 2D image; and
generating, based on an output of the trained machine-learning model, the reference 3D model to represent the identified portion of the 2D image in the virtual space, wherein the reference 3D model is an untextured block or planar representation of the physical structure.
17. The computer-program product of claim 15 , wherein determining the 3D orientation of the 3D planar surface further comprises:
extracting a plurality of lines from the identified subset of pixels;
classifying, using a trained machine-learning model, each of the plurality of lines as a structural feature of the physical structure;
determining that an angular difference between two classified lines is a predetermined angle, the two classified lines forming the 3D planar surface;
computing a cross product of the two classified lines to generate a surface normal of the 3D planar surface; and
comparing the surface normal of the 3D planar surface with a vertical vector associated with the 2D image to determine a pitch of the 3D planar surface.
18. The computer-program product of claim 15 , wherein generating the reference 3D model further comprises:
extracting one or more lines from the set of pixels of the 2D image;
building a virtual wire frame in the virtual space using the extracted one or more lines, the virtual wire frame representing one or more edges of the physical structure;
identifying a portion of the virtual wire frame that corresponds to the identified portion of the 2D image; and
supplementing the portion of the virtual wire frame with the synthetic image data.
19. The computer-program product of claim 15 , wherein the image capturing device is a Light Detecting and Ranging (LiDAR) depth camera, and wherein generating the 3D model further comprises:
generating a 3D point cloud using the LiDAR depth camera;
identifying a portion of the 3D point cloud that models the identified portion of the 2D image, the portion of the 3D point cloud being characterized by a surface orientation;
identifying three 3D points from amongst the 3D point cloud, each of the three 3D points being associated with a vector;
computing a cross product of vectors between the three 3D points, the cross product resulting in a surface normal of the identified portion of the 3D point cloud;
determining a 3D surface orientation of the identified portion of the 3D point cloud, the 3D surface orientation being determined using the surface normal;
retrieving one or more image swatches; and
warping each image swatch of the one or more image swatches according to the surface orientation of the portion of the 3D point cloud.
20. The computer-program product of claim 15 , wherein rendering the photorealistic image further comprises:
determining a scene effect associated with the 2D image by detecting at least one of the following from the 2D image or the metadata:
a color cast from the 2D image;
a film grain associated with the 2D image;
a chromatic aberration;
a weather condition;
a direction of a light source; or
a lens or color effect;
modifying the synthetic image data applied to the reference 3D model, wherein the modification is based on the determined scene effect; and
rendering the photorealistic image using the modified synthetic image data applied to the reference 3D model.