Generating scale fields indicating pixel-to-metric distances relationships in digital images via neural networks
The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify two-dimensional images via scene-based editing using three-dimensional representations of the two-dimensional images. For instance, in one or more embodiments, the disclosed systems utilize three-dimensional representations of two-dimensional images to generate and modify shadows in the two-dimensional images according to various shadow maps. Additionally, the disclosed systems utilize three-dimensional representations of two-dimensional images to modify humans in the two-dimensional images. The disclosed systems also utilize three-dimensional representations of two-dimensional images to provide scene scale estimation via scale fields of the two-dimensional images. In some embodiments, the disclosed systems utilizes three-dimensional representations of two-dimensional images to generate and visualize 3D planar surfaces for modifying objects in two-dimensional images. The disclosed systems further use three-dimensional representations of two-dimensional images to customize focal points for the two-dimensional images.
1 . A computer-implemented method comprising:
generating, by at least one processor utilizing one or more neural networks, a feature representation of a two-dimensional image;
generating, by the at least one processor utilizing the one or more neural networks and based on the feature representation, a scale field for the two-dimensional image comprising pixel-to-metric ratios between:
pixel distances from pixels to a horizon line in the two-dimensional image; and
metric distances from corresponding three-dimensional points to a three-dimensional horizon line in a three-dimensional space corresponding to the two-dimensional image; and
performing at least one of:
generating, by the at least one processor, a metric distance of content portrayed in the two-dimensional image utilizing the pixel-to-metric ratios of the scale field of the two-dimensional image; or
modifying, by the at least one processor, the two-dimensional image utilizing the pixel-to-metric ratios of the scale field of the two-dimensional image.
2 . The computer-implemented method of claim 1 , further comprising generating, utilizing the one or more neural networks and based on the feature representation, a plurality of ground-to-horizon vectors in the three-dimensional space according to the horizon line of the two-dimensional image projected into the three-dimensional space.
3 . The computer-implemented method of claim 2 , wherein generating the plurality of ground-to-horizon vectors comprises generating a ground-to-horizon vector indicating a distance and a direction from a three-dimensional point corresponding to a pixel of the two-dimensional image to the three-dimensional horizon line in the three-dimensional space.
4 . The computer-implemented method of claim 1 , wherein generating the scale field for the two-dimensional image comprises generating, for a pixel of the two-dimensional image, a pixel-to-metric ratio between a pixel distance in the two-dimensional image and a corresponding three-dimensional distance in the three-dimensional space relative to a camera height of the two-dimensional image.
5 . The computer-implemented method of claim 1 , wherein generating the metric distance of the content portrayed in the two-dimensional image comprises:
determining a pixel distance between a first pixel corresponding to the content and a second pixel corresponding to the content; and
generating the metric distance based on the pixel distance and the pixel-to-metric ratios between the pixel distances in the two-dimensional image and the metric distances in the three-dimensional space.
6 . The computer-implemented method of claim 5 , wherein generating the metric distance of the content portrayed in the two-dimensional image comprises:
determining a value of the scale field corresponding to the first pixel; and
converting the value of the scale field corresponding to the first pixel to the metric distance based on the pixel distance between the first pixel and the second pixel.
7 . The computer-implemented method of claim 1 , wherein modifying the two-dimensional image comprises:
determining a pixel position of an object placed within the two-dimensional image; and
determining a scale of the object based on the pixel position and the scale field.
8 . The computer-implemented method of claim 7 , wherein determining the scale of the object comprises:
determining an initial size of the object; and
inserting the object at the pixel position with a modified size based on a ratio indicated by a value from the scale field at the pixel position of the object.
9 . The computer-implemented method of claim 1 , further comprising learning parameters of the one or more neural networks by:
generating, for an additional two-dimensional image, estimated depth values for a plurality of pixels of the additional two-dimensional image projected to a corresponding three-dimensional space;
determining, for the additional two-dimensional image, an additional horizon line according to an estimated camera height of the additional two-dimensional image;
generating, for the additional two-dimensional image, a ground-truth scale field based on a plurality of ground-to-horizon vectors in the corresponding three-dimensional space according to the estimated depth values for the plurality of pixels and the additional horizon line; and
modifying parameters of the one or more neural networks based on the ground-truth scale field of the additional two-dimensional image.
10 . A non-transitory computer readable medium storing executable instructions which, when executed by a processing device, cause the processing device to perform operations comprising:
generating, utilizing one or more neural networks comprising parameters learned from a plurality of digital images with annotated horizon lines and ground-to-horizon vectors, a feature representation of a two-dimensional image;
generating, utilizing the one or more neural networks and based on the feature representation, a scale field for the two-dimensional image comprising a plurality of values indicating ratios of pixel distances relative to a camera height of the two-dimensional image; and
performing at least one of:
generating a metric distance of an object portrayed in the two-dimensional image according to the scale field of the two-dimensional image; or
modifying the two-dimensional image according to the scale field of the two-dimensional image.
11 . The non-transitory computer readable medium of claim 10 , wherein generating the scale field comprises generating, for a pixel of the two-dimensional image, a value representing a ratio between a pixel distance from the pixel to a horizon line of the two-dimensional image and a camera height of the two-dimensional image.
12 . The non-transitory computer readable medium of claim 10 , wherein modifying the two-dimensional image comprises inserting an object at a location of the two-dimensional image by:
determining a pixel corresponding to the location of the two-dimensional image;
determining a scaled size of the object based on a value from the scale field for the pixel corresponding to the location of the two-dimensional image; and
inserting the object at the location of the two-dimensional image according to the scaled size of the object.
13 . A system comprising:
one or more memory devices comprising a two-dimensional image; and
one or more processors configured to cause the system to:
generate, utilizing one or more neural networks, a feature representation of the two-dimensional image;
generate, utilizing the one or more neural networks and based on the feature representation, a scale field for the two-dimensional image comprising pixel-to-metric ratios between:
pixel distances from pixels to a horizon line in the two-dimensional image; and
metric distances from corresponding three-dimensional points to a three-dimensional horizon line in a three-dimensional space corresponding to the two-dimensional image; and
perform at least one of:
generating a metric distance of content portrayed in the two-dimensional image utilizing the pixel-to-metric ratios of the scale field of the two-dimensional image; or
modifying the two-dimensional image utilizing the pixel-to-metric ratios of the scale field of the two-dimensional image.
14 . The system of claim 13 , wherein the one or more processors are further configured to cause the system to generate, utilizing the one or more neural networks and based on the feature representation, a plurality of ground-to-horizon vectors in the three-dimensional space according to the horizon line of the two-dimensional image projected into the three-dimensional space.
15 . The system of claim 14 , wherein the one or more processors are further configured to cause the system to generate the plurality of ground-to-horizon vectors by generating a ground-to-horizon vector indicating a distance and a direction from a three-dimensional point corresponding to a pixel of the two-dimensional image to the three-dimensional horizon line in the three-dimensional space.
16 . The system of claim 13 , wherein the one or more processors are further configured to cause the system to generate the scale field for the two-dimensional image by generating, for a pixel of the two-dimensional image, a pixel-to-metric ratio between a pixel distance in the two-dimensional image and a corresponding three-dimensional distance in the three-dimensional space relative to a camera height of the two-dimensional image.
17 . The system of claim 13 , wherein the one or more processors are further configured to cause the system to generate the metric distance of the content portrayed in the two-dimensional image by:
determining a pixel distance between a first pixel corresponding to the content and a second pixel corresponding to the content; and
generating the metric distance based on the pixel distance and the pixel-to-metric ratios between the pixel distances in the two-dimensional image and the metric distances in the three-dimensional space.
18 . The system of claim 17 , wherein the one or more processors are further configured to cause the system to generate the metric distance of the content portrayed in the two-dimensional image by:
determining a value of the scale field corresponding to the first pixel; and
converting the value of the scale field corresponding to the first pixel to the metric distance based on the pixel distance between the first pixel and the second pixel.
19 . The system of claim 13 , wherein the one or more processors are further configured to cause the system to modify the two-dimensional image by:
determining a pixel position of an object placed within the two-dimensional image; and
determining a scale of the object based on the pixel position and the scale field.
20 . The system of claim 19 , wherein the one or more processors are further configured to cause the system to determine the scale of the object by:
determining an initial size of the object; and
inserting the object at the pixel position with a modified size based on a ratio indicated by a value from the scale field at the pixel position of the object.