Image stitching with color harmonization for surround view systems and applications
In various examples, color statistic(s) from ground projections are used to harmonize color between reference and target frames representing an environment. The reference and target frames may be projected onto a representation of the ground (e.g., a ground plane) of the environment, an overlapping region between the projections may be identified, and the portion of each projection that lands in the overlapping region may be taken as a corresponding ground projection. Color statistics (e.g., mean, variance, standard deviation, kurtosis, skew, correlation(s) between color channels) may be computed from the ground projections (or a portion thereof, such as a majority cluster) and used to modify the colors of the target frame to have updated color statistics that match those from the ground projection of the reference frame, thereby harmonizing color across the reference and target frames.
1. A method comprising:
determining, using frames of image data that represent overlapping views of an environment surrounding an ego-object, a reference frame and a target frame;
generating a ground projection of the reference frame;
computing a reference color statistic using the ground projection of the reference frame; and
based at least on a determination of whether an overlapping region associated with the reference frame and the target frame has a threshold amount of pixels that correspond to a detected object, transferring the reference color statistic from the ground projection of the reference frame to the target frame.
2. The method of claim 1 , wherein the determining of the reference frame comprises selecting one of the frames of image data based at least on one or more of: a direction of ego-motion of the ego-object, an active viewport into the environment, or a gaze of an operator of the ego-object.
3. The method of claim 1 , wherein the generating of the ground projection of the reference frame comprises projecting the reference frame onto a three-dimensional (3D) bowl to generate a projected 3D bowl representation, and using a portion of the projected 3D bowl representation corresponding to a ground plane of the environment as the ground projection of the reference frame.
4. The method of claim 1 , wherein the reference and target frames represent different views of the environment in a same time slice, and the transferring the reference color statistic from the ground projection of the reference frame to the target frame is based at least on a determination that the overlapping region between the ground projection of the reference frame and a second ground projection of the target frame has less than or equal to a threshold number of pixels that belong to the detected object.
5. The method of claim 1 , wherein the reference frame represents at least a portion of the environment from a preceding time slice and the target frame represents at least a portion of the environment from a subsequent time slice, and the transferring the reference color statistic comprises transferring the reference color statistic from the ground projection of the reference frame based at least on a determination that the overlapping region between a second ground projection of a candidate reference frame from the subsequent time slice and a third ground projection of the target frame includes more than a threshold number or percentage of pixels that correspond to the detected object.
6. The method of claim 1 , wherein the computing of the reference color statistic from the ground projection of the reference frame comprises computing the reference color statistic from a majority cluster of the ground projection of the reference frame, and the transferring the reference color statistic comprises transferring the reference color statistic from the majority cluster of the ground projection of the reference frame to the target frame based at least on a determination that the overlapping region between the ground projection of the reference frame and a second ground projection of the target frame includes less than a threshold number or percentage of pixels that belong to the detected object.
7. The method of claim 1 , wherein the transferring of the reference color statistic from the ground projection of the reference frame to the target frame generates a modified target frame, and the method further comprises stitching at least the reference frame and the modified target frame into a stitched image and causing presentation of a visualization based at least on the stitched image.
8. The method of claim 1 , wherein the method is performed by at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing real-time streaming;
a system for presenting one or more of augmented reality content, virtual reality content,
or mixed reality content;
a system for performing digital twin operations;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for generating synthetic data; or
a system implemented at least partially using cloud computing resources.
9. A processor comprising:
one or more processing units to:
determine a reference frame and a target frame; and
based at least on a determination of whether an overlapping region associated with the reference frame and the target frame has a threshold amount of pixels that correspond to a detected object, use a first ground projection of the reference frame and a second ground projection of the target frame to transfer a reference color statistic from the first ground projection of the reference frame to the target frame.
10. The processor of claim 9 , wherein the reference and target frames are generated using cameras disposed on an ego-object in an environment, and the one or more processing units are further to determine the reference frame based at least on one or more of: a direction of ego-motion of the ego-object, an active viewport into the environment, or a gaze of an operator of the ego-object.
11. The processor of claim 9 , wherein the one or more processing units are further to generate the first ground projection of the reference frame based at least on projecting the reference frame onto a three-dimensional (3D) bowl to generate a projected 3D bowl representation and using a portion of the projected 3D bowl representation corresponding to a ground plane as the first ground projection of the reference frame.
12. The processor of claim 9 , wherein the reference and target frames represent different views of an environment in a same time slice, and the one or more processing units are further to transfer the reference color statistic from the first ground projection of the reference frame to the target frame based at least on a determination that the overlapping region between the first ground projection of the reference frame and the second ground projection of the target frame does not include any detected objects.
13. The processor of claim 9 , wherein the reference frame represents a preceding time slice and the target frame represents a subsequent time slice, and the one or more processing units are further to transfer the reference color statistic from the reference frame from the preceding time slice based on a determination that the overlapping region between a third ground projection of a candidate reference frame from the subsequent time slice and the second ground projection of the target frame includes more than a threshold number or percentage of pixels that belong to the detected object.
14. The processor of claim 9 , the one or more processing units are further to determine to transfer the reference color statistic from a majority cluster of the reference frame to the target frame based on a determination that the overlapping region between the first ground projection of the reference frame and the second ground projection of the target frame includes less than a threshold number or percentage of pixels that belong to the detected object.
15. The processor of claim 9 , wherein the processor is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing deep learning operations;
a system for performing real-time streaming;
a system for presenting one or more of augmented reality content, virtual reality content,
or mixed reality content;
a system implemented using an edge device;
a system implemented using a robot;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for generating synthetic data; or
a system implemented at least partially using cloud computing resources.
16. A system comprising:
one or more processing units to:
identify a reference image and a target image of an environment;
generate a ground projection of the reference image;
compute a reference color statistic from the ground projection of the reference image; and
based at least on a determination of whether an overlapping region associated with the reference image and the target image has a threshold amount of pixels that correspond to a detected object, modify one or more colors of the target image to correspond to the reference color statistic from the ground projection of the reference image.
17. The system of claim 16 , wherein the reference and target images represent different views of the environment in a same time slice, and the one or more processing units are further to modify the colors of the target image to correspond to the reference color statistic from the ground projection of the reference image based on a determination that the overlapping region between the ground projection of the reference image and a second ground projection of the target image has less than or equal to a threshold number of pixels that belong to the detected object.
18. The system of claim 16 , wherein the reference image represents the environment from a preceding time slice and the target image represents the environment from a subsequent time slice, and the one or more processing units are further to modify the colors of the target image to correspond to the reference color statistic from the ground projection of the reference image from the preceding time slice based on a determination that the overlapping region between a second ground projection of a candidate reference image from the subsequent time slice and a third ground projection of the target image includes more than a threshold number or percentage of pixels that belong to the detected object.
19. The system of claim 16 , wherein the computing of the reference color statistic from the ground projection of the reference image comprises computing the reference color statistic from a majority cluster of the ground projection of the reference image, and the one or more processing units are further to modify the colors of the target image to correspond to the reference color statistic from the majority cluster of the ground projection of the reference image based on a determination that the overlapping region between the ground projection of the reference image and a second ground projection of the target image includes less than a threshold number or percentage of pixels that belong to the detected object.
20. The system of claim 16 , wherein the system is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing deep learning operations;
a system for performing real-time streaming;
a system for presenting one or more of augmented reality content, virtual reality content,
or mixed reality content;
a system implemented using an edge device;
a system implemented using a robot;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for generating synthetic data; or
a system implemented at least partially using cloud computing resources.