Image harmonization for image stitching systems and applications
In various examples, metadata-based image harmonization for image stitching systems and applications are disclosed. Systems and methods are disclosed that preprocess images with respect to rendering parameters, with the effect of blending those parameters at a border between images to facilitate a smooth rendering when those images are stitched together. An image signal processing (ISP) parameter harmonization function may input metadata parameters associated with a set of images to match and blend one or more of the rendering parameters across an overlapping border between images prior to applying those images to a stitching algorithm. A scaling of the metadata parameter may be performed using a parameter gain function. Pixels in both images located along the border are adjusted to the same boundary metadata parameter value, and smoothed based on the parameter gain function. A discontinuity in rendering parameters is avoided, substantially avoiding corresponding artifacts in the resulting stitched image.
1 . One or more processors comprising one or more processing units to:
determine a first parameter value associated with rendering a first image and a second parameter value associated with rendering a second image, wherein the first image at least partially overlaps at an image border with the second image;
compute a boundary parameter value based at least on the first parameter value and the second parameter value;
apply a first parameter gain function that adjusts the first parameter value in at least a portion of the first image directionally toward the boundary parameter value at the image border;
apply a second parameter gain function that adjusts the second parameter value in at least a portion of the second image directionally toward the boundary parameter value at the image border; and
stitch the first image to the second image at the image border.
2 . The one or more processors of claim 1 , wherein the first parameter value and the second parameter value are associated with a rendering parameter comprising at least one of: white balance, tone mapping, exposure time, brightness, contrast, gamma, hue, noise reduction, saturation, sharpness, color filter array (CFA) pattern, lens shading correction, lens distortion correction, focal length correction, barrel distortion correction, or pincushion distortion.
3 . The one or more processors of claim 1 , wherein the one or more processing units are further to compute the boundary parameter value at the image border between the first and second images based at least on an average computed using the first parameter value and the second parameter value.
4 . The one or more processors of claim 1 , wherein the one or more processing units are further to:
determine the first parameter value based on metadata corresponding to the first image; and
determine the second parameter value based on metadata corresponding to the second image.
5 . The one or more processors of claim 1 , wherein the one or more processing units are further to:
determine the first parameter value based on metadata associated with a characteristic of a first camera used to capture the first image; and
determine the second parameter value based on metadata associated with the characteristic of a second camera used to capture the second image.
6 . The one or more processors of claim 1 , wherein at least one of the first parameter gain function or the second parameter gain function comprises a linear function, a non-linear curve function, an s-curve function, or a logistic curve function.
7 . The one or more processors of claim 1 , wherein the one or more processing units are further to adjust a rate of change of at least one of the first parameter gain function or the second parameter gain function based at least on a difference in lighting between the first and second images.
8 . The one or more processors of claim 1 , wherein the one or more processing units are further to generate at least one of:
a surround-view stitched image based at least on the first image and the second image;
a fisheye-view stitched image based at least on the first image and the second image; and
a panoramic-view stitched image based at least on the first image and the second image.
9 . The one or more processors of claim 1 , wherein the first image is captured by a first camera and the second image is captured by a second camera, the first and second cameras having an at least partially overlapping field of view.
10 . The one or more processors of claim 1 , wherein the one or more processors are comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for three-dimensional assets;
a system for performing deep learning operations;
a system for performing remote operations;
a system for performing real-time streaming;
a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system implementing one or more language models;
a system implementing one or more large language models (LLMs);
a system for generating synthetic data;
a system for generating synthetic data using AI;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
11 . A system comprising:
one or more processing units to:
determine a first parameter value associated with a metadata parameter for a first image of a plurality of images;
compute a first boundary value based at least on the first parameter value and a second parameter value associated with the metadata parameter for a second image of the plurality of images;
compute a second boundary value based at least on the first parameter value and a third parameter value associated with the metadata parameter for a third image of the plurality of images;
apply a first parameter gain function that adjusts the first parameter value in at least a first portion of the first image directionally toward the first boundary value at a first image border of the first image; and
apply a second parameter gain function that adjusts the first parameter value in at least a second portion of the first image directionally toward the second boundary value at a second image border of the first image.
12 . The system of claim 11 , wherein the one or more processing units are further to:
stitch the first image to the second image based on an overlapping image border between the first image and the second image; and
stitch the first image to the third image based on an overlapping image border between the first image and the third image.
13 . The system of claim 11 , wherein the first parameter value and the second parameter value are associated with a rendering parameter comprising at least one of: white balance, tone mapping, exposure time, brightness, contrast, gamma, hue, noise reduction, saturation, sharpness, color filter array (CFA) pattern, lens shading correction, lens distortion correction, focal length correction, barrel distortion correction, or pincushion distortion.
14 . The system of claim 11 , wherein the one or more processing units are further to:
compute the first boundary value at the first image border between the first and second images based at least on an average computed using the first parameter value and the second parameter value; and
compute the second boundary value at the second image border between the first and third images based at least on an average computed using the first parameter value and the third parameter value.
15 . The system of claim 11 , wherein the one or more processing units are further to:
determine the first parameter value based on metadata corresponding to the first image;
determine the second parameter value based on metadata corresponding to the second image; and
determine the third parameter value based on metadata corresponding to the third image.
16 . The system of claim 11 , wherein the one or more processing units are further to:
determine the first parameter value based on metadata associated with a characteristic of a first camera used to capture the first image;
determine the second parameter value based on metadata associated with the characteristic of a second camera used to capture the second image; and
determine the third parameter value based on metadata associated with the characteristic of a third camera used to capture the second image.
17 . The system of claim 11 , wherein at least one of the first parameter gain function or the second parameter gain function comprises a linear function, a non-linear curve function, an s-curve function, or a logistic curve function.
18 . The system of claim 11 , wherein the one or more processing units are further to adjust a rate of change of at least one of the first parameter gain function or the second parameter gain function based at least on a difference in lighting between the first image and at least one of the second or third images.
19 . The system of claim 11 , wherein the one or more processing units are comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for three-dimensional assets;
a system for performing deep learning operations;
a system for performing remote operations;
a system for performing real-time streaming;
a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system implementing one or more language models;
a system implementing one or more large language models (LLMs);
a system for generating synthetic data;
a system for generating synthetic data using AI;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
20 . A method comprising:
stitching a first image to a second image that at least partially overlaps with the first image at an image border, the stitching being based at least on adjusting the first image by blending a metadata parameter of the first image directionally toward the image border from a first value of the metadata parameter at a first position within the first image to a second value of the metadata parameter at a second position at the image border, wherein the second value is computed based at least on the first value of the metadata parameter and a third value of the metadata parameter associated with the second image.