Distortion-based image rendering
Synthesizing high-resolution input for rendering a digital human includes generating, with a generative artificial intelligence (AI) model, a distorted image of the digital human by enhancing a region of interest (ROI) within the distorted image relative to other regions of the distorted image. The generative AI model is previously trained against a distorted control image generated using a distortion function to distort a control image used to guide image generation by the generative AI model. The distorted control image is generated by reconfiguring and augmenting pixels of the control image. An undistorted image of the digital human is generated using a reverse distortion function to reverse distortion of the distorted image.
1 . A computer-implemented method of rendering a digital human using generative artificial intelligence (AI), the method comprising:
generating, with a generative AI model, a distorted image of the digital human by enhancing a region of interest (ROI) within the distorted image relative to other regions of the distorted image;
wherein the generative AI model is trained against a distorted control image generated using a distortion function to distort a control image used to guide image generation by the generative AI model, the distorted control image generated by reconfiguring pixels of the control image with the distortion function and augmenting the pixels with pixels generated by interpolation; and
generating an undistorted image of the digital human using a reverse distortion function to reverse distortion of the distorted image.
2 . The computer-implemented method of claim 1 , wherein the generative AI model learns to generate the distorted image from data input to the generative AI model, wherein the data comprises a distorted contour image generated from landmarks of the control image in which the landmarks are distorted by the distortion function.
3 . The computer-implemented method of claim 1 , wherein the generative AI model learns to generate the distorted image from data input to the generative AI model, wherein the data comprises a distorted contour image, and wherein a mouth of the distorted image is generated using audio data.
4 . The computer-implemented method of claim 1 , wherein the distortion function is a continuous monotonically non-decreasing function that distorts the ROI by generating at least one of a spline or radial expansion from an approximate center of the ROI.
5 . The computer-implemented method of claim 1 , wherein the reverse distortion function is generated by fitting a polynomial to sampled points of the distorted image.
6 . The computer-implemented method of claim 1 , wherein the control image includes one or more additional ROIs, the method further comprising:
sequentially distorting the control image using the distortion function to distend each of the one or more additional ROIs by re-aligning and augmenting pixels corresponding to each of the one or more additional ROIs; and
generating the undistorted image by applying the reverse distortion function to the ROI and the one or more additional ROIs in a reverse sequence of the distorting the ROI and the one or more additional ROIs.
7 . A computer-implemented method of training a generative artificial intelligence (AI) model, the method comprising:
identifying a region of interest (ROI) within a control image of a human;
generating a distorted control image by distorting the control image using a distortion function that distends the ROI by reconfiguring and augmenting pixels of the control image corresponding to the ROI to thereby expand the ROI relative to other regions of the control image; and
generating, using a generative AI model, a distorted image, wherein the generative AI model learns to generate the distorted image against the distorted control image as distorted by the distortion function.
8 . The computer-implemented method of claim 7 , wherein the generative AI model learns to generate the distorted image from data input to the generative AI model, wherein the data comprises a distorted contour image generated from landmarks of the control image in which the landmarks are distorted by the distortion function.
9 . The computer-implemented method of claim 7 , wherein the generative AI model learns to generate the distorted AI image using multimodal data input to the generative AI model.
10 . The computer-implemented method of claim 9 , wherein the multimodal data includes a distorted contour image, and wherein a mouth of the distorted image is generated using audio data.
11 . The computer-implemented method of claim 7 , wherein the distortion function is a monotonically non-decreasing function that distorts the ROI by generating at least one of a spline or radial expansion from an approximate center of the ROI.
12 . The computer-implemented method of claim 7 , wherein the generative AI model is a generative adversarial network.
13 . The computer-implemented method of claim 7 , wherein the control image includes one or more additional ROIs, the method further comprising:
sequentially distorting the control image using the distortion function to distend each of the one or more additional ROIs by reconfiguring and augmenting pixels corresponding to each of the one or more additional ROIs; and
generating an undistorted image by applying a reverse distortion function to the ROI and to the one or more additional ROIs in a reverse sequence of the distorting of the ROI and the one or more additional ROIs.
14 . A system, comprising:
one or more processors configured to execute operations including:
generating, with a generative AI model, a distorted image of a digital human by enhancing a region of interest (ROI) within the distorted image relative to other regions of the distorted image;
wherein the generative AI model is trained against a distorted control image generated using a distortion function to distort a control image used to guide image generation by the generative AI model, the distorted control image generated by reconfiguring pixels of the control image with the distortion function and augmenting the pixels with pixels generated by interpolation; and
generating an undistorted image of the digital human using a reverse distortion function to reverse distortion of the distorted image.
15 . The system of claim 14 , wherein the generative AI model learns to generate the distorted image from data input to the generative AI model, wherein the data comprises a distorted contour image generated from landmarks of the control image in which the landmarks are distorted by the distortion function.
16 . The system of claim 14 , wherein the generative AI model learns to generate the distorted image using multimodal data input to the generative AI model.
17 . The system of claim 16 , wherein the multimodal data includes a distorted contour image, and wherein a mouth of the distorted image is generated using audio data.
18 . The system of claim 14 , wherein the distortion function is a monotonically non-decreasing function that distorts the ROI by generating at least one of a spline or radial expansion from an approximate center of the ROI.
19 . The system of claim 14 , wherein the reverse distortion function is generated by fitting a polynomial to sampled points of the distorted image.
20 . The system of claim 14 , wherein the control image includes one or more additional ROIs, wherein the one or more processors are configured to execute operations further comprising:
sequentially distorting the control image of the human using the distortion function to distend each of the one or more additional ROIs by reconfiguring and augmenting pixels corresponding to each of the one or more additional ROIs; and
generating the undistorted image by applying the reverse distortion function to the ROI and to the one or more additional ROIs in a reverse sequence of the distorting of the ROI and the additional ROIs.