Granted Patent
B2
US 12,340,484 · App. 16/813,589 · Granted Jun 24, 2025
Techniques to use a neural network to expand an image
Inventors:
Guilin Liu (San Jose, CA); Andrew Tao (Los Altos, CA); Bryan Christopher Catanzaro (Los Altos Hills, CA); Ting-Chun Wang (Santa Clara, CA); Zhiding Yu (Santa Clara, CA); Shiqiu Liu (Santa Clara, CA); Fitsum Reda (Santa Clara, CA); Karan Sapra (Santa Clara, CA); Brandon Rowlett (Cedar Park, TX)
Assignee:
NVIDIA Corporation
G06T3/4038G06N3/08G06T3/4046G06T7/40G06V10/454G06V10/54G06V10/776G06V10/82G06T2207/20081G06T2207/20084
View Patent ↗
Loading inventors, assignments & file history…
Abstract
Apparatuses, systems, and techniques for texture synthesis from small input textures in images using convolutional neural networks. In at least one embodiment, one or more convolutional layers are used in conjunction with one or more transposed convolution operations to generate a large textured output image from a small input textured image while preserving global features and texture, according to various novel techniques described herein.
Claims (10)
1. A processor comprising:
one or more circuits to use one or more neural networks to generate a second image based, at least in part, on one or more feature maps corresponding to a first image, wherein the first image is smaller than the second image; wherein:
the one or more feature maps are generated by the one or more neural networks from the first image;
one or more shifted feature maps are generated from the one or more feature maps;
one or more weights are computed based, at least in part, on features shared between the one or more shifted feature maps and the one or more feature maps;
one or more combined feature maps are generated based, at least in part, on combining the one or more shifted feature maps and the one or more feature maps according to the one or more weights; and
the second image is generated by aggregating and upsampling the one or more combined feature maps.
2. The processor of claim 1 , wherein each of the one or more feature maps is scaled from the first image using one or more convolutional layers.
3. The processor of claim 1 , wherein the one or more weights are computed based, at least in part, on which corresponding feature of the one or more shifted feature maps and the one or more feature maps is more prominent.
4. The processor of claim 1 , wherein upsampling comprises zero-padding a smaller first feature map into a larger second feature map.
Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST
Recorded May 11, 2020
From: LIU, GUILIN; TAO, ANDREW; CATANZARO, BRYAN CHRISTOPHER; WANG, TING-CHUN; YU, ZHIDING; LIU, SHIQIU; REDA, FITSUM; SAPRA, KARAN; ROWLETT, BRANDON
To: NVIDIA CORPORATION
Reel/Frame 052628/0844 →
Continuity (1)
References Cited (79)
US 20200118249A1
· Kim
· 2020
[cited by applicant]
CN 108830827A
· 2018
[cited by applicant]
CN 109993200A
· 2019
[cited by applicant]
WO 2018194863A1
· 2018
[cited by applicant]
WO 2019008519A1
· 2019
[cited by applicant]
WO 2019238712A1
· 2019
[cited by applicant]
WO 2020020146A1
· 2020
[cited by applicant]
WO 2020028382A1
· 2020
[cited by applicant]
Zhang et al StackGAN: Text to photo-realistic image synthesis with Stacked Generative Adversarial Networks, arXiv:1612.03242v2, (Aug. 5, 2017).
[cited by examiner]
He et al, Deep Residual Learning for Image Recognition, CVPR (Year: 2016).
[cited by examiner]
Jason Brownlee PhD, How to Visualize Filters and Feature Maps in Convolutional Neural Networks, Deep Learning for Computer Vision, https://machinelearningmastery.com/how-to-visualize-filters-and-feature-maps-in-convolut…
[cited by examiner]
Zhu et al., “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks,” Nov. 24, 2017, 20 pages.
[cited by applicant]
Lai et al., “Fast and Accurate Image Super-Resolution with Deep Laplacian Pyramid Networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, Aug. 9, 2018, 16 pages.
[cited by applicant]
Song et al., “Imaging Inpainting using Multi-Scale Feature Image Translation,” Nov. 23, 2017, 10 pages.
[cited by applicant]
United Kingdom Combined Search and Examination Report for Patent Application No. 2103223.0 dated Oct. 26, 2021, 10 pages.
[cited by applicant]
Abdelmounaime et al., “New Brodatzbased Image Databases for Grayscale Color and Multiband Texture Analysis,” Hindawi Publishing Corporation, 2013, 15 pages.
[cited by applicant]
Alanov et al., “Usercontrollable Multi-texture Synthesis with Generative Adversarial Networks,” Apr. 24, 2019, 27 pages.
[cited by applicant]
Barnes et al., “Patchmatch: A Randomized Correspondence Algorithm for Structural Image Editing,” ACM Transactions on Graphics, vol. 28, 2009, 21 pages.
[cited by applicant]
Bergmann et al., “Learning Texture Manifolds with the Periodic Spatial GAN,” Proceedings of the 34th International Conference on Machine Learning, vol. 70, 2017, 9pages.
[cited by applicant]
Bonet, “Multiresolution Sampling Procedure for Analysis and Synthesis of Texture Images,” Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, 1997, 2 pages.
[cited by applicant]
Burghouts et al., “Materialspecific Adaptation of Color Invariant Features,” Pattern Recognition Letters, 30(3): 2009, 9 pages.
[cited by applicant]
Cimpoi et al., “Describing Textures in the Wild,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, 8 pages.
[cited by applicant]
Dai et al., “The Synthesizability of Texture Examples,” In IEEE Conference on Computer Vision and Pattern Recognition, 2014, 8 pages.
[cited by applicant]
De Bonet et al., “Multiresolution Sampling Procedure for Analysis and Synthesis of Texture Images,” In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, 1997, 8 pages.
[cited by applicant]
Dumoulin et al., “A Guide to Convolution Arithmetic for Deep Learning,” Mar. 23, 2016, 28 pages.
[cited by applicant]
Efros et al., “Image Quilting for Texture Synthesis and Transfer,” Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, ACM, 2001, 6 pages.
[cited by applicant]
Efros et al., “Texture Synthesis by Non-parametric Sampling,” Proceedings of the Seventh IEEE International Conference on Computer Vision, vol. 2, IEEE, Sep. 1999, 6 pages.
[cited by applicant]
Fritz et al., The KTH-TIPS Database, 2004, 7 pages.
[cited by applicant]
Frühstück et al., “TileGAN: Synthesis of Large-Scale Non-Homogeneous Textures,” Apr. 29, 2019, 11 pages.
[cited by applicant]
Gatys et al., “A Neural Algorithm of Artistic Style,” Sep. 2, 2015, 16 pages.
[cited by applicant]
Gatys et al., “Texture Synthesis using Convolutional Neural Networks,” Advances in Neural Information Processing Systems, May 27, 2015, 9 pages.
[cited by applicant]
Goodfellow et al., “Generative Adversarial Nets,” In Advances in Neural Information Processing Systems, Jun. 10, 2014, 9 pages.
[cited by applicant]
Heusel et al., “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium,” Nov. 8, 2017, 38 pages.
[cited by applicant]
IEEE, “IEEE Standard 754-2008 (Revision of IEEE Standard 754-1985): IEEE Standard for Floating-Point Arithmetic,” Aug. 29, 2008, 70 pages.
[cited by applicant]
Ioffe et al., “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift,” Mar. 2, 2015, 11 pages.
[cited by applicant]
Isola et al., “Image-to-Image Translation with Conditional Adversarial Networks,” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul. 21, 2017, 10 pages.
[cited by applicant]
Jetchev et al., “Texture Synthesis with Spatial Generative Adversarial Networks,” Dec. 1, 2016, 11 pages.
[cited by applicant]
Johnson et al., “Perceptual Losses for Real-Time Style Transfer and Super-Resolution,” Department of Computer Science, 2016, 17 pages.
[cited by applicant]
Kaspar et al., “Self Tuning Texture Optimization,” In Computer Graphics Forum, vol. 34, Wiley Online Library, Nov. 2, 2015, 11 pages.
[cited by applicant]
Kwatra et al., “Graphcut Textures: Image and Video Synthesis Using Graph Cuts,” ACM Transactions on Graphics, vol. 22, 2003, 10 pages.
[cited by applicant]
Kwatra et al., “Texture Optimization for Example-Based Synthesis,” ACM, 2005, 8 pages.
[cited by applicant]
Kwitt et al., “Salzburg Texture Image Database,” 2 pages.
[cited by applicant]
Ledig et al., “Photo-Realistic Single Image Superresolution Using a Generative Adversarial Network,” CVPR, 2016, 10 pages.
[cited by applicant]
Li et al., “Diversified Texture Synthesis with Feed-forward Networks,” CVPR, 2017, 9 pages.
[cited by applicant]
Li et al., “Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks,” Apr. 15, 2016, 17 pages.
[cited by applicant]
Li et al., “Universal Style Transfer via Feature Transforms,” Nov. 17, 2017, 11 pages.
[cited by applicant]
Liu et al., “Texture Synthesis through Convolutional Neural Networks and Spectrum Constraints,” ICPR, Dec. 4-8, 2016, 6 pages.
[cited by applicant]
Mallikarjuna et al., “The KTH-TIP2 database,” Jul. 2006, 10 pages.
[cited by applicant]
Picard et al., “Vistex Vision Texture Database,” Jan. 2010, 2 pages.
[cited by applicant]
Portilla et al., “A Parametric Texture Model Based on Joint Statistics of Complex Wavelet Coefficients,” International Journal of Computer Vision, 40(1): Oct. 2000, 23 pages.
[cited by applicant]
Rosenberger et al., “Layered Shape Synthesis: Automatic Generation of Control Maps for Non-Stationary Textures,” ACM, 2009, 10 pages.
[cited by applicant]
Rosenholtz et al., Material Perception: What can you see in a brief glance? vol. 9, Aug. 2009, 2 pages.
[cited by applicant]
Sendik et al., “Deep Correlations for Texture Synthesis,” ACM Transactions on Graphics, 36(5):161, 2017, 15 pages.
[cited by applicant]
Shaham et al., “SinGAN: Learning a Generative Model from a Single Natural Image,” ICCV, 11 pages.
[cited by applicant]
Society of Automotive Engineers On-Road Automated Vehicle Standards Committee, “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles,” Standard No. J3016-201609, issued Jan…
[cited by applicant]
Society of Automotive Engineers On-Road Automated Vehicle Standards Committee, “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles,” Standard No. J3016-201806, issued Jan…
[cited by applicant]
Texture Library “Texture Library,” retrieved from Internet on May 5, 2021, from http://textures.forrest.cz/?spgmGal=2009_forest, Aug. 2009, 2 pages.
[cited by applicant]
TextureKing “TeaxtureKing,” retireved from Internet on May 5, 2021, from https://www.textureking.com/, 1 page.
[cited by applicant]
Ulyanov, “Texture Networks: Feed-forward Synthesis of Textures and Stylized Images,” Mar. 10, 2016, 16 pages.
[cited by applicant]
Wang et al., “High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs,” CVPR, 2018, 10 pages.
[cited by applicant]
Wang et al., “Image Quality Assessment: From Error Visibility to Structural Similarity,” IEEE Transactions on Image Processing, 13(4): Apr. 2004, 14 pages.
[cited by applicant]
Wei et al., “Fast Texture Synthesis using Tree-structured Vector Quantization,” Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, 2000, 11 pages.
[cited by applicant]
Yu et al., “Attributing Fake Images to GANs: Learning and Analyzing GAN Fingerprints,” ICCV, 2019, 11 pages.
[cited by applicant]
Zhang et al., “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,” CVPR, 2018, 10 pages.
[cited by applicant]
Zhou et al., “Non-Stationary Texture Synthesis by Adversarial Expansion,” May 11, 2018, 13 pages.
[cited by applicant]
Korinevskaya et al., “Fast Depth Map Super-Resolution using Deep Neural Network,” IEEE International Symposium on Mixed and Augmented Reality Adjunct, 2018, 6 pages.
[cited by applicant]
Notice of Intention to Grant for Chinese Application No. 202110255809.9, mailed Jul. 25, 2024, 6 pages.
[cited by applicant]
Notice of Intention to Grant for United Kingdom Application No. GB2304886.1, mailed Jun. 5, 2024, 2 pages.
[cited by applicant]
Office Action for Chinese Application No. 202110255809.9, mailed Jan. 17, 2024, 33 pages.
[cited by applicant]
United Kingdom Combined Search and Examination Report for Application No. GB2304886.1, mailed Oct. 24, 2023, 3 pages.
[cited by applicant]
Wikipedia, “Residual Neural Network,” retrieved from <https://en.wikipedia.ord/wiki/Residual_neural_network,> 2024, 7 pages.
[cited by applicant]
Zhang et al., “StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks,” PAMI, Jun. 28, 2018, 16 pages.
[cited by applicant]
Cited By (1)