IP Library › Granted Patent US 12,307,576
Granted Patent B2
US 12,307,576 · App. 17/993,854 · Granted May 20, 2025

Neural networks to render textured materials on curved surfaces

Inventors: Krishna Bhargava Mullia Lakshminarayana (San Francisco, CA); Zexiang Xu (San Jose, CA); Milos Hasan (Lafayette, CA); Fujun Luan (San Jose, CA); Alexandr Kuznetsov (San Diego, CA); Xuezheng Wang (San Diego, CA); Ravi Ramamoorthi (Carlsbad, CA)
Assignees: Adobe Inc.; The Regents of the University of California
G06T15/06G06T15/04G06T15/506G06T2215/12
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,307,576
App. No.
17/993,854
Granted
May 20, 2025
Kind
B2
Abstract

A scene modeling system accesses a three-dimensional (3D) scene including a 3D object. The scene modeling system applies a silhouette bidirectional texture function (SBTF) model to the 3D object to generate an output image of a textured material rendered as a surface of the 3D object. Applying the SBTF model includes determining a bounding geometry for the surface of the 3D object. Applying the SBTF model includes determining, for each pixel of the output image, a pixel value based on the bounding geometry. The scene modeling system displays, via a user interface, the output image based on the determined pixel values.

Claims (64)

1. A method comprising:

accessing a three-dimensional (3D) scene including one or more 3D objects;

applying a silhouette bidirectional texture function (SBTF) model to a 3D object to generate an output image of a textured material rendered as a surface of the 3D object, wherein applying the SBTF model comprises:

determining a bounding geometry for the surface of the 3D object; and

for each pixel of the output image, determining a pixel value based on the bounding geometry wherein the pixel value comprises a reflectance value, and wherein determining the pixel value for each pixel comprises:

projecting a ray through the pixel into the scene;

determining that the ray encounters the bounding geometry or does not encounter the bounding geometry;

responsive to determining that the ray encounters the bounding geometry, determining the reflectance value for the pixel based in part on a curvature value of the ray, the curvature value filtered by taking a minimum curvature along a curve on the bounding geometry, and applying the pixel to the surface of the 3D object; and

displaying, via a user interface, the output image based on the determined pixel values.

2. The method of claim 1 , wherein the pixel value further comprises an opacity value, and determining the pixel value for each pixel further comprises:

responsive to determining that the ray does not encounter the bounding geometry, assign an opacity value of zero to the pixel.

3. The method of claim 1 , wherein determining the reflectance value further comprises:

determining a UV offset for the pixel based on a location on a surface of the bounding geometry; and

determining the reflectance value based on the location on the surface of the bounding geometry and the UV offset.

4. The method of claim 1 , wherein the SBTF model comprises:

an alpha network configured to determine an opacity value, wherein determining that the ray encounters the bounding geometry comprises determining a non-zero opacity value, wherein determining that the ray does not encounter the bounding geometry comprises determining a zero opacity value.

5. The method of claim 4 , wherein determining the opacity value further comprises:

determining a silhouette cosine value;

if a cosine of the ray is less than the silhouette cosine value, determining a zero value for the opacity value; or

if the cosine of the ray is greater than or equal to the silhouette cosine value, determining a value of one for the opacity value.

6. The method of claim 1 , further comprising training the SBTF model, wherein training the SBTF model comprises:

generating a training dataset of cylindrical patches of varying radii;

applying, to the cylindrical patches of the training dataset, one or more of random rotations, random camera directions, random light directions, or random translations to a UV mapping; and

sampling rays incident upon each of the cylindrical patches from different directions perpendicular to a cylinder axis.

7. A system comprising:

a memory component; and

a processing device coupled to the memory component, the processing device to perform operations comprising:

accessing a three-dimensional (3D) scene including one or more 3D objects;

applying a silhouette bidirectional texture function (SBTF) model to a 3D object to generate an output image of a textured material rendered as a surface of the 3D object, wherein the SBTF model comprises:

an alpha network configured to determine an opacity value for a pixel of the output image based at least in part on a bounding geometry of the surface of the 3D object;

an offset network configured to determine a reflectance value for the pixel wherein the offset network determines the reflectance value for the pixel by:

projecting a ray through the pixel into the scene;

determining the ray encounters the bounding geometry of the surface of the 3D object or does not encounter the bounding geometry of the bounding geometry;

responsive to determining that the ray encounters the bounding geometry, determining the reflectance value for the pixel based in part on a curvature value of the ray, the curvature value filtered by taking a minimum curvature along a curve on the bounding geometry, and applying the pixel to the surface of the 3D object; and

a color network configured to determine a reflectance value for the pixel if the opacity is a non-zero value; and

displaying, via a user interface, the output image based on the determined pixel values.

8. The system of claim 7 , wherein one or more of the alpha network, the offset network, and the color network comprise a fully connected network including a multi-layer perceptron (MLP) that uses a rectified linear unit (ReLU) activation function.

9. The system of claim 7 , wherein the non-zero opacity value indicates that a ray projected through the pixel into the scene encounters the bounding geometry, wherein a zero opacity value indicates that the ray does not encounter the bounding geometry.

10. The system of claim 7 , wherein the color network does not determine the reflectance value if the opacity value is zero.

11. The system of claim 7 , wherein determining the opacity value comprises:

determining a silhouette cosine value;

if a cosine of a ray projected through the pixel into the scene is less than the silhouette cosine value, determining a zero value for the opacity value; or

if the cosine of the ray is greater than or equal to the silhouette cosine value, determining a value of one for the opacity value.

12. The system of claim 7 , wherein the operations further comprise training the SBTF model, wherein training the SBTF model comprises:

generating a training dataset of cylindrical patches of varying radii;

applying, to the cylindrical patches of the training dataset, random rotations and translations to a UV mapping; and

sampling rays incident upon each of the cylindrical patches from different directions perpendicular to a cylinder axis.

13. A non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:

accessing a three-dimensional (3D) scene including one or more 3D objects;

applying a silhouette bidirectional texture function (SBTF) model to a 3D object to generate an output image of a textured material rendered as a surface of the 3D object, wherein applying the SBTF model comprises:

determining a bounding geometry for the surface of the 3D object; and

for each pixel of the output image, determining a silhouette cosine value based on a location on a surface of the bounding geometry of a ray projected through the pixel into the scene;

determine an opacity value based on comparing the silhouette cosine value to a cosine value of the ray; and

determine a pixel value based at least in part on the opacity value and a reflectance value wherein determining the reflectance value comprises:

determining the ray encounters the bounding geometry or does not encounter the bounding geometry;

responsive to determining that the ray encounters the bounding geometry, determining the reflectance value for the pixel based in part on a curvature value of the ray, the curvature value filtered by taking a minimum curvature along a curve on the bounding geometry, and applying the pixel to the surface of the 3D object; and

displaying, via a user interface, the output image based on the determined pixel values.

14. The non-transitory computer-readable medium of claim 13 , wherein the SBTF model determines a zero value for the opacity value if a cosine of the ray is less than the silhouette cosine value and wherein the SBTF model determines a non-zero value for the opacity value if the cosine of the ray is greater than or equal to the silhouette cosine value.

15. The non-transitory computer-readable medium of claim 14 , wherein the non-zero opacity value indicates that the ray encounters the bounding geometry, and wherein the zero opacity value indicates that the ray does not encounter the bounding geometry.

16. The non-transitory computer-readable medium of claim 15 , wherein the reflectance value comprises a Red-Green-Blue (RGB) color value.

17. The non-transitory computer-readable medium of claim 13 , the operations further comprising training the SBTF model, wherein training the SBTF model comprises:

generating a training dataset of cylindrical patches of varying radii;

applying, to the cylindrical patches of the training dataset, random rotations and translations to a UV mapping; and

sampling rays incident upon each of the cylindrical patches from different directions perpendicular to a cylinder axis.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 3, 2025
From: RAMAMOORTHI, RAVI; KUZNETSOV, ALEXANDR; WANG, XUEZHENG
To: THE REGENTS OF THE UNIVERSITY OF CALIFORNIA
Reel/Frame 070385/0400 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 23, 2022
From: LAKSHMINARAYANA, KRISHNA BHARGAVA MULLIA; XU, ZEXIANG; HASAN, MILOS; LUAN, FUJUN
To: ADOBE INC.
Reel/Frame 061869/0417 →
Continuity (1)
Related Publication 20240169653A1 · May 23, 2024
References Cited (60)
US 7430490B2 · Tong · 2008 [cited by examiner]
US 11816779B2 · Lakshminarayana et al. · 2023 [cited by applicant]
US 20190043242A1 · Risser · 2019 [cited by applicant]
US 20190236830A1 · Akenine-Moller et al. · 2019 [cited by applicant]
US 20230237738A1 · Rump · 2023 [cited by examiner]
US 20230260193A1 · Rump · 2023 [cited by examiner]
US 20230316627A1 · Lanfer · 2023 [cited by examiner]
US 20230343051A1 · Rump · 2023 [cited by examiner]
M. Yengui, et al., “Simulated bidirectional texture functions with Silhouette details”, Proceedings of the Graphics Interface 2013, Toronto, Ont., Canada, May 2013. p. 45-54, at https://dl.acm.org/doi/pdf/10.5555/253212… [cited by examiner]
J. Wang, et. et., “Capturing and Rendering Geometry Details for BTF mapped surfaces”, the Visual Computer, Aug. 31, 2005, vol. 21, pp. 559-568 (Year: 2005). [cited by examiner]
A. Kuznetsov, et al., “NeuMIP: Multi-resolution Neural Materials”, ACM Transactions on Graphics (TOG), vol. 40, No. 4, Jul. 19, 2021, pp. 1-13 (Year: 2021). [cited by examiner]
A. Kuznetsov, et al., “Rendering Neural Materials on Curved Surfaces”, SIGGRAPH '22 Conf. Proc., Aug. 7-11, 2022, Vancouver, BC, Canada, pp. 1-12 (Year: 2022). [cited by examiner]
Mitsuba 3, Mitsuba Renderer, Available Online at: http://www.mitsuba-renderer.org., 2010, 1 page. [cited by applicant]
Principal Curvature, Available Online at: https://en.wikipedia.org/wiki/Principal_curvature, Accessed from Internet on Jun. 12, 2023, 3 pages. [cited by applicant]
Substance Source Library, Available Online at: https://source.substance3d.com, Accessed from Internet on Jun. 12, 2023, 8 pages. [cited by applicant]
U.S. Appl. No. 17/538,311, Notice of Allowance mailed on Mar. 23, 2023, 20 pages. [cited by applicant]
Baatz et al., NeRF-Tex: Neural Reflectance Field Textures, Eurographics Symposium on Rendering—DL-Only Track, Adrien Bousseau and Morgan McGuire (Eds.), vol. 40, No. 4, Jun. 22, 2021, pp. 1-13. [cited by applicant]
Dana et al., Reflectance and Texture of Real-World Surfaces, ACM Transactions on Graphics, vol. 18, No. 1, Jan. 1999, pp. 1-34. [cited by applicant]
Dufort et al., Interactive Rendering of Meso-Structure Surface Details using Semi-Transparent 3D Textures, In Proceeding of Vision, Modeling and Visualization, Nov. 16-18, 2005, 9 pages. [cited by applicant]
Dupuy et al., Linear Efficient Antialiased Displacement and Reflectance Mapping, ACM Transactions on Graphics, vol. 32, No. 6, Nov. 2013, pp. 211:1-121:11. [cited by applicant]
Fan et al., Neural BRDFs: Representation and Operations, ACM Transactions on Graphics, vol. 1, No. 1, Nov. 2021, 13 pages. [cited by applicant]
Filip et al., Bidirectional Texture Function Modeling: A State of the Art Survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, No. 11, Nov. 2009, pp. 1921-1940. [cited by applicant]
Han et al., Frequency Domain Normal Map Filtering, ACM Transactions on Graphics, vol. 26, No. 3, Jul. 2007, pp. 28-1-28-11. [cited by applicant]
Jacob et al., Discrete Stochastic Microfacet Models, ACM Transactions on Graphics, vol. 33, No. 4, Jul. 2014, 10 pages. [cited by applicant]
Jacob, Mitsuba2: Physically Based Renderer, Available Online at: http://www.mitsuba-renderer.org/, Accessed from Internet on Dec. 1, 2021, 1 page. [cited by applicant]
Jacobson et al., Fibigl—A simple C++ Geometry Processing Library, Available Online at: https://libigl.github.io/, Accessed from Internet on Jun. 12, 2023, 7 pages. [cited by applicant]
Kaneko et al., Detailed Shape Representation with Parallax Mapping, In Proceedings of the ICAT, Dec. 5-7, 2001, 4 pages. [cited by applicant]
Kaplanyan et al., Filtering Distributions of Normals for Shading Antialiasing, Proceedings of High Performance Graphics, Jun. 2016, 12 pages. [cited by applicant]
Koudelka et al., Acquisition, Compression, and Synthesis of Bidirectional Texture Functions, In 3rd International Workshop on Texture Analysis and Synthesis, Jan. 2003, 5 pages. [cited by applicant]
Kuznetsov et al., Learning Generative Models for Rendering Specular Microgeometry, ACM Transactions on Graphics, vol. 38, No. 6, Nov. 2019, pp. 225:1-225:14. [cited by applicant]
Kuznetsov et al., NeuMIP: Multi-Resolution Neural Materials, ACM Transactions on Graphics, vol. 40, No. 4, Article 175, Aug. 2021, pp. 1-13. [cited by applicant]
Kuznetsov et al., Rendering Neural Materials on Curved Surfaces, In Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings, Aug. 7-11, 2022, 9 pages. [cited by applicant]
Maximov et al., Deep Appearance Maps, Proceedings of the IEEE/CVF International Conference on Computer Vision, Oct. 29, 2019, 10 pages. [cited by applicant]
Mildenhall et al., NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis, European Conference on Computer Vision, Aug. 3, 2020, pp. 1-25. [cited by applicant]
Muller et al., Compression and Real-Time Rendering of Measured BTFs Using Local PCA, Proceedings of the Vision, Modeling, and Visualization Conference, Jan. 2003, pp. 271-279. [cited by applicant]
Muller et al., Neural Control Variates, ACM Transactions on Graphics, vol. 39, No. 6, Dec. 2020, pp. 243:1-243:19. [cited by applicant]
Muller et al., Neural Importance Sampling, ACM Transactions on Graphics, vol. 38, No. 5, Oct. 2019, pp. 145:1-145:19. [cited by applicant]
Nalbach et al., Deep Shading: Convolutional Neural Networks for Screen-Space Shading, Eurographics Symposium on Rendering, vol. 36, No. 4, Jul. 2017, 14 pages. [cited by applicant]
Olano et al., Lean Mapping, In Proceedings of the 2010 ACM SIGGRAPH symposium on Interactive 3D Graphics and Games, Feb. 19-21, 2010, pp. 181-188. [cited by applicant]
Oliveira et al., An Efficient Representation for Surface Details, UFRGS Technical Report, Jan. 26, 2005, 9 pages. [cited by applicant]
Oliveira, Relief Texture Mapping, In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, 2000, 10 pages. [cited by applicant]
Pharr et al., PBRT V3, Available Online at: https://github.com/mmp/pbrt-v3/, Accessed from Internet on Dec. 1, 2021, pp. 1-4. [cited by applicant]
Policarpo et al., Real-Time Relief Mapping on Arbitrary Polygonal Surfaces, Proceedings of the 2005 Symposium on Interactive 3D Graphics and Games, Apr. 2005, pp. 155-162. [cited by applicant]
Porumbescu et al., Shell Maps, ACM Transactions on Graphics, vol. 24, No. 3, Jul. 2005, 8 pages. [cited by applicant]
Rainer et al., Neural BTF Compression and Interpolation, Computer Graphics Forum, vol. 38, No. 2, May 2019, 10 pages. [cited by applicant]
Rainer et al., Unified Neural Encoding of BTFs, Computer Graphics Forum (Proceedings of Eurographics), vol. 39, No. 2, May 2020, pp. 167-178. [cited by applicant]
Rematas et al., Deep Reflectance Maps, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Jun. 27-30, 2016, pp. 4508-4516. [cited by applicant]
Szirmay-Kalos et al., Displacement Mapping on the GPU—State of the Art, Computer Graphics Forum, vol. 25, No. 3, 2006, pp. 1-24. [cited by applicant]
Thies et al., Deferred Neural Rendering: Image Synthesis Using Neural Textures, ACM Transactions on Graphics, vol. 38, No. 4, Apr. 28, 2019, 12 pages. [cited by applicant]
Thonat et al., Tessellation-Free Displacement Mapping for Ray Tracing, ACM Transactions on Graphics, vol. 40, No. 6, Article 282, Dec. 14, 2021, 2 pages. [cited by applicant]
Wang et al., Capturing and Rendering Geometry Details for BTF-Mapped Surfaces, The Visual Computer, vol. 21, No. 8, Sep. 2005, pp. 1-9. [cited by applicant]
Wang et al., Generalized Displacement Maps, The Eurographics Association, 2004, 8 pages. [cited by applicant]
Wang et al., View-Dependent Displacement Mapping, ACM Transactions on Graphics (TOG), vol. 22, No. 2, 2003, pp. 334-339. [cited by applicant]
Weinmann et al., Material Classification Based on Training Data Synthesized Using a BTF Database, European Conference on Computer Vision, vol. 8691, Sep. 2014, pp. 156-171. [cited by applicant]
Williams, Pyramidal Parametrics, Siggraph, ACM, vol. 17, No. 3, Jul. 1983, pp. 1-11. [cited by applicant]
Wu et al., Accurate Appearance Preserving Prefiltering for Rendering Displacement-Mapped Surfaces, ACM Transactions on Graphics, vol. 38, No. 4, Jul. 2019, pp. 137:1-137:14. [cited by applicant]
Yan et al., A Bssrdf Model for Efficient Rendering of Fur with Global Illumination, ACM Transactions on Graphics, vol. 36, No. 6, Nov. 2017, pp. 208:1-208:13. [cited by applicant]
Zhao et al., Downsampling Scattering Parameters for Rendering Anisotropic Media, ACM Transactions on Graphics, vol. 35, No. 6, Nov. 2016, pp. 166:1-166:11. [cited by applicant]
U.S. Appl. No. 17/538,311, “Notice of Allowance”, Jul. 7, 2023, 18 pages. [cited by applicant]
Zhao et al., “Building Volumetric Appearance Models of Fabric Using Micro CT Imaging”, ACM Transactions on Graphics, vol. 30, Issue 4, Article No. 44, 2014, pp. 1-4. [cited by applicant]