IP Library › Granted Patent US 12,190,452
Granted Patent B2
US 12,190,452 · App. 17/527,255 · Granted Jan 7, 2025

Method and system for enforcing smoothness constraints on surface meshes from a graph convolutional neural network

Inventors: Udaranga Pamuditha Wickramasinghe (Ecublens, CH); Pascal Fua (Vaux-sur-Morges, CH)
Assignee: ECOLE POLYTECHNIQUE FEDERALE DE LAUSANNE (EPFL)
G06T17/205G06N3/04G06V10/754
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,190,452
App. No.
17/527,255
Granted
Jan 7, 2025
Kind
B2
Abstract

A method for enforcing smoothness constraints on surface meshes produced by a Graph Convolutional Neural Network (GCNN) including the steps of reading image data from a memory, the image data including two-dimensional image data representing a three-dimensional object or a three-dimensional image stack of the three-dimensional object, performing a GCNN mesh deformation step on the image data to obtain an approximation of a surface of the three-dimensional object, the surface represented by triangulated surface meshes, at least some vertices of the triangulated surface meshes having a different number of neighboring vertices compared to other vertices in a same triangulated surface mesh, and performing a deep active surface model (DASM) transformation step on the triangulated surface meshes to obtain a corrected representation of the surface of three-dimensional object to improve smoothness of the surface.

Claims (34)

1. A method for enforcing smoothness constraints on surface meshes produced by a Graph Convolutional Neural Network (GCNN), the method performed on a data processor of a computer, the method comprising the steps of:

reading image data from a memory, the image data including two-dimensional image data representing a three-dimensional object or a three-dimensional image stack of the three-dimensional object;

performing a GCNN mesh deformation step on the image data to obtain an approximation of a surface of the three-dimensional object, the surface represented by triangulated surface meshes, at least some vertices of the triangulated surface meshes having a different number of neighboring vertices compared to other vertices in a same triangulated surface mesh; and

performing a deep active surface model (DASM) transformation step on the triangulated surface meshes to obtain a corrected representation of the surface of the three-dimensional object to improve smoothness of the surface,

wherein the DASM transformation step uses a first order and second order derivative at locations of each vertex of the surface of the three-dimensional object to filter out rough edges of the surface.

2. The method of claim 1 , wherein the DASM transformation step is performed after a graph convolution step of the GCNN mesh deformation step.

3. The method of claim 2 , wherein the step of performing the GCNN mesh deformation step and the step of performing the DASM transformation step are repeated for iterative correction of the surface of the three-dimensional object.

4. The method of claim 1 , wherein the DASM transformation step further comprises:

computing different discrete positions around a vertex of triangulated surface meshes, a number of discrete positions being constant irrespective of a number of neighboring vertices of the vertex;

computing surface derivates at the vertex by using the different discrete positions;

defining a cost function by using the computed surface derivatives and the triangulated surface meshes from the step of performing the GCNN mesh deformation;

minimizing the cost function to obtain a corrected representation of the surface.

5. The method of claim 4 , wherein the step of minimizing includes a step of iteratively solving a Euler-Lagrange equation associated with the cost function to determine the corrected representation of the surface.

6. The method of claim 1 , wherein the DASM transformation step performs a mapping of the approximation of the surface towards the corrected representation of the surface, in which vertices are moved towards the surface that satisfy an associated Euler-Lagrange equation associated with the cost function.

7. The method of claim 1 , wherein in the step of reading image data from the memory, the data of three-dimensional image stacks of the three-dimensional object are read, and the three-dimensional image stack includes image body slice data from a medical imaging device.

8. The method of claim 1 , further comprising the step of:

displaying a rendered two-dimensional representation or three-dimensional representation of the corrected representation of the surface of the three-dimensional object on a display device.

9. A non-transitory computer readable medium, the computer readable medium having computer-executable code recorded thereon, the computer-executable code configured to perform the method of claim 1 when executed on a computer device.

10. A computer system including a data processor and memory, the computer system configured to perform the method of claim 1 .

11. A method for enforcing smoothness constraints on surface meshes produced by a Graph Convolutional Neural Network (GCNN), the method performed on a data processor of a computer, the method comprising the steps of:

reading image data from a memory, the image data including two-dimensional image data representing a three-dimensional object or a three-dimensional image stack of the three-dimensional object;

performing a GCNN mesh deformation step on the image data to obtain an approximation of a surface of the three-dimensional object, the surface represented by triangulated surface meshes, at least some vertices of the triangulated surface meshes having a different number of neighboring vertices compared to other vertices in a same triangulated surface mesh; and

performing a deep active surface model (DASM) transformation step on the triangulated surface meshes to obtain a corrected representation of the surface of the three-dimensional object to improve smoothness of the surface,

wherein the DASM transformation step further comprises:

computing different discrete positions around a vertex of triangulated surface meshes, a number of discrete positions being constant irrespective of a number of neighboring vertices of the vertex;

computing surface derivates at the vertex by using the different discrete positions;

defining a cost function by using the computed surface derivatives and the triangulated surface meshes from the step of performing the GCNN mesh deformation;

minimizing the cost function to obtain a corrected representation of the surface.

12. The method of claim 11 , wherein the step of minimizing includes a step of iteratively solving a Euler-Lagrange equation associated with the cost function to determine the corrected representation of the surface.

13. A method for enforcing smoothness constraints on surface meshes produced by a Graph Convolutional Neural Network (GCNN), the method performed on a data processor of a computer, the method comprising the steps of:

reading image data from a memory, the image data including two-dimensional image data representing a three-dimensional object or a three-dimensional image stack of the three-dimensional object;

performing a GCNN mesh deformation step on the image data to obtain an approximation of a surface of the three-dimensional object, the surface represented by triangulated surface meshes, at least some vertices of the triangulated surface meshes having a different number of neighboring vertices compared to other vertices in a same triangulated surface mesh; and

performing a deep active surface model (DASM) transformation step on the triangulated surface meshes to obtain a corrected representation of the surface of the three-dimensional object to improve smoothness of the surface,

wherein the DASM transformation step performs a mapping of the approximation of the surface towards the corrected representation of the surface, in which vertices are moved towards the surface that satisfy an associated Euler-Lagrange equation associated with the cost function.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 24, 2021
From: WICKRAMASINGHE, UDARANGA PAMUDITHA; FUA, PASCAL
To: ECOLE POLYTECHNIQUE FEDERALE DE LAUSANNE (EPFL)
Reel/Frame 058201/0625 →
Continuity (1)
Related Publication 20230169732A1 · Jun 1, 2023
References Cited (49)
US 20200027269A1 · Jiang · 2020 [cited by examiner]
Nanyang Wang et al., Pixel2Mesh: 3D Mesh Model Generation via Image Guided Deformation, IEEE Transactions on Pattern Analysis and Machine Intelligence ( vol. 43, Issue: 10, Apr. 2, 2020, pp. 3600-3613 (Year: 2010). [cited by examiner]
Andrei Jalba et al., An Efficient Morphological Active Surface Model for Volumetric Image Segmentation, Mathematical Morphology and Its Application to Signal and Image Processing pp. 193-204, 2009 (Year: 2009). [cited by examiner]
Noha El-Zehiry et al,. An active surface model for volumetric image segmentation, 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro, (Year: 2009). [cited by examiner]
Udaranga Wickramasinghe et al., Deep Active Surface Models, Computer Vision and Pattern Recognition (cs.CV); Graphics (cs.GR); Machine Learning (cs.LG), arXiv:2011.08826v5, Nov. 2020 (Year: 2020). [cited by examiner]
Acuna, D., Kar, A., & Fidler, S. (2019). Devil is in the edges: Learning semantic boundaries from noisy annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11075-11083). [cited by applicant]
Chang, A. X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., . . . & Yu, F. (2015). Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012. [cited by applicant]
Chen, Z., & Zhang, H. (2019). Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 5939-5948). [cited by applicant]
Cheng, D., Liao, R., Fidler, S., & Urtasun, R. (2019). Darnet: Deep active ray network for building segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7431-7439). [cited by applicant]
Chibane, J., Alldieck, T., & Pons-Moll, G. (2020). Implicit functions in feature space for 3d shape reconstruction and completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp… [cited by applicant]
Choy, C. B., Xu, D., Gwak, J., Chen, K., & Savarese, S. (Oct. 2016). 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In European conference on computer vision (pp. 628-644). Springer, Cha… [cited by applicant]
Çiçek, Ö., Abdulkadir, A., Lienkamp, S. S., Brox, T., & Ronneberger, O. (Oct. 2016). 3D U-Net: Learning dense volumetric segmentation from sparse annotation. In International conference on medical image computing and co… [cited by applicant]
Cohen, L. D., & Cohen, I. (1993). Finite-element methods for active contour models and balloons for 2-D and 3-D images. IEEE Transactions on Pattern Analysis and machine intelligence, 15(11), 1131-1147. [cited by applicant]
Dong, S., Luo, G., Wang, K., Cao, S., Li, Q., & Zhang, H. (2018). A combined fully convolutional networks and deformable model for automatic left ventricle segmentation based on 3D echocardiography. BioMed research inte… [cited by applicant]
Duchi, J., Hazan, E., & Singer, Y. (2011). Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7). [cited by applicant]
Fua, P. (1996). Model-based optimization: Accurate and consistent site modeling. In 18th Congress of International Society for Photogrammetry and Remote Sensing (No. CONF). [cited by applicant]
Fua, P., & Leclerc, Y. G. (1995). Object-centered surface reconstruction: Combining multi-image stereo and shading. International Journal of Computer Vision, 16(1), 35-56. [cited by applicant]
Gkioxari, G., Malik, J., & Johnson, J. (2019). Mesh r-cnn. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 9785-9795). [cited by applicant]
Hatamizadeh, A., Sengupta, D., & Terzopoulos, D. (Aug. 2020). End-to-end trainable deep active contour models for automated image segmentation: Delineating buildings in aerial imagery. In European Conference on Computer… [cited by applicant]
He, L., Peng, Z., Everding, B., Wang, X., Han, C. Y., Weiss, K. L., & Wee, W. G. (2008). A comparative study of deformable contour methods on medical image segmentation. Image and vision computing, 26(2), 141-163. [cited by applicant]
Iglovikov, V., & Shvets, A. (2018). Ternausnet: U-net with vgg 11 encoder pre-trained on imagenet for image segmentation. arXiv preprint arXiv:1801.05746. [cited by applicant]
Jorstad, A., Nigro, B., Cali, C., Wawrzyniak, M., Fua, P., & Knott, G. (2015). NeuroMorph: a toolset for the morphometric analysis and visualization of 3D models derived from electron microscopy image stacks. Neuroinfor… [cited by applicant]
Kass, M., Witkin, A., & Terzopoulos, D. (1988). Snakes: Active contour models. International journal of computer vision, 1(4), 321-331. [cited by applicant]
Kavur, A. E., Gezer, N. S., Bar, M., Aslan, S., Conze, P. H., Groza, V., . . . & Selver, M. A. (2021). CHAOS challenge-combined (CT-MR) healthy abdominal organ segmentation. Medical Image Analysis, 69, 101950. [cited by applicant]
Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. [cited by applicant]
Lengagne, R., Fua, P., & Monga, O. (2000). 3D stereo reconstruction of human faces driven by differential constraints. Image and Vision Computing, 18(4), 337-343. [cited by applicant]
Lengagne, R., Monga, O., & Fua, P. (1997, June). Using differential constraints to reconstruct complex surfaces from stereo. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition … [cited by applicant]
Leventon, M. E., Grimson, W. E. L., & Faugeras, O. (Jun. 2000). Statistical Shape Influence in Geodesic Active Contours. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No. PR0… [cited by applicant]
Liang, J., Homayounfar, N., Ma, W. C., Xiong, Y., Hu, R., & Urtasun, R. (2020). Polytransform: Deep polygon transformer for instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern… [cited by applicant]
Ling, H., Gao, J., Kar, A., Chen, W., & Fidler, S. (2019). Fast interactive object annotation with curve-gcn. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 5257-5266). [cited by applicant]
Lorensen, W. E., & Cline, H. E. (1987). Marching cubes: A high resolution 3D surface construction algorithm. ACM siggraph computer graphics, 21(4), 163-169. [cited by applicant]
Marcos, D., Tuia, D., Kellenberger, B., Zhang, L., Bai, M., Liao, R., & Urtasun, R. (2018). Learning deep structured active contours end-to-end. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog… [cited by applicant]
McInerney, T., & Terzopoulos, D. (1995). A dynamic finite element surface model for segmentation and tracking in multidimensional medical images with application to cardiac 4D image analysis. Computerized medical imagin… [cited by applicant]
Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., & Geiger, A. (2019). Occupancy networks:. Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec… [cited by applicant]
Milletari, F., Navab, N., & Ahmadi, S. A. (2016). V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV) (pp. 565-571). IEEE. [cited by applicant]
Newman, T. S., & Yi, H. (2006). A survey of the marching cubes algorithm. Computers & Graphics, 30(5), 854-879. [cited by applicant]
Pan, J., Han, X., Chen, W., Tang, J., & Jia, K. (2019). Deep mesh reconstruction from single RGB images via topology modification networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp.… [cited by applicant]
Park, J. J., Florence, P., Straub, J., Newcombe, R., & Lovegrove, S. (2019). Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF Conference on Computer Vision … [cited by applicant]
Peng, S., Jiang, W., Pi, H., Li, X., Bao, H., & Zhou, X. (2020). Deep snake for real-time instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8533-8542). [cited by applicant]
Prevost, R., Cuingnet, R., Mory, B., Cohen, L. D., & Ardon, R. (Sep. 2013). Incorporating shape variability in image segmentation via implicit template deformation. In International Conference on Medical Image Computing… [cited by applicant]
Remelli, E., Lukoianov, A., Richter, S. R., Guillard, B., Bagautdinov, T., Baque, P., & Fua, P. (2020). Meshsdf: Differentiable iso-surface extraction. arXiv preprint arXiv:2006.03997. [cited by applicant]
Shvets, A. A., Rakhlin, A., Kalinin, A. A., & Iglovikov, V. I. (Dec. 2018). Automatic instrument segmentation in robot-assisted surgery using deep learning. In 2018 17th IEEE International Conference on Machine Learning… [cited by applicant]
Terzopoulos, D., Witkin, A., & Kass, M. (1988). Constraints on deformable models: Recovering 3D shape and nonrigid motion. Artificial intelligence, 36(1), 91-123. [cited by applicant]
Terzopoulos, D., Witkin, A., & Kass, M. (1988). Symmetry-seeking models and 3D object reconstruction. International Journal of Computer Vision, 1(3), 211-221. [cited by applicant]
Wang, N., Zhang, Y., Li, Z., Fu, Y., Liu, W., & Jiang, Y. G. (2018). Pixel2mesh: Generating 3d mesh. Models from single rgb images. In Proceedings of the European Conference on Computer Vision (ECCV) (pp. 52-67). [cited by applicant]
Wen, C., Zhang, Y., Li, Z., & Fu, Y. (2019). Pixel2mesh++: Multi-view 3d mesh generation via deformation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 1042-1051). [cited by applicant]
Wickramasinghe, U., Fua, P., & Knott, G. (2021). Deep Active Surface Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11652-11661). [cited by applicant]
Wickramasinghe, U., Remelli, E., Knott, G., & Fua, P. (Oct. 2020). Voxel2mesh: 3d mesh model generation from volumetric data. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp… [cited by applicant]
Xu, Q., Wang, W., Ceylan, D., Mech, R., & Neumann, U. (2019). DISN: Deep implicit surface network for high-quality single-view 3d reconstruction. arXiv preprint arXiv:1905.10711. [cited by applicant]