IP Library › Granted Patent US 12,602,921
Granted Patent B2
US 12,602,921 · App. 16/885,170 · Granted Apr 14, 2026

Techniques for modifying and training a neural network

Inventors: Wentao Zhu (Bethesda, MD); Daguang Xu (Potomac, MD); Can Zhao (Rockville, MD); Ziyue Xu (Reston, VA); Holger Reinhard Roth (Rockville, MD)
Assignee: NVIDIA Corporation
G06V10/94G06F9/3877G06F18/214G06N3/04G06N3/08G06T7/0012G06T7/10G06V10/764G06V10/82
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,602,921
App. No.
16/885,170
Granted
Apr 14, 2026
Kind
B2
Abstract

Apparatuses, systems, and techniques are described herein to speed up inferencing in a neural network by copying output from one layer of the neural network to another computing resource based on dependencies among layers in the network. In at least one embodiment, a processor comprising one or more circuits causes two or more subsequent layers of one or more neural networks to be performed on separate computing resources from a previous layer of the one or more neural networks.

Claims (43)

1 . A processor, comprising:

one or more circuits to cause two or more subsequent layers of one or more neural networks to be performed on separate computing resources from a precedent layer of the one or more neural networks based, at least in part, on whether a same output of the precedent layer is to be provided to both the two or more subsequent layers, wherein the same output of the precedent layer is to be provided to both the two or more subsequent layers by:

determining that both the two or more subsequent layers are dependent on the same output from precedent layer;

copying the same output of the precedent layer; and

providing the same output of the precedent layer to a first subsequent layer of the two or more subsequent layers and a copied output to a second subsequent layer of the two or more subsequent layers.

2 . The processor of claim 1 , further comprising:

the one or more circuits to:

modify the one or more neural networks to reduce an amount of dependencies between the precedent layer and the two or more subsequent layers of the one or more neural networks; and

partition the modified one or more neural networks to be trained using pipeline parallelism by:

copying data from a component of the one or more neural networks from a computing resource to another computing resource, wherein the computing resource from which data is being copied performs the component of the one or more neural networks dependent on the data and the another computing resource to which the data is copied performs another component of the one or more neural networks dependent on the data.

3 . The processor of claim 1 , wherein both the two or more subsequent layers comprises at least an encoder in the first subsequent layer and a decoder in the second subsequent layer.

4 . The processor of claim 3 , wherein the same output is provided to the encoder in the first subsequent layer and the copied output is provided to the decoder in the second subsequent layer.

5 . The processor of claim 4 , wherein the first subsequent layer is to be performed by the processor and the second subsequent layer is to be performed by a different processor.

6 . The processor of claim 1 , further comprising:

the one or more circuits to cause both the two or more subsequent layers of the one or more neural networks to be performed, at a same time, using the same output of the precedent layer.

7 . The processor of claim 1 , wherein the separate computing resources comprises:

graphical processing units (GPUs), parallel processing units (PPUs), computer systems, pipelines, multiprocessors, other circuits of a different processor, and/or other circuits of a same processor.

8 . A system, comprising:

one or more computers having one or more processors to cause two or more subsequent layers of one or more neural networks to be performed on separate computing resources from a previous layer of the one or more neural networks based, at least in part, on whether a same output of the previous layer is to be provided to both the two or more subsequent layers, wherein the same output of the previous layer is to be provided to both the two or more subsequent layers by:

determining that both the two or more subsequent layers are dependent on output from the previous layer;

copying the output of the previous layer; and

providing the output of the previous layer to a first subsequent layer of the two or more subsequent layers and a copied output to a second subsequent layer of the two or more subsequent layers.

9 . The system of claim 8 , wherein the previous layer comprises a layer of the one or more neural networks that is performed prior to both the two or more subsequent layers.

10 . The system of claim 8 , wherein the same output of the previous layer is to be provided to both the two or more subsequent layers further comprises:

generating a duplicated copy of the output of the previous layer; and

providing the duplicated copy to one of the two or more subsequent layers.

11 . The system of claim 8 , wherein the previous layer comprises an encoder and both the two or more subsequent layers comprise a second encoder in a first subsequent layer and a decoder in a second subsequent layer.

12 . The system of claim 8 , further comprising the one or more computers having one or more processors to modify a parallel block from the one or more neural networks to remove a dependency within the parallel block.

13 . The system of claim 8 , further comprising the one or more computers having one or more processors to train the one or more neural networks by reducing dependencies of the output from the previous layer with the two or more subsequent layers.

14 . The system of claim 8 , further comprising the one or more computers having one or more processors to train the one or more neural networks to perform medical image segmentation.

15 . The system of claim 8 , further comprise the one or more computers having one or more processors to partition the one or more neural networks to be trained using pipeline parallelism.

16 . A non-transitory machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to cause two or more subsequent layers of one or more neural networks to be performed on separate computing resources from a preceding layer of the one or more neural networks based, at least in part, on whether a same output of the preceding layer is to be provided to both the two or more subsequent layers, wherein the same output of the preceding layer is to be provided to both the two or more subsequent layers by:

determining that both the two or more subsequent layers are dependent on output from preceding layer;

copying the output of the preceding layer; and

providing the output of the preceding layer to a first subsequent layer of the two or more subsequent layers and a copied output to a second subsequent layer of the two or more subsequent layers.

17 . The non-transitory machine-readable medium of claim 16 , wherein the set of instructions, which if performed by the one or more processors, further cause the one or more processors to:

modify the one or more neural networks by modifying a first parallel block to reduce an amount of dependencies between layers of the first parallel block; and

partition the modified one or more neural networks to be trained using pipeline parallelism.

18 . The non-transitory machine-readable medium of claim 17 , wherein the set of instructions, which if performed by the one or more processors, further cause the one or more processors to generate a second parallel block, wherein layers in the second parallel block is to be performed by a different computing resource than layers in the first parallel block.

19 . The non-transitory machine-readable medium of claim 18 , wherein the set of instructions, which if performed by the one or more processors, further cause the one or more processors to copy data from a component of the one or more neural networks from the first parallel block to the second parallel block such that the different computing resource performs another component of the one or more neural networks using the copied data.

20 . The non-transitory machine-readable medium of claim 19 , wherein the output of the preceding layer from the first parallel block is provided to a second encoder in a first subsequent layer of the two or more subsequent layers and a copy of the output is provided to a decoder in the second parallel block.

21 . The non-transitory machine-readable medium of claim 16 , wherein the set of instructions, which if performed by the one or more processors, further cause the one or more processors to cause both the two or more subsequent layers of the one or more neural networks to be performed, simultaneously, using the same output of the preceding layer.

22 . The non-transitory machine-readable medium of claim 16 , wherein the set of instructions, which if performed by the one or more processors, further cause the one or more processors to train the one or more neural networks by using a combination of training data parallelism and model parallelism.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 24, 2020
From: ZHU, WENTAO; XU, DAGUANG; ZHAO, CAN; XU, ZIYUE; ROTH, HOLGER REINHARD
To: NVIDIA CORPORATION
Reel/Frame 053030/0506 →
Continuity (1)
Related Publication 20210374518A1 · Dec 2, 2021
References Cited (50)
US 10106153B1 · Xiao · 2018 [cited by examiner]
US 20180314250A1 · Lewis · 2018 [cited by examiner]
US 20190205747A1 · Srivastava · 2019 [cited by applicant]
US 20200026988A1 · Guo et al. · 2020 [cited by applicant]
US 20200074318A1 · Chen · 2020 [cited by applicant]
US 20200175396A1 · Boyce · 2020 [cited by examiner]
US 20200211231A1 · Brownlee · 2020 [cited by examiner]
US 20200234130A1 · Yan · 2020 [cited by examiner]
CN 110383292A · 2019 [cited by applicant]
CN 110582748A · 2019 [cited by applicant]
NPL—Device Placement Optimization with Reinforcement Learning by Mirhoseini et al.(Year: 2017). [cited by examiner]
NPL—TernausNet: U-Net with VGG11 Encoder Pre-trained on ImageNet for Image Segmentation by Iglovikov et al. (Year: 2018). [cited by examiner]
Harlap et al., “PipeDream: Fast and Efficient Pipeline Parallel DNN Training,” Jun. 8, 2018, 14 pages. [cited by applicant]
Mirhoseini et al., “Device Placement Optimization with Reinforcement Learning,” Jun. 25, 2017, 11 pages. [cited by applicant]
United Kingdom Combined Search and Examination Report for Patent Application No. 2107525.4 dated Oct. 25, 2021, 6 pages. [cited by applicant]
Zhu et al., “LAMP: Large Deep Nets with Automated Model Parallelism for Image Segmentation,” Jun. 26, 2020, 12 pages. [cited by applicant]
Bengio et al., “Curriculum Learning,” Proceedings of the 26th Annual International Conference on Machine Learning, 2009, 8 pages. [cited by applicant]
Blumberg et al., “Deeper Image Quality Transfer: Training Low-Memory Neural Networks for 3D Images,” International Conference on Medical Image Computing and Computer-Assisted Intervention, Aug. 16, 2018, 3 pages. [cited by applicant]
Brügger et al., “A Partially Reversible U-Net for Memory-Efficient Volumetric Image Segmentation,” International Conference on Medical Image Computing and Computer-Assisted Intervention, Jun. 20, 2019, 8 pages. [cited by applicant]
Chen et al., “Training Deep Nets with Sublinear Memory Cost,” Apr. 22, 2016, 12 pages. [cited by applicant]
Clark et al., “The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository,” Journal of digital imaging 26(6): 2013, 14 pages. [cited by applicant]
Deng et al. “Imagenet: A Large-Scale Hierarchical Image Database,” ICLR, 2009, 8 pages. [cited by applicant]
Devlin et al., “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” Oct. 11, 2018, 14 pages. [cited by applicant]
Gomez et al., “The Reversible Residual Network: Backpropagation without Storing Activations,” Advances in Neural Information Processing Systems, 2017, 11 pages. [cited by applicant]
Hu et al., “Squeeze-and-Excitation Networks,” CVPR, 2018, 10 pages. [cited by applicant]
Huang et al., “GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism,” Advances in Neural Information Processing Systems, 2019, 10 pages. [cited by applicant]
IEEE, “IEEE Standard 754-2008 (Revision of IEEE Standard 754-1985): IEEE Standard for Floating-Point Arithmetic,” Aug. 29, 2008, 70 pages. [cited by applicant]
Isensee et al., “nnU-Net: Self-Adapting Framework for U-Net-Based Medical Image Segmentation,” arXiv:1809.10486, Sep. 27, 2018, 11 pages. [cited by applicant]
Jesson et al., “CASED: Curriculum Adaptive Sampling for Extreme Data Imbalance,” International Conference on Medical Image Computing and Computer-Assisted Intervention, Jul. 27, 2018, 8 pages. [cited by applicant]
Kervadec et al., “Boundary Loss for Highly Unbalanced Segmentation,” International Conference on Medical Imaging with Deep Learning, 2019, 12 pages. [cited by applicant]
Lee et al., “A GPipe Implementation in PyTorch,” retrieved from https://github.com/kakaobrain/torchgpipe, 2019, 6 pages. [cited by applicant]
Martens et al., “Training Deep and Recurrent Networks with Hessian-Free Optimization,” Neural networks: Tricks of the trade, 2012, 63 pages. [cited by applicant]
Micikevicius et al., “Mixed Precision Training,” Oct. 12, 2017, 14 pages. [cited by applicant]
Narayanan et al., “Pipedream: Generalized Pipeline Parallelism for DNN Training,” Proceedings of the 27th ACM Symposium on Operating Systems Principles, 2019, 15 pages. [cited by applicant]
Radford et al., “Language Models are Unsupervised Multitask Learners,” OpenAI Blog 1(8), 2019, 24 pages. [cited by applicant]
Rajbhandari et al., “Zero: Memory Optimization Towards Training a Trillion Parameter Models,” Oct. 7, 2019, 17 pages. [cited by applicant]
Raudaschl et al.: Evaluation of Segmentation Methods on Head and Neck CT: Auto-Segmentation Challenge, Medical physics, 2017, 17 pages. [cited by applicant]
Ronneberger et al., “U-net: Convolutional networks for biomedical image segmentation,” International Conference on Medical Image Computing and Computer-Assisted Intervention, Oct. 5, 2015, 8 pages. [cited by applicant]
Shazeer et al., “Mesh-TensorFlow: Deep Learning for Supercomputers,” Advances in Neural Information Processing Systems, 2018, 10 pages. [cited by applicant]
Shoeybi et al., “Megatron-LM: Training Multi-Billion Parameter Language Models Using GPU Model Parallelism,” Oct. 5, 2019, 15 pages. [cited by applicant]
Simpson et al.: A Large Annotated Medical Image Dataset for the Development and Evaluation of Segmentation Algorithms, Feb. 25, 2019, 15 pages. [cited by applicant]
Society of Automotive Engineers On-Road Automated Vehicle Standards Committee, “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles,” Standard No. J3016-201609, issued Jan… [cited by applicant]
Society of Automotive Engineers On-Road Automated Vehicle Standards Committee, “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles,” Standard No. J3016-201806, issued Jan… [cited by applicant]
Sudre et al., Generalised Dice Overlap as a Deep Learning Loss Function for Highly Unbalanced Segmentations, Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, Jul. 14, 2017, … [cited by applicant]
Tajbakhsh et al., “Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?” IEEE Transactions on Medical Imaging, 35(5), Jun. 2, 2017, 17 pages. [cited by applicant]
Tieleman et al., “Lecture 6.5-rmsprop: Divide the Gradient by a Running Average of Its Recent Magnitude,” COURSERA: Neural Networks for Machine Learning, 4(2), 2012, 31 pages. [cited by applicant]
Valliéres, et al., “Radiomics Strategies for Risk Assessment of Tumour Failure in Head-and-Neck Cancer,” Scientific Reports, 7(1): 2017, 14 pages. [cited by applicant]
Zhu et al., “Anatomynet: Deep Learning for Fast and Fully Automated Whole-Volume Segmentation of Head and Neck Anatomy,” Medical Physics 46(2): 2019, 13 pages. [cited by applicant]
Zhuang et al., “Invertible Network for Classification and Biomarker Selection for ASD.,” International Conference on Medical Image Computing and Computer-Assisted Intervention, 2019, 16 pages. [cited by applicant]
Office Action for Chinese Application No. 202110577146.2, mailed Feb. 20, 2024, 24 pages. [cited by applicant]