IP Library › Granted Patent US 12,393,847
Granted Patent B2
US 12,393,847 · App. 18/661,377 · Granted Aug 19, 2025

Gradient adversarial training of neural networks

Inventors: Ayan Tuhinendu Sinha (San Francisco, CA); Andrew Rabinovich (San Francisco, CA); Zhao Chen (Mountain View, CA); Vijay Badrinarayanan (Mountain View, CA)
Assignee: MAGIC LEAP, INC.
G06N3/088G06N3/045G06N3/084
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,393,847
App. No.
18/661,377
Filed
May 10, 2024
Granted
Aug 19, 2025
Kind
B2
Art Unit
2127
USPC
706/25
Abstract

Systems and methods for gradient adversarial training of a neural network are disclosed. In one aspect of gradient adversarial training, an auxiliary neural network can be trained to classify a gradient tensor that is evaluated during backpropagation in a main neural network that provides a desired task output. The main neural network can serve as an adversary to the auxiliary network in addition to a standard task-based training procedure. The auxiliary neural network can pass an adversarial gradient signal back to the main neural network, which can use this signal to regularize the weight tensors in the main neural network. Gradient adversarial training of the neural network can provide improved gradient tensors in the main network. Gradient adversarial techniques can be used to train multitask networks, knowledge distillation networks, and adversarial defense networks.

Claims (45)

1. A head mounted display system comprising:

non-transitory memory configured to store executable instructions, and a neural network that determines a task output associated with the head mounted display system,

wherein the neural network is trained using an auxiliary neural network trained by an auxiliary loss function programmed to receive gradient tensors evaluated during backpropagation of the neural network and to generate an adversarial gradient signal, and

wherein weights of the neural network are updated using the adversarial gradient signal from the auxiliary neural network;

a display;

a sensor; and

a hardware processor in communication with the non-transitory memory and the display, the hardware processor programmed by the executable instructions to:

receive a sensor datum captured by the sensor;

determine the task output using the neural network with the sensor datum as input;

cause the display to show information related to a task of the task output to a user of the head mounted display system;

perform an analysis of the information using the neural network trained by the auxiliary neural network trained by the auxiliary loss function; and

perform the task based on the analysis and the neural network trained by the auxiliary neural network trained by the auxiliary loss function.

2. The system of claim 1 , wherein the sensor comprises an outward-facing camera and the task comprises a computer vision task.

3. The system of claim 2 , wherein the computer vision task comprises one or more of face recognition, visual search, gesture identification, room layout estimation, cuboid detection, semantic segmentation, object detection, lighting detection, simultaneous localization and mapping, or relocalization.

4. The system of claim 1 , wherein the neural network comprises a multitask network, a knowledge distillation network, or an adversarial defense network.

5. The system of claim 1 , wherein a gradient of a main loss function with respect to a weight tensor in each layer of the neural network is evaluated for each of the gradient tensors.

6. The system of claim 1 , wherein a gradient tensor in the auxiliary neural network is processed and the adversarial gradient signal is passed back to the neural network via a gradient reversal procedure.

7. The system of claim 1 , wherein a signal for a layer in the neural network that is based at least partly on weights in preceding layers of the neural network is determined to provide the neural network the adversarial gradient signal.

8. The system of claim 1 , wherein the weights are updated to regularize the weights in the neural network based at least in part on the adversarial gradient signal.

9. The system of claim 1 , wherein the neural network comprises a multitask network, the task comprises a plurality of tasks, and the multitask network comprises:

a shared encoder;

a plurality of task-specific decoders associated with a respective task from the plurality of tasks; and

a plurality of gradient-alignment layers (GALs), each GAL of the plurality of GALs located after the shared encoder and before at least one of the task-specific decoders.

10. The system of claim 9 , wherein each GAL of the plurality of GALs is trained using a reversed gradient signal from the auxiliary neural network.

11. A method in a head mounted display system, the method comprising:

storing executable instructions, and a neural network that determines a task output associated with the head mounted display system,

wherein the neural network is trained using an auxiliary neural network trained by an auxiliary loss function programmed to receive gradient tensors evaluated during backpropagation of the neural network and to generate an adversarial gradient signal, and

wherein weights of the neural network are updated using the adversarial gradient signal from the auxiliary neural network;

receiving a sensor datum captured by a sensor of the head mounted display system;

determining the task output using the neural network with the sensor datum as input;

causing a display of the head mounted display system to show information related to a task of the task output to a user of the head mounted display system;

analyzing the information using the neural network trained by the auxiliary neural network trained by the auxiliary loss function; and

performing the task based on the analyzing and the neural network trained by the auxiliary neural network trained by the auxiliary loss function.

12. The method of claim 11 , wherein the sensor comprises an outward-facing camera and the task comprises a computer vision task.

13. The method of claim 12 , wherein the computer vision task comprises one or more of face recognition, visual search, gesture identification, room layout estimation, cuboid detection, semantic segmentation, object detection, lighting detection, simultaneous localization and mapping, or relocalization.

14. The method of claim 11 , wherein the neural network comprises a multitask network, a knowledge distillation network, or an adversarial defense network.

15. The method of claim 11 , wherein a gradient of a main loss function with respect to a weight tensor in each layer of the neural network is evaluated for each of the gradient tensors.

16. The method of claim 11 , wherein a gradient tensor in the auxiliary neural network is processed and the adversarial gradient signal is passed back to the neural network via a gradient reversal procedure.

17. The method of claim 11 , wherein a signal for a layer in the neural network that is based at least partly on weights in preceding layers of the neural network is determined to provide the neural network the adversarial gradient signal.

18. The method of claim 11 , wherein the weights are updated to regularize the weights in the neural network based at least in part on the adversarial gradient signal.

19. The method of claim 11 , wherein the neural network comprises a multitask network, the task comprises a plurality of tasks, and the multitask network comprises:

a shared encoder;

a plurality of task-specific decoders associated with a respective task from the plurality of tasks; and

a plurality of gradient-alignment layers (GALs), each GAL of the plurality of GALs located after the shared encoder and before at least one of the task-specific decoders.

20. The method of claim 19 , wherein each GAL of the plurality of GALs is trained using a reversed gradient signal from the auxiliary neural network.

Assignments (3)
SECURITY INTEREST Recorded Nov 3, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073480/0540 →
SECURITY INTEREST Recorded Oct 24, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073255/0581 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 31, 2024
From: SINHA, AYAN TUHINENDU; RABINOVICH, ANDREW; CHEN, ZHAO; BADRINARAYANAN, VIJAY
To: MAGIC LEAP, INC.
Reel/Frame 067585/0168 →
Continuity (3)
Continuation 17051982
Provisional Application 62673116 · May 17, 2018
Related Publication 20240330691A1 · Oct 3, 2024
References Cited (57)
US 6850221B1 · Tickle · 2005 [cited by applicant]
US 10521718B1 · Szegedy · 2019 [cited by examiner]
US 20060028436A1 · Armstrong · 2006 [cited by applicant]
US 20070081123A1 · Lewis · 2007 [cited by applicant]
US 20120127062A1 · Bar-Zeev et al. · 2012 [cited by applicant]
US 20120162549A1 · Gao et al. · 2012 [cited by applicant]
US 20130082922A1 · Miller · 2013 [cited by applicant]
US 20130117377A1 · Miller · 2013 [cited by applicant]
US 20130125027A1 · Abovitz · 2013 [cited by applicant]
US 20130208234A1 · Lewis · 2013 [cited by applicant]
US 20130242262A1 · Lewis · 2013 [cited by applicant]
US 20140071539A1 · Gao · 2014 [cited by applicant]
US 20140177023A1 · Gao et al. · 2014 [cited by applicant]
US 20140218468A1 · Gao et al. · 2014 [cited by applicant]
US 20140267420A1 · Schowengerdt et al. · 2014 [cited by applicant]
US 20150016777A1 · Abovitz et al. · 2015 [cited by applicant]
US 20150103306A1 · Kaji et al. · 2015 [cited by applicant]
US 20150178939A1 · Bradski et al. · 2015 [cited by applicant]
US 20150205126A1 · Schowengerdt · 2015 [cited by applicant]
US 20150309263A2 · Abovitz et al. · 2015 [cited by applicant]
US 20150326570A1 · Publicover et al. · 2015 [cited by applicant]
US 20150346495A1 · Welch et al. · 2015 [cited by applicant]
US 20160011419A1 · Gao · 2016 [cited by applicant]
US 20160026253A1 · Bradski et al. · 2016 [cited by applicant]
US 20170351952A1 · Zhang et al. · 2017 [cited by applicant]
US 20180012411A1 · Richey et al. · 2018 [cited by applicant]
US 20180137642A1 · Malisiewicz et al. · 2018 [cited by applicant]
US 20180268220A1 · Lee et al. · 2018 [cited by applicant]
US 20180268292A1 · Choi · 2018 [cited by examiner]
US 20190130110A1 · Lee · 2019 [cited by examiner]
US 20190130275A1 · Chen et al. · 2019 [cited by applicant]
WO WO2019222401A2 · 2019 [cited by applicant]
Ros, “Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing Their Input Gradients”, The Thirty-Second AAAI Conference on Artificial Intelligence, Apr. 25, 2018. (Year: 2018). [cited by examiner]
Andrychowicz et al, “Leaming to Learn by Gradient Descent by Gradient Descent,” arXiv:1606.04474v2 [cs.NE], Nov. 2016. (17 pages). [cited by applicant]
ARToolKit: htpps://web.archive.org/web/20051013062315/http://www.hitl.washington.edu:80/artoolkit/documentation/hardware.htm, archived Oct. 13, 2005. [cited by applicant]
Azuma, “A Survey of Augmented Reality,” In Presence: Teleoperators and Virtual Environments 6(4):355-385, http://www.cs.unc.edu/˜azuma, Aug. 1997. [cited by applicant]
Azuma, “Predictive Tracking for Augmented Reality,” Dissertation, TR95-007, Doctor of Philosophy, Department of Computer Science, UNC-Chapel Hill, NC, Feb. 1995. (262 pages). [cited by applicant]
Bimber et al., “Spatial Augmented Reality-Merging Real and Virtual Worlds,” 2005. (393 pages). [cited by applicant]
Chen et al., “GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks,” arXiv:1711.02257v1 [cs.CV], Nov. 2017. [cited by applicant]
Chollet, “Deep Learning with Python,” Manning Publications, Version 6, pp. 1398-1410, 2017. [cited by applicant]
Drucker et al., “Double Backpropagation Increasing Generalization Performance,” In [cited by applicant]
Ehmann et al., “Transferring Information Between Neural Networks,” [cited by applicant]
Ganin et al., “Domain-Adversarial Training of Neural Networks,” [cited by applicant]
Gomes, “Convolutional Neural Networks,” [cited by applicant]
Goodfellow et al., “Advances in Neural Information Processing Systems 27,” Curran Associates, Inc., via NIPS Generative Adversarial nets Paper, pp. 2672-2680, 2014. [cited by applicant]
Goodfellow et al., “Explaining and Harnessing Adversarial Examples,” [cited by applicant]
He et al., “Deep Residual Learning for Image Recognition,” [cited by applicant]
Huang et al., “Adversarial Attacks on Neural Network Policies,” arXiv:1702.02284v1 [cs.LG], Feb. 2017. (10 pages). [cited by applicant]
International Preliminary Report on Patentability, dated Nov. 17, 2020, for International Application No. PCT/US19/32486. (5 pages). [cited by applicant]
International Search Report and Written Opinion, dated Nov. 13, 2019, for International Application No. PCT/US19/32486. (14 pages). [cited by applicant]
Jacob, “Eye Tracking in Advanced Interface Design,” [cited by applicant]
Jaderberg et al, “Decoupled Neural Interfaces using Synthetic Gradients,” [cited by applicant]
Nøkland “Improving Back-Propagation by Adding an Adversarial Gradient,” arXiv:1510.04189v2 [stat.ML], Apr. 2016. (8 pages). [cited by applicant]
Selvaraju et al., “Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization,” [cited by applicant]
Sinha et al. “Gradient Adversarial Training of Neural Networks,” arXiv:1806.08028v1 [cs.LG], Jun. 2018. (13 pages). [cited by applicant]
Srinivas et al., “Knowledge Transfer with Jacobian Matching,” [cited by applicant]
Tanriverdi et al., “Interacting With Eye Movements in Virtual Environments,” Department of Electrical Engineering and Computer Science, Tufts University, Medford, MA, [cited by applicant]