IP Library › Granted Patent US 12,223,734
Granted Patent B2
US 12,223,734 · App. 17/151,001 · Granted Feb 11, 2025

Systems and methods for training machine-learned models with deviating intermediate representations

Inventors: Xuanyuan Tu (Milton, CA); Raquel Urtasun (Toronto, CA); Tsun-Hsuan Wang (Cambridge, CA); Sivabalan Manivasagam (Toronto, CA); Jingkang Wang (Toronto, CA); Mengye Ren (Toronto, CA)
Assignee: AURORA OPERATIONS, INC.
G06V20/56G01S17/931G05D1/0088G05D1/0221G06F18/2163G06F18/217G06F18/24G06N20/00G06V10/764G06V10/82
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,223,734
App. No.
17/151,001
Granted
Feb 11, 2025
Kind
B2
Abstract

Systems and methods for vehicle-to-vehicle communications are provided. An adverse system can obtain sensor data representative of an environment proximate to a targeted system. The adverse system can generate an intermediate representation of the environment and a representation deviation for the intermediate representation. The representation deviation can be designed to disrupt a machine-learned model associated with the target system. The adverse system can communicate the intermediate representation modified by the representation deviation to the target system. The target system can train the machine-learned model associated with the target system to detect the modified intermediate representation. Detected modified intermediate representations can be discarded before disrupting the machine-learned model.

Claims (62)

1. A computer-implemented method, the method comprising:

obtaining sensor data representative of a secondary environment proximate to an autonomous vehicle;

generating an intermediate representation for the autonomous vehicle based, at least in part, on the sensor data, wherein the intermediate representation is descriptive of at least a portion of the secondary environment;

determining an intermediate representation deviation for the intermediate representation based, at least in part, on the intermediate representation and a machine-learned model associated with the autonomous vehicle, by:

obtaining, via the machine-learned model, one or more ground truth bounding box proposals based, at least in part, on the intermediate representation,

obtaining, via the machine-learned model, one or more deviating bounding box proposals, and

determining the intermediate representation deviation for the intermediate representation based, at least in part, on a comparison between the one or more ground truth bound box proposals and the one or more deviating bounding box proposals;

generating data indicative of a modified intermediate representation based, at least in part, on the intermediate representation and the intermediate representation deviation; and

communicating the data indicative of the modified intermediate representation to a vehicle computing system associated with the autonomous vehicle.

2. The computer-implemented method of claim 1 , wherein the machine-learned model associated with the autonomous vehicle comprises a machine-learned model utilized by the vehicle computing system to detect one or more objects within a surrounding environment of the autonomous vehicle.

3. The computer-implemented method of claim 2 , wherein the machine-learned model is configured to output one or more bounding box proposals indicative of one or more objects within the surrounding environment of the autonomous vehicle based, at least in part, on one or more intermediate representations.

4. The computer-implemented method of claim 1 , wherein the intermediate representation deviation is based at least in part on an adversarial loss.

5. The computer-implemented method of claim 4 , wherein modifying the intermediate representation deviation for the intermediate representation based, at least in part, on the comparison between the one or more ground truth bound box proposals and the one or more deviating bounding box proposals comprise:

determining the adversarial loss for the intermediate representation deviation based, at least in part, on the one or more ground truth bound box proposals and the one or more deviating bounding box proposals; and

modifying the intermediate representation deviation based, at least in part, on the adversarial loss, wherein the intermediate representation deviation is modified to minimize the adversarial loss over the one or more deviating bounding box proposals.

6. The computer-implemented method of claim 5 , wherein each respective ground truth bounding box proposal of the one or more ground truth bounding box proposals comprises a respective ground truth class score indicative of respective ground truth object classification and one or more respective ground truth bounding box parameters indicative of a respective ground truth spatial location and one or more respective ground truth dimensions of the respective ground truth object classification, and

wherein each respective deviating bounding box proposal of the one or more respective deviating bounding box proposals comprises a respective deviating class score indicative of a respective deviating object classification and one or more respective deviating bounding box parameters indicative of a respective deviating spatial location and one or more respective deviating dimensions of the respective deviating object classification.

7. The computer-implemented method of claim 6 , wherein the adversarial loss is determined based, at least in part, on a difference between a ground truth class score corresponding to at least one ground truth bounding box proposal and a deviating class score corresponding to a deviating bounding box proposal corresponding to the at least one ground truth bounding box proposal.

8. The computer-implemented method of claim 6 , wherein the adversarial loss is determined based, at least in part, on a difference between one or more ground truth bounding box parameters corresponding to at least one ground truth bounding box proposal and one or more deviating bounding box parameters corresponding to a deviating bounding box proposal corresponding to the at least one ground truth bounding box proposal.

9. The computer-implemented method of claim 4 , wherein the computing system is onboard a transmitting autonomous vehicle physically located proximate to the autonomous vehicle, and wherein the intermediate representation deviation is associated with a first time.

10. The computer-implemented method of claim 9 , further comprising:

obtaining movement data indicative of a motion of the transmitting autonomous vehicle from the first time to a second time;

obtaining second sensor data representative of the secondary environment proximate to the autonomous vehicle at the second time;

generating a second intermediate representation for the autonomous vehicle based, at least in part, on the second sensor data; and

determining a second intermediate representation deviation for the second intermediate representation based, at least in part, on the intermediate representation deviation associated with the first time and the movement data.

11. A computing system comprising:

one or more processors; and

one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the system to perform operations, the operations comprising:

obtaining a plurality of intermediate representations associated with an autonomous vehicle, wherein each intermediate representation is descriptive of at least a portion of a secondary environment proximate to the autonomous vehicle at a plurality of times;

generating a surrogate machine-learned model based, at least in part, on the plurality of intermediate representations;

obtaining a target intermediate representation from the plurality of intermediate representations;

determining an intermediate representation deviation for the target intermediate representation based, at least in part, on the target intermediate representation and the surrogate machine-learned model by:

obtaining, via the surrogate machine-learned model, one or more ground truth bounding box proposals based, at least in part, on the target intermediate representation,

obtaining, via the surrogate machine-learned model, one or more deviating bounding box proposals, and

determining the intermediate representation deviation for the target intermediate representation based, at least in part, on a comparison between the one or more ground truth bound box proposals and the one or more deviating bounding box proposals;

generating data indicative of a modified intermediate representation based, at least in part, on the target intermediate representation and the intermediate representation deviation; and

communicating the data indicative of the modified intermediate representation to a vehicle computing system associated with the autonomous vehicle.

12. The computing system of claim 11 , wherein each intermediate representation of the plurality of intermediate representations is generated by a first portion of a machine-learned model associated with the autonomous vehicle, and wherein a first portion of the surrogate machine-learned model is trained to output a surrogate intermediate representation, wherein the surrogate intermediate representation comprises one or more similarities to the plurality of intermediate representations.

13. The computing system of claim 11 , wherein generating the surrogate machine-learned model based, at least in part, on the plurality of intermediate representations comprises:

obtaining sensor data representative of surrogate environment proximate to the autonomous vehicle; and

generating the surrogate machine-learned model based, at least in part, on the plurality of intermediate representations and the sensor data.

14. The computing system of claim 13 , wherein generating the surrogate machine-learned model based, at least in part, on the plurality of intermediate representations further comprises:

generating, via a first portion of the surrogate machine-learned model, a surrogate intermediate representation based, at least in part, on the sensor data;

generating, via a machine-learned discriminator model, a discriminator loss based, at least in part, on the surrogate intermediate representation and at least one of the plurality of intermediate representations; and

training the surrogate machine-learned model to minimize the discriminator loss.

15. The computing system of claim 14 , wherein the discriminator loss is indicative of a difference between the surrogate intermediate representation and the at least one intermediate representation.

16. The computing system of claim 11 , wherein a second portion of the surrogate machine-learned model is configured to output one or more bounding box proposals indicative of one or more objects within the secondary environment proximate to the autonomous vehicle based, at least in part, on the target intermediate representation.

17. The computing system of claim 16 , wherein the intermediate representation deviation is based at least in part on an adversarial loss.

18. An autonomous vehicle comprising:

one or more sensors;

one or more processors; and

one or more tangible, non-transitory, computer readable media that collectively store instructions that when executed by the one or more processors cause the one or more processors to perform operations, the operations comprising:

obtaining, via the one or more sensors, sensor data representative of a surrounding environment of the autonomous vehicle;

generating, via a first portion of a machine-learned model, an intermediate representation based, at least in part, on the sensor data, wherein the intermediate representation is descriptive of at least a portion of the surrounding environment of the autonomous vehicle;

determining an intermediate representation deviation for the intermediate representation based, at least in part, on the intermediate representation and the machine-learned model, by:

obtaining, via the machine-learned model, one or more ground truth bounding box proposals based, at least in part, on the intermediate representation,

obtaining, via the machine-learned model, one or more deviating bounding box proposals, and

determining the intermediate representation deviation for the intermediate representation based, at least in part, on a comparison between the one or more ground truth bound box proposals and the one or more deviating bounding box proposals;

generating, data indicative of a modified intermediate representation based, at least in part, on the intermediate representation and the intermediate representation deviation; and

communicating the data indicative of the modified intermediate representation to one or more devices associated with a target autonomous vehicle.

19. The autonomous vehicle of claim 18 , wherein the target autonomous vehicle is configured to utilize a second portion of the machine-learned model to detect one or more objects within a surrounding environment of the target autonomous vehicle.

20. The autonomous vehicle of claim 18 , wherein the machine-learned model is trained to detect the modified intermediate representation.

Assignments (6)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 14, 2024
From: UATC, LLC
To: AURORA OPERATIONS, INC.
Reel/Frame 067733/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 7, 2022
From: UBER TECHNOLOGIES, INC.
To: UATC, LLC
Reel/Frame 058962/0140 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 29, 2021
From: URTASUN SOTIL, RAQUEL
To: UBER TECHNOLOGIES, INC.
Reel/Frame 056969/0695 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 24, 2021
From: TU, XUANYUAN
To: UBER TECHNOLOGIES, INC.
Reel/Frame 055703/0732 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 16, 2021
From: UBER TECHNOLOGIES, INC.
To: UATC, LLC
Reel/Frame 055267/0625 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 12, 2021
From: WANG, TSUN-HSUAN; MANIVASAGAM, SIVABALAN; WANG, JINGKANG; REN, MENGYE
To: UBER TECHNOLOGIES, INC.
Reel/Frame 055247/0988 →
Continuity (3)
Provisional Application 63132780 · Dec 31, 2020
Provisional Application 62985865 · Mar 5, 2020
Related Publication 20210279640A1 · Sep 9, 2021
References Cited (68)
US 10664722B1 · Sharma · 2020 [cited by examiner]
US 11010907B1 · Bagwell · 2021 [cited by examiner]
US 20180130324A1 · Yu · 2018 [cited by examiner]
US 20180272963A1 · Meyhofer · 2018 [cited by examiner]
US 20190049987A1 · Djuric · 2019 [cited by examiner]
US 20200174490A1 · Ogale · 2020 [cited by examiner]
US 20200201351A1 · Armstrong-Crews · 2020 [cited by examiner]
US 20210037044A1 · Achanta · 2021 [cited by examiner]
US 20210157912A1 · Kruthiveti Subrahmanyeswara Sai · 2021 [cited by examiner]
US 20220126864A1 · Moustafa · 2022 [cited by examiner]
Gindele, Tobias, Sebastian Brechtel, and Rudiger Dillmann. “Learning driver behavior models from traffic observations for decision making and planning.” IEEE Intelligent Transportation Systems Magazine 7.1 (2015): 69-79… [cited by examiner]
Hubschneider, Christian, et al. “Integrating end-to-end learned steering into probabilistic autonomous driving.” 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2017. (Year: 2… [cited by examiner]
Drews, Paul, et al. “Aggressive deep driving: Model predictive control with a cnn cost model.” arXiv preprint arXiv:1707.05303 (2017): 1-11 (Year: 2017). [cited by examiner]
Liu, Yanpei, et al. “Delving into transferable adversarial examples and black-box attacks.” arXiv preprint arXiv:1611.02770 v3 (2017): 1-24 (Year: 2017). [cited by examiner]
Amini, Alexander, et al. “Variational autoencoder for end-to-end control of autonomous driving with novelty detection and training de-biasing.” 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (I… [cited by examiner]
Cao, Yulong, et al. “Adversarial sensor attack on lidar-based perception in autonomous driving.” Proceedings of the 2019 ACM SIGSAC conference on computer and communications security. 2019: 2267-2281 (Year: 2019). [cited by examiner]
Qayyum, Adnan, et al. “Securing Connected & Autonomous Vehicles: Challenges Posed by Adversarial Machine Learning and The Way Forward.” arXiv preprint arXiv:1905.12762 (2019): 1-29 (Year: 2019). [cited by examiner]
Deng, Yao, et al. “An Analysis of Adversarial Attacks and Defenses on Autonomous Driving Models.” arXiv preprint arXiv:2002.02175 (Feb. 6, 2020). (Year: 2020). [cited by examiner]
Yang, Luona, et al. “Real-to-virtual domain unification for end-to-end autonomous driving.” Proceedings of the European conference on computer vision (ECCV). 2018. (Year: 2018). [cited by examiner]
Ilievski, Marko, et al. “Design space of behaviour planning for autonomous driving.” arXiv preprint arXiv:1908.07931 (2019). (Year: 2019). [cited by examiner]
Bonawitz et al., “Towards Federated Learning at Scale: System Design”, arXiv:1902.01046v2, Mar. 22, 2019, 15 pages. [cited by applicant]
Boreselius, “Mobile agent security”, Electronics and Communication Engineering Journal, vol. 14, No. 5, Aug. 2002, 11 pages. [cited by applicant]
Brendel et al., “Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models”, International Conference on Learning Representations, Apr. 30-May 3, 2018, Vancouver, Canada, 12 pages. [cited by applicant]
Brunner et al., “Guessing Smart: Biased Sampling for Efficient Black-Box Adversarial Attacks”, arXiv:1812.09803v3, May 5, 2019. [cited by applicant]
Cao et al., “Adversarial Sensor Attack on LiDAR-based Perception in Autonomous Driving”, ACM Conference on Computer and Communications Security, Nov. 11-15, 2019, London, UK, pp. 2267-2281. [cited by applicant]
Carlini et al., “Towards Evaluating the Robustness of Neural Networks”, IEEE Symposium on Security and Privacy, May 22-25, 2017, San Jose, CA, 19 pages. [cited by applicant]
Chen et al., “Cooper: Cooperative Perception for Connected Autonomous Vehicles based on 3D Point Clouds”, IEEE 39 [cited by applicant]
Chen et al., “HopSkipJumpAttack: A Query-Efficient Decision-Based Attack”, arXiv:1904.02144v3, Jun. 10, 2019, 26 pages. [cited by applicant]
Chen et al., “ZOO: Zeroth Order Optimization Based Black-box Attacks to Deep Neural Networks without Training Substitute Models”, ACM SIGSAC Conference on Computer and Communications Security, Nov. 3, 2017, Dallas, TX, … [cited by applicant]
Cheng et al., “Improving Black-box Adversarial Attacks with a Transfer-based Prior”, Conference on Neural Information Processing Systems, Dec. 8-14, 2019, Vancouver, Canada, 11 pages. [cited by applicant]
Cheng et al., “Query-Efficient Hard-Label Black-Box Attack: An Optimization-Based Approach”, International Conference on Learning Representations, May 6-9, 2019, New Orleans, LA, 14 pages. [cited by applicant]
Cheng et al., “Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples”, arXiv:1803.01128v1, Mar. 3, 2018, 16 pages. [cited by applicant]
Dillon et al., “Cloud Computing: Issues and Challenges”, IEEE International Conference on Advanced Information Networking and Applications, Apr. 20-23, 2010, Perth, Western Australia, pp. 27-33. [cited by applicant]
Dong et al., “Boosting Adversarial Attacks with Momentum”, Conference on Computer Vision and Pattern Recognition, Jun. 18-22, 2018, Salt Lake City, UT, pp. 9185-9193. [cited by applicant]
Eshratifar et al., “Energy and Performance Efficient Computation Offloading for Deep Neural Networks in a Mobile Cloud Computing Environment”, ACM Great Lakes Symposium on VLSI, May 23-25, 2018, Chicago, Illinois, pp. 1… [cited by applicant]
Gil et al., “White-to-Black: Efficient Distillation of Black-Box Adversarial Attacks”, arXiv:1904.02405v1, Apr. 4, 2019, 7 pages. [cited by applicant]
Goodfellow et al., “Explaining and Harnessing Adversarial Examples”, arXiv:1412.6572v3, Mar. 20, 2015, 11 pages. [cited by applicant]
Heusel et al., “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium”, Conference on Neural Information Processing Systems, Dec. 4-9, 2017, Long Beach, CA 12 pages. [cited by applicant]
Huang et al., “Adversarial Attacks on Neural Network Policies”, 5th International Conference on Learning Representations, Apr. 24-26, 2017, Toulon, France, 7 pages. [cited by applicant]
Huang et al., “Enhancing Adversarial Example Transferability with an Intermediate Level Attack”, arXiv:1907.10823v2, Oct. 6, 2019, 20 pages. [cited by applicant]
Ilyas et al., “Black-box Adversarial Attacks with Limited Queries and Information”, International Conference on Machine Learning, Jul. 10-15, 2018, Stockholm, Sweden, 10 pages. [cited by applicant]
Jiang et al., “Black-box Adversarial Attacks on Video Recognition Models”, ACM Multimedia, Oct. 21-25, 2019, Nice, France, pp. 864-872. [cited by applicant]
Konecny et al., “Federated Learning: Strategies for Improving Communication Efficiency”, arXiv:1610.05492v1, Oct. 18, 2016, 5 pages. [cited by applicant]
Liu et al., “Delving into Transferable Adversarial Examples and Black-Box Attacks”, 5th International Conference on Learning Representations, Apr. 24-26, 2017, Toulon, France, 14 pages. [cited by applicant]
Madry et al., “Towards Deep Learning Models Resistant to Adversarial Attacks”, arXiv:1706.06083v3, Nov. 9, 2017, 27 pages. [cited by applicant]
Manivasagam et al., “LiDARsim: Realistic LiDAR Simulation by Leveraging the Real World”, Conference on Computer Vision and Pattern Recognition, Jun. 14-19, 2020, Virtual, pp. 11167-11176. [cited by applicant]
Miyato et al., “Spectral Normalization for Generative Adversarial Networks”, arXiv:1802.05957v1, Feb. 16, 2018, 26 pages. [cited by applicant]
Nakamoto, “Bitcoin: A Peer-to-Peer Electronic Cash System”, https://bitcoin.org/bitcoin.pdf?, 2019, 9 pages. [cited by applicant]
Novak et al., “Communication Security in Multi-agent Systems”, CEEMAS Lecture Notes in Computer Science, vol. 2691, 2003, pp. 454-463. [cited by applicant]
Obst et al., “Multi-Sensor Data Fusion for Checking Plausibility of V2V Communications by Vision-based Multiple-Object Tracking”, Dec. 3-5, 2014, Paderborn, Germany, pp. 143-150. [cited by applicant]
Papernot et al., “Crafting Adversarial Input Sequences for Recurrent Neural Networks”, The 35th Military Communications Conference (MILCOM 2016), Nov. 1-3, 2016, Baltimore, MD, 6 pages. [cited by applicant]
Papernot et al., “Practical Black-Box Attacks against Machine Learning”, Asia CCS, Apr. 2-6, 2017, Abu Dhabi, United Arab Emirates, pp. 506-519. [cited by applicant]
Rauch et al., “Car2X-Based Perception in a High-Level Fusion Architecture for Cooperative Perception Systems”, 2012 Intelligent Vehicles Symposium, Jun. 3-7, 2012, Alcala, Spain, pp. 270-275. [cited by applicant]
Rawashdeh et al., “Collaborative Automated Driving: A Machine Learning-based Method to Enhance the Accuracy of Shared Information”, 21st International Conference on Intelligent Transportation Systems (ITSC), Nov. 4-7, 2… [cited by applicant]
Rockl et al., “V2V Communications in Automotive Multi-sensor Multi-target Tracking”, 68th IEEE Vehicular Technology Conference, Sep. 21-24, 2008, Calgary, Canada, 5 pages. [cited by applicant]
Sato et al., “Interpretable Adversarial Perturbation in Input Embedding Space for Text”, International Joint Conference on Artificial Intelligence, Jul. 13-19, 2018, Stockholm, Sweden, pp. 4323-4330. [cited by applicant]
Szegedy et al., “Intriguing properties of neural networks”, 2nd International Conference on Learning Representations, ICLR 2014, Apr. 14-16, 2014, Banff, AB, Canada, 9 pages. [cited by applicant]
Tramer et al., “Ensemble Adversarial Training: Attacks and Defenses”, Sixth International Conference on Learning Representations, Apr. 30-May 3, 2018, Vancouver, Canada, 20 pages. [cited by applicant]
Tu et al., “Physically Realizable Adversarial Examples for LiDAR Object Detection”, arXiv:2004.00543v2, Apr. 2, 2020, 10pages. [cited by applicant]
Wang et al., “V2VNet: Vehicle-to-Vehicle Communication for Joint Perception and Prediction”, arXiv:2008.07519v1, Aug. 17, 2020, 17 pages. [cited by applicant]
Wei et al., “Sparse Adversarial Perturbations for Videos”, Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19), Jan. 27-Feb. 1, 2019, Honolulu, Hawaii, pp. 8973-8980. [cited by applicant]
Wong et al., “Adding Security and Trust to Multiagent Systems”, Applied Artificial Intelligence, vol. 14, 2000, pp. 927-941. [cited by applicant]
Wu et al., “Adversarial Training for Relation Extraction”, Conference on Empirical Methods in Natural Language Processing 2017 (EMNLP), Sep. 7-11, 2017, Copenhagen, Denmark, 6 pages. [cited by applicant]
Xie et al., “Adversarial Examples for Semantic Segmentation and Object Detection”, International Conference on Computer Vision, Oct. 22-29, 2017, Venice, Italy, pp. 1369-1378. [cited by applicant]
Xie et al., “Improving Transferability of Adversarial Examples with Input Diversity”, Conference on Computer Vision and Pattern Recognition, Jun. 16-20, 2019, Long Beach, CA, pp. 2730-2739. [cited by applicant]
Zeng et al., “Wireless Communications and Control for Swarms of Cellular-Connected UAVs”, Asilomar Conference on Signals, Systems, and Computers, Oct. 28-31, 2018, Pacific Grove, CA, pp. 719-723. [cited by applicant]
Zhang et al., “Adversarial Attacks on Deep Learning Models in Natural Language Processing: A Survey”, arXiv:1901.06796v3, Apr. 11, 2019, 40 pages. [cited by applicant]
Zhu et al., “FreeLB: Enhanced Adversarial Training for Natural Language Understanding”, Eighth International Conference on Learning Representations, Apr. 26-May 1, 2020, Virtual, 14 pages. [cited by applicant]