IP Library › Granted Patent US 12,722,661
Granted Patent B2
US 12,722,661 · App. 18/345,501 · Granted Sep 1, 2026

Proactive risk mitigation with generalized virtual vehicles

Inventors: Qizhan Tam (San Jose, CA); Christopher Ostafew (Mountain View, CA); Manh Huynh (Sunnyvale, CA); Huiching Chen (San Jose, CA)
Assignee: Nissan North America, Inc.
B60W60/0027B60W2554/20B60W2554/4041
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,722,661
App. No.
18/345,501
Granted
Sep 1, 2026
Kind
B2
Abstract

Proactively mitigating risk to a vehicle traversing a vehicle transportation network includes identifying a location for a virtual vehicle. The virtual vehicle is added to a world object model maintained with respect to the vehicle. A trajectory is predicted for the virtual vehicle. The vehicle is autonomously controlled according to an adjusted trajectory that is based on the trajectory for the virtual vehicle. The adjusted trajectory includes at least one of a lateral constraint or a speed constraint. The location for the virtual vehicle is identified based on a lane in map data, a trajectory of a vehicle, and a perceptible area by sensors of the vehicle. The virtual vehicle is a hypothetical vehicle that is not observed by sensors of the vehicle.

Claims (58)

1 . A method, comprising:

identifying a location for a virtual vehicle,

wherein the location is identified based on a lane in map data, a trajectory of a vehicle, and a perceptible area by sensors of the vehicle,

wherein the virtual vehicle is a hypothetical vehicle that is not observed by sensors of the vehicle, and

wherein identifying the location for the virtual vehicle comprises:

searching backwards, starting at a location along the lane corresponding to a current location of the vehicle, along a predicted trajectory of the virtual vehicle until a location outside the perceptible area of the sensors is identified;

setting a location of the virtual vehicle, as a determined trajectory, along the predicted trajectory that is not visible to the sensors; and

adjusting the location of the virtual vehicle to an adjusted location, that is along the determined trajectory, at a time of instantiating the virtual vehicle to account for a processing delay associated with the sensors of the vehicle so that the virtual vehicle is within the perceptible area,

wherein the processing delay comprises perception delays, tracking delays, and prediction delays in converting sensor data into classified objects, and

wherein the adjusted location is positioned further along a predicted path of the virtual vehicle in accordance with the processing delay;

including the virtual vehicle at the adjusted location to a world object model maintained with respect to the vehicle;

predicting a trajectory for the virtual vehicle; and

autonomously controlling the vehicle according to an adjusted trajectory that is based on the determined trajectory for the virtual vehicle, wherein the adjusted trajectory includes at least one of a lateral constraint or a speed constraint.

2 . The method of claim 1 , wherein the lane is a same lane on which the vehicle is traveling, the virtual vehicle is a leading virtual vehicle that would be ahead of the vehicle on the trajectory of the vehicle, and wherein the searching backwards comprises searching along the trajectory of the vehicle until a location outside the perceptible area is identified.

3 . The method of claim 2 , wherein the leading virtual vehicle is instantiated as a static virtual vehicle.

4 . The method of claim 1 , wherein the lane is an oncoming lane, the virtual vehicle is an oncoming virtual vehicle, and wherein the predicted trajectory is a trajectory that would bring the oncoming virtual vehicle toward the vehicle within lane geometry obtained from the map data.

5 . The method of claim 4 , wherein after identifying the location outside the perceptible area, the oncoming virtual vehicle is located at an adjusted location that accounts for predicted movement of the oncoming virtual vehicle during the processing delay.

6 . The method of claim 5 , wherein the processing delay comprises delays specific to sensor fusion algorithms used to integrate data from multiple sensor types.

7 . The method of claim 1 , wherein the lane is a crossing lane, the virtual vehicle is a crossing virtual vehicle, and wherein the predicted trajectory would cause the crossing virtual vehicle to intersect the trajectory of the vehicle.

8 . The method of claim 7 , wherein after identifying the location outside the perceptible area, the crossing virtual vehicle is located at an adjusted location that accounts for predicted movement of the crossing virtual vehicle during the processing delay.

9 . A vehicle, comprising:

one or more memories; and

one or more processors, the one or more processors configured to execute instructions stored in the one or more memories to:

identify a location for a virtual vehicle,

wherein the location is identified based on a lane in map data, a trajectory of a vehicle, and a perceptible area by sensors of the vehicle,

wherein the virtual vehicle is a hypothetical vehicle that is not observed by sensors of the vehicle,

and wherein, to identify the location for the virtual vehicle, the one or more processors configured to execute instructions stored in the one or more memories to:

search backwards, starting at a location along the lane corresponding to a current location of the vehicle, along a predicted trajectory of the virtual vehicle until a location outside the perceptible area of the sensors is identified;

set a location of the virtual vehicle, as a determined trajectory, along the predicted trajectory that is not visible to the sensors; and

adjust the location of the virtual vehicle to an adjusted location, that is along the determined trajectory, at a time of instantiating the virtual vehicle to account for a processing delay associated with the sensors of the vehicle so that the virtual vehicle is within the perceptible area,

 wherein the processing delay comprises perception delays, tracking delays, and prediction delays in converting sensor data into classified objects, and

 wherein the adjusted location is positioned further along a predicted path of the virtual vehicle in accordance with the processing delay;

include the virtual vehicle at the adjusted location to a world object model maintained with respect to the vehicle;

predict a trajectory for the virtual vehicle; and

autonomously control the vehicle according to an adjusted trajectory that is based on the determined trajectory for the virtual vehicle, wherein the adjusted trajectory includes at least one of a lateral constraint or a speed constraint.

10 . The vehicle of claim 9 , wherein the lane is a same lane on which the vehicle is traveling, the virtual vehicle is a leading virtual vehicle that would be ahead of the vehicle on the trajectory of the vehicle, and wherein, to search backwards, the one or more processors configured to execute instructions stored in the one or more memories to search along the trajectory of the vehicle until a location outside the perceptible area is identified.

11 . The vehicle of claim 10 , wherein the leading virtual vehicle is instantiated as a static virtual vehicle.

12 . The vehicle of claim 9 , wherein the lane is an oncoming lane, the virtual vehicle is an oncoming virtual vehicle, and wherein the predicted trajectory is a trajectory that would bring the oncoming virtual vehicle toward the vehicle within lane geometry obtained from the map data.

13 . The vehicle of claim 12 , wherein after identifying the location outside the perceptible area, the oncoming virtual vehicle is located at an adjusted location that accounts for predicted movement of the oncoming virtual vehicle, via the determined trajectory, during the processing delay.

14 . The vehicle of claim 13 , wherein the processing delay comprises delays specific to sensor fusion algorithms used to integrate data from multiple sensor types.

15 . The vehicle of claim 9 , wherein the lane is a crossing lane, the virtual vehicle is a crossing virtual vehicle, and wherein the predicted trajectory would cause the crossing virtual vehicle to intersect the trajectory of the vehicle.

16 . The vehicle of claim 15 , wherein after identifying the location outside the perceptible area, the crossing virtual vehicle is located at an adjusted location that accounts for predicted movement of the crossing virtual vehicle during the processing delay.

17 . One or more non-transitory computer-readable media including instructions that, when executed by one or more processors, perform operations comprising:

identifying a location for a virtual vehicle,

wherein the location is identified based on a lane in map data, a trajectory of a vehicle, and a perceptible area by sensors of the vehicle,

wherein the virtual vehicle is a hypothetical vehicle that is not observed by sensors of the vehicle, and

wherein identifying the location for the virtual vehicle comprises:

searching backwards, starting at a location along the lane corresponding to a current location of the vehicle, along a predicted trajectory of the virtual vehicle until a location outside the perceptible area of the sensors is identified;

setting a location of the virtual vehicle, as a determined trajectory, along the predicted trajectory that is not visible to the sensors; and

adjusting the location of the virtual vehicle to an adjusted location, that is along the determined trajectory, at a time of instantiating the virtual vehicle to account for a processing delay associated with the sensors of the vehicle so that the virtual vehicle is within the perceptible area,

wherein the processing delay comprises perception delays, tracking delays, and prediction delays in converting sensor data into classified objects, and

wherein the adjusted location is positioned further along a predicted path of the virtual vehicle in accordance with the processing delay;

including the virtual vehicle at the adjusted location to a world object model maintained with respect to the vehicle;

predicting a trajectory for the virtual vehicle; and

autonomously controlling the vehicle according to an adjusted trajectory that is based on the determined trajectory for the virtual vehicle, wherein the adjusted trajectory includes at least one of a lateral constraint or a speed constraint.

18 . The one or more non-transitory computer-readable media of claim 17 , wherein the lane is a same lane on which the vehicle is traveling, the virtual vehicle is a leading virtual vehicle that would be ahead of the vehicle on the trajectory of the vehicle, and wherein the searching backwards comprises searching along the trajectory of the vehicle until a location outside the perceptible area is identified.

19 . The one or more non-transitory computer-readable media of claim 18 , wherein the leading virtual vehicle is instantiated as a static virtual vehicle.

20 . The one or more non-transitory computer-readable media of claim 17 , wherein the lane is an oncoming lane, the virtual vehicle is an oncoming virtual vehicle, and wherein the predicted trajectory is a trajectory that would bring the oncoming virtual vehicle toward the vehicle within lane geometry obtained from the map data.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 5, 2023
From: TAM, QIZHAN; OSTAFEW, CHRISTOPHER; HUYNH, MANH; CHEN, HUICHING
To: NISSAN NORTH AMERICA, INC.
Reel/Frame 064153/0436 →
Continuity (1)
Related Publication 20250002049A1 · Jan 2, 2025
References Cited (117)
US 8559673B2 · Fairfield et al. · 2013 [cited by applicant]
US 8761991B1 · Ferguson et al. · 2014 [cited by applicant]
US 9779314B1 · Wendel et al. · 2017 [cited by applicant]
US 9990548B2 · Wellington et al. · 2018 [cited by applicant]
US 10210408B2 · Ferguson et al. · 2019 [cited by applicant]
US 10269257B1 · Gohl et al. · 2019 [cited by applicant]
US 10303178B1 · Gutmann · 2019 [cited by applicant]
US 10467487B1 · Wang et al. · 2019 [cited by applicant]
US 10525903B2 · Laur et al. · 2020 [cited by applicant]
US 10698407B2 · Ostafew · 2020 [cited by applicant]
US 10936902B1 · Bagwell et al. · 2021 [cited by applicant]
US 11227409B1 · Wu et al. · 2022 [cited by applicant]
US 11345335B1 · Luders · 2022 [cited by examiner]
US 11433922B1 · Van Heukelom et al. · 2022 [cited by applicant]
US 11720116B1 · Gutmann · 2023 [cited by applicant]
US 12017632B1 · Valverde et al. · 2024 [cited by applicant]
US 12071127B2 · Tam et al. · 2024 [cited by applicant]
US 12179795B2 · Henke · 2024 [cited by examiner]
US 12194991B2 · Kobashi et al. · 2025 [cited by applicant]
US 20040119610A1 · Maemura et al. · 2004 [cited by applicant]
US 20090254235A1 · Kuroda · 2009 [cited by applicant]
US 20100026555A1 · Whittaker et al. · 2010 [cited by applicant]
US 20100235033A1 · Yamamoto et al. · 2010 [cited by applicant]
US 20130325344A1 · Yester · 2013 [cited by applicant]
US 20140379179A1 · Goossen et al. · 2014 [cited by applicant]
US 20170124781A1 · Douillard et al. · 2017 [cited by applicant]
US 20170329332A1 · Pilarski et al. · 2017 [cited by applicant]
US 20170336795A1 · Wei et al. · 2017 [cited by applicant]
US 20180012088A1 · Matsuo et al. · 2018 [cited by applicant]
US 20180151075A1 · Claesson · 2018 [cited by applicant]
US 20180154723A1 · Anderson et al. · 2018 [cited by applicant]
US 20180178784A1 · Ohta · 2018 [cited by applicant]
US 20180194354A1 · Takeda · 2018 [cited by applicant]
US 20190009784A1 · Takeda · 2019 [cited by examiner]
US 20190025843A1 · Wilkinson et al. · 2019 [cited by applicant]
US 20190047584A1 · Donnelly · 2019 [cited by applicant]
US 20190065871A1 · Pogorelik · 2019 [cited by applicant]
US 20190129831A1 · Goldberg · 2019 [cited by examiner]
US 20190243371A1 · Nister · 2019 [cited by examiner]
US 20190332875A1 · Vallespi-Gonzalez et al. · 2019 [cited by applicant]
US 20190355257A1 · Caldwell et al. · 2019 [cited by applicant]
US 20190369616A1 · Ostafew · 2019 [cited by applicant]
US 20190383631A1 · Bigio et al. · 2019 [cited by applicant]
US 20190384302A1 · Silva et al. · 2019 [cited by applicant]
US 20190391587A1 · Uvarov et al. · 2019 [cited by applicant]
US 20200026288A1 · Herz et al. · 2020 [cited by applicant]
US 20200081450A1 · Creusot et al. · 2020 [cited by applicant]
US 20200086855A1 · Packer et al. · 2020 [cited by applicant]
US 20200133280A1 · Seccamonte et al. · 2020 [cited by applicant]
US 20200148223A1 · Fukatani et al. · 2020 [cited by applicant]
US 20200191972A1 · Zhao et al. · 2020 [cited by applicant]
US 20200211394A1 · King et al. · 2020 [cited by applicant]
US 20200216091A1 · Vladimerou · 2020 [cited by applicant]
US 20200269877A1 · Mortazavi et al. · 2020 [cited by applicant]
US 20200278681A1 · Gier et al. · 2020 [cited by applicant]
US 20200284607A1 · Mangal et al. · 2020 [cited by applicant]
US 20200292322A1 · Ozog et al. · 2020 [cited by applicant]
US 20200310467A1 · Birkedahl et al. · 2020 [cited by applicant]
US 20200341474A1 · Zuo · 2020 [cited by applicant]
US 20200364892A1 · Ko et al. · 2020 [cited by applicant]
US 20200410062A1 · O'Malley · 2020 [cited by examiner]
US 20210004363A1 · Bailly et al. · 2021 [cited by applicant]
US 20210031760A1 · Ostafew et al. · 2021 [cited by applicant]
US 20210035314A1 · Shu et al. · 2021 [cited by applicant]
US 20210055741A1 · Kawanai et al. · 2021 [cited by applicant]
US 20210061269A1 · Petroff · 2021 [cited by examiner]
US 20210086795A1 · Zhao et al. · 2021 [cited by applicant]
US 20210125431A1 · Mcgill et al. · 2021 [cited by applicant]
US 20210166044A1 · Ferguson et al. · 2021 [cited by applicant]
US 20210191419A1 · Xu et al. · 2021 [cited by applicant]
US 20210211568A1 · Zhou et al. · 2021 [cited by applicant]
US 20210261152A1 · Meijburg et al. · 2021 [cited by applicant]
US 20210390349A1 · Hotson et al. · 2021 [cited by applicant]
US 20210404841A1 · Das et al. · 2021 [cited by applicant]
US 20220076037A1 · Hochman et al. · 2022 [cited by applicant]
US 20220097736A1 · Lin · 2022 [cited by examiner]
US 20220379917A1 · Henke · 2022 [cited by examiner]
US 20230012853A1 · Tam et al. · 2023 [cited by applicant]
US 20230121388A1 · Khan · 2023 [cited by examiner]
US 20230124314A1 · Foster et al. · 2023 [cited by applicant]
US 20230143805A1 · Baba · 2023 [cited by examiner]
US 20230145472A1 · Ehmann · 2023 [cited by applicant]
US 20230159026A1 · Pan · 2023 [cited by examiner]
US 20230159056A1 · Cheng et al. · 2023 [cited by applicant]
US 20230227031A1 · Kobashi · 2023 [cited by examiner]
US 20240109533A1 · Tam et al. · 2024 [cited by applicant]
US 20240149912A1 · Tam et al. · 2024 [cited by applicant]
US 20240208531A1 · Schier et al. · 2024 [cited by applicant]
US 20240300482A1 · Aizawa · 2024 [cited by applicant]
US 20250002049A1 · Tam et al. · 2025 [cited by applicant]
CN 113682300A · 2021 [cited by applicant]
CN 114475645A · 2022 [cited by applicant]
CN 115097857A · 2022 [cited by applicant]
EP 3611069A1 · 2020 [cited by applicant]
EP 3650294A1 · 2020 [cited by applicant]
JP 2009217692A · 2009 [cited by applicant]
JP 2011198247A · 2011 [cited by applicant]
JP 2012188021A · 2012 [cited by applicant]
JP 2016146113A · 2016 [cited by applicant]
JP 2017114405A · 2017 [cited by applicant]
JP 2018049496A · 2018 [cited by applicant]
JP 2019040427A · 2019 [cited by applicant]
WO 2018225575A1 · 2018 [cited by applicant]
WO 2019098216A1 · 2019 [cited by applicant]
WO 2022016941A1 · 2022 [cited by applicant]
Alsterda et al., Contingency Model Predictive Control for Autmotaed Vehicles; Cornell University; arXiv:1903.08818v1 [cs.SY];Mar. 21, 2019; 6 pages. [cited by applicant]
Kim et al., An Efficient Color Space for Deep-Learning Based Traffic Light Recognition; JOurnal of Advanced Transportation; vol. 2018; Artice ID 2365414; 13 pages. [cited by applicant]
Krylov et al., Object Geolocation Using MRF Based Multi-Sensor Fusion; ADAPT Centre, School of Computer Science and Statistics, Trinity College Dublin, Dublin, Ireland; Oct. 2018; 5 pages. [cited by applicant]
Levinson et al., Robust Real-Time Tracking Combining 3D Shape, Color, and Motion; The International Journal of Robotics Research; 2015; 28 pages. [cited by applicant]
Charette et al., Traffic Light Recognition Using Image Processing Compared to Learning Processes; 2009 IEEE/RSH International Conference on Intelligent Robots and Systems; Oct. 2009; 6 pages. [cited by applicant]
Trehard et al., Tracking Both Pose and Status of a Traffic Light Via an Interacting Multiple Model Filter; Jul. 2014; 7 pages. [cited by applicant]
Thrun et al., The GraphSLAM Algorithm with Applications to Large-Scale Mapping of Urban Structures; The International Journal of Robotics Research, vol. 25, No. 5-6, May-Jun. 2006; pp. 403-429. [cited by applicant]
Fairfield et al., Traffic Light Mapping and Detection; 2011 IEEE International Conference on Robotics and Automoation; May 2011; 6 pages. [cited by applicant]
Levinson et al., Traffic Light Mapping, Localization, and State Detetion for Autonomous Vehicles; 2011 IEEE International Conference on Robotics and Automation, Shanghai, 2011, pp. 5784-5791. [cited by applicant]
Kim et al., Traffic Light Recognition Based on Binary Semantic Segmentation Network; Sensors 2019, 19, 1700; 15 pages. [cited by applicant]
Basich et al., Learning to Optimize Autonomy in Competence-Aware Systems; Proc. of the 19th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2020); May 2020; 9 pages. [cited by applicant]
Wray et al., Online Decision-Making for Scalable Autonomous Systems; Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence; AI and autonomy track, 2017. pp. 4768-4774. [cited by applicant]