IP Library Granted Patent US 12,240,494
Granted Patent B2
US 12,240,494 · App. 18/086,832 · Granted Mar 4, 2025

Method and system for remote assistance of an autonomous agent

Inventors: Melinda Kothbauer (Ann Arbor, MI); Jeff Sterniak (Ann Arbor, MI); Edwin B. Olson (Ann Arbor, MI)
Assignee: May Mobility, Inc.
B60W60/0011B60W40/02G05D1/0011G05D1/223B60W2554/20B60W2554/4041B60W2556/45
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,240,494
App. No.
18/086,832
Granted
Mar 4, 2025
Kind
B2
Abstract

A system 100 for remote assistance of an autonomous agent can include and/or interface with any or all of: a sensor suite 110 , a computing system 120 , a communication interface 130 , and/or any other suitable components. The system can further optionally include a set of infrastructure devices 140 , a teleoperator platform 150 , and/or any other suitable components. The system 100 functions to enable information to be exchanged between an autonomous agent and a tele-assist. Additionally or alternatively, the system 100 can function to operate the autonomous agent (e.g., based on remote inputs received from a teleoperator, indirectly, etc.) and/or can perform any other suitable functions.

Claims (39)

1. A method for consideration of teleoperator input in the operation of an autonomous vehicle, the method comprising:

with a computing subsystem onboard the autonomous vehicle:

detecting satisfaction of a trigger condition;

receiving the teleoperator input, wherein the teleoperator input comprises a set of waypoints associated with the trigger condition;

based on the set of waypoints, determining a plurality of predefined autonomously electable policies, each comprising a set of rules for decision-making;

evaluating a set of aggregated decision options, the set of aggregated decision options comprising:

the set of waypoints;

the set of predefined autonomously electable policies;

selecting one of the set of aggregated decision options based on the evaluation, wherein the evaluation comprises simulating each of the set of predefined autonomously electable policies; and

with a control subsystem in communication with the computing subsystem, operating the autonomous vehicle in accordance with the selected decision option.

2. The method of claim 1 , wherein the trigger condition is associated with the autonomous vehicle's environment.

3. The method of claim 2 , wherein the trigger condition comprises exceeding a predetermined probability threshold.

4. The method of claim 3 , wherein the probability threshold is associated with a probability of the autonomous vehicle being stopped by an obstacle in the autonomous vehicle's environment.

5. The method of claim 1 , further comprising providing a set of input options to a teleoperator interface in response to detecting satisfaction of the trigger condition, wherein the teleoperator input is received in response to providing the set of input options.

6. The method of claim 5 , wherein the set of input options comprises a waypoint provision interface, wherein the set of waypoints is received at the waypoint provision interface.

7. The method of claim 6 , wherein the set of inputs options further comprises a set of teleoperator-electable policy options, wherein in an event that the teleoperator input comprises a selected teleoperator-electable policy option from the set of teleoperator-electable policy options, the set of aggregated decision options further comprises the selected teleoperator-electable policy option.

8. The method of claim 1 , wherein evaluating the set of aggregated decision options comprises performing a set of simulations.

9. The method of claim 8 , wherein the selected decision option is determined based on a set of parameter values produced from the set of simulations.

10. The method of claim 9 , wherein the set of parameters values comprises a score associated with the set of waypoints and a set of scores associated with each of the set of predefined autonomously electable policies.

11. A system for consideration of teleoperator input in the operation of an autonomous vehicle while navigating the autonomous vehicle along a target path, the system comprising:

a computing subsystem onboard the autonomous vehicle, wherein the computing subsystem is configured to:

detect satisfaction of a trigger condition;

receive the teleoperator input, wherein the teleoperator input comprises a set of waypoints associated with the trigger condition;

update the target path based on the set of waypoints;

evaluate a set of aggregated decision options, the set of aggregated decision options comprising:

the set of waypoints;

a set of autonomously electable policies, each comprising a set of rules for decision-making;

select one of the set of aggregated decision options based on the evaluation and the updated target path; and

a control subsystem in communication with the computing subsystem, wherein the control subsystem is configured to operate the autonomous vehicle in accordance with the selected decision option.

12. The system of claim 11 , wherein the trigger condition is associated with the autonomous vehicle's environment.

13. The system of claim 12 , wherein the trigger condition comprises exceeding a predetermined probability threshold.

14. The system of claim 13 , wherein the probability threshold is associated with a probability of the autonomous vehicle being stopped by an obstacle in the autonomous vehicle's environment.

15. The system of claim 11 , wherein the computing subsystem is further configured to provide a set of input options to a teleoperator interface in response to detecting satisfaction of the trigger condition, wherein the teleoperator input is received in response to providing the set of input options.

16. The system of claim 15 , wherein the set of input options comprises a waypoint provision interface, wherein the set of waypoints is received at the waypoint provision interface.

17. The system of claim 16 , wherein the set of inputs options further comprises a set of teleoperator-electable policy options, wherein in an event that the teleoperator input comprises a selected teleoperator-electable policy option from the set of teleoperator-electable policy options, the set of aggregated decision options further comprises the selected teleoperator-electable policy option.

18. The system of claim 11 , wherein the computing subsystem is further configured to, in evaluating the set of aggregated decision options, perform a plurality of simulations for each aggregated decision option of the set.

19. The system of claim 18 , wherein the selected decision option is determined based on a set of parameter values produced from the set of simulations.

20. The system of claim 19 , wherein the set of parameters values comprises a score associated with the set of waypoints and a set of scores associated with each of the set of autonomously electable policies.

21. The method of claim 1 , wherein the set of aggregated decision options comprises policies determined before satisfaction of the trigger condition and policies determined after satisfaction of the trigger condition.

Assignments (2)
SECURITY INTEREST Recorded May 15, 2026
From: MAY MOBILITY, INC.
To: ACP REDSTONE CREDIT, LLC
Reel/Frame 075610/0696 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 22, 2022
From: KOTHBAUER, MELINDA; STERNIAK, JEFF; OLSON, EDWIN B.
To: MAY MOBILITY, INC.
Reel/Frame 062181/0157 →
Continuity (3)
Continuation 17826655 · May 27, 2022
Provisional Application 63195774 · Jun 2, 2021
Related Publication 20230127379A1 · Apr 27, 2023
References Cited (234)
US 5544282A · Chen et al. · 1996 [cited by applicant]
US 6199013B1 · Oshea · 2001 [cited by applicant]
US 9129519B2 · Aoude et al. · 2015 [cited by applicant]
US 9274525B1 · Ferguson et al. · 2016 [cited by applicant]
US 9368026B1 · Herbach et al. · 2016 [cited by applicant]
US 9495874B1 · Zhu et al. · 2016 [cited by applicant]
US 9507346B1 · Levinson · 2016 [cited by examiner]
US 9618938B2 · Olson et al. · 2017 [cited by applicant]
US 9646428B1 · Konrardy · 2017 [cited by examiner]
US 9720412B1 · Zhu et al. · 2017 [cited by applicant]
US 9811760B2 · Richardson et al. · 2017 [cited by applicant]
US 9914452B1 · Ferguson · 2018 [cited by examiner]
US 10012981B2 · Gariepy · 2018 [cited by examiner]
US 10062294B2 · Kunzi et al. · 2018 [cited by applicant]
US 10156848B1 · Konrardy et al. · 2018 [cited by applicant]
US 10235882B1 · Aoude et al. · 2019 [cited by applicant]
US 10248120B1 · Siegel et al. · 2019 [cited by applicant]
US 10386856B2 · Wood et al. · 2019 [cited by applicant]
US 10467891B1 · Bart et al. · 2019 [cited by applicant]
US 10518770B2 · Kroop · 2019 [cited by examiner]
US 10518783B2 · Tanimichi · 2019 [cited by examiner]
US 10540892B1 · Fields et al. · 2020 [cited by applicant]
US 10558224B1 · Lin · 2020 [cited by examiner]
US 10564641B2 · Vozar et al. · 2020 [cited by applicant]
US 10564643B2 · Lui · 2020 [cited by examiner]
US 10571916B2 · Tschanz · 2020 [cited by examiner]
US 10586254B2 · Singhal · 2020 [cited by applicant]
US 10599155B1 · Konrardy · 2020 [cited by examiner]
US 10614709B2 · Vozar et al. · 2020 [cited by applicant]
US 10642276B2 · Huai · 2020 [cited by applicant]
US 10654476B2 · Wray · 2020 [cited by examiner]
US 10671076B1 · Kobilarov et al. · 2020 [cited by applicant]
US 10717445B2 · Michalakis · 2020 [cited by examiner]
US 10761542B1 · Fairfield · 2020 [cited by examiner]
US 10796581B2 · Herbach et al. · 2020 [cited by applicant]
US 10860019B2 · Censi · 2020 [cited by examiner]
US 10969470B2 · Voorheis et al. · 2021 [cited by applicant]
US 11003916B2 · Hummelshøj · 2021 [cited by applicant]
US 11086318B1 · Davis · 2021 [cited by examiner]
US 11087200B2 · Olson · 2021 [cited by examiner]
US 11215982B2 · Urano · 2022 [cited by examiner]
US 11242054B2 · Isele · 2022 [cited by examiner]
US 11260855B2 · Zhang · 2022 [cited by examiner]
US 11269332B2 · Vozar et al. · 2022 [cited by applicant]
US 11281213B2 · Cross et al. · 2022 [cited by applicant]
US 11300957B2 · Wray · 2022 [cited by examiner]
US 11352023B2 · Fairley et al. · 2022 [cited by applicant]
US 11396302B2 · Ye · 2022 [cited by examiner]
US 11472436B1 · Patel et al. · 2022 [cited by applicant]
US 11472444B2 · Goeddel et al. · 2022 [cited by applicant]
US 11565717B2 · Kothbauer et al. · 2023 [cited by applicant]
US 11748085B2 · Fields et al. · 2023 [cited by applicant]
US 11780466B1 · Shah · 2023 [cited by examiner]
US 20020062207A1 · Faghri · 2002 [cited by applicant]
US 20040100563A1 · Sablak et al. · 2004 [cited by applicant]
US 20050004723A1 · Duggan et al. · 2005 [cited by applicant]
US 20060184275A1 · Hosokawa et al. · 2006 [cited by applicant]
US 20060200333A1 · Dalal et al. · 2006 [cited by applicant]
US 20070193798A1 · Allard et al. · 2007 [cited by applicant]
US 20070276600A1 · King et al. · 2007 [cited by applicant]
US 20080033684A1 · Vian et al. · 2008 [cited by applicant]
US 20100114554A1 · Misra · 2010 [cited by applicant]
US 20100204867A1 · Longstaff · 2010 [cited by applicant]
US 20110142283A1 · Huang et al. · 2011 [cited by applicant]
US 20120089275A1 · Yao-Chang et al. · 2012 [cited by applicant]
US 20130054106A1 · Schmudderich · 2013 [cited by examiner]
US 20130141576A1 · Lord et al. · 2013 [cited by applicant]
US 20130253816A1 · Caminiti et al. · 2013 [cited by applicant]
US 20140195138A1 · Stelzig et al. · 2014 [cited by applicant]
US 20140244198A1 · Mayer · 2014 [cited by applicant]
US 20140309815A1 · Ricci et al. · 2014 [cited by applicant]
US 20150105961A1 · Callow · 2015 [cited by applicant]
US 20150284010A1 · Beardsley et al. · 2015 [cited by applicant]
US 20150302756A1 · Guehring et al. · 2015 [cited by applicant]
US 20150316928A1 · Guehring et al. · 2015 [cited by applicant]
US 20150321337A1 · Stephens · 2015 [cited by applicant]
US 20150344030A1 · Damerow · 2015 [cited by examiner]
US 20160005333A1 · Naouri · 2016 [cited by applicant]
US 20160209840A1 · Kim · 2016 [cited by applicant]
US 20160314224A1 · Wei et al. · 2016 [cited by applicant]
US 20160334230A1 · Ross · 2016 [cited by examiner]
US 20170031361A1 · Olson et al. · 2017 [cited by applicant]
US 20170032671A1 · Toyama et al. · 2017 [cited by applicant]
US 20170072853A1 · Matsuoka et al. · 2017 [cited by applicant]
US 20170123419A1 · Levinson et al. · 2017 [cited by applicant]
US 20170155696A1 · Dong et al. · 2017 [cited by applicant]
US 20170192426A1 · Rust · 2017 [cited by applicant]
US 20170199523A1 · Barton-Sweeney et al. · 2017 [cited by applicant]
US 20170268896A1 · Bai et al. · 2017 [cited by applicant]
US 20170289341A1 · Rodriguez et al. · 2017 [cited by applicant]
US 20170291560A1 · Schroeder et al. · 2017 [cited by applicant]
US 20170301111A1 · Zhao et al. · 2017 [cited by applicant]
US 20170320500A1 · Yoo · 2017 [cited by examiner]
US 20170323568A1 · Inoue et al. · 2017 [cited by applicant]
US 20170356748A1 · Iagnemma · 2017 [cited by applicant]
US 20180011485A1 · Ferren · 2018 [cited by applicant]
US 20180046182A1 · Joyce et al. · 2018 [cited by applicant]
US 20180047291A1 · Konishi et al. · 2018 [cited by applicant]
US 20180053102A1 · Martinson et al. · 2018 [cited by applicant]
US 20180065625A1 · Tijerina et al. · 2018 [cited by applicant]
US 20180070056A1 · Deangelis et al. · 2018 [cited by applicant]
US 20180082596A1 · Whitlow · 2018 [cited by applicant]
US 20180089563A1 · Redding · 2018 [cited by examiner]
US 20180100743A1 · Diaz · 2018 [cited by examiner]
US 20180136651A1 · Levinson · 2018 [cited by examiner]
US 20180183873A1 · Wang et al. · 2018 [cited by applicant]
US 20180184352A1 · Lopes et al. · 2018 [cited by applicant]
US 20180196427A1 · Majumdar et al. · 2018 [cited by applicant]
US 20180220283A1 · Condeixa et al. · 2018 [cited by applicant]
US 20180224851A1 · Park · 2018 [cited by applicant]
US 20180251126A1 · Linscott et al. · 2018 [cited by applicant]
US 20180267550A1 · Kopetz et al. · 2018 [cited by applicant]
US 20180268281A1 · Olson et al. · 2018 [cited by applicant]
US 20180293537A1 · Kwok · 2018 [cited by applicant]
US 20180299898A1 · Luo · 2018 [cited by examiner]
US 20180330481A1 · Watanabe et al. · 2018 [cited by applicant]
US 20180365908A1 · Liu et al. · 2018 [cited by applicant]
US 20180367997A1 · Shaw et al. · 2018 [cited by applicant]
US 20190027034A1 · Xu et al. · 2019 [cited by applicant]
US 20190039545A1 · Kumar · 2019 [cited by examiner]
US 20190042859A1 · Schubert et al. · 2019 [cited by applicant]
US 20190066399A1 · Jiang et al. · 2019 [cited by applicant]
US 20190096244A1 · Guruva Reddiar · 2019 [cited by examiner]
US 20190101914A1 · Coleman et al. · 2019 [cited by applicant]
US 20190101919A1 · Kobilarov et al. · 2019 [cited by applicant]
US 20190106117A1 · Goldberg · 2019 [cited by applicant]
US 20190113919A1 · Englard et al. · 2019 [cited by applicant]
US 20190113929A1 · Mukadam · 2019 [cited by examiner]
US 20190130878A1 · Bradley · 2019 [cited by applicant]
US 20190138524A1 · Singh et al. · 2019 [cited by applicant]
US 20190147610A1 · Frossard et al. · 2019 [cited by applicant]
US 20190163176A1 · Wang et al. · 2019 [cited by applicant]
US 20190180529A1 · Smith · 2019 [cited by applicant]
US 20190188493A1 · Tiziani · 2019 [cited by examiner]
US 20190196465A1 · Hummelshøj · 2019 [cited by applicant]
US 20190220011A1 · Della Penna · 2019 [cited by applicant]
US 20190227553A1 · Kentley-Klay et al. · 2019 [cited by applicant]
US 20190235516A1 · Zhang et al. · 2019 [cited by applicant]
US 20190236950A1 · Li et al. · 2019 [cited by applicant]
US 20190258246A1 · Liu · 2019 [cited by examiner]
US 20190258251A1 · Ditty et al. · 2019 [cited by applicant]
US 20190265059A1 · Warnick et al. · 2019 [cited by applicant]
US 20190271554A1 · Colgate et al. · 2019 [cited by applicant]
US 20190271981A1 · Oba · 2019 [cited by applicant]
US 20190291726A1 · Shalev-Shwartz et al. · 2019 [cited by applicant]
US 20190329771A1 · Wray · 2019 [cited by examiner]
US 20190331758A1 · Malkes et al. · 2019 [cited by applicant]
US 20190332106A1 · Belloni Mourao · 2019 [cited by examiner]
US 20190337509A1 · Shalev-Shwartz et al. · 2019 [cited by applicant]
US 20200004241A1 · Levinson · 2020 [cited by examiner]
US 20200017114A1 · Santoni et al. · 2020 [cited by applicant]
US 20200020226A1 · Stenneth et al. · 2020 [cited by applicant]
US 20200026286A1 · Vozar et al. · 2020 [cited by applicant]
US 20200057441A1 · Wang et al. · 2020 [cited by applicant]
US 20200086837A1 · Le Cornec · 2020 [cited by applicant]
US 20200094850A1 · Chi et al. · 2020 [cited by applicant]
US 20200097003A1 · Wray · 2020 [cited by examiner]
US 20200098269A1 · Wray et al. · 2020 [cited by applicant]
US 20200110411A1 · Vozar et al. · 2020 [cited by applicant]
US 20200122830A1 · Anderson et al. · 2020 [cited by applicant]
US 20200124447A1 · Schwindt et al. · 2020 [cited by applicant]
US 20200150661A1 · Vozar et al. · 2020 [cited by applicant]
US 20200159227A1 · Cohen et al. · 2020 [cited by applicant]
US 20200189731A1 · Mistry et al. · 2020 [cited by applicant]
US 20200209846A1 · Chen · 2020 [cited by applicant]
US 20200209853A1 · Leach et al. · 2020 [cited by applicant]
US 20200209864A1 · Chen · 2020 [cited by examiner]
US 20200233060A1 · Lull et al. · 2020 [cited by applicant]
US 20200255027A1 · Kulkarni et al. · 2020 [cited by applicant]
US 20200269843A1 · Wissing · 2020 [cited by examiner]
US 20200290619A1 · Mehdi · 2020 [cited by examiner]
US 20200293038A1 · Laurent et al. · 2020 [cited by applicant]
US 20200293041A1 · Palanisamy · 2020 [cited by applicant]
US 20200294401A1 · Kerecsen · 2020 [cited by applicant]
US 20200309543A1 · Voznesensky · 2020 [cited by applicant]
US 20200310417A1 · Pedersen et al. · 2020 [cited by applicant]
US 20200339151A1 · Batts et al. · 2020 [cited by applicant]
US 20200346666A1 · Wray et al. · 2020 [cited by applicant]
US 20200355820A1 · Zeng et al. · 2020 [cited by applicant]
US 20200369294A1 · Jeon et al. · 2020 [cited by applicant]
US 20200379457A1 · Ostafew et al. · 2020 [cited by applicant]
US 20200400781A1 · Voorheis et al. · 2020 [cited by applicant]
US 20200401135A1 · Chen et al. · 2020 [cited by applicant]
US 20210018916A1 · Thakur et al. · 2021 [cited by applicant]
US 20210042535A1 · Abbott et al. · 2021 [cited by applicant]
US 20210116907A1 · Altman · 2021 [cited by examiner]
US 20210132606A1 · Basich et al. · 2021 [cited by applicant]
US 20210163021A1 · Frazzoli et al. · 2021 [cited by applicant]
US 20210181758A1 · Das et al. · 2021 [cited by applicant]
US 20210197864A1 · Oltmann et al. · 2021 [cited by applicant]
US 20210208244A1 · Voorheis et al. · 2021 [cited by applicant]
US 20210229697A1 · Lee · 2021 [cited by examiner]
US 20210237759A1 · Wray et al. · 2021 [cited by applicant]
US 20210245785A1 · Suzuki · 2021 [cited by examiner]
US 20210269063A1 · Lee et al. · 2021 [cited by applicant]
US 20210286651A1 · Ho · 2021 [cited by examiner]
US 20210323573A1 · Gogna · 2021 [cited by examiner]
US 20210339741A1 · Rezvan Behbahani et al. · 2021 [cited by applicant]
US 20210365701A1 · Eshet · 2021 [cited by examiner]
US 20220066440A1 · Sucan et al. · 2022 [cited by applicant]
US 20220073097A1 · Sucan et al. · 2022 [cited by applicant]
US 20220076032A1 · Jain et al. · 2022 [cited by applicant]
US 20220081005A1 · Brown · 2022 [cited by examiner]
US 20220126878A1 · Moustafa · 2022 [cited by examiner]
US 20220185325A1 · Chen et al. · 2022 [cited by applicant]
US 20220194419A1 · Houshmand · 2022 [cited by examiner]
US 20220204010A1 · Zhu · 2022 [cited by examiner]
US 20220227389A1 · Tam · 2022 [cited by applicant]
US 20220348222A1 · Zhao et al. · 2022 [cited by applicant]
US 20220348223A1 · Zhao et al. · 2022 [cited by applicant]
US 20230060435A1 · Funke · 2023 [cited by examiner]
US 20230067887A1 · Garg et al. · 2023 [cited by applicant]
US 20230166764A1 · Johnson · 2023 [cited by examiner]
US 20230174103A1 · Patel · 2023 [cited by examiner]
US 20230176573A1 · Kumavat · 2023 [cited by examiner]
US 20230185300A1 · Wray · 2023 [cited by examiner]
US 20230192134A1 · Winter et al. · 2023 [cited by applicant]
US 20230194286A1 · Winter · 2023 [cited by examiner]
US 20230256991A1 · Johnson · 2023 [cited by examiner]
JP 2011100492A · 2011 [cited by applicant]
JP 2015083417A · 2015 [cited by applicant]
JP 2016091039A · 2016 [cited by applicant]
JP 2016184276A · 2016 [cited by applicant]
JP 2019185279A · 2019 [cited by applicant]
WO 2015160900A1 · 2015 [cited by applicant]
WO 2020264144A1 · 2020 [cited by applicant]
Cunningham, A. , et al., “MPDM: Multipolicy Decision-Making in Dynamic, Uncertain Environments for Autonomous Driving”, Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) (2015). [cited by applicant]
Mehta, D., et al., “Autonomous Navigation in Dynamic Social Environments Using Multi-Policy Decision Making”, Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (2016). [cited by applicant]
Mehta, D., et al., “Fast Discovery of Influential Outcomes For Risk-Aware MPDM”, Proceedings go the IEEE International Conference on Robotics and Automation (ICRA) (2017). [cited by applicant]
Neumeier Stefan, et al., “Towards a Driver Support System for Teleoperated Driving”, 2019 IEEE Intelligent Transportation Systems Conference (ITSC) (Year: 2019). [cited by applicant]
Paden, B., et al., “A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehicles”, IEEE Transactions on Intelligent Vehicles, vol. 1, Ids. 1, (Jun. 13, 2016). [cited by applicant]
Straub, J., “Comparing The Effect Of Pruning On A Best Path and a Naive-approach Blackboard Solver”, International Journal of Automation and Computing (Oct. 2015). [cited by applicant]
Wuthishuwong, Chairit, et al., “Vehicle to Infrastructure based Safe Trajectory Planning for Autonomous Intersection Management”, 2013 13th International Conference on ITS Telecommunications (ITST) (Year: 2013). [cited by applicant]
Crossman, Jacob, et al., “Method and System for Assessing and Mitigating Risks Encounterable by an Autonomous Vehicle”, U.S. Appl. No. 18/538,312, filed Dec. 13, 2023. [cited by applicant]