IP Library Granted Patent US 12,321,170
Granted Patent B2
US 12,321,170 · App. 18/396,135 · Granted Jun 3, 2025

Robotic process automation for achieving an optimized margin of vehicle operational safety

Inventor: Charles Howard Cella (Pembroke, MA)
Assignee: STRONG FORCE TP PORTFOLIO 2022, LLC
G05D1/0022B60W40/08G01C21/3438G01C21/3461G01C21/3469G01C21/3617G05B13/027G05D1/0088G05D1/0212G05D1/0287G05D1/224G05D1/225G05D1/226G05D1/227G05D1/228G05D1/229G05D1/24G05D1/646G05D1/69G05D1/692G05D1/81G06F40/40G06N3/0418G06N3/045G06N3/08G06N3/086G06N20/00G06Q30/0208G06Q50/188G06Q50/40G06V10/764G06V10/82G06V20/56G06V20/59G06V20/597G06V20/64G07C5/006G07C5/008G07C5/02G07C5/08G07C5/0808G07C5/0816G07C5/0866G07C5/0891G10L15/16G10L25/63B60W2040/0881G06N3/02G06Q30/0281G06Q50/01
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,321,170
App. No.
18/396,135
Granted
Jun 3, 2025
Kind
B2
Abstract

A system may include an operator data collection module to capture human operator interactions with a vehicle control system interface and may include a vehicle data collection module to capture vehicle response and operating conditions associated at least contemporaneously with the human operator interaction. An artificial intelligence system learns to control the vehicle with an optimized margin of safety while mimicking the human operator, where the artificial intelligence system is responsive to the robotic process automation system and the artificial intelligence system is to detect data indicative of at least one of a plurality of the instances of environmental information associated with the contemporaneously captured vehicle response and operating conditions. The optimized margin of safety is to be achieved by training the artificial intelligence system to control the vehicle based on the set of human operator interaction data collected at the robotic process automation system.

Claims (29)

1. A transportation system for optimizing a margin of safety when mimicking human operation of a vehicle, the transportation system comprising:

a robotic process automation system comprising:

an operator data collection module to capture human operator interaction data with a vehicle control system interface;

a vehicle data collection module to capture vehicle response and operating conditions associated at least contemporaneously with the human operator interaction; and

an environment data collection module to capture instances of environmental information associated at least contemporaneously with the human operator interactions; and

an artificial intelligence system to learn to control the vehicle with an optimized margin of safety while mimicking the human operator, wherein the artificial intelligence system is responsive to the robotic process automation system, wherein the artificial intelligence system is to detect data indicative of at least one of a plurality of the instances of environmental information associated with the contemporaneously captured vehicle response and operating conditions, wherein the optimized margin of safety is to be achieved by training the artificial intelligence system to control the vehicle based on the human operator interaction data collected at the robotic process automation system, and controlling the vehicle with the optimized margin of safety.

2. The transportation system of claim 1 , wherein the robotic process automation system further comprises a feedback module configured to provide corrective suggestions to the human operator based on an assessment by the artificial intelligence system of the human operator interactions and vehicle responses.

3. The transportation system of claim 1 , wherein the operator data collection module includes a camera system configured to capture visual data of the human operator interactions with the vehicle control system interface.

4. The transportation system of claim 1 , wherein the vehicle data collection module is further configured to capture data related to powertrain performance of the vehicle during the human operator interaction.

5. The transportation system of claim 1 , wherein the environment data collection module includes sensors for detecting road surface conditions and adjusting driving parameters of the vehicle to maintain the optimized margin of safety.

6. The transportation system of claim 1 , wherein the artificial intelligence system comprises a neural network trained to recognize patterns in the human operator interactions and to predict potential safety risks based on the patterns.

7. The transportation system of claim 1 , wherein the artificial intelligence system is further configured to adjust operational parameters of the vehicle in real-time to maintain the optimized margin of safety during autonomous operation.

8. The transportation system of claim 1 , wherein the artificial intelligence system utilizes machine learning algorithms wherein the machine learning algorithms are trained on a dataset including historical vehicle operation data and environmental conditions.

9. The transportation system of claim 1 , wherein the robotic process automation system is further configured to integrate data from a navigation system of the vehicle to anticipate changes in the environment and to adjust operation of the vehicle accordingly.

10. The transportation system of claim 1 , wherein the robotic process automation system includes an interface for the human operator to review and adjust operational parameters used by the artificial intelligence system for controlling the vehicle.

11. A method of robotic process automation for achieving an optimized margin of vehicle operational safety, the method comprising:

tracking expert human vehicle control interactions with a vehicle control-facilitating interface;

recording the tracked expert human vehicle control interactions in a robotic process automation system training data structure;

tracking vehicle operational state information of a vehicle; recording vehicle operational state information in the robotic process automation system training data structure;

training, via at least one neural network, the vehicle to operate with the optimized margin of vehicle operational safety in a manner consistent with the expert human vehicle control interactions based on the expert human vehicle control interactions and the vehicle operational state information in the robotic process automation system training data structure; and

controlling at least one aspect of the vehicle with a trained artificial intelligence system.

12. The method of claim 11 , wherein the vehicle control-facilitating interface comprises at least one of: a touch screen, a virtual assistant, an entertainment system interface, a communication interface, or a navigation interface.

13. The method of claim 11 , wherein the expert human vehicle control interactions include at least one of: braking patterns, follow-behind distance, approach to curve acceleration patterns, lane preferences, or passing preferences.

14. The method of claim 11 , wherein the vehicle operational state information includes data streams indicating states and changes in state in at least one of: steering, braking, acceleration, forward looking images, or rear-looking images.

15. The method of claim 11 , wherein the at least one neural network comprises a convolutional neural network.

16. The method of claim 11 , wherein the training further comprises applying deep learning to optimize the margin of vehicle operational safety by structured variation in the controlling of the at least one aspect of the vehicle and processing feedback from the controlling of the at least one aspect of the vehicle with machine learning.

17. The method of claim 11 , wherein a robotic process automation system includes an operator data collection module to capture human operator interaction with a vehicle control system interface, a vehicle data collection module to capture vehicle response and operating conditions associated at least contemporaneously with the human operator interaction, and an environment data collection module to capture instances of environmental information associated at least contemporaneously with the human operator interactions.

18. The method of claim 11 , wherein a robotic process automation system facilitates automation of a decision-making workflow employed by the artificial intelligence system.

19. The method of claim 11 , wherein a robotic process automation system facilitates automation of a remote control workflow, wherein the artificial intelligence system employs the automated remote control workflow to remotely control the vehicle.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 24, 2024
From: CELLA, CHARLES HOWARD
To: STRONG FORCE TP PORTFOLIO 2022, LLC
Reel/Frame 066235/0543 →
Continuity (6)
Continuation 17977698 · Oct 31, 2022
Continuation 16887583 · May 29, 2020
Continuation 16803356 · Feb 27, 2020
Continuation PCTUS2019053857 · Sep 30, 2019
Provisional Application 62739335 · Sep 30, 2018
Related Publication 20240142974A1 · May 2, 2024
References Cited (119)
US 5400734A · Doyon · 1995 [cited by applicant]
US 5566072A · Momose et al. · 1996 [cited by applicant]
US 6058352A · Lu et al. · 2000 [cited by applicant]
US 6188945B1 · Graf et al. · 2001 [cited by applicant]
US 7982620B2 · Prokhorov et al. · 2011 [cited by applicant]
US 8630897B1 · Prada Gomez et al. · 2014 [cited by applicant]
US 8825374B2 · Dimitriadis · 2014 [cited by applicant]
US 8862317B2 · Shin et al. · 2014 [cited by applicant]
US 8996303B1 · Bogovich et al. · 2015 [cited by applicant]
US 9293042B1 · Wasserman · 2016 [cited by applicant]
US 9451396B2 · Takatsuji · 2016 [cited by applicant]
US 9527508B1 · Lee · 2016 [cited by applicant]
US 9649974B1 · Arumugasamy et al. · 2017 [cited by applicant]
US 9744363B2 · Grill, Jr. et al. · 2017 [cited by applicant]
US 9827993B2 · Prokhorov · 2017 [cited by applicant]
US 9889770B2 · Hotary et al. · 2018 [cited by applicant]
US 9914475B2 · Sun et al. · 2018 [cited by applicant]
US 10117109B2 · Madaiah et al. · 2018 [cited by applicant]
US 10122692B2 · MacCarthaigh · 2018 [cited by applicant]
US 10126743B2 · Fukumoto · 2018 [cited by applicant]
US 10147319B1 · Chen · 2018 [cited by applicant]
US 10235859B1 · Hiles · 2019 [cited by applicant]
US 10289618B2 · Fink et al. · 2019 [cited by applicant]
US 10322727B1 · Chan et al. · 2019 [cited by applicant]
US 10322728B1 · Porikli et al. · 2019 [cited by applicant]
US 10346415B1 · Smith et al. · 2019 [cited by applicant]
US 10373522B2 · Byron et al. · 2019 [cited by applicant]
US 10387205B2 · Boutnaru · 2019 [cited by applicant]
US 10387417B1 · Hansen et al. · 2019 [cited by applicant]
US 10387429B2 · Smith-Mickelson et al. · 2019 [cited by applicant]
US 10832261B1 · Chan et al. · 2020 [cited by applicant]
US 10850693B1 · Pertsel et al. · 2020 [cited by applicant]
US 11049383B1 · Shahamad et al. · 2021 [cited by applicant]
US 20050143845A1 · Kaji · 2005 [cited by applicant]
US 20090055824A1 · Rychtyckyj et al. · 2009 [cited by applicant]
US 20090058683A1 · Becker · 2009 [cited by applicant]
US 20090292528A1 · Kameyama · 2009 [cited by applicant]
US 20100134302A1 · Ahn et al. · 2010 [cited by applicant]
US 20100293033A1 · Hall et al. · 2010 [cited by applicant]
US 20110213511A1 · Msconti et al. · 2011 [cited by applicant]
US 20110313259A1 · Hatakeyama et al. · 2011 [cited by applicant]
US 20130070043A1 · Geva et al. · 2013 [cited by applicant]
US 20130188513A1 · Vasseur et al. · 2013 [cited by applicant]
US 20140171752A1 · Park et al. · 2014 [cited by applicant]
US 20140172910A1 · Jung et al. · 2014 [cited by applicant]
US 20140201004A1 · Parundekar et al. · 2014 [cited by applicant]
US 20140218187A1 · Chun et al. · 2014 [cited by applicant]
US 20140257920A1 · Gilman et al. · 2014 [cited by applicant]
US 20140277902A1 · Koch · 2014 [cited by applicant]
US 20140278029A1 · Tonguz et al. · 2014 [cited by applicant]
US 20140278910A1 · Visintainer et al. · 2014 [cited by applicant]
US 20140279021A1 · MacNeille et al. · 2014 [cited by applicant]
US 20150112800A1 · Binion et al. · 2015 [cited by applicant]
US 20150166069A1 · Engelman et al. · 2015 [cited by applicant]
US 20150220991A1 · Butts et al. · 2015 [cited by applicant]
US 20150254955A1 · Fields et al. · 2015 [cited by applicant]
US 20150262239A1 · Goralnick · 2015 [cited by applicant]
US 20150292894A1 · Goddard et al. · 2015 [cited by applicant]
US 20150301602A1 · Dow et al. · 2015 [cited by applicant]
US 20150342007A1 · Anderson et al. · 2015 [cited by applicant]
US 20150344036A1 · Kristinsson et al. · 2015 [cited by applicant]
US 20150379530A1 · Bostick et al. · 2015 [cited by applicant]
US 20160063561A1 · MacNeille et al. · 2016 [cited by applicant]
US 20160104486A1 · Penilla et al. · 2016 [cited by applicant]
US 20160282132A1 · Bostick et al. · 2016 [cited by applicant]
US 20160354027A1 · Benson et al. · 2016 [cited by applicant]
US 20160356617A1 · Verosub et al. · 2016 [cited by applicant]
US 20170026790A1 · Flitsch et al. · 2017 [cited by applicant]
US 20170098231A1 · Dietrich · 2017 [cited by applicant]
US 20170105667A1 · Wei et al. · 2017 [cited by applicant]
US 20170171178A1 · Reynders · 2017 [cited by applicant]
US 20170285642A1 · Rander · 2017 [cited by applicant]
US 20170289184A1 · C et al. · 2017 [cited by applicant]
US 20170299399A1 · Yamaguchi et al. · 2017 [cited by applicant]
US 20170329331A1 · Gao · 2017 [cited by applicant]
US 20170355377A1 · Kumar et al. · 2017 [cited by applicant]
US 20170370732A1 · Bender et al. · 2017 [cited by applicant]
US 20180006822A1 · Brickell · 2018 [cited by applicant]
US 20180009444A1 · Grimm et al. · 2018 [cited by applicant]
US 20180032955A1 · Lindawati · 2018 [cited by applicant]
US 20180066833A1 · Draajer et al. · 2018 [cited by applicant]
US 20180113461A1 · Potnis et al. · 2018 [cited by applicant]
US 20180118218A1 · Miloser · 2018 [cited by applicant]
US 20180143635A1 · Zijderveld et al. · 2018 [cited by applicant]
US 20180174457A1 · Taylor · 2018 [cited by applicant]
US 20180281812A1 · Tochioka et al. · 2018 [cited by applicant]
US 20180297602A1 · Richmond et al. · 2018 [cited by applicant]
US 20190033085A1 · Ogale et al. · 2019 [cited by applicant]
US 20190035394A1 · Tsai · 2019 [cited by applicant]
US 20190049267A1 · Huang · 2019 [cited by applicant]
US 20190049957A1 · Healey et al. · 2019 [cited by applicant]
US 20190049969A1 · Qawami et al. · 2019 [cited by applicant]
US 20190187705A1 · Ganguli et al. · 2019 [cited by applicant]
US 20190193591A1 · Migneco et al. · 2019 [cited by applicant]
US 20190197430A1 · Arditi · 2019 [cited by applicant]
US 20190212967A1 · Stanley et al. · 2019 [cited by applicant]
US 20190222885A1 · Cho et al. · 2019 [cited by applicant]
US 20190225232A1 · Blau · 2019 [cited by applicant]
US 20190232974A1 · Reiley et al. · 2019 [cited by applicant]
US 20190265060A1 · Han et al. · 2019 [cited by applicant]
US 20190357834A1 · Aarts et al. · 2019 [cited by applicant]
US 20190373472A1 · Smith et al. · 2019 [cited by applicant]
US 20190391581A1 · Vardaro et al. · 2019 [cited by applicant]
US 20200023846A1 · Husain et al. · 2020 [cited by applicant]
US 20200070840A1 · Gunaratne · 2020 [cited by applicant]
US 20200074266A1 · Peake et al. · 2020 [cited by applicant]
US 20200097754A1 · Tawari et al. · 2020 [cited by applicant]
US 20200331465A1 · Herman et al. · 2020 [cited by applicant]
US 20210113129A1 · Huang · 2021 [cited by applicant]
US 20210374710A1 · Kuehne · 2021 [cited by applicant]
US 20210390581A1 · Chan et al. · 2021 [cited by applicant]
CN 107364368A · 2017 [cited by applicant]
KR 20150050134A · 2015 [cited by applicant]
WO 2009061687A1 · 2009 [cited by applicant]
WO 2019004468A1 · 2019 [cited by applicant]
Bojarksi et al., “End to End Learning for Self-Driving Cars”, arXiv preprint arXiv: 1604.07316, Apr. 25, 2016. [cited by applicant]
Author Unknown, “The Difference Between Robotic Process Automation and Artificial Intelligence”, CFB Bots, retrieved from https://medium.com/@cfb_bots, 7 pages, Apr. 20, 2018. [cited by applicant]
Popa, Alexandra, et al., “Intelligent Autonomous Driving”, 2018 22nd International Conference on System Theory, Control and Computing (ICSTCC), pp. 53-58, 2018. [cited by applicant]
Kocic et al., “Sim-to-Real Autonomous Vehicle Lane Keeping using Vision”, 2021 29th Telecommunications Forum (TELFOR), pp. 1-8, 2021. [cited by applicant]