IP Library Granted Patent US 12,639,189
Granted Patent B2
US 12,639,189 · App. 17/118,671 · Granted May 26, 2026

Selecting automation scripts using reinforced learning

Inventors: Marcelo Mota Manhaes (Curitiba, BR); Rogerio Baldini Das Neves (Belo Horizonte, BR); Sergio Varga (Campinas, BR); Igor Monteiro Vieira (Hortolandia, BR); Joao Luiz Todari (Sao Caetano do Sul, BR)
Assignee: Kyndryl, Inc.
G06F11/3457G06F11/3428G06F40/20G06N5/02
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,639,189
App. No.
17/118,671
Granted
May 26, 2026
Kind
B2
Abstract

A system can evaluate multiple candidate scripts. The system receives a problem statement and a sample solution script. The system selects an additional script based on the sample solution script, and compiles a list of candidates including the sample and additional scripts. Then, for each of the candidates, the system simulates execution of the script and scores performance of the script. The system then presents results of the execution.

Claims (121)

1 . A method, comprising:

receiving a problem statement describing a problem;

receiving a sample solution script from a user;

selecting, based on the sample solution script, a selected solution script;

compiling a list of candidate scripts, the list including the sample solution script and the selected solution script;

optimizing the candidate scripts by replacing at least one command in a candidate script with an alternative command from a command mapping database, wherein the command mapping database comprises at least one entry for a given command, wherein the at least one entry comprises a timestamp denoting the given command has been searched and a lack of an alternative command for the given command in response to searching for the given command in a predefined resource, wherein the at least one entry having the timestamp prevents a redundant search for the given command in the predefined resource;

identifying execution rules for each of the candidate scripts;

executing each of the candidate scripts in a simulation based on the execution rules;

scoring, based on the simulation and the execution rules, performance of each of the candidate scripts in addressing the problem, resulting in scored candidate scripts; and

presenting the scored candidate scripts;

wherein the simulating includes: defining a state-transition-reward routine; setting a first set of state checkpoints in a first script included within the candidate scripts; replicating a production environment, resulting in a replicated environment; and executing the first script within the replicated environment, the executing including:

executing a command included in the first script;

detecting a first state checkpoint included within the first set of state checkpoints; and

determining, in response to the detecting, a state of the first script, wherein, for the first script, the scoring is based further on a reward associated with a transition defined in the state-transition-reward routine, the transition associated with the state and a previous state, wherein the reward is a negative value applied to the transition, wherein the scoring comprises summing the negative value for occurrences of the transition to result in a final score for the first script, wherein a second script within the candidate scripts has another final score, wherein the state is one of a plurality of states comprising a ready state, a running state, a waiting state, a completed state, a terminated state, a succeeded state, and a failed state such that the reward is according to the transition among the plurality of states, wherein the negative value for the transition to the waiting state is more negative than the negative value for the transition to the running state, the completed state, or the succeeded state, wherein the negative value for the transition to the terminated state is more negative than the negative value for the transition to the waiting state, wherein the negative value for the transition to the failed state is more negative than the negative value for the transition the terminated state.

2 . The method of claim 1 , further comprising:

querying, based on the sample solution script, an external resource on the Internet identified in a knowledge base;

receiving, in response to the query, one or more found scripts; and

evaluating similarity of the one or more found scripts to the sample solution script, wherein:

the selecting is based on the evaluating; and

the selected solution script is included in a set of found scripts of the one or more found scripts.

3 . The method of claim 2 , wherein, for a first found script included in the set of found scripts, the evaluating includes:

comparing the first found script to the sample solution script; and

calculating, based on the comparing, a similarity score for the first found script, wherein the selecting is based further on the similarity score.

4 . The method of claim 2 , wherein the querying is performed via one or more Natural Language Processing (NLP) techniques based further on information included in a knowledge base.

5 . The method of claim 1 , further comprising optimizing the candidate scripts, the optimizing including, for a first script included in the candidate scripts:

extracting a first set of commands of the first script, the first set of commands including a first command;

determining that the first command is listed in an entry included in the command mapping database;

identifying, based on the first command and the entry included in the command mapping database, a first preferable alternative command; and

modifying the first script, the modifying including:

removing the first command from the first script; and

inserting the first preferable alternative command into the first script, resulting in a first optimized script.

6 . The method of claim 5 , wherein the optimizing further includes, for a second script included in the candidate scripts:

extracting a second set of commands of the second script, the second set of commands including a second command;

determining that the second command is not listed in the command mapping database;

searching, based on the determining that the second command is not listed in the command mapping database, an external resource for preferable alternative commands;

identifying, based on the searching, a second preferable alternative command;

updating, based on the second preferable alternative command, the command mapping database; and

modifying the second script, the modifying including:

removing the second command from the second script; and

inserting the second preferable alternative command into the second script, resulting in a second optimized script.

7 . A system comprising:

a memory; and

a central processing unit (CPU) coupled to the memory, the CPU configured to:

receive a problem statement describing a problem;

receive a sample solution script from a user;

select, based on the sample solution script, a selected solution script;

compile a list of candidate scripts, the list including the sample solution script and the selected solution script;

optimizing the candidate scripts by replacing at least one command in a candidate script with an alternative command from a command mapping database, wherein the command mapping database comprises at least one entry for a given command, wherein the at least one entry comprises a timestamp denoting the given command has been searched and a lack of an alternative command for the given command in response to searching for the given command in a predefined resource, wherein the at least one entry having the timestamp prevents a redundant search for the given command in the predefined resource;

identify execution rules of each of the candidate scripts;

executing each of the candidate scripts based on the execution rules;

score, based on the simulation and the execution rules, performance of each of the candidate scripts in addressing the problem, resulting in scored candidate scripts; and

present the scored candidate scripts;

wherein the simulating includes: defining a state-transition-reward routine; setting a first set of state checkpoints in a first script included within the candidate scripts; replicating a production environment, resulting in a replicated environment; and executing the first script within the replicated environment, the executing including:

executing a command included in the first script;

detecting a first state checkpoint included within the first set of state checkpoints; and

determining, in response to the detecting, a state of the first script, wherein, for the first script, the scoring is based further on a reward associated with a transition defined in the state-transition-reward routine, the transition associated with the state and a previous state, wherein the reward is a negative value applied to the transition, wherein the scoring comprises summing the negative value for occurrences of the transition to result in a final score for the first script, wherein a second script within the candidate scripts has another final score, wherein the state is one of a plurality of states comprising a ready state, a running state, a waiting state, a completed state, a terminated state, a succeeded state, and a failed state such that the reward is according to the transition among the plurality of states, wherein the negative value for the transition to the waiting state is more negative than the negative value for the transition to the running state, the completed state, or the succeeded state, wherein the negative value for the transition to the terminated state is more negative than the negative value for the transition to the waiting state, wherein the negative value for the transition to the failed state is more negative than the negative value for the transition the terminated state.

8 . The system of claim 7 , wherein the CPU is further configured to:

query, based on the sample solution script, an external resource on the Internet identified in a knowledge base;

receive, in response to the query, one or more found scripts; and

evaluate similarity of the one or more found scripts to the sample solution script, wherein:

the selecting is based on the evaluating; and

the selected solution script is included in a set of found scripts of the one or more found scripts.

9 . The system of claim 8 , wherein, for a first found script included in the set of found scripts, the evaluating includes:

comparing the first found script to the sample solution script; and

calculating, based on the comparing, a similarity score for the first found script, wherein the selecting is based further on the similarity score.

10 . The system of claim 8 , wherein the querying is performed via one or more Natural Language Processing (NLP) techniques based further on information included in a knowledge base.

11 . The system of claim 7 , wherein the CPU is further configured to optimize the candidate scripts, the optimizing including, for a first script included in the candidate scripts:

extracting a first set of commands of the first script, the first set of commands including a first command;

determining that the first command is listed in an entry included in the command mapping database;

identifying, based on the first command and the entry included in the command mapping database, a first preferable alternative command; and

modifying the first script, the modifying including:

removing the first command from the first script; and

inserting the first preferable alternative command into the first script, resulting in a first optimized script.

12 . The system of claim 11 , wherein the optimizing further includes, for a second script included in the candidate scripts:

extracting a second set of commands of the second script, the second set of commands including a second command;

determining that the second command is not listed in the command mapping database;

searching, based on the determining that the second command is not listed in the command mapping database, an external resource for preferable alternative commands;

identifying, based on the searching, a second preferable alternative command;

updating, based on the second preferable alternative command, the command mapping database; and

modifying the second script, the modifying including:

removing the second command from the second script; and

inserting the second preferable alternative command into the second script, resulting in a second optimized script.

13 . A computer program product, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to:

receive a problem statement describing a problem;

receive a sample solution script from a user;

select, based on the sample solution script, a selected solution script;

compile a list of candidate scripts, the list including the sample solution script and the selected solution script;

optimize the candidate scripts by replacing at least one command in a candidate script with an alternative command from a command mapping database, wherein the command mapping database comprises at least one entry for a given command, wherein the at least one entry comprises a timestamp denoting the given command has been searched and a lack of an alternative command for the given command in response to searching for the given command in a predefined resource, wherein the at least one entry having the timestamp prevents a redundant search for the given command in the predefined resource;

identify execution rules of each of the candidate scripts;

execute each of the candidate scripts in a simulation based on the execution rules;

score, based on the simulation and the execution rules, performance of each of the candidate scripts in addressing the problem, resulting in scored candidate scripts; and

present the scored candidate scripts;

wherein the simulating includes: defining a state-transition-reward routine; setting a first set of state checkpoints in a first script included within the candidate scripts; replicating a production environment, resulting in a replicated environment; and executing the first script within the replicated environment, the executing including:

executing a command included in the first script;

detecting a first state checkpoint included within the first set of state checkpoints; and

determining, in response to the detecting, a state of the first script, wherein, for the first script, the scoring is based further on a reward associated with a transition defined in the state-transition-reward routine, the transition associated with the state and a previous state, wherein the reward is a negative value applied to the transition, wherein the scoring comprises summing the negative value for occurrences of the transition to result in a final score for the first script, wherein a second script within the candidate scripts has another final score, wherein the state is one of a plurality of states comprising a ready state, a running state, a waiting state, a completed state, a terminated state, a succeeded state, and a failed state such that the reward is according to the transition among the plurality of states, wherein the negative value for the transition to the waiting state is more negative than the negative value for the transition to the running state, the completed state, or the succeeded state, wherein the negative value for the transition to the terminated state is more negative than the negative value for the transition to the waiting state, wherein the negative value for the transition to the failed state is more negative than the negative value for the transition the terminated state.

14 . The computer program product of claim 13 , wherein the instructions further cause the computer to:

query, based on the sample solution script, an external resource on the Internet identified in a knowledge base;

receive, in response to the query, one or more found scripts; and

evaluate similarity of the one or more found scripts to the sample solution script, wherein:

the selecting is based on the evaluating; and

the selected solution script is included in a set of found scripts of the one or more found scripts.

15 . The computer program product of claim 14 , wherein, for a first found script included in the set of found scripts, the evaluating includes:

comparing the first found script to the sample solution script; and

calculating, based on the comparing, a similarity score for the first found script, wherein the selecting is based further on the similarity score.

16 . The computer program product of claim 13 , wherein the instructions further cause the computer to optimize the candidate scripts, the optimizing including, for a first script included in the candidate scripts:

extracting a first set of commands of the first script, the first set of commands including a first command;

determining that the first command is listed in an entry included in the command mapping database;

identifying, based on the first command and the entry included in the command mapping database, a first preferable alternative command; and

modifying the first script, the modifying including:

removing the first command from the first script; and

inserting the first preferable alternative command into the first script, resulting in a first optimized script.

17 . The computer program product of claim 16 , wherein the optimizing further includes, for a second script included in the candidate scripts:

extracting a second set of commands of the second script, the second set of commands including a second command;

determining that the second command is not listed in the command mapping database;

searching, based on the determining that the second command is not listed in the command mapping database, an external resource for preferable alternative commands;

identifying, based on the searching, a second preferable alternative command;

updating, based on the second preferable alternative command, the command mapping database; and

modifying the second script, the modifying including:

removing the second command from the second script; and

inserting the second preferable alternative command into the second script, resulting in a second optimized script.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 18, 2021
From: INTERNATIONAL BUSINESS MACHINES CORPORATION
To: KYNDRYL, INC.
Reel/Frame 058213/0912 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 11, 2020
From: MOTA MANHAES, MARCELO; BALDINI DAS NEVES, ROGERIO; VARGA, SERGIO; MONTEIRO VIEIRA, IGOR; TODARI, JOAO LUIZ
To: INTERNATIONAL BUSINESS MACHINES CORPORATION
Reel/Frame 054612/0819 →
Continuity (1)
Related Publication 20220188213A1 · Jun 16, 2022
References Cited (50)
US 8266592B2 · Beto et al. · 2012 [cited by applicant]
US 8688676B2 · Rush · 2014 [cited by examiner]
US 8874498B2 · Modha · 2014 [cited by applicant]
US 8972372B2 · Elbaum · 2015 [cited by examiner]
US 9412075B2 · Padala et al. · 2016 [cited by applicant]
US 10033406B2 · Huang et al. · 2018 [cited by applicant]
US 10063406B2 · Tapia et al. · 2018 [cited by applicant]
US 10374919B2 · Pai · 2019 [cited by examiner]
US 10922357B1 · Chennuru · 2021 [cited by examiner]
US 11042369B1 · Kimball · 2021 [cited by examiner]
US 11580445B2 · Liu · 2023 [cited by examiner]
US 20090265681A1 · Beto et al. · 2009 [cited by applicant]
US 20130164727A1 · Dzakula · 2013 [cited by applicant]
US 20160063062A1 · Yahav · 2016 [cited by examiner]
US 20170019315A1 · Tapia et al. · 2017 [cited by applicant]
US 20180165604A1 · Minkin · 2018 [cited by examiner]
US 20220138240A1 · Bahrami · 2022 [cited by examiner]
CN 103248693A · 2013 [cited by applicant]
CN 105355111A · 2016 [cited by applicant]
Tesauro, Gerald, Nicholas K. Jong, Rajarshi Das, and Mohamed N. Bennani. “On the use of hybrid reinforcement learning for autonomic resource allocation.” Cluster Computing 10 (2007): 287-299. (Year: 2007). [cited by examiner]
Heo, Kihong, Woosuk Lee, Pardis Pashakhanloo, and Mayur Naik. “Effective program debloating via reinforcement learning.” In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, pp. 380-… [cited by examiner]
Starlinger, Johannes, Sarah Cohen-Boulakia, Sanjeev Khanna, Susan B. Davidson, and Ulf Leser. “Effective and efficient similarity search in scientific workflow repositories.” Future Generation Computer Systems 56 (2016)… [cited by examiner]
Fursin, Grigori, Anton Lokhmotov, Dmitry Savenko, and Eben Upton. “A collective knowledge workflow for collaborative research into multi-objective autotuning and machine learning techniques.” arXiv preprint arXiv:1801.0… [cited by examiner]
Battina, Dhaya Sindhu. “AI-Augmented Automation for DevOps, a Model-Based Framework for Continuous Development in Cyber-Physical Systems.” International Journal of Creative Research Thoughts (IJCRT), ISSN (2016): 2320-2… [cited by examiner]
Saini, Vaibhav, Hitesh Sajnani, Jaewoo Kim, and Cristina Lopes. “Sourcerercc and sourcerercc-i: tools to detect clones in batch mode and during software development.” In Proceedings of the 38th international conference … [cited by examiner]
Sajnani, Hitesh, Vaibhav Saini, Jeffrey Svajlenko, Chanchal K. Roy, and Cristina V. Lopes. “Sourcerercc: Scaling code clone detection to big-code.” In Proceedings of the 38th international conference on software enginee… [cited by examiner]
Alici, Sadiye, Ismail Sengor Altingovde, Rifat Ozcan, Berkant Barla Cambazoglu, and Özgür Ulusoy. “Timestamp-based result cache invalidation for web search engines.” In Proceedings of the 34th international ACM SIGIR co… [cited by examiner]
Chen, Yan, Jaylin Herskovitz, Walter S. Lasecki, and Steve Oney. “Bashon: A Hybrid Crowd-Machine Workflow for Shell Command Synthesis.” In 2020 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), pp… [cited by examiner]
Chakraborti, Tathagata, et al. “UbuntuWorld 1.0 LTS—a platform for automated problem solving and troubleshooting in the Ubuntu OS.” In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31, No. 2, pp. 4… [cited by examiner]
Reid, Brittany, Christoph Treude, and Markus Wagner. “Optimising the fit of stack overflow code snippets into existing code.” In Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion, pp. 194… [cited by examiner]
Zagalsky, Alexey, Ohad Barzilay, and Amiram Yehudai. “Example overflow: Using social media for code recommendation.” In 2012 Third International Workshop on Recommendation Systems for Software Engineering (RSSE), pp. 38… [cited by examiner]
Greco, Chase D. “A Behavior-Driven Recommendation System for Stack Overflow Posts.” Virginia Commonwealth University, Thesis (2018). Downloaded from <https://scholarscompass.vcu.edu/etd/5396> (Year: 2018). [cited by examiner]
Dai, Ting, Alexei Karve, Grzegorz Koper, and Sai Zeng. “Automatically detecting risky scripts in infrastructure code.” In Proceedings of the 11th ACM Symposium on Cloud Computing, pp. 358-371. 2020. (Year: 2020). [cited by examiner]
Agarwal, Mayank, Jorge J. Barroso, Tathagata Chakraborti, Eli M. Dow, Kshitij Fadnis, Borja Godoy, Madhavan Pallan, and Kartik Talamadupula. “Project clai: Instrumenting the command line as a new environment for ai agen… [cited by examiner]
Kim, Kisub, Dongsun Kim, Tegawendé F. Bissyandé, Eunjong Choi, Li Li, Jacques Klein, and Yves Le Traon. “FaCoY: a code-to-code search engine.” In Proceedings of the 40th International Conference on Software Engineering,… [cited by examiner]
Nascimento, André, Victor Olimpio, Vitor Silva, Aline Paes, and Daniel de Oliveira. “A reinforcement learning scheduling strategy for parallel cloud-based workflows.” In 2019 IEEE international parallel and distributed … [cited by examiner]
McKeeman, William M. “Peephole optimization.” Communications of the ACM 8, No. 7 (1965): 443-444. (Year: 1965). [cited by examiner]
Massalin, Henry. “Superoptimizer: a look at the smallest program.” ACM SIGARCH Computer Architecture News 15, No. 5 (1987): 122-126. (Year: 1987). [cited by examiner]
Bansal, Sorav, and Alex Aiken. “Automatic generation of peephole superoptimizers.” ACM SIGARCH Computer Architecture News 34, No. 5 (2006): 394-403. (Year: 2006). [cited by examiner]
Schkufza, Eric, Rahul Sharma, and Alex Aiken. “Stochastic superoptimization.” ACM SIGARCH Computer Architecture News 41, No. 1 (2013): 305-316. (Year: 2013). [cited by examiner]
Bunel, Rudy, Alban Desmaison, M. Pawan Kumar, Philip HS Torr, and Pushmeet Kohli. “Learning to superoptimize programs.” arXiv preprint arXiv:1611.01787 (2016). Submitted on Nov. 6, 2016 (v1), last revised Jun. 28, 2017 … [cited by examiner]
“Train Reinforcement Learning Agent in MDP Environment,” MATLAB & Simulink, Jul. 16, 2020, 5 pages, https://www.mathworks.com/help/reinforcement-learning/ug/train-reinforcement-learning-agent-in-mdp-environment.html. [cited by applicant]
“System and Method for Automated Solution Identification of an Organization IT Platform Issue through Machine Learning,” IP.Com, An IP.com Prior Art Database Technical Disclosure, IP.com No. IPCOM000260437D, IP.com Elec… [cited by applicant]
Agarwal et al., “Automatic problem extraction and analysis from unstructured text in IT tickets,” IBM J. Res. & Dev. vol. 61 No. 1 Paper 4 Jan./Feb. 2017, 12 pages, https://ieeexplore.ieee.org/document/7877279. [cited by applicant]
“What is AI Ops and How is it Changing IT Operations,” Logic Monitor, Printed Dec. 8, 2020, 5 pages https://www.logicmonitor.com/blog/what-is-aiops-and-how-is-it-changing-it-operations. [cited by applicant]
“10 good shell scripting practices,” The UNIX School, Mar. 29, 2012, 6 pages https://www.theunixschool.com/2012/03/10-good-shell-scripting-practices.html. [cited by applicant]
Rowe et al., “Toward Automated Scenario Generation with Deep Reinforcement Learning in GIFT,” North Carolina State University, Intelligent Automation, Inc., Printed Dec. 8, 2020, 10 pages. [cited by applicant]
Rodriguez et al., “Continuous Action Reinforcement Learning Automata,” Performance and Convergence, Proceedings of the 3rd International Conference on Agents and Artificial Intelligence (ICAART-2011), pp. 473-478. [cited by applicant]
Arabnejad et al., “A Comparison of Reinforcement Learning Techniques for Fuzzy Cloud Auto-Scaling,” Research Gate, May 2017, 11 pages. [cited by applicant]
“IBM Watson AIOps,” An AI-powered IT incident resolution application fueled by your own data, printed Dec. 8, 2020, 5 pages. [cited by applicant]