IP Library › Granted Patent US 12,190,222
Granted Patent B2
US 12,190,222 · App. 16/696,920 · Granted Jan 7, 2025

Active optimization approach for rapid and efficient design space exploration using ensemble machine learning

Inventors: Opeoluwa Olawale Owoyele (Aurora, IL); Pinaki Pal (Chicago, IL)
Assignee: UChicago Argonne, LLC
G06N3/045G06F7/582G06N3/08
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,190,222
App. No.
16/696,920
Granted
Jan 7, 2025
Kind
B2
Abstract

A design optimization method based on active learning, which involves dynamic exploration and exploitation of the design space of interest using an ensemble of machine learning algorithms. In this approach, a hybrid methodology incorporating an explorative weak learner which fits high-level information about the response surface, and an exploitative strong learner (based on committee machine) that fits finer details around promising regions identified by the weak learner, is employed. For each design iteration, an aristocratic approach is used to select a set of nominees, where points that meet a threshold merit value as predicted by the weak learner are selected to be evaluated using function evaluation. In addition to these points, the global optimum as predicted by the strong learner is also evaluated to enable rapid convergence to the actual global optimum once the most promising region has been identified by the optimizer.

Claims (64)

1. A method for design space optimization, comprising:

generating, by one or more processors, a plurality of first data points by evaluating a function;

training, by the one or more processors, a weak learner model using the plurality of first data points;

training, by the one or more processors, a strong learner model using the plurality of first data points, the strong learner model having a greater measure of fit to the plurality of first data points than the weak learner model;

defining, by one or more processors, a plurality of monitor points within a design space;

in a first iteration, generating, by the one or more processors using the weak learner model, a first count of second data points; generating, by the one or more processors, using the strong learner model, a second count of second data points using an optimizer; and generating, by the one or more processors, a first count of a plurality of output monitor points using the plurality of monitor points and the weak learner model;

in a second iteration, generating, by the one or more processors using the weak learner model, a third count of second data points; generating, by the one or more processors using the strong learner model, a fourth count of second data points using the optimizer; and generating, by the one or more processors, a second count of a plurality of output monitor points using the plurality of monitor points and the weak learner model;

determining, by the one or more processors, a change of the plurality of output monitor points above a predetermined threshold between the first iteration and the second iteration;

in a third iteration, generating by the one or more processors using the weak learner model, a fifth count of second data points, and generating, by the one or more processors using the strong learner model, a sixth count of second data points using the optimizer, wherein the fifth count of second data points and the sixth count of second data points are each determined based on the change of the plurality of output monitor points above the predetermined threshold;

evaluating, by the one or more processors using the function, input values corresponding to the second data points to generate a candidate optimum output; and

outputting, by the one or more processors, the candidate optimum output responsive to an output condition being satisfied.

2. The method of claim 1 , wherein generating, by the one or more processors, the plurality of first data points includes applying randomly selected inputs to the function.

3. The method of claim 1 , wherein training, by the one or more processors, the weak learner model includes providing the plurality of first data points as input to at least one of a regression model or a support vector machine.

4. The method of claim 1 , wherein training, by the one or more processors, the strong learner model includes providing the plurality of first data points as input to at least one of a neural network or a random forest model.

5. The method of claim 1 , further comprising:

generating, by the one or more processors, a plurality of candidate second data points; and

selecting, by the one or more processors, the second data points from the plurality of candidate second data points by determining that each second data point satisfies the optimization condition based on a candidate second data point meeting or exceeding a threshold percentile relative to the plurality of candidate second data points.

6. The method of claim 5 , further comprising selecting, by the one or more processors, input values to generate the plurality of candidate second data points using a pseudorandom number generator and based on a distance between each input value.

7. The method of claim 1 , further comprising determining, by the one or more processors, the output condition to be satisfied responsive to at least one of a threshold number of iterations or the candidate optimum output being within a threshold of an expected optimum of the function.

8. The method of claim 1 , wherein an optimum of the function is a maximum value or a minimum value.

9. The method of claim 1 , further comprising:

training, by the one or more processors, the strong learner model to include a plurality of neural networks, each neural network of the plurality of neural networks provided with at least one of different weights or different biases; and

generating, by the one or more processors, the sixth count of second data points using each of the plurality of neural networks.

10. The method of claim 1 , further comprising updating, by the one or more processors responsive to the output condition not being satisfied, the weak learner model and the strong learner model using a plurality of second data points.

11. The method of claim 1 , further comprising using the function to optimize a combustion process.

12. A system, comprising:

one or more processors configured to:

generate a plurality of first data points by evaluating a function;

train a weak learner model using the plurality of first data points;

train a strong learner model using the plurality of first data points, the strong learner model having a greater measure of fit to the plurality of first data points than the weak learner model;

define a plurality of monitor points within a design space;

in a first iteration, generate, using the weak learner model, a first count of second data points; generate, using the strong learner model, a second count of second data points; and generate a first count of a plurality of output monitor points using the plurality of monitor points and the weak learner model and;

in a second iteration, generate, using the weak learner model, a third count of second data points; generate, using the strong learner model, a fourth count of second data points; and generate a second count of a plurality of output monitor points using the plurality of monitor points and the weak learner model;

detect a change of the plurality of output monitor points above a predetermined threshold between the first iteration and the second iteration;

in a third iteration, generate using the weak learner model, a fifth count of second data points, and generate, using the strong learner model, a sixth count of second data points, wherein the fifth count of second data points and the sixth count of second data points are each determined based on the change of the plurality of output monitor points above the predetermined threshold;

evaluate, using the function, input values corresponding to the second data points to generate a candidate optimum output; and

output the candidate optimum output responsive to an output condition being satisfied.

13. The system of claim 12 , wherein the one or more processors are configured to:

train the weak learner model by providing plurality of first data points as input to at least one of a regression model or a support vector machine; and

train the strong learner model by providing the plurality of first data points as input to at least one of a neural network or a random forest model.

14. The system of claim 12 , wherein the one or more processors are configured to:

generate a plurality of candidate second data points using the weak learner model; and

determine that each second data point satisfies the optimization condition based on a candidate second data point meeting or exceeding a threshold percentile relative to the plurality of candidate second data points.

15. The system of claim 12 , wherein the one or more processors are configured to determine the output condition to be satisfied responsive to at least one of a threshold number of iterations or the candidate optimum output being within a threshold of an expected optimum of the function.

16. The system of claim 12 , wherein the one or more processors are configured to:

train the strong learner model to include a plurality of neural networks, each neural network of the plurality of neural networks provided with at least one of different weights or different biases; and

generate the sixth count of second data points using each of the plurality of neural networks.

17. The system of claim 12 , wherein the one or more processors are configured to generate, using the weak learner model, in a fourth iteration, a seventh count of second data points.

18. A method for optimizing a design space comprising:

(i) populating a design space with random N design points evaluated by a function evaluator and populating the design space with random X monitor points;

(ii) training a weak learner model using available data;

(iii) training a strong learner model using the available data, the strong learner model having a greater measure of fit to the available data than the weak learner model;

(iv) randomly sampling first points that the weak learner model predicts will be above a selected objective value (λ) based on an objective value function, the number of random sampling first points, where the first points are increased or decreased over iterations based at least on a change in output monitor points generated by the weak learner model using the random X monitor points output over iterations of the weak learner model;

(v) finding optimum points predicted by the strong learner model using a global optimization scheme, the optimum points increasing when the first points decrease and the optimum points decreasing when the first points increase;

(vi) adding the optimum points identified in step five to the first points to select N points to be evaluated by a design function;

(vii) performing design function evaluations on the N points to determine N design solutions; and

(viii) adding the N design solutions to a database and repeating steps 2-7 until the method converges or maximum number of iterations is reached.

19. The method of claim 1 , wherein:

the fifth count of second data points is smaller than the third count of second data points responsive to the change of the plurality of output monitor points below the predetermined threshold;

the fifth count of second data points is larger than the third count of second data points responsive to the change of the plurality of output monitor points below the predetermined threshold; and

the sixth count of second data points is larger than the fourth count of second data points responsive to the fifth count of second data points being smaller than the third count of second data points and the sixth count of second data points is smaller than the fourth count of second data points responsive to the fifth count of second data points being larger than the third count of second data points.

20. The method of claim 5 , wherein the plurality of candidate second data points is a first plurality of candidate second data points; and

further comprising generating, by the one or more processors, a second plurality of candidate second data points at a distance from the first plurality of candidate second data points defined by

X new=argmax( d min).

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 26, 2021
From: PAL, PINAKI; OWOYELE, OPEOLUWA OLAWALE
To: UCHICAGO ARGONNE, LLC
Reel/Frame 056974/0980 →
CONFIRMATORY LICENSE Recorded Feb 21, 2020
From: UCHICAO ARGONNE, LLC
To: UNITED STATES DEPARTMENT OF ENERGY
Reel/Frame 051990/0872 →
Continuity (2)
Provisional Application 62884502 · Aug 8, 2019
Related Publication 20210042609A1 · Feb 11, 2021
References Cited (52)
US 20070005313A1 · Sevastyanov · 2007 [cited by examiner]
US 20110078100A1 · Goel · 2011 [cited by examiner]
US 20160048771A1 · Chen · 2016 [cited by examiner]
US 20190026466A1 · Krasser · 2019 [cited by examiner]
US 20190095805A1 · Tristan · 2019 [cited by examiner]
US 20200210812A1 · Baker · 2020 [cited by examiner]
US 20200410090A1 · Baker · 2020 [cited by examiner]
US 20210042297A1 · Urbanke · 2021 [cited by examiner]
Nardi, L. et al., “Practical Design Space Exploration” (Year: 2019). [cited by examiner]
Cheng, Y. et al., “ThermalNet: A deep reinforcement learning-based combustion optimization system for coal-fired boiler” (Year: 2018). [cited by examiner]
Zhang, C. et al., “Deep Belief Networks Ensemble with Multi-objective Optimization for Failure Diagnosis” (Year: 2015). [cited by examiner]
Liu, H. et al., “On Learning-Based Methods for Design-Space Exploration with High-Level Synthesis” (Year: 2013). [cited by examiner]
Sewell, M., “Ensemble Learning” (Year: 2008). [cited by examiner]
Nardi, L. et al., “Practical Design Space Exploration”, https://arxiv.org/abs/1810.05236 (Year: 2019). [cited by examiner]
Cheng, Y. et al., “ThermalNet: A deep reinforcement learning-based combustion optimization system for coal-fired boiler”, https://www.sciencedirect.com/science/article/pii/S0952197618301477 (Year: 2018). [cited by examiner]
Sewell, M., “Ensemble Learning”, http://machine-learning.martinsewell.com/ensembles/ensemble-learning.pdf (Year: 2008). [cited by examiner]
Nardi, L. et al., “Practical Design Space Exploration”, https://arxiv.org/abs/1810.05236, Jul. 2019 (Year: 2019). [cited by examiner]
Natekin, A. et al., “Gradient boosting machines, a tutorial”, https://www.frontiersin.org/articles/10.3389/fnbot.2013.00021/full (Year: 2013). [cited by examiner]
Ceperic, V. et al., “Recurrent sparse support vector regression machines trained by active learning in the time-domain”, https://www.sciencedirect.com/science/article/pii/S0957417412005015 (Year: 2012). [cited by examiner]
Buche, D. et al., “Multiobjective Evolutionary Algorithm for the Optimization of Noisy Combustion Processes”, https://ieeexplore.ieee.org/abstract/document/1176895 (Year: 2002). [cited by examiner]
Abadi, et al., “TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems,” Cornell University Computer Science: Distributed, Parallel, and Cluster Computing, arXiv:1603.04467, 19 pages (2016). [cited by applicant]
Bergmeir, et al., “Memetic Algorithms with Local Search Chains in R: The Rmalschains Package,” Journal of Statistical Software 75(4), pp. 1-33 (2016). [cited by applicant]
Clerc, “Beyond Standard Particle Swarm Optimisation,” Innovations and Developments of Swarm Intelligence Applications, pp. 1-19 (2012). [cited by applicant]
Holmstrom, et al., “An adaptive radial basis algorithm (ARBF) for expensive black-box mixed-integer constrained global optimization,” Optimization and Engineering 9(4), pp. 311-339 (2008). [cited by applicant]
Jones, “A Taxonomy of Global Optimization Methods Based on Response Surfaces,” Journal of Global Optimization 21(4), pp. 345-383 (2001). [cited by applicant]
Kennedy & Eberhart, “Particle swarm optimization,” Proceedings of ICNN'95—International Conference on Neural Networks, pp. 1942-1948 (1995). [cited by applicant]
Moiz, et al., “A Machine Learning-Genetic Algorithm (ML-GA) Approach for Rapid Optimization Using High-Performance Computing,” SAE International Journal of Commercial Vehicles 11(5), pp. 291-306 (2018). [cited by applicant]
Muller, “MISO: mixed-integer surrogate optimization framework,” Optimization and Engineering 17(1), pp. 177-203 (2016). [cited by applicant]
Pal, et al., “Numerical Investigation of a Gasoline-Like Fuel in a Heavy-Duty Compression Ignition Engine Using Global Sensitivity Analysis,” SAE International Journal of Fuels and Lubricants 10(1), pp. 56-68 (2017). [cited by applicant]
Pedregosa, et al., “Scikit-learn: Machine Learning in Python,” Journal of Machine Learning Research 12, pp. 2825-2830 (2011). [cited by applicant]
Pei, et al., “CFD-Guided Combustion System Optimization of a Gasoline Range Fuel in a Heavy-Duty Compression Ignition Engine Using Automatic Piston Geometry Generation and a Supercomputer,” SAE International Journal of … [cited by applicant]
Rezaei, et al., “Performance prediction of HCCI engines with oxygenated fuels using artificial neural networks,” Applied Energy 138, pp. 460-473 (2015). [cited by applicant]
Samadani, et al., “A Method for Pre-Calibration of DI Diesel Engine Emissions and Performance Using Neural Network and Multi-Objective Genetic Algorithm,” Iranian Journal of Chemistry and Chemical Engineering 28(4), pp.… [cited by applicant]
Senecal, “Numerical optimization using the GEN4 micro-genetic algorithm code,” University of Wisconsin-Madison: Engine Research Center, 18 pages (2000). [cited by applicant]
Validi, et al., “HCCI Intelligent Rapid Modeling by Artificial Neural Network and Genetic Algorithm,” Journal of Combustion 2012, 854393, 11 pages (2012). [cited by applicant]
Van Der Laan, et al., “Super Learner,” Statistical Applications in Genetics and Molecular Biology 6(1), 23 pages (2007). [cited by applicant]
Vaughan & Bohac, “An Extreme Learning Machine Approach to Predicting Near Chaotic HCCI Combustion Phasing in Real-Time,” Cornell University Computer Science: Machine Learning, arXiv:1310.3567, 11 pages (2015). [cited by applicant]
Wang, et al., “An evaluation of adaptive surrogate modeling based optimization with two benchmark problems,” Environmental Modelling & Software 60, pp. 167-197 (2014). [cited by applicant]
Bertram, et al., “A novel particle swarm and genetic algorithm hybrid method for diesel engine performance optimization,” International Journal of Engine Research 17(7), pp. 732-747 (2016). [cited by applicant]
Brahma, et al., “A New Approach to System Level Soot Modeling,” SAE Technical Paper 2005-01-1122, 16 pages (2005). [cited by applicant]
Hanson, et al., “Piston Bowl Optimization for RCCI Combustion in a Light-Duty Multi-Cylinder Engine,” SAE International Journal of Engines 5(2), pp. 286-299 (2012). [cited by applicant]
He & Rutland, “Modeling of a Turbocharged DI Diesel Engine Using Artificial Neural Networks,” SAE Technical Paper 2002-01-2772, 14 pages (2002). [cited by applicant]
He & Rutland, “Neural Cylinder Model and Its Transient Results,” SAE Technical Paper 2003-01-3232, 17 pages (2003). [cited by applicant]
Joly, et al., “Machine Learning Enabled Adaptive Optimization of a Transonic Compressor Rotor With Precompression,” Journal of Turbomachinery 141(5), 051011, 9 pages (2019). [cited by applicant]
Kavuri & Kokjohn, “Exploring the potential of machine learning in reducing the computational time/expense and improving the reliability of engine optimization studies,” International Journal of Engine Research, 14680874… [cited by applicant]
Krishnakumar, “Micro-Genetic Algorithms For Stationary And Non-Stationary Function Optimization,” SPIE Proceedings 1196: Intelligent Control and Adaptive Systems, pp. 289-296 (1990). [cited by applicant]
Probst, et al., “Optimization and Uncertainty Analysis of a Diesel Engine Operating Point Using CFD,” ASME 2016 Internal Combustion Engine Division Fall Technical Conference, 13 pages (2016). [cited by applicant]
Shi & Reitz, “Optimization of a heavy-duty compression-ignition engine fueled with diesel and gasoline-like fuels,” Fuel 89(11), pp. 3416-3430 (2010). [cited by applicant]
Wickman, et al., “Diesel Engine Combustion Chamber Geometry Optimization Using Genetic Algorithms and Multi-Dimensional Spray and Combustion Modeling,” SAE Transactions 110(2001), Section 3: Journal of Engines, pp. 487-… [cited by applicant]
Wu, et al., “Numerical optimization of natural gas and diesel dual-fuel combustion for a heavy-duty engine operated at a medium load,” International Journal of Engine Research 19(6), pp. 682-696 (2018). [cited by applicant]
Zhang, et al., “A comparative study of biodiesel engine performance optimization using enhanced hybrid PSO-GA and basic GA,” Applied Energy 165, pp. 676-684 (2016). [cited by applicant]
Owoyele & Pal, “A novel machine learning-based optimization algorithm (ActivO) for accelerating simulation-driven engine design,” Applied Energy 285:116455, 10 pages (2021). [cited by applicant]