IP Library › Granted Patent US 12,373,699
Granted Patent B2
US 12,373,699 · App. 18/320,758 · Granted Jul 29, 2025

Systems and methods for an accelerated tuning of hyperparameters of a model using a machine learning-based tuning service

Inventors: Michael McCourt (San Francisco, CA); Ben Hsu (San Francisco, CA); Patrick Hayes (San Francisco, CA); Scott Clark (San Francisco, CA)
Assignee: INTEL CORPORATION
G06N3/082G06N20/00G06N20/20
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,373,699
App. No.
18/320,758
Granted
Jul 29, 2025
Kind
B2
Abstract

A system and method for accelerated tuning of hyperparameters includes receiving a multi-task tuning work request for tuning hyperparameters of a model, wherein the multi-task tuning work request includes: a full tuning task for tuning hyperparameters, wherein the full tuning task includes a first set of tuning parameters governing a first tuning operation; a partial tuning task for tuning the hyperparameters of the model, wherein the partial tuning task includes a second distinct set of tuning parameters governing a second tuning operation; executing the first tuning operation and the second tuning operation; generating a first suggestion set and a second suggestion set of one or more proposed values for the hyperparameters based on the execution of the full tuning task and the partial tuning task; and setting the partial tuning task as a proxy for the full tuning task thereby accelerating a tuning of the hyperparameters of the model.

Claims (50)

1. A system comprising:

memory;

machine-readable instructions; and

programmable circuitry, the machine-readable instructions to cause the programmable circuitry to:

access a tuning work request to tune hyperparameters of a model, the tuning work request to include:

a full tuning task and a corresponding first number of datapoints;

a partial tuning task and a corresponding second number of datapoints, the second number of datapoints being less than the first number of datapoints;

execute a full tuning of the model based on the full tuning task;

execute a partial tuning of the model based on the partial tuning task;

evaluate a measured performance associated with the partial tuning of the model and a measured performance associated with the full tuning of the model;

select hyperparameter values associated with the partial tuning of the model based on a determination that the measured performance associated with the partial tuning of the model satisfies a performance value;

determine an importance of a first one of the hyperparameters of the model; and

based on the importance of the first hyperparameter, rank the first hyperparameter relative to other ones of the hyperparameters of the model.

2. The system of claim 1 , wherein the programmable circuitry is to, based on a budget, determine to use fewer than the second number of datapoints for the partial tuning of the model.

3. The system of claim 2 , wherein the programmable circuitry is to set a tuning cost parameter for the partial tuning of the model.

4. The system of claim 1 , wherein the first number of datapoints for the full tuning task is at least 1000 samples, and the second number of datapoints for the partial tuning task is at least 500 samples.

5. The system of claim 1 , wherein the model is a Latin Hypercube Sampling (LHS) model.

6. The system of claim 5 , wherein the partial tuning refers to a number of interchanges.

7. The system of claim 1 , wherein the model performs a dimensionality reduction method.

8. At least one of a random access memory (RAM), a read only memory (ROM), a flash memory, an electrically erasable programmable read only memory (EEPROM), an optical disc, or a hard drive comprising instructions to cause programmable circuitry to at least:

access a tuning work request to tune hyperparameters of a model, the tuning work request to include:

a full tuning task and a corresponding first number of datapoints;

a partial tuning task and a corresponding second number of datapoints, the second number of datapoints being less than the first number of datapoints;

execute a full tuning of the model based on the full tuning task;

execute a partial tuning of the model based on the partial tuning task;

evaluate a measured performance associated with the partial tuning of the model and a measured performance associated with the full tuning of the model;

select hyperparameter values associated with the partial tuning of the model based on a determination that the measured performance associated with the partial tuning of the model satisfies a performance value;

determine an importance of a first one of the hyperparameters of the model; and

prioritize the first hyperparameter relative to other ones of the hyperparameters of the model based on the importance of the first hyperparameter.

9. The at least one of the RAM, the ROM, the flash memory, the EEPROM, the optical disc, or the hard drive of claim 8 , wherein the instructions are to cause the programmable circuitry to, based on a budget, determine to use fewer than the second number of datapoints for the partial tuning of the model.

10. The at least one of the RAM, the ROM, the flash memory, the EEPROM, the optical disc, or the hard drive of claim 9 , wherein the instructions are to cause the programmable circuitry to set a tuning cost parameter for the partial tuning of the model.

11. The at least one of the RAM, the ROM, the flash memory, the EEPROM, the optical disc, or the hard drive of claim 8 , wherein the first number of datapoints for the full tuning task is at least 1000 samples, and the second number of datapoints for the partial tuning task is at least 500 samples.

12. The at least one of the RAM, the ROM, the flash memory, the EEPROM, the optical disc, or the hard drive of claim 8 , wherein the model is a Latin Hypercube Sampling (LHS) model.

13. The at least one of the RAM, the ROM, the flash memory, the EEPROM, the optical disc, or the hard drive of claim 12 , wherein the partial tuning refers to a number of interchanges.

14. The at least one of the RAM, the ROM, the flash memory, the EEPROM, the optical disc, or the hard drive of claim 8 , wherein the model performs a dimensionality reduction method.

15. A method to tune hyperparameters of a model based on a tuning work request, the method comprising:

accessing the tuning work request to tune the hyperparameters of the model, the tuning work request to include:

a full tuning task and a corresponding first number of datapoints;

a partial tuning task and a corresponding second number of datapoints, the second number of datapoints being less than the first number of datapoints;

executing, by executing an instruction with programmable circuitry, a full tuning of the model based on the full tuning task;

executing, by the programmable circuitry, a partial tuning of the model based on the partial tuning task;

evaluating, by the programmable circuitry, a measured performance associated with the partial tuning of the model and a measured performance associated with the full tuning of the model; and

selecting, by the programmable circuitry, hyperparameter values associated with the partial tuning of the model based on a determination that the measured performance associated with the partial tuning of the model satisfies a performance value;

determining an importance of a first one of the hyperparameters of the model; and

ranking the first hyperparameter relative to other ones of the hyperparameters of the model based on the importance of the first hyperparameter.

16. The method of claim 15 , including determining, based on a budget, to use fewer than the second number of datapoints for the partial tuning of the model.

17. The method of claim 16 , further including setting a tuning cost parameter for the partial tuning of the model.

18. The method of claim 15 , wherein the first number of datapoints for the full tuning task is at least 1000 samples, and the second number of datapoints for the partial tuning task is at least 500 samples.

19. The method of claim 15 , wherein the model is a Latin Hypercube Sampling (LHS) model.

20. The method of claim 19 , wherein the partial tuning refers to a number of interchanges.

Continuity (3)
Continuation 16511320 · Jul 15, 2019
Provisional Application 62697578 · Jul 13, 2018
Related Publication 20230325672A1 · Oct 12, 2023
References Cited (155)
US 7363281B2 · Jin et al. · 2008 [cited by applicant]
US 8364613B1 · Lin et al. · 2013 [cited by applicant]
US 9786036B2 · Annapureddy · 2017 [cited by applicant]
US 10217061B2 · Hayes et al. · 2019 [cited by applicant]
US 10282237B1 · Johnson et al. · 2019 [cited by applicant]
US 10360517B2 · Koch et al. · 2019 [cited by applicant]
US 10379913B2 · Johnson et al. · 2019 [cited by applicant]
US 10445150B1 · Johnson et al. · 2019 [cited by applicant]
US 10528891B1 · Cheng et al. · 2020 [cited by applicant]
US 10558934B1 · Cheng et al. · 2020 [cited by applicant]
US 10565025B2 · Johnson et al. · 2020 [cited by applicant]
US 10607159B2 · Hayes et al. · 2020 [cited by applicant]
US 10621514B1 · Cheng et al. · 2020 [cited by applicant]
US 10740695B2 · Cheng et al. · 2020 [cited by applicant]
US 11301781B2 · Hayes et al. · 2022 [cited by applicant]
US 11334813B2 · Ura · 2022 [cited by examiner]
US 11704567B2 · McCourt · 2023 [cited by examiner]
US 20070019065A1 · Mizes · 2007 [cited by applicant]
US 20080183648A1 · Goldberg et al. · 2008 [cited by applicant]
US 20090244070A1 · Mattikalli et al. · 2009 [cited by applicant]
US 20100083196A1 · Liu · 2010 [cited by applicant]
US 20150288573A1 · Baughman et al. · 2015 [cited by applicant]
US 20160010657A1 · Kimura et al. · 2016 [cited by applicant]
US 20160110657A1 · Gibiansky et al. · 2016 [cited by applicant]
US 20160132787A1 · Drevo · 2016 [cited by examiner]
US 20160232540A1 · Gao et al. · 2016 [cited by applicant]
US 20170124487A1 · Szeto et al. · 2017 [cited by applicant]
US 20180121797A1 · Prabhu et al. · 2018 [cited by applicant]
US 20180121814A1 · Yu et al. · 2018 [cited by applicant]
US 20180129892A1 · Bahl et al. · 2018 [cited by applicant]
US 20180240041A1 · Koch et al. · 2018 [cited by applicant]
US 20180356949A1 · Wang et al. · 2018 [cited by applicant]
US 20190019108A1 · Eads · 2019 [cited by applicant]
US 20190095785A1 · Sarkar et al. · 2019 [cited by applicant]
US 20190220755A1 · Carbune et al. · 2019 [cited by applicant]
US 20190236487A1 · Huang et al. · 2019 [cited by applicant]
US 20190370684A1 · Gunes et al. · 2019 [cited by applicant]
US 20200019888A1 · Mcourt et al. · 2020 [cited by applicant]
US 20200097847A1 · Convertino · 2020 [cited by examiner]
US 20200151029A1 · Johnson et al. · 2020 [cited by applicant]
US 20200226496A1 · Basu · 2020 [cited by examiner]
US 20200302234A1 · Walters · 2020 [cited by examiner]
US 20200302342A1 · Cheng et al. · 2020 [cited by applicant]
US 20210042656A1 · Watson · 2021 [cited by examiner]
US 20210390466A1 · Varadarajan · 2021 [cited by examiner]
WO 2018213119A1 · 2018 [cited by applicant]
Mockus et al., “The Application of Bayesian Methods for Seeking the Extremum,” Towards Global Optimisation, Elsevier, 1978, 7 pages. [cited by applicant]
O'Hagan, “On Outlier Rejection Phenomena in Bayes Interference,” Journal of the Royal Statistical Society, Series B (Methodological), 1979, 10 pages. [cited by applicant]
Lange et al., “Robust Statistical Modeling Using the t Distribution,” Journal of the American Statistical Association, 1989, 16 pages. [cited by applicant]
O'Hagan, “Some Bayesian Numerical Analysis,” Bayesian Statistics, 1992, 19 pages. [cited by applicant]
Williams, “Simple Statistical Gradient—Following Algorithms for Connectionist Reinforcement Learning,” Machine Learning, 1992, 27 pages. [cited by applicant]
Kennedy et al., “Particle Swarm Optimization,” Proceedings of the 1995 IEEE International Conference on Neural Networks, vol. 4, Australia, IEEE Service Center, NJ, 1995, 7 pages. [cited by applicant]
Moore et al., “Memory-Based Stochastic Optimization,” Advances in Neural Information Processing Systems, vol. 8, The MIT Press, 1996, 7 pages. [cited by applicant]
Neal, “Monte Carlo Implementation of Gaussian Process Models for Bayesian Regression and Classification,” Technical Report 9702, Department of Statistics and Department of Computer Science, University of Toronto, 1997, … [cited by applicant]
Sutton et al., “Reinforcement Learning: An Introduction,” The MIT Press, 1998, 548 pages. [cited by applicant]
Jones et al., “Efficient Global Optimization of Expensive Black-Box Functions,” Journal of Global Optimization, 1998, 38 pages. [cited by applicant]
Williams et al., “Sequential Design of Computer Experiments to Minimize Integrated Response Functions,” Statistica Sinica, 2000, 20 pages. [cited by applicant]
Dwork et al., “Rank Aggregation Methods for the Web,” Proeedings of the 10th International Conference on World Wide Web, ACM, 2001, 28 pages. [cited by applicant]
Santner et al., “The Design and Analysis of Computer Experiments,” Springer-Verlag, 2003, 236 pages. [cited by applicant]
Kohl et al., “Policy Gradient Reinforcement Learning for Fast Quadrupedal Locomotion,” Proceedings of the IEEE International Conference on Robotics & Automation, 2004, 6 pages. [cited by applicant]
Hodge et al., “A Survey of Outlier Detection Methodologies,” Artificial Intelligence Review, Oct. 2004, 43 pages. [cited by applicant]
Rousseeuw et al., “Robust Regression and Outlier Detection,” vol. 589, John Wiley & Sons, 2005, 341 pages. [cited by applicant]
Rasmussen et al., “Gaussian Processes for Machine Learning,” The MIT Press, Cambridge, Massachusetts, 2006, 266 pages. [cited by applicant]
Huang et al., “Global Optimization of Stochastic Black-Box Systems Via Sequential Kriging Meta-Models,” Journal of Global Optimization, 2006, 35 pages. [cited by applicant]
Peters et al., “Policy Gradient Methods for Robotics,” IEEE/RSJ International Conference on Intelligent Robots and Systems, 2006, 7 pages. [cited by applicant]
Demŝar, “Statistical Comparisons of Classifiers Over Multiple Data Sets,” The Journal of Machine Learning Research, 2006, 30 pages. [cited by applicant]
Cook et al., “Creating a Consensus Ranking of Proposals From Reviewers Partial Ordinal Rankings,” Computers & Operations Research, 2007, 22 pages. [cited by applicant]
Lizotte et al., “Automatic Gait Optimization with Gaussian Process Regression,” IJCAI, 2007, 6 pages. [cited by applicant]
Martinez-Cantin et al., “Active Policy Learning for Robot Planning and Exploration under Uncertainty,” Robotics: Science and Systems, 2007, 8 pages. [cited by applicant]
Zhou et al., “Combining Global and Local Surrogate Models to Accelerate Evolutionary Optimization,” IEEE Transactions on Systems, Man, and Cybernetics, Part C, vol. 37, Issue 1, Jan. 2007, 11 pages. [cited by applicant]
Brochu et al., “Active Preference Learning with Discrete Choice Data,” Advances in Neural Information Processing Systems, 2008, 8 pages. [cited by applicant]
Ginsbourger et al., “A Multi-Points Criterion for Deterministic Parallel Global Optimization Based on Gaussian Processes,” Technical Report, Mar. 2008, 30 pages. [cited by applicant]
Vanhatalo et al., “Gaussian Process Regression with Student-t Likelihood,” Advances in Neural Information Processing Systems 22, 2009, 9 pages. [cited by applicant]
Martinez-Cantin et al., “A Bayesian Exploration-Exploitation Approach for Optimal Online Sensing and Planning with a Visually Guided Mobile Robot,” Autonomous Robots—Special Issue on Robot Learning, 2009, 10 pages. [cited by applicant]
Dick et al., “Digital Nets and Sequences: Discrepancy Theory and Quasi-Monte Carlo Integration,” Cambridge University Press, 2010, 627 pages. [cited by applicant]
Korolev et al., “On the Upper Bound for the Absolute Constant in the Berry-Esseen Inequality,” Theory of Probability & Its Applications, 2010, 21 pages. [cited by applicant]
Brochu et al., “A Bayesian Interactive Optimization Approach to Procedural Animation Design,” Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Eurographics Association, 2010, 10 pages. [cited by applicant]
Okuma et al., “An Adaptive Interface for Active Localization,” International Conference on Computer Vision Theory and Applications, 2011, 12 pages. [cited by applicant]
Hutter et al., “SMAC: Sequential Model-Based Algorithm Configuration,” http://www.cs.ubc.ca/labs/beta/Projects/SMAC/, 2011, 1 page. [cited by applicant]
Bergstra et al., “Algorithms for Hyper-Parameter Optimization,” Advances in Neural Information Processing Systems. 2011, 9 pages. [cited by applicant]
Hutter et al., “Sequential Model-Based Optimization for General Algorithm Configuration,” Proceedings of Learning and Intelligent Optimization, 2011, 15 pages. [cited by applicant]
Tesch et al., “Adapting Control Policies for Expensive Systems to Changing Environments,” IEEE/RSJ International Conference on Intelligent Robots and Systems, 2011, 5 pages. [cited by applicant]
Jylänki et al., “Robust Gaussian Process Regression with a Student-t Likelihood,” Journal of Machine Learning Research, Nov. 2011, 31 pages. [cited by applicant]
Kulesza et al., “Determinantal Point Processes for Machine Learning,” Foundations and Trends in Machine Learning, 2012, 120 pages. [cited by applicant]
Hennig et al., “Entropy Search for Information-Efficient Global Optimization,” Journal of Machine Learning Research, 2012, 29 pages. [cited by applicant]
Bergstra et al., “Random Search for Hyper-Parameter Optimization,” The Journal of Machine Learning Research, 2012, 25 pages. [cited by applicant]
Snoek et al., “Practical Bayesian Optimization of Machine Learning Algorithms,” Advances in Neural Information Processing Systems, 2012, 12 pages. [cited by applicant]
Bergstra et al., “Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures,” International Conference on Machine Learning, 2013, 9 pages. [cited by applicant]
Sutskever et al., “On the Importance of Initialization and Momentum in Deep Learning,” Proceedings of the 30th International Conference on Machine Learning, 2013, 14 pages. [cited by applicant]
Gavana, “AMPGO Global Optimization Benchmark Functions,” https://github.com/andyfaff/ampgo, 2013, 2 pages. [cited by applicant]
Eggensperger et al., “Towards an Empirical Foundation for Assessing Bayesian Optimization of Hyperparameters,” NIPS Workshop on Bayesian Optimization in Theory and Practice, 2013, 5 pages. [cited by applicant]
Kuindersma et al., “Variable Risk Control Via Stochastic Optimization,” The International Journal of Robotics Research, 2013, 29 pages. [cited by applicant]
Deinsroth et al., “A Survey on Policy Search for Robotics,” Foundations and Trends in Robotics, 2013, 147 pages. [cited by applicant]
Levine et al., “Learning Neural Network Policies with Guided Policy Search Under Unknown Dynamics,” Advances in Neural Information Processing Systems, 2014, 9 pages. [cited by applicant]
Marchant et al., “Bayesian Optimisation for Informative Continuous Path Planning,” IEEE International Conference on Robotics and Automation (ICRA), 2014, 5 pages. [cited by applicant]
Mehmani, “A Novel Approach to Simultaneous Selection of Surrogate Models, Constitutive Kernels, and Hyper-parameter Values,” 10th AIAA Multidisciplinary Design Optimization Conference, Jan. 13-17, 2014, 18 pages. [cited by applicant]
United States Patent and Trademark Office, “Final Office Action,” issued in connection with U.S. Appl. No. 17/561,480, dated Mar. 26, 2024, 36 pages. [cited by applicant]
United States Patent and Trademark Office, “Advisory Action,” issued in connection with U.S. Appl. No. 17/561,480, dated Jun. 3, 2024, 2 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance,” issued in U.S. Appl. No. 16/511,320, on Mar. 15, 2023, 6 pages. [cited by applicant]
United States Patent and Trademark Office, “Corrected Notice of Allowability,” issued in U.S. Appl. No. 16/511,320, on Mar. 23, 2023, 3 pages. [cited by applicant]
European Patent Office, “Communication Pursuant to Article 94(3) EPC,” issued in connection with European Application No. 18801697.6, dated Apr. 5, 2023, 5 pages. [cited by applicant]
United States Patent and Trademark Office, “Corrected Notice of Allowability,” issued in U.S. Appl. No. 16/511,320, on May 15, 2023, 2 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 17/561,480, dated Oct. 18, 2023, 23 pages. [cited by applicant]
Martinez-Cantin, “Bayesopt: A Bayesian Optimization Library for Nonlinear Optimization, Experimental Design and Bandits,” Journal of Machine Learning Research, 2014, 5 pages. [cited by applicant]
Shah et al., “Student-t Processes as Alternatives to Gaussian Processes,” Artificial Intelligence and Statistics, 2014, 9 pages. [cited by applicant]
Snoek et al., “Input Warping for Bayesian Optimization of Non-Stationary Functions,” International Conference on Machine Learning, 2014, 11 pages. [cited by applicant]
Lee, “PYSWARM: Particle Swarm Optimization (PSO) with Constraint Support,” https://github.com/tisimet/pyswarm, 2014, 7 pages. [cited by applicant]
Guenther et al., “Finding and Choosing Among Multiple Optima,” Applied Mathematics, 2014, 18 pages. [cited by applicant]
Gelman et al., “Bayesian Data Analysis,” CRC Press, FL, vol. 2, 2014, 573 pages. [cited by applicant]
Bergstra et al., Preliminary Evaluation of Hyperopt Algorithms on HPOLib, ICML Workshop on AutoML, 2014, 7 pages. [cited by applicant]
Snoek et al., “Spearmint: Bayesian Optimization Software,” https://github.com/HIPS/Spearmint, 2014, 5 pages. [cited by applicant]
Clark et al., “MOE: A Global Black Box Optimization Engine for Real World Metric Optimization,” https://github.com/Yelp/MOE, 2014, 6 pages. [cited by applicant]
Falkner, “PYSMAC: Simple Python Interface to SMAC,” https://github.com/automl/pysmac, 2014, 3 pages. [cited by applicant]
Hernández-Lobato et al., “Predictive Entropy Search for Bayesian Optimization with Unknown Constraints,” Proceedings of the 32nd international Conference on Machine Learning, 2015, 9 pages. [cited by applicant]
González et al., “GLASSES: Relieving the Myopia of Bayesian Optimisation,” NIPS Workshop on Bayesian Optimization, 2015, 12 pages. [cited by applicant]
Wong et al., “Evolutionary Multimodal Optimization: A Short Survey,” CoRR, abs/1508.00457, URL:https://arxiv.org/abs/1508.00457, 2015, 17 pages. [cited by applicant]
Snoek et al., “Scalable Bayesian Optimization Using Deep Neural networks,” International Conference on Machine Learning, 2015, 13 pages. [cited by applicant]
Fasshauer et al., “Kernel-Based Approximation Methods Using Matlab,” World Scientific Publishing Co. Inc., vol. 19, 2015, 23 pages. [cited by applicant]
Eggensperger et al., “Efficient Benchmarking of Hyperparameter Optimizers Via Surrogates,” Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015, 7 pages. [cited by applicant]
Calandra et al., “Bayesian Optimization for Leaning Gaits Under Uncertainty,” Annals of Mathematics and Artificial Intelligence (AMAI), 2015, 19 pages. [cited by applicant]
Cully et al., “Robots That Can Adapt Like Animals,” Nature, 2015, 26 pages. [cited by applicant]
Daniel et al., “Active Reward Learning with a Novel Acquisition Function,” Autonomous Robots, 2015, 17 pages. [cited by applicant]
Bergstra et al., “Hyperopt: A Python Library for Model Selection and Hyperparameter Optimization”, Computational Science & Discovery, Aug. 2015, 25 pages. [cited by applicant]
Nogueira et al., “Unscented Bayesian Optimization for Safe Robot Grasping,” Proeedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, 2016, 16 pages. [cited by applicant]
González et al., “Bayesian Optimisation with Pairwise Preferential Returns,” NIPS Workshop in Bayesian Optimization, 2016, 5 pages. [cited by applicant]
Tang et al., “Student-t Process Regression with Dependent Student-t Noise,” European Conference on Artificial Intelligence, 2016, 8 pages. [cited by applicant]
Kaelbling et al., Implicit Belief-Space Pre-Images for Hierarchical Planning and Execution. IEEE Conference on Robotics and Automation, ICRA, 2016, 8 pages. [cited by applicant]
Dewancker et al., Interactive Preference Learning of Utility Functions for Multi-Objective Optimization, NIPS Future of Interactive Learning Machines Workshop, 2016, 8 pages. [cited by applicant]
Nguyen et al., “Think Globally, Act Locally: A Local Strategy for Bayesian Optimization,” NIPS Workshop on Bayesian Optimization, 2016, 5 pages. [cited by applicant]
McCourt, “Optimization Test Functions,” https://github.com/sigopt/evalset, 2016, 2 pages. [cited by applicant]
Springenberg et al., “Bayesian Optimization with Robust Bayesian Neural Networks,” Advances in Neural Information Processing Systems, 2016, 9 pages. [cited by applicant]
Shahriari et al., “Taking the Human Out of the Loop: A Review of Bayesian Optimization,” Proceedings of the IEEE, 2016, 24 pages. [cited by applicant]
Wang et al., “Max-Value Entropy Search for Efficient Bayesian Optimization,” Proceedings of the 34th International Conference on Machine Learning, vol. 70, 2017, 12 pages. [cited by applicant]
Chollet, “Variational Autoencoder with Keras,” https://github.com/fchollet/keras/blob/master/examples/variational_{} autoencoder.py, 2017, 18 pages. [cited by applicant]
Tang et al., “Student-t Process Regression with Student-t Likelihood,” Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, 2017, 7 pages. [cited by applicant]
Atkeson, “Using Function to Find Policies: Walking,” http://www.cs.cmu.edu/˜cga/dynopt/ass2/, 2017, 3 pages. [cited by applicant]
Martinez-Cantin, “Bayesian Optimization with Adaptive Kernels for Robot Control,” IEEE International conference on Robotics and Automation, 2017, 7 pages. [cited by applicant]
Thatte et al., “A Sample-Efficient Black-Box Optimizer to Train Policies for Human-in-the-Loop Systems with User Preferences,” IEEE Robotics and Automation Letters, 2017, 8 pages. [cited by applicant]
Klein et al., “Learning Curve Prediction with Bayesian Neural Networks,” International Conference on Learning Representations (ICLR), 2017 Conference Track, Apr. 2017, 16 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 15/977,168, on Jul. 5, 2018, 13 pages. [cited by applicant]
International Searching Authority, “International Search Report,” issued in connection with International Patent Application No. PCT/US2018/032248, dated Jul. 30, 2018, 2 pages. [cited by applicant]
International Searching Authority, “Written Opinion of the International Searching Authority,” issued in connection with International Patent Application No. PCT/US2018/032248, dated Jul. 30, 2018, 7 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance,” issued in connection with U.S. Appl. No. 15/977,168, on Nov. 28, 2018, 5 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action” issued in U.S. Appl. No. 16/243,361, on Sep. 11, 2019, 10 pages. [cited by applicant]
International Searching Authority, “International Preliminary Report on Patentability,” issued in connection with PCT No. PCT/US2018/032248, Nov. 19, 2019, 8 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance,” issued in U.S. Appl. No. 16/243,361 on Dec. 31, 2019, 7 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 16/796,489, on Aug. 25, 2020, 10 pages. [cited by applicant]
United States Patent and Trademark Office, “Final Office Action,” issued in connection with U.S. Appl. No. 16/796,489, on Mar. 12, 2021, 10 pages. [cited by applicant]
European Patent Office, “European Search Report” issued in connection with European patent application No. 18801697.6, Mar. 24, 2021, 9 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance,” issued in connection with U.S. Appl. No. 16/796,489, on Sep. 9, 2021, 5 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance,” issued in connection with U.S. Appl. No. 16/796,489 on Dec. 7, 2021, 5 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in U.S. Appl. No. 16/511,320, on Jun. 14, 2022, 15 pages. [cited by applicant]
United States Patent and Trademark Office, “Final Office Action,” issued in U.S. Appl. No. 16/511,320, on Dec. 2, 2022, 17 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 17/561,480, dated Jul. 24, 2024, 5 pages. [cited by applicant]
European Patent Office, “Communication pursuant to Article 94(3) EPC,” issued in connection with European Patent Application No. 18801697.6, dated Sep. 25, 2024, 4 pages. [cited by applicant]