IP Library › Granted Patent US 12,423,595
Granted Patent B2
US 12,423,595 · App. 16/682,813 · Granted Sep 23, 2025

Systems for time-series predictive data analytics, and related methods and apparatus

Inventors: Jeremy Achin (Boston, MA); Thomas DeGodoy (Medford, MA); Xavier Conort (Singapore, SG); Mark L. Steadman (Watertown, MA); Peter Prettenhofer (Vienna, AT); Timothy Owen (Newton, MA)
Assignee: DataRobot, Inc.
G06N5/04G06F9/5011G06N5/02G06N5/046G06N20/00G06N20/10G06N20/20G06Q10/04G06Q10/06Y02P90/80
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,423,595
App. No.
16/682,813
Granted
Sep 23, 2025
Kind
B2
Abstract

A predictive modeling method may include determining a time interval of time-series data; identifying one or more variables of the data as targets; determining a forecast range and a skip range associated with a prediction problem represented by the data; generating training data and testing data from the time-series data; fitting a predictive model to the training data; and testing the fitted model on the testing data. The forecast range may indicate a duration of a period for which values of the targets are to be predicted. The skip range may indicate a temporal lag between the time period corresponding to the data used to make predictions and the time period corresponding to the predictions. The skip range may separate input data subsets representing model inputs from subsets representing model outputs, and separate test data subsets representing model inputs from subsets representing validation data.

Claims (70)

1. A computer-implemented modeling method comprising:

obtaining a plurality of data sets, wherein each data set includes a plurality of observations, wherein each observation includes an indication of a time of the observation and values of one or more variables, wherein at least one of the variables is a target;

setting values of a plurality of temporal parameters of a modeling procedure, the plurality of temporal parameters including a forecast range parameter and a skip range parameter, wherein the forecast range parameter indicates a duration of a period of time for prediction of one or more values of the target, and wherein the skip range parameter indicates a duration of a time period between a time of an earliest-in-time prediction in the forecast range and a time of a latest-in-time observation upon which the earliest-in-time prediction in the forecast range is based;

segmenting at least one of the data sets into training-input data and training-output data based, at least in part, on the values of the forecast range parameter and the skip range parameter, wherein the training-input data include a first subset of the observations of the at least one data set, the training-output data include a second subset of the observations of the at least one data set, a range of the times of the observations in the second subset matches the value of the forecast range parameter, and a range from a latest-in-time observation in the first subset to an earliest-in-time observation in the second subset matches the value of the skip range parameter; and

adapting a model to solve a prediction problem represented by the plurality of data sets and the values of the forecast range and skip range parameters, including training the model using the training-input data and the training-output data.

2. The method of claim 1 , wherein at least a group of the observations of the plurality of data sets include respective values of a first variable, and wherein the method further comprises, prior to training the model using the training-input data and the training-output data:

determining that the values of the first variable comprise time values;

for each observation in the group, generating a respective value of a second variable, wherein the respective value of the second variable comprises an offset between the respective time value of the first variable and a reference time value; and

adding the respective values of the second variable to the respective observations in the group.

3. The method of claim 2 , further comprising: removing the values of the first variable from the observations in the group.

4. The method of claim 1 , further comprising determining the value of the forecast range parameter based, at least in part, on (1) a time interval of the plurality of data sets, (2) a number of observations included in the plurality of data sets, (3) a time period corresponding to the plurality of data sets, and/or (4) a natural time period selected from the group consisting of microseconds, milliseconds, seconds, minutes, hours, days, weeks, months, quarters, seasons, years, decades, centuries, and millennia.

5. The method of claim 1 , further comprising determining the value of the skip range parameter based, at least in part, on latency in collection of the plurality of data sets, latency in communication of the plurality of data sets, latency in analyzing the plurality of data sets, latency in communication of analyses of the plurality of data sets and/or latency of implementing actions based on the analyses of the plurality of data sets.

6. The method of claim 1 , further comprising:

selecting validation data from the plurality of data sets based on the value of the forecast range parameter and the value of the skip range parameter, wherein the validation data include a subset of observations of at least one of the data sets; and

performing cross-validation of the model using the validation data.

7. The method of claim 6 , wherein the model is trained and/or cross-validated using sliding windows, wherein the sliding windows correspond to sequential subsets of observations from at least one of the data sets.

8. The method of claim 1 , further comprising:

selecting validation data from the plurality of data sets based on the value of the forecast range parameter and the value of the skip range parameter, wherein the validation data include a subset of observations of at least one of the data sets; and

performing nested cross-validation of the model using the validation data.

9. The method of claim 1 , wherein the indication of the time of each observation comprises a time at which the respective observation was made or another time associated with the respective observation.

10. The method of claim 1 , wherein the one or more variables include a first variable and a second variable, and wherein the method further comprises:

determining that changes in the values of the first and second variables are correlated with a duration of a time period between the changes in the value of the first variable and the correlated changes in the value of the second variable; and

displaying, via a graphical user interface, graphical content indicating the duration of the time period between the changes in the value of the first variable and the correlated changes in the value of the second variable.

11. The method of claim 1 , further comprising:

determining suitabilities of a plurality of modeling procedures for the prediction problem based, at least in part, on characteristics of the prediction problem and/or on attributes of the respective modeling procedures;

selecting one or more modeling procedures from the plurality of modeling procedures based on the determined suitabilities of the selected modeling procedures for the prediction problem; and

performing the one or more modeling procedures.

12. The method of claim 11 , wherein performing the one or more modeling procedures comprises:

transmitting instructions to a plurality of processing nodes, the instructions comprising a resource allocation schedule allocating resources of the processing nodes for execution of the one or more modeling procedures, the resource allocation schedule being based, at least in part, on the suitabilities of the one or more modeling procedures for the prediction problem;

receiving results of the execution of the one or more modeling procedures by the plurality of processing nodes in accordance with the resource allocation schedule, wherein the results include one or more models generated by the one or more modeling procedures, and/or scores of the generated one or more models for data associated with the prediction problem; and

selecting, from the generated one or more models, a particular model based, at least in part, on the score of the particular model.

13. The method of claim 1 , further comprising deploying the trained model, wherein the trained model is deployed to one or more servers, wherein other trained models are also deployed to the one or more servers, and wherein requests to the trained model and the other trained models are allocated among the servers based, at least in part, on (1) an estimate of an amount of time used by each of the trained models to generate a response, and/or (2) an estimate of a frequency with which requests for each of the trained models are received.

14. The method of claim 13 , wherein each request is assigned to a respective thread, wherein each request has an associated latency-sensitivity value, and wherein a number of threads executing on a particular server is determined based, at least in part, on the latency-sensitivity values of the threads executing on the particular server.

15. The method of claim 1 , further comprising:

determining a value of a metric that indicates an interaction strength between two or more non-target variables included in the plurality of data sets; and

responsive to the value of the metric exceeding a threshold value, generating values of a new non-target variable based on the values of the two or more non-target variables and adding the new non-target variable to the plurality of data sets.

16. An apparatus comprising:

a memory configured to store a machine-executable module encoding a modeling procedure, and

at least one processor configured to execute the machine-executable module, wherein executing the machine-executable module causes the apparatus to perform the modeling procedure, including:

obtaining a plurality of data sets, wherein each data set includes a plurality of observations, wherein each observation includes an indication of a time of the observation and values of one or more variables, wherein at least one of the variables is a target;

setting values of a plurality of temporal parameters of the modeling procedure, the plurality of temporal parameters including a forecast range parameter and a skip range parameter, wherein the forecast range parameter indicates a duration of a period of time for prediction of one or more values of the target, and wherein the skip range parameter indicates a duration of a time period between a time of an earliest-in-time prediction in the forecast range and a time of a latest-in-time observation upon which the earliest-in-time prediction in the forecast range is based;

segmenting at least one of the data sets into training-input data and training-output data based, at least in part, on the values of the forecast range parameter and the skip range parameter, wherein the training-input data include a first subset of the observations of the at least one data set, the training-output data include a second subset of the observations of the at least one data set, a range of the times of the observations in the second subset matches the value of the forecast range parameter, and a range from a latest-in-time observation in the first subset to an earliest-in-time observation in the second subset matches the value of the skip range parameter, and

adapting a model to solve a prediction problem represented by the plurality of data sets and the values of the forecast range and skip range parameters, including training the model using the training-input data and the training-output data.

17. The apparatus of claim 16 , wherein at least a group of the observations of the plurality of data sets include respective values of a first variable, and wherein the modeling procedure further includes, prior to training the model using the training-input data and the training-output data:

determining that the values of the first variable comprise time values;

for each observation in the group, generating a respective value of a second variable, wherein the respective value of the second variable comprises an offset between the respective time value of the first variable and a reference time value; and

adding the respective values of the second variable to the respective observations in the group.

18. The apparatus of claim 16 , wherein performing the modeling procedure further includes determining the value of the forecast range parameter based, at least in part, on (1) a time interval of the plurality of data sets, (2) a number of observations in the pluralities of observations included in the plurality of data sets, (3) a time period corresponding to the plurality of data sets, and/or (4) a natural time period selected from the group consisting of microseconds, milliseconds, seconds, minutes, hours, days, weeks, months, quarters, seasons, years, decades, centuries, and millennia.

19. The apparatus of claim 16 , wherein performing the modeling procedure further includes determining the value of the skip range parameter based, at least in part, on latency in collection of the plurality of data sets, latency in communication of the plurality of data sets, latency in analyzing the plurality of data sets, latency in communication of analyses of the plurality of data sets and/or latency of implementing actions based on the analyses of the plurality of data sets.

20. The apparatus of claim 16 , wherein the one or more variables include a first variable and a second variable, and wherein the modeling procedure further includes:

determining that changes in the values of the first and second variables are correlated with a duration of a time period between the changes in the value of the first variable and the correlated changes in the value of the second variable; and

displaying, via a graphical user interface, graphical content indicating the duration of the time period between the changes in the value of the first variable and the correlated changes in the value of the second variable.

21. The apparatus of claim 16 , wherein the modeling procedure further includes:

determining suitabilities of a plurality of modeling procedures for the prediction problem based, at least in part, on characteristics of the prediction problem and/or on attributes of the respective modeling procedures;

selecting one or more modeling procedures from the plurality of modeling procedures based on the determined suitabilities of the selected modeling procedures for the prediction problem; and

performing the one or more modeling procedures.

22. The apparatus of claim 21 , wherein performing the one or more modeling procedures comprises:

transmitting instructions to a plurality of processing nodes, the instructions comprising a resource allocation schedule allocating resources of the processing nodes for execution of the one or more modeling procedures, the resource allocation schedule being based, at least in part, on the suitabilities of the one or more modeling procedures for the prediction problem;

receiving results of the execution of the one or more modeling procedures by the plurality of processing nodes in accordance with the resource allocation schedule, wherein the results include one or more models generated by the one or more modeling procedures, and/or scores of the generated one or more models for data associated with the prediction problem; and

selecting, from the generated one or more models, a particular model based, at least in part, on the score of the particular model.

23. The apparatus of claim 16 , wherein the modeling procedure further includes deploying the trained model, wherein the trained model is deployed to one or more servers, wherein other trained models are also deployed to the one or more servers, and wherein requests to the trained model and the other trained models are allocated among the servers based, at least in part, on (1) an estimate of an amount of time used by each of the trained models to generate response, and/or (2) an estimate of a frequency with which requests for each of the trained models are received.

24. The apparatus of claim 23 , wherein each request is assigned to a respective thread, wherein each request has an associated latency-sensitivity value, and wherein a number of threads executing on a particular server is determined based, at least in part, on the latency-sensitivity values of the threads executing on the particular server.

25. The apparatus of claim 16 , wherein the modeling procedure further includes:

determining a value of a metric that indicates an interaction strength between two or more non-target variables included in the plurality of data sets; and

responsive to the value of the metric exceeding a threshold value, generating values of a new non-target variable based on the values of the two or more non-target variables and adding the new non-target variable to the plurality of data sets.

26. A non-transitory computer-readable storage medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform operations including:

obtaining a plurality of data sets, wherein each data set includes a plurality of observations, wherein each observation includes an indication of a time of the observation and values of one or more variables, wherein at least one of the variables is a target;

setting values of a plurality of temporal parameters of a modeling procedure, the plurality of temporal parameters including a forecast range parameter and a skip range parameter, wherein the forecast range parameter indicates a duration of a period of time for prediction of one or more values of the target, and wherein the skip range parameter indicates a duration of a time period between a time of an earliest-in-time prediction in the forecast range and a time of a latest-in-time observation upon which the earliest-in-time prediction in the forecast range is based;

segmenting at least one of the data sets into training-input data and training-output data based, at least in part, on the values of the forecast range parameter and the skip range parameter, wherein the training-input data include a first subset of the observations of the at least one data set, the training-output data include a second subset of the observations of the at least one data set, a range of the times of the observations in the second subset matches the value of the forecast range parameter, and a range from a latest-in-time observation in the first subset to an earliest-in-time observation in the second subset matches the value of the skip range parameter; and

adapting a model to solve a prediction problem represented by the plurality of data sets and the values of the forecast range and skip range parameters, including training the model using the training-input data and the training-output data.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 15, 2025
From: ACHIN, JEREMY; DEGODOY, THOMAS; OWEN, TIMOTHY; CONORT, XAVIER; YURGENSON, SERGEY; STEADMAN, MARK L.; KOUNDRY, GLEN; PRETTENHOFER, PETER
To: DATAROBOT, INC.
Reel/Frame 070847/0903 →
RELEASE OF SECURITY INTEREST Recorded Apr 7, 2025
From: CITIBANK, N.A.
To: DATAROBOT, INC.; ALGORITHMIA, INC.; DULLES RESEARCH, LLC
Reel/Frame 070750/0866 →
SECURITY INTEREST Recorded Mar 22, 2023
From: DATAROBOT, INC.; ALGORITHMIA, INC.; DULLES RESEARCH, LLC
To: CITIBANK, N.A.
Reel/Frame 063263/0926 →
Continuity (7)
Continuation 15790803 · Oct 23, 2017
Continuation In Part 15331797 · Oct 21, 2016
Continuation In Part 15217626 · Jul 22, 2016
Continuation 14720079 · May 22, 2015
Provisional Application 62411526 · Oct 21, 2016
Provisional Application 62002469 · May 23, 2014
Related Publication 20200257992A1 · Aug 13, 2020
References Cited (217)
US 5123057A · Verly et al. · 1992 [cited by applicant]
US 5761442A · Barr et al. · 1998 [cited by applicant]
US 7072863B1 · Phillips et al. · 2006 [cited by applicant]
US 7117185B1 · Aliferis et al. · 2006 [cited by applicant]
US 7580852B2 · Ouimet et al. · 2009 [cited by applicant]
US 8024216B2 · Aronowich et al. · 2011 [cited by applicant]
US 8108243B2 · Solotorevsky et al. · 2012 [cited by applicant]
US 8180664B2 · Shan · 2012 [cited by applicant]
US 8280903B2 · Broder et al. · 2012 [cited by applicant]
US 8370280B1 · Lin et al. · 2013 [cited by applicant]
US 8645421B2 · Meric et al. · 2014 [cited by applicant]
US 8676629B2 · Chien et al. · 2014 [cited by applicant]
US 8682709B2 · Coldren et al. · 2014 [cited by applicant]
US 8751273B2 · Pinto et al. · 2014 [cited by applicant]
US 8782037B1 · Barad et al. · 2014 [cited by applicant]
US 8799009B2 · Mellin et al. · 2014 [cited by applicant]
US 8843427B1 · Lin et al. · 2014 [cited by applicant]
US 8924269B1 · Seubert et al. · 2014 [cited by applicant]
US 9489630B2 · Achin et al. · 2016 [cited by applicant]
US 9495641B2 · Schmidt · 2016 [cited by applicant]
US 9524473B2 · Schmidt · 2016 [cited by applicant]
US 9652714B2 · Achin et al. · 2017 [cited by applicant]
US 9659254B2 · Achin et al. · 2017 [cited by applicant]
US 10102483B2 · Schmidt · 2018 [cited by applicant]
US 10318874B1 · Duncan et al. · 2019 [cited by applicant]
US 10366335B2 · Schmidt · 2019 [cited by applicant]
US 10366346B2 · Achin et al. · 2019 [cited by applicant]
US 10387900B2 · Bledsoe et al. · 2019 [cited by applicant]
US 10496927B2 · Achin et al. · 2019 [cited by applicant]
US 10558924B2 · Achin et al. · 2020 [cited by applicant]
US 10984367B2 · Achin et al. · 2021 [cited by applicant]
US 20020144178A1 · Castelli et al. · 2002 [cited by applicant]
US 20040030777A1 · Reedy et al. · 2004 [cited by applicant]
US 20040054997A1 · Katragadda et al. · 2004 [cited by applicant]
US 20050154861A1 · Arimilli et al. · 2005 [cited by applicant]
US 20050183073A1 · Reynolds · 2005 [cited by applicant]
US 20050192937A1 · Barsness et al. · 2005 [cited by applicant]
US 20050234762A1 · Pinto et al. · 2005 [cited by applicant]
US 20060101014A1 · Forman et al. · 2006 [cited by applicant]
US 20060190285A1 · Harris et al. · 2006 [cited by applicant]
US 20070133848A1 · McNutt et al. · 2007 [cited by applicant]
US 20080059284A1 · Solotorevsky et al. · 2008 [cited by applicant]
US 20080097802A1 · Ladde et al. · 2008 [cited by applicant]
US 20080307399A1 · Zhou et al. · 2008 [cited by applicant]
US 20090138286A1 · Linder et al. · 2009 [cited by applicant]
US 20100049340A1 · Smits et al. · 2010 [cited by applicant]
US 20100100877A1 · Greene et al. · 2010 [cited by applicant]
US 20100131314A1 · Lo Yuk Ting et al. · 2010 [cited by applicant]
US 20100312718A1 · Rosenthal et al. · 2010 [cited by applicant]
US 20110072138A1 · Canturk et al. · 2011 [cited by applicant]
US 20110119100A1 · Ruhl et al. · 2011 [cited by applicant]
US 20110296406A1 · Bhandari et al. · 2011 [cited by applicant]
US 20120078678A1 · Pradhan et al. · 2012 [cited by applicant]
US 20120144325A1 · Mital et al. · 2012 [cited by applicant]
US 20120151062A1 · Gusev et al. · 2012 [cited by applicant]
US 20120192051A1 · Rothschiller et al. · 2012 [cited by applicant]
US 20120202240A1 · Deigner et al. · 2012 [cited by applicant]
US 20130073061A1 · Mu et al. · 2013 [cited by applicant]
US 20130096892A1 · Essa et al. · 2013 [cited by applicant]
US 20130290226A1 · Dokken · 2013 [cited by applicant]
US 20140052882A1 · Waters et al. · 2014 [cited by applicant]
US 20140074829A1 · Schmidt · 2014 [cited by applicant]
US 20140136452A1 · Wellman et al. · 2014 [cited by applicant]
US 20140172773A1 · Schmidt · 2014 [cited by applicant]
US 20140258189A1 · Schmidt · 2014 [cited by applicant]
US 20140359560A1 · Avadhanula et al. · 2014 [cited by applicant]
US 20140372172A1 · Martinez et al. · 2014 [cited by applicant]
US 20150088606A1 · Tyagi · 2015 [cited by applicant]
US 20150154619A1 · Grichnik et al. · 2015 [cited by applicant]
US 20150170049A1 · Mann et al. · 2015 [cited by applicant]
US 20150317589A1 · Anderson et al. · 2015 [cited by applicant]
US 20150339572A1 · Achin et al. · 2015 [cited by applicant]
US 20150356576A1 · Malaviya et al. · 2015 [cited by applicant]
US 20160005055A1 · Sarferaz · 2016 [cited by applicant]
US 20160048766A1 · McMahon et al. · 2016 [cited by applicant]
US 20160217390A1 · Shoaib et al. · 2016 [cited by applicant]
US 20160301624A1 · Gonzalez et al. · 2016 [cited by applicant]
US 20160335550A1 · Achin et al. · 2016 [cited by applicant]
US 20160364647A1 · Achin et al. · 2016 [cited by applicant]
US 20160379244A1 · Kalish et al. · 2016 [cited by applicant]
US 20170193398A1 · Schmidt · 2017 [cited by applicant]
US 20170243140A1 · Achin et al. · 2017 [cited by applicant]
US 20170330078A1 · Koduru · 2017 [cited by applicant]
US 20180046926A1 · Achin et al. · 2018 [cited by applicant]
US 20180060738A1 · Achin et al. · 2018 [cited by applicant]
US 20180060744A1 · Achin et al. · 2018 [cited by applicant]
US 20180300737A1 · Bledsoe et al. · 2018 [cited by applicant]
US 20190220772A1 · Schmidt · 2019 [cited by applicant]
US 20200090075A1 · Achin et al. · 2020 [cited by applicant]
US 20200134489A1 · Achin et al. · 2020 [cited by applicant]
US 20200257992A1 · Achin et al. · 2020 [cited by applicant]
CN 101226484A · 2008 [cited by applicant]
CN 104281492A · 2015 [cited by applicant]
JP 05081228A · 1991 [cited by applicant]
JP 2005135287A · 2005 [cited by applicant]
KR 101369261B1 · 2014 [cited by applicant]
WO 2007033203A2 · 2007 [cited by applicant]
WO WO2010044683A1 · 2010 [cited by applicant]
WO WO2015179778A1 · 2015 [cited by applicant]
“Machine Learning Techniques for Multimedia” Cord et al (Year: 2008). [cited by examiner]
A. Lorbert et al., Descent Methods for Tuning Parameter Refinement, Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS), 2010, pp. 469-476. [cited by applicant]
A. Rudi et al., Adaptive Optimization for Cross Validation, Proceedings of the European Symposium on Artificial Neural Networks, Computational Intelligence, and Machine Learning, 2012, pp. 435-440. [cited by applicant]
B. Boukhatem et al., Predicting concrete properties using neural networks (NN) with principal component analysis (PCA) technique, Computers and Concrete, vol. 10, No. 6, 2012, pp. 1-17. [cited by applicant]
B. Flyvbjerg et al., What Causes Cost Overruns in Transport Infrastructure Projects?, Transport Reviews, vol. 24, 2004, pp. 1-40. [cited by applicant]
B.K. Behera et al., Aabric Quality Evaluation By Objective Measurement, Indian Journal of Fibre and Textile Research, vol. 19, 1994, pp. 168-171. [cited by applicant]
Barresse, Microsoft Excel 2013—Flash Fill. Microsoft Excel and Access Experts Blog. Jul. 28, 2012. retrieved from https://excelandaccess.wordpress.com/2012/07/28/excel-2013-flash-fill/[Mar. 3, 2016 1:45:25PM]. [cited by applicant]
C. Bordat et al., An Analysis of Cost Overruns and Time Delays of INDOT Projects, Final Report, FHWA/IN/JTRP-2004/7, Joint Transportation Research Program, Purdue University, 2004, 191 pages. [cited by applicant]
Ekart et al. 2000. A Metric for Genetic Programs and Fitness Sharing. Genetic Programming. Springer Berlin Heidelberg. EuroGP 2000, LNCS 1802. 2000. 259-70. [cited by applicant]
Ensemble Learning; https://en.wikipedia.org/wiki/Ensemble_learing; Creative Commons Attribution—ShareAlike License, Aug. 2, 2017; pp. 1-8. [cited by applicant]
F. Xiao et al., Prediction of Fatigue Life of Rubberized Asphalt Concrete Mixtures Containing Reclaimed Asphalt Pavement Using Artificial Neural Networks, Journal of Materials in Civil Engineering, 2007, 41 pages. [cited by applicant]
G. Biau et al., COBRA: A Nonlinear Aggregation Strategy, Cornell University Library, Nov. 2013, 40 pages. [cited by applicant]
G. Bortolin et al., On modeling of curl in multi-ply paperboard, Journal of Process Control, vol. 16, 2006, pp. 419-429. [cited by applicant]
G. Bortolin, On Modeling and Estimation of Curl and Twist in Multi-ply Paperboard, Licentiate Thesis, Optimization and Systems Theory, Department of Mathematics, Royal Institute of Technology, Stockholm, Sweden, 2002, 1… [cited by applicant]
Google Cloud Platform—Smart Autofill Spreadsheets Add On—Predication API. Apr. 23, 2015. retrieved from https://cloud.google.com/prediction/docs/smart_autofill_add_on. [Aug. 23, 2015], 9 pages. [cited by applicant]
I.M. Alsmadi et al., Evaluation of Cost Estimation Metrics: Towards a Unified Terminology, Journal of Computing and Information Technology, vol. 21, Mar. 2013, pp. 23-34. [cited by applicant]
Information Criterion; https://en.wikipedia.org/w/index.php?title=Informationcriterion&oldid=793390961; Creative Commons Attribution—ShareAlike License; Aug. 1, 2017; 1 pg. [cited by applicant]
International Preliminary Report on Patentability for International Application No. PCT/US2015/032203 dated Nov. 29, 2016. (9 pages). [cited by applicant]
International Search Report and Written Opinion in PCT/US2015/032203 dated Jul. 22, 2015, 11 pages. [cited by applicant]
International Search Report and Written Opinion in PCT/US2017/057753 dated Feb. 12, 2018, 14 pages. [cited by applicant]
K. Valkili et al., Finding Regression Outliers with FastRCS, Cornell University Library, Feb. 2014, 23 pages. [cited by applicant]
M. Claesen et al., Hyperparameter tuning in Python using Optunity, International Workshop on Technical Computing for Machine Learning and Mathematical Engineering (TCMM), Sep. 2014, 2 pages. [cited by applicant]
Mitra et al., 2006. Multi-objective evolutionary biclustering of gene expression data. Pattern Recognition. 2006;39(12):2464-77. [cited by applicant]
N. Deshpande et al., Modelling Compressive Strength of Recycled Aggregate Concrete by Artificial Neural Network, Model Tree and Non-linear Regression, International Journal of Sustainable Built Environment, Dec. 2014, p… [cited by applicant]
N. Deshpande et al., Modelling Compressive Strength of Recycled Aggregate Concrete Using Neural Networks and Regression, Concrete Research Letters, vol. 4(2), Jun. 2013, pp. 580-590. [cited by applicant]
N. Sharma et al., Incorporating Data Mining Techniques on Software Cost Estimation: Validation and Improvement, International Journal of Emerging Technology and Advanced Engineering, vol. 2, Mar. 2012, pp. 301-309. [cited by applicant]
P. Ramesh, Prediction of Cost Overruns Using Ensemble Methods in Data Mining and Text Mining Algorithms, Master's Thesis, Graduate Program in Civil and Environmental Engineering, Rutgers University, Jan. 2014, 50 pages. [cited by applicant]
P.J. Edwards et al., The application of neural networks to the paper-making industry, Proceedings of the European Symposium on Artificial Neural Networks, Apr. 1999, 6 pages. [cited by applicant]
R. Strapasson et al., Tensile and impact behavior of polypropylene/low density polyethylene blends, Polymer Testing 24, 2005, pp. 468-473. [cited by applicant]
S. Zheng, Boosting Based Conditional Quantile Estimation for Regression and Binary Classification, Proceedings of the 9th Mexican international conference on Artificial intelligence, 2010, pp. 67-79. [cited by applicant]
S.C. Lhee et al., Development of a two-step neural network-based model to predict construction cost contingency, Journal of Information Technology in Construction, vol. 19, Sep. 2014, pp. 399-411. [cited by applicant]
S.T. Yousif et al., Artificial Neural Network Model for Predicting Compressive Strength of Concrete, Tikrit Journal of Engineering Sciences, vol. 16, 2009, pp. 55-63. [cited by applicant]
Schmidt, Michael D. et al. “Automated refinement and inference of analytical models for metabolic networks,” Aug. 10, 2011, Physical Biology, vol. 8, No. 5. 36 pages. [cited by applicant]
Schmidt, Michael et al., “Distilling Free-Form Natural Laws from Experimental Data,” Apr. 3, 2009, Science, vol. 324, No. 5923, pp. 81-85. [cited by applicant]
T. Kraska et al. MLbase: A distributed machine-learning system. In Proceedings of 6th Biennial Conference on Innovative Data Systems Research (CIDR'13), Jan. 6, 2013 (7 pages). Available at <http://cidrdb.org/cidr2013/P… [cited by applicant]
V. Chandwani et al., Applications of Soft Computing in Civil Engineering: A Review, International Journal of Computer Applications, vol. 81, Nov. 2013, pp. 13-20. [cited by applicant]
V. Chandwani et al., Modeling Slump of Ready Mix Concrete Using Genetically Evolved Artificial Neural Networks, Advances in Artificial Neural Systems, Nov. 2014, 9 pages. [cited by applicant]
Vladislavleva et al., Order of Nonlinearity as a Complexity Measure for Models Generated by Symbolic Regression via Pareto Genetic Programming. IEEE Transactions on Evolutionary Computation. 2008; 13(2): 333-49. [cited by applicant]
X. He et al., Practical Lessons from Predicting Clicks on Ads at Facebook, Proceedings of ADKDD'14, Aug. 2014, 9 pages. [cited by applicant]
Y. Shan et al., Machine Learning of Poorly Predictable Ecological Data, Ecological Modeling, vol. 195, 2006, pp. 129-138. [cited by applicant]
Y.X. Zhao et al., Concrete cracking process induced by steel corrosion—A review, Proceedings of the Thirteenth East Asia-Pacific Conference on Structural Engineering and Construction, Sep. 2013, pp. 1-10. [cited by applicant]
Z.A. Khalifelu et al., Comparison and evaluation of data mining techniques with algorithmic models in software cost estimation, Procedia Technology, vol. 1, 2012, pp. 65-71. [cited by applicant]
Arnaldo et al., Multiple Regression Genetic Programming, Proceedings of the 2014 Annual Conf. Genetic and Evolutionary Comp., ACM, Jul. 12-16, 2014, pp. 879-886. [cited by applicant]
Schmidt et al., Comparison of Tree and Graph Encodings as Function of Problem Complexity, in Proceedings of the 9th Annual Conf. Genetic and Evolutionary Comp., ACM, Jul. 7-11, 2007, pp. 1674-1679. [cited by applicant]
Vanneschi et al., Measuring Bloat, Overfitting and Functional Complexity in Genetic Programming, in Proceedings of the 12th Annual Conf. Genetic and Evolutionary Comp., ACM, Jul. 7-11, 2010, DD. 877-884. [cited by applicant]
J. Du, The ‘Weight’ of Models and Complexity, in Complexity, vol. 21, No. 3, Sep. 2014, pp. 21-35. [cited by applicant]
M. Kommenda, Complexity Measures for Multi-Objective Symbolic Regression, in Int'l Conf. Computer Aided Systems Theory, 2015, pp. 409-412, 415-416. [cited by applicant]
J. Koza, Genetic Programming: On the Programming of Computers by Means of Natural Selection, vol. 1, MIT Press, 1992, pp. i-xl. N. Nikolaev, Regularization Approach to Inductive Genetic Programming, IEEE, vol. 5, No. 4.… [cited by applicant]
N. Nikolaev, Regularization Approach to Inductive Genetic Programming, IEEE, vol. 5, No. 4., Aug. 4, 2011, pp. 359-375. [cited by applicant]
G. Smits, Pareto-Front Exploitation in Symbolic Regression, in Genetic Programming Theory and Practice II, 2005, pp. 283-299. [cited by applicant]
D.N. Stivers et al., Organ-specific differences in gene expression and Unigene annotations describing source material, in Methods of Microarray Data Analysis III, K.F. Johnson and S.M. Lin, eds., Kluwer Academic Publish… [cited by applicant]
F. Provost et al. Data Science for Business, Reilly Media, Aug. 2013, 414 pages. [cited by applicant]
G. James et al. An Introduction to Statistical Learning, 2013, 426 pages. [cited by applicant]
I-C. Yeh, Modeling of strength of high performance concrete using artificial neural networks, Cement and Concrete Research, vol. 28, No. 12, 1998, pp. 1797-1808. [cited by applicant]
I-C. Yeh, Modeling slump flow of concrete using second-order regressions and artificial neural networks, Cement and Concrete Composites, vol. 29, 2007, pp. 474-480. [cited by applicant]
P. Flach. Machine Learning: The Art and Science of Algorithms That Make Sense of Data, Cambridge University Press, Nov. 12, 2012, 409 pages. [cited by applicant]
P. Love et al., Determining the Probability of Project Cost Overruns, Journal of Construction Engineering and Management, vol. 139, Mar. 2013, pp. 321-330. [cited by applicant]
R.K. Pearson, Mining Imperfect Data, Society for Industrial and Applied Mathematics, 2005, 305 pages. [cited by applicant]
S.C. Lhee et al., Prediction of Financial Contingency for Asphalt Resurfacing Projects using Artificial Neural Networks, Journal of Construction Engineering and Management, vol. 138, Jan. 2012, pp. 22-30. [cited by applicant]
T. Erl. Cloud Computing, Prentice Hall, May 2, 2013, 528 pages. [cited by applicant]
T. Hastie et al., The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed., Feb. 2009, 758 pages, available at http://web.stanford.edu/˜hastie/local. ftp/Springer/OLD/ESLI I pri nt4. pdf. [cited by applicant]
T.P. Williams et al., Predicting construction cost overruns using text mining, numerical data and ensemble classifiers, Automation in Construction, vol. 43, 2014, pp. 23-29. [cited by applicant]
U.S. Appl. No. 17/112,729, filed Dec. 4, 2020, Achin et al. [cited by applicant]
U.S. Appl. No. 16/722,831, filed Dec. 20, 2019, Achin et al. [cited by applicant]
U.S. Appl. No. 16/506,219, filed Jul. 9, 2019, Bledsoe et al. [cited by applicant]
Yang, F. et al., “Forecasting time series with genetic programming based on least square method,” J. Syst. Sci. Complex, vol. 27 (2014), pp. 117-129. [cited by applicant]
Zamani et al., “Benchmark problems solving with the complexity of genetic programming”, 1.1 Int'l J. Applies Res. Computer Sci. Info. Tech., pp. 13-26 (2012). [cited by applicant]
G. James et al. An Introduction to Statistical Learning with Applications in R, 2013, 441 pages. [cited by applicant]
Communication on EP Appl. Ser. No. 17801798.4 dated May 20, 2020 (10 pages). [cited by applicant]
Examination Report on GB Appl. Ser. No. 1621870.3 dated May 24, 2021 (5 pages). [cited by applicant]
Examination Report on GB Appl. Ser. No. 1621870.3 dated Oct. 12, 2020 (4 pages). [cited by applicant]
Examination Report on GB Appl. Ser. No. 1621870.3 dated Oct. 15, 2021 (4 pages). [cited by applicant]
Examination Report on GB Appl. Ser. No. 1907147.1 dated Sep. 23, 2021 (7 pages). [cited by applicant]
Final Office Action on U.S. Appl. No. 15/331,797 dated Dec. 21, 2017 (12 pages). [cited by applicant]
Final Office Action on U.S. Appl. No. 15/587,951 dated Jul. 12, 2018 (18 pages). [cited by applicant]
Final Office Action on U.S. Appl. No. 15/587,951 dated Jun. 4, 2020 (13 pages). [cited by applicant]
Final Office Action on U.S. Appl. No. 15/790,756 dated Dec. 31, 2018 (13 pages). [cited by applicant]
International Preliminary Report on Patentability on PCT Appl. Ser. No. PCT/US2017/057763 dated Apr. 23, 2019 (8 pages). [cited by applicant]
Non-Final Office Action on U.S. Appl. No. 14/720,079 dated Nov. 2, 2015 (24 pages). [cited by applicant]
Non-Final Office Action on U.S. Appl. No. 15/217,626 dated Nov. 16, 2016 (18 pages). [cited by applicant]
Non-Final Office Action on U.S. Appl. No. 15/217,640 dated Nov. 16, 2016 (22 pages). [cited by applicant]
Non-Final Office Action on U.S. Appl. No. 15/331,797 dated Mar. 8, 2017 (19 pages). [cited by applicant]
Non-Final Office Action on U.S. Appl. No. 15/587,951 dated Sep. 22, 2017 (38 pages). [cited by applicant]
Non-Final Office Action on U.S. Appl. No. 15/587,951 dated Sep. 3, 2019 (9 pages). [cited by applicant]
Non-Final Office Action on U.S. Appl. No. 15/790,756 dated Mar. 28, 2018 (9 pages). [cited by applicant]
Non-Final Office Action on U.S. Appl. No. 15/790,803 dated Mar. 1, 2018 (9 pages). [cited by applicant]
Notice of Allowance on U.S. Appl. No. 14/720,079 dated Mar. 30, 2016 (10 pages). [cited by applicant]
Notice of Allowance on U.S. Appl. No. 15/217,626 dated Apr. 17, 2017 (4 pages). [cited by applicant]
Notice of Allowance on U.S. Appl. No. 15/217,626 dated Mar. 23, 2017 (7 pages). [cited by applicant]
Notice of Allowance on U.S. Appl. No. 15/217,640 dated Apr. 25, 2017 (4 pages). [cited by applicant]
Notice of Allowance on U.S. Appl. No. 15/217,640 dated Mar. 27, 2017 (5 pages). [cited by applicant]
Notice of Allowance on U.S. Appl. No. 15/331,797 dated Jun. 19, 2019 (2 pages). [cited by applicant]
Notice of Allowance on U.S. Appl. No. 15/331,797 dated Mar. 13, 2019 (11 pages). [cited by applicant]
Notice of Allowance on U.S. Appl. No. 15/587,951 dated Dec. 18, 2020 (9 pages). [cited by applicant]
Notice of Allowance on U.S. Appl. No. 15/790,756 dated Aug. 6, 2019 (7 pages). [cited by applicant]
Notice of Allowance on U.S. Appl. No. 15/790,756 dated Nov. 27, 2019 (6 pages). [cited by applicant]
Notice of Allowance on U.S. Appl. No. 15/790,756 dated Oct. 2, 2019 (2 pages). [cited by applicant]
Notice of Allowance on U.S. Appl. No. 15/790,803 dated Jul. 25, 2019 (5 pages). [cited by applicant]
Notice of Allowance on U.S. Appl. No. 15/790,803 dated Nov. 13, 2018 (12 pages). [cited by applicant]
Notice of Final Rejection on JP Appl. Ser. No. 2019-520991 dated Dec. 10, 2021 (7 pages). [cited by applicant]
Notice of Reasons for Refusal on JP Appl. Ser. No. 2017-513620 dated Jul. 2, 2018 (4 pages). [cited by applicant]
Notice of Reasons for Refusal on JP Appl. Ser. No. 2018-187956 dated Dec. 15, 2020 (6 pages). [cited by applicant]
Notice of Reasons for Refusal on JP Appl. Ser. No. 2018-187956 dated Jan. 27, 2020 (4 pages). [cited by applicant]
Notice of Reasons for Refusal on JP Appl. Ser. No. 2020-126434 dated Jun. 8, 2021 (4 pages). [cited by applicant]
Notice of Reasons for Rejection on JP Appl. Ser. No. 2019-520991 dated May 31, 2021 (13 pages). [cited by applicant]
Notice of Reasons for Rejection on JP Appl. Ser. No. 2020-126434 dated Jun. 8, 2021 (5 pages). [cited by applicant]
Notification of Reason for Refusal on KR Appl. Ser. No. 10-2019-7014598 dated Jan. 18, 2022 (14 pages). [cited by applicant]
Oral Proceedings Preliminary Opinion for EP Appl. Ser. No. 17801798.4 dated Jan. 18, 2022 (7 pages). [cited by applicant]
Reasons for Refusal on JP Appl. Ser. No. 2019-520991 dated Dec. 10, 2021 (4 pages). [cited by applicant]
Summons to Attend Oral Proceedings on EP Appl. Ser. No. 17801798.4 dated Feb. 9, 2021 (16 pages). [cited by applicant]
Written Opinion on JP Appl. Ser. No. 2017-513620 dated Oct. 3, 2018 (2 pages). [cited by applicant]
Written Opinion on JP Appl. Ser. No. 2018-187956 dated Jul. 27, 2020 (2 pages). [cited by applicant]
Written Opinion on JP Appl. Ser. No. 2018-187956 dated Jun. 15, 2021 (4 pages). [cited by applicant]
Examination Report No. 1 in AU Application No. 2017345796, dated Sep. 6, 2022 (3 Pages). [cited by applicant]
Examination Report in GB Application No. 1907147.1, dated Sep. 23, 2021 (6 Pages), by: Mr. David Maskery. [cited by applicant]
Examination Report in GB Application No. 1907147.1, dated Feb. 7, 2022 (2 Pages), by: Mr. David Maskery. [cited by applicant]
Examination Report in GB Application No. 1907147.1, dated May 26, 2022 (4 Pages), by: Mr. David Maskery. [cited by applicant]
Search Report in GB Application No. 2211852.5, dated Aug. 31, 2022 (1 Page), by: Mr. David Maskery. [cited by applicant]