IP Library › Granted Patent US 12,566,993
Granted Patent B2
US 12,566,993 · App. 16/447,924 · Granted Mar 3, 2026

Systems and techniques for determining the predictive value of a feature

Inventors: Jeremy Achin (Boston, MA); Thomas DeGodoy (Medford, MA); Xavier Conort (Singapore, SG); Sergey Yurgenson (Marlborough, MA); Mark L. Steadman (Watertown, MA); Glen Koundry (Waltham, MA)
Assignee: DataRobot, Inc.
G06N20/00G06Q10/04G06Q10/06G06Q10/067G06Q30/0201
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,566,993
App. No.
16/447,924
Granted
Mar 3, 2026
Kind
B2
Abstract

A method for determining the predictive value of a feature may include: (a) performing predictive modeling procedures associated with respective predictive models, wherein performing each modeling procedure includes fitting the associated model to an initial dataset representing an initial prediction problem; (b) determining a first accuracy score of each of the fitted models, representing an accuracy with which the fitted model predicts an outcome of the initial prediction problem; (c) shuffling values of a feature across observations included in the initial dataset, thereby generating a modified dataset representing a modified prediction problem; (d) determining a second accuracy score of each of the fitted models, representing an accuracy with which the fitted model predicts an outcome of the modified prediction problem; and (e) determining a model-specific predictive value of the feature for each of the fitted models based on the first and second accuracy scores of the fitted model.

Claims (58)

1 . A computer-implemented predictive modeling method comprising:

(a) performing a predictive modeling procedure, wherein the predictive modeling procedure is associated with a predictive model, wherein performing the modeling procedure comprises fitting the associated predictive model to an initial dataset, and wherein the initial dataset includes a plurality of observations and each observation includes respective values for a plurality of features;

(b) determining a model-specific predictive value of a feature for the fitted model;

(c) based at least in part on the model-specific predictive value of the feature, generating a modified dataset, wherein generating the modified dataset includes pruning the feature from the initial dataset and/or creating a first derived feature based on the feature and at least one other feature in the initial dataset and adding the first derived feature to the modified dataset; and

(d) performing the predictive modeling procedure again, wherein performing the modeling procedure again comprises fitting the associated predictive model to the modified dataset,

wherein the feature is a first feature, wherein the initial dataset includes the first feature and a plurality of second features, and wherein the method further comprises detecting an interaction between at least two of the second features, and

wherein generating the modified dataset further comprises creating a second derived feature based on the at least two second features and adding the second derived feature to the modified dataset.

2 . The method of claim 1 , wherein the method further comprises determining model-specific predictive values of the second features of the initial dataset.

3 . The method of claim 2 , further comprising displaying, via a graphical user interface, graphical content identifying the first and second features of the initial dataset and the model-specific predictive values of the first and second features.

4 . The method of claim 2 , wherein generating the modified dataset further includes performing feature engineering on the initial dataset based on the model-specific predictive values of the first feature and the second features.

5 . The method of claim 4 , wherein performing the feature engineering comprises:

generating a third derived feature based on two or more features of the initial dataset having high model-specific predictive values; and

adding the third derived feature to the modified dataset.

6 . The method of claim 5 , further comprising determining that the model-specific predictive values of the two or more features are high based on the model-specific predictive values of the two or more features being higher than a threshold value and/or based on the model-specific predictive values of the two or more features being in a particular percentile of the particular model-specific predictive values for the first and second features of the initial dataset.

7 . The method of claim 2 , further comprising:

(e) displaying, via a graphical user interface, graphical content ranking the first feature and the plurality of second features based on the respective model-specific predictive values specific to the first predictive model.

8 . The method of claim 1 , further comprising detecting one or more statistically significant interactions between two or more features included in the first and second features, and providing the one or more statistically significant interactions as input to a modeling procedure selected from the group consisting of a random forest modeling procedure, a generalized additive modeling procedure, and a support vector machine modeling procedure.

9 . The method of claim 1 , wherein generating the modified dataset further comprises determining that two or more of the second features are highly correlated with each other and pruning at least one of the two or more highly correlated second features.

10 . The method of claim 1 , wherein the model-specific predictive value of the first feature for the fitted model is determined using an analysis technique applicable to any predictive model.

11 . The method of claim 1 , wherein determining the model-specific predictive value of the first feature for the fitted model comprises analyzing a significance of the first feature in predicting a target based on conditional expectations.

12 . The method of claim 1 , wherein determining the model-specific predictive value of the first feature for the fitted model comprises:

determining a first accuracy score of the fitted predictive model representing an accuracy with which the fitted model predicts one or more outcomes based on the initial dataset;

shuffling values of the first feature across respective observations included in the initial dataset, thereby generating a second dataset; and

determining a second accuracy score of the fitted predictive model representing an accuracy with which the fitted model predicts one or more outcomes based on the second dataset.

13 . The method of claim 1 , wherein the predictive modeling procedure is a first predictive modeling procedure, the predictive model is a first predictive model, and the model-specific predictive value is a first model-specific predictive value, the method further comprising:

performing at least one second predictive modeling procedure, wherein the second predictive modeling procedure is associated with a second predictive model, wherein performing the second modeling procedure comprises fitting the second predictive model to the initial dataset; and

determining a second model-specific predictive value of the first feature for the fitted second model.

14 . The method of claim 13 , further comprising determining a model-independent predictive value of the first feature based on the first and second model-specific predictive values of the first feature.

15 . The method of claim 14 , wherein determining the model-independent predictive value of the first feature comprises calculating a statistical measure of a center and/or a spread of the first and second model-specific predictive values of the first feature.

16 . The method of claim 15 , wherein the statistical measure of the center is selected from the group consisting of a mean, a median, and a mode of the first and second model-specific predictive values, and wherein the statistical measure of the spread is selected from the group consisting of a range, a variance, and a standard deviation of the first and second model-specific predictive values.

17 . The method of claim 14 , wherein determining the model-independent predictive value of the first feature comprises calculating a combination of the first and second model-specific predictive values of the first feature.

18 . The method of claim 14 , wherein generating the modified dataset based at least in part on the model-specific predictive value of the first feature comprises generating the modified dataset based at least in part on the model-independent predictive value of the first feature.

19 . The method of claim 13 , further comprising, prior to performing the at least one second modeling procedure:

selecting the second modeling procedure based on a suitability of the selected modeling procedure for the initial dataset,

wherein a suitability of a particular predictive modeling procedure for the initial dataset is determined based, at least in part, on characteristics of one or more particular features of the initial dataset having high model-specific predictive values for the particular predictive modeling procedure.

20 . The method of claim 13 , wherein the first and second predictive modeling procedures are selected from the group consisting of a random forest modeling procedure, a generalized additive modeling procedure, and a support vector machine modeling procedure.

21 . The method of claim 13 , wherein the first modeling procedure is selected from a first family of modeling procedures and the second modeling procedure is selected from a second family of modeling procedures.

22 . The method of claim 13 , further comprising:

generating a blended predictive model by combining two or more of the fitted predictive models.

23 . The method of claim 22 , wherein the two or more fitted models are complementary models.

24 . A predictive modeling apparatus comprising:

a memory configured to store processor-executable instructions; and

a processor configured to execute the processor-executable instructions to cause the apparatus to perform operations including:

(a) performing a predictive modeling procedure, wherein the predictive modeling procedure is associated with a predictive model, wherein performing the modeling procedure comprises fitting the associated predictive model to an initial dataset, and wherein the initial dataset includes a plurality of observations and each observation includes respective values for a plurality of features;

(b) determining a model-specific predictive value of a feature for the fitted model;

(c) based at least in part on the model-specific predictive value of the feature, generating a modified dataset, wherein generating the modified dataset includes (i) pruning the feature from the initial dataset and/or (ii) creating a first derived feature based on the feature and at least one other feature in the initial dataset and adding the first derived feature to the modified dataset; and

(d) performing the predictive modeling procedure again, wherein performing the modeling procedure again comprises fitting the associated predictive model to the modified dataset,

wherein the feature is a first feature, wherein the initial dataset includes the first feature and a plurality of second features, and wherein the operations further comprise detecting an interaction between at least two of the second features, and

wherein generating the modified dataset further comprises creating a second derived feature based on the at least two second features and adding the second derived feature to the modified dataset.

25 . The apparatus of claim 24 , wherein determining the model-specific predictive value of the first feature for the fitted model comprises analyzing a significance of the first feature in predicting a target based on conditional expectations.

26 . The apparatus of claim 24 , wherein the model-specific predictive value of the first feature for the fitted model is determined using an analysis technique applicable to any predictive model.

27 . An article of manufacture having computer-readable instructions stored thereon that, when executed by a processor, cause the processor to perform operations including:

(a) performing a predictive modeling procedure, wherein the predictive modeling procedure is associated with a predictive model, wherein performing the modeling procedure comprises fitting the associated predictive model to an initial dataset, and wherein the initial dataset includes a plurality of observations and each observation includes respective values for a plurality of features;

(b) determining a model-specific predictive value of a feature for the fitted model;

(c) based at least in part on the model-specific predictive value of the feature, generating a modified dataset, wherein generating the modified dataset includes (i) pruning the feature from the initial dataset and/or (ii) creating a first derived feature based on the feature and at least one other feature in the initial dataset and adding the first derived feature to the modified dataset; and

(d) performing the predictive modeling procedure again, wherein performing the modeling procedure again comprises fitting the associated predictive model to the modified dataset,

wherein the feature is a first feature, wherein the initial dataset includes the first feature and a plurality of second features, and wherein the operations further comprise detecting an interaction between at least two of the second features, and

wherein generating the modified dataset further comprises creating a second derived feature based on the at least two second features and adding the second derived feature to the modified dataset.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 29, 2025
From: ACHIN, JEREMY; DEGODOY, THOMAS; CONORT, XAVIER; YURGENSON, SERGEY; STEADMAN, MARK L.; KOUNDRY, GLEN
To: DATAROBOT, INC.
Reel/Frame 072402/0148 →
RELEASE OF SECURITY INTEREST Recorded Apr 7, 2025
From: CITIBANK, N.A.
To: DATAROBOT, INC.; ALGORITHMIA, INC.; DULLES RESEARCH, LLC
Reel/Frame 070750/0866 →
SECURITY INTEREST Recorded Mar 22, 2023
From: DATAROBOT, INC.; ALGORITHMIA, INC.; DULLES RESEARCH, LLC
To: CITIBANK, N.A.
Reel/Frame 063263/0926 →
Continuity (6)
Continuation 15331797 · Oct 21, 2016
Continuation In Part 15217626 · Jul 22, 2016
Continuation 14720079 · May 22, 2015
Provisional Application 62411526 · Oct 21, 2016
Provisional Application 62002469 · May 23, 2014
Related Publication 20200090075A1 · Mar 19, 2020
References Cited (205)
US 5123057A · Verly et al. · 1992 [cited by applicant]
US 5761442A · Barr et al. · 1998 [cited by applicant]
US 7072863B1 · Phillips et al. · 2006 [cited by applicant]
US 7117185B1 · Aliferis et al. · 2006 [cited by applicant]
US 7580852B2 · Ouimet et al. · 2009 [cited by applicant]
US 7788195B1 · Subramanian · 2010 [cited by examiner]
US 8024216B2 · Aronowich et al. · 2011 [cited by applicant]
US 8108243B2 · Solotorevsky et al. · 2012 [cited by applicant]
US 8180664B2 · Shan · 2012 [cited by applicant]
US 8280903B2 · Broder et al. · 2012 [cited by applicant]
US 8370280B1 · Lin et al. · 2013 [cited by applicant]
US 8645421B2 · Meric et al. · 2014 [cited by applicant]
US 8676629B2 · Chien et al. · 2014 [cited by applicant]
US 8682709B2 · Coldren et al. · 2014 [cited by applicant]
US 8751273B2 · Pinto et al. · 2014 [cited by applicant]
US 8782037B1 · Barad et al. · 2014 [cited by applicant]
US 8799009B2 · Mellin et al. · 2014 [cited by applicant]
US 8843427B1 · Lin et al. · 2014 [cited by applicant]
US 8924269B1 · Seubert et al. · 2014 [cited by applicant]
US 9489630B2 · Achin et al. · 2016 [cited by applicant]
US 9495641B2 · Schmidt · 2016 [cited by applicant]
US 9524473B2 · Schmidt · 2016 [cited by applicant]
US 9652714B2 · Achin et al. · 2017 [cited by applicant]
US 9659254B2 · Achin et al. · 2017 [cited by applicant]
US 10102483B2 · Schmidt · 2018 [cited by applicant]
US 10318874B1 · Duncan et al. · 2019 [cited by applicant]
US 10366335B2 · Schmidt · 2019 [cited by applicant]
US 10366346B2 · Achin et al. · 2019 [cited by applicant]
US 10387900B2 · Bledsoe et al. · 2019 [cited by applicant]
US 10496927B2 · Achin et al. · 2019 [cited by applicant]
US 10558924B2 · Achin et al. · 2020 [cited by applicant]
US 10984367B2 · Achin et al. · 2021 [cited by applicant]
US 20020144178A1 · Castelli et al. · 2002 [cited by applicant]
US 20040030777A1 · Reedy et al. · 2004 [cited by applicant]
US 20040054997A1 · Katragadda et al. · 2004 [cited by applicant]
US 20050183073A1 · Reynolds · 2005 [cited by applicant]
US 20050234762A1 · Pinto et al. · 2005 [cited by applicant]
US 20060101014A1 · Forman et al. · 2006 [cited by applicant]
US 20060190285A1 · Harris et al. · 2006 [cited by applicant]
US 20070133848A1 · McNutt et al. · 2007 [cited by applicant]
US 20080059284A1 · Solotorevsky et al. · 2008 [cited by applicant]
US 20080097802A1 · Ladde et al. · 2008 [cited by applicant]
US 20080307399A1 · Zhou et al. · 2008 [cited by applicant]
US 20090138286A1 · Linder et al. · 2009 [cited by applicant]
US 20100049340A1 · Smits et al. · 2010 [cited by applicant]
US 20100131314A1 · Lo Yuk Ting et al. · 2010 [cited by applicant]
US 20100312718A1 · Rosenthal et al. · 2010 [cited by applicant]
US 20110022359A1 · Thorat · 2011 [cited by examiner]
US 20110119100A1 · Ruhl et al. · 2011 [cited by applicant]
US 20120078678A1 · Pradhan et al. · 2012 [cited by applicant]
US 20120144325A1 · Mital et al. · 2012 [cited by applicant]
US 20120192051A1 · Rothschiller et al. · 2012 [cited by applicant]
US 20120202240A1 · Deigner et al. · 2012 [cited by applicant]
US 20130073061A1 · Mu et al. · 2013 [cited by applicant]
US 20130096892A1 · Essa et al. · 2013 [cited by applicant]
US 20130290226A1 · Dokken · 2013 [cited by applicant]
US 20140074829A1 · Schmidt · 2014 [cited by applicant]
US 20140136452A1 · Wellman et al. · 2014 [cited by applicant]
US 20140172773A1 · Schmidt · 2014 [cited by applicant]
US 20140258189A1 · Schmidt · 2014 [cited by applicant]
US 20140359560A1 · Avadhanula et al. · 2014 [cited by applicant]
US 20140372172A1 · Nez et al. · 2014 [cited by applicant]
US 20150088606A1 · Tyagi · 2015 [cited by applicant]
US 20150154619A1 · Grichnik et al. · 2015 [cited by applicant]
US 20150170049A1 · Mann et al. · 2015 [cited by applicant]
US 20150317589A1 · Anderson et al. · 2015 [cited by applicant]
US 20150339572A1 · Achin et al. · 2015 [cited by applicant]
US 20150356576A1 · Malaviya et al. · 2015 [cited by applicant]
US 20160005055A1 · Sarferaz · 2016 [cited by applicant]
US 20160048766A1 · McMahon et al. · 2016 [cited by applicant]
US 20160217390A1 · Shoaib et al. · 2016 [cited by applicant]
US 20160301624A1 · Gonzalez et al. · 2016 [cited by applicant]
US 20160335550A1 · Achin et al. · 2016 [cited by applicant]
US 20160364647A1 · Achin et al. · 2016 [cited by applicant]
US 20160379244A1 · Kalish et al. · 2016 [cited by applicant]
US 20170193398A1 · Schmidt · 2017 [cited by applicant]
US 20170243140A1 · Achin et al. · 2017 [cited by applicant]
US 20170330078A1 · Koduru · 2017 [cited by applicant]
US 20180046926A1 · Achin et al. · 2018 [cited by applicant]
US 20180060738A1 · Achin et al. · 2018 [cited by applicant]
US 20180060744A1 · Achin et al. · 2018 [cited by applicant]
US 20180300737A1 · Bledsoe et al. · 2018 [cited by applicant]
US 20190220772A1 · Schmidt · 2019 [cited by applicant]
US 20200090075A1 · Achin et al. · 2020 [cited by applicant]
US 20200134489A1 · Achin et al. · 2020 [cited by applicant]
US 20200257992A1 · Achin et al. · 2020 [cited by applicant]
JP 05081228A · 1991 [cited by applicant]
JP 2005135287A · 2005 [cited by applicant]
KR 101369261B1 · 2014 [cited by applicant]
WO WO2010044683A1 · 2010 [cited by applicant]
WO WO2015179778A1 · 2015 [cited by applicant]
“GA-ACE: Alternating conditional expectations regression with selection of significant predictors by genetic algorithms” Esteban-Diez et al (Year: 2005). [cited by examiner]
“Visualization Concepts to Enhance Quantitative Decision Making in Drug Development” Dykstra et al (Year: 2010). [cited by examiner]
Communication on EP Appl. Ser. No. 17801798.4 dated May 20, 2020 (10 pages). [cited by applicant]
Erl, T. et al., “Cloud Computing”, Prentice Hall, May 2, 2013, 528 pages. [cited by applicant]
Examination Report on GB Appl. Ser. No. 1621870.3 dated May 24, 2021 (5 pages). [cited by applicant]
Examination Report on GB Appl. Ser. No. 1621870.3 dated Oct. 12, 2020 (4 pages). [cited by applicant]
Examination Report on GB Appl. Ser. No. 1621870.3 dated Oct. 15, 2021 (4 pages). [cited by applicant]
Examination Report on GB Appl. Ser. No. 1907147.1 dated Sep. 23, 2021 (7 pages). [cited by applicant]
Final Office Action on U.S. Appl. No. 15/331,797 dated Dec. 21, 2017 (12 pages). [cited by applicant]
Final Office Action on U.S. Appl. No. 15/587,951 dated Jul. 12, 2018 (18 pages). [cited by applicant]
Final Office Action on U.S. Appl. No. 15/587,951 dated Jun. 4, 2020 (13 pages). [cited by applicant]
Final Office Action on U.S. Appl. No. 15/790,756 dated Dec. 31, 2018 (13 pages). [cited by applicant]
Flach, Peter, “Machine Learning: The Art and Science of Algorithms That Make Sense of Data”, Cambridge University Press, Nov. 12, 2012, 409 pages. [cited by applicant]
International Preliminary Report on Patentability on PCT Appl. Ser. No. PCT/US2017/057763 dated Apr. 23, 2019 (8 pages). [cited by applicant]
Lhee, Sang C. et al., “Prediction of Financial Contingency for Asphalt Resurfacing Projects using Artificial Neural Networks”, Journal of Construction Engineering and Management, vol. No. 138, Issue No. 1, Jan. 2012, pp… [cited by applicant]
Non-Final Office Action on U.S. Appl. No. 14/720,079 dated Nov. 2, 2015 (24 pages). [cited by applicant]
Non-Final Office Action on U.S. Appl. No. 15/217,626 dated Nov. 16, 2016 (18 pages). [cited by applicant]
Non-Final Office Action on U.S. Appl. No. 15/217,640 dated Nov. 16, 2016 (22 pages). [cited by applicant]
Non-Final Office Action on U.S. Appl. No. 15/331,797 dated Mar. 8, 2017 (19 pages). [cited by applicant]
Non-Final Office Action on U.S. Appl. No. 15/587,951 dated Sep. 22, 2017 (38 pages). [cited by applicant]
Non-Final Office Action on U.S. Appl. No. 15/587,951 dated Sep. 3, 2019 (9 pages). [cited by applicant]
Non-Final Office Action on U.S. Appl. No. 15/790,756 dated Mar. 28, 2018 (9 pages). [cited by applicant]
Non-Final Office Action on U.S. Appl. No. 15/790,803 dated Mar. 1, 2018 (9 pages). [cited by applicant]
Notice of Allowance on U.S. Appl. No. 14/720,079 dated Mar. 30, 2016 (10 pages). [cited by applicant]
Notice of Allowance on U.S. Appl. No. 15/217,626 dated Apr. 17, 2017 (4 pages). [cited by applicant]
Notice of Allowance on U.S. Appl. No. 15/217,626 dated Mar. 23, 2017 (7 pages). [cited by applicant]
Notice of Allowance on U.S. Appl. No. 15/217,640 dated Apr. 25, 2017 (4 pages). [cited by applicant]
Notice of Allowance on U.S. Appl. No. 15/217,640 dated Mar. 27, 2017 (5 pages). [cited by applicant]
Notice of Allowance on U.S. Appl. No. 15/331,797 dated Jun. 19, 2019 (2 pages). [cited by applicant]
Notice of Allowance on U.S. Appl. No. 15/331,797 dated Mar. 13, 2019 (11 pages). [cited by applicant]
Notice of Allowance on U.S. Appl. No. 15/587,951 dated Dec. 18, 2020 (9 pages). [cited by applicant]
Notice of Allowance on U.S. Appl. No. 15/790,756 dated Aug. 6, 2019 (7 pages). [cited by applicant]
Notice of Allowance on U.S. Appl. No. 15/790,756 dated Nov. 27, 2019 (6 pages). [cited by applicant]
Notice of Allowance on U.S. Appl. No. 15/790,756 dated Oct. 2, 2019 (2 pages). [cited by applicant]
Notice of Allowance on U.S. Appl. No. 15/790,803 dated Jul. 25, 2019 (5 pages). [cited by applicant]
Notice of Allowance on U.S. Appl. No. 15/790,803 dated Nov. 13, 2018 (12 pages). [cited by applicant]
Notice of Final Rejection on JP Appl. Ser. No. 2019-520991 dated Dec. 10, 2021 (7 pages). [cited by applicant]
Notice of Reasons for Refusal on JP Appl. Ser. No. 2017-513620 dated Jul. 2, 2018 (4 pages). [cited by applicant]
Notice of Reasons for Refusal on JP Appl. Ser. No. 2018-187956 dated Dec. 15, 2020 (6 pages). [cited by applicant]
Notice of Reasons for Refusal on JP Appl. Ser. No. 2018-187956 dated Jan. 27, 2020 (4 pages). [cited by applicant]
Notice of Reasons for Refusal on JP Appl. Ser. No. 2020-126434 dated Jun. 8, 2021 (4 pages). [cited by applicant]
Notice of Reasons for Rejection on JP Appl. Ser. No. 2019-520991 dated May 31, 2021 (13 pages). [cited by applicant]
Notice of Reasons for Rejection on JP Appl. Ser. No. 2020-126434 dated Jun. 8, 2021 (5 pages). [cited by applicant]
Notification of Reason for Refusal on KR Appl. Ser. No. 10-2019-7014598 dated Jan. 18, 2022 (14 pages). [cited by applicant]
Oral Proceedings Preliminary Opinion for EP Appl. Ser. No. 17801798.4 dated Jan. 18, 2022 (7 pages). [cited by applicant]
Pearson, Ronald K., “Mining Imperfect Data: Dealing with Contamination and Incomplete Records”, 2005, 305 pages. [cited by applicant]
Provost, Foster et al., “Data Science for Business”, O'Reilly Media, Inc., Aug. 2013, 414 pages. [cited by applicant]
Reasons for Refusal on JP Appl. Ser. No. 2019-520991 dated Dec. 10, 2021 (4 pages). [cited by applicant]
Stivers, David N. et al., “Organ-Specific Differences in Gene Expression and Unigene Annotations Describing Source Material”, Methods of Microarray Data Analysis III, K.F. Johnson and S.M. Lin, eds., Kluwer Academic Pub… [cited by applicant]
Summons to Attend Oral Proceedings on EP Appl. Ser. No. 17801798.4 dated Feb. 9, 2021 (16 pages). [cited by applicant]
Williams, Trefor et al., “Predicting construction cost overruns using text mining, numerical data and ensemble classifiers”, Automation in Construction, vol. No. 43, Jul. 2014, pp. 23-29. [cited by applicant]
Written Opinion on JP Appl. Ser. No. 2017-513620 dated Oct. 3, 2018 (2 pages). [cited by applicant]
Written Opinion on JP Appl. Ser. No. 2018-187956 dated Jul. 27, 2020 (2 pages). [cited by applicant]
Written Opinion on JP Appl. Ser. No. 2018-187956 dated Jun. 15, 2021 (4 pages). [cited by applicant]
Yeh, I.-C., “Modeling slump flow of concrete using second-order regressions and artificial neural networks”, Cement and Concrete Composites, vol. No. 29, Issue No. 6, Jul. 2007, pp. 474-480. [cited by applicant]
A. Lorbert et al., Descent Methods for Tuning Parameter Refinement, Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS), 2010, pp. 469-476. [cited by applicant]
A. Rudi et al., Adaptive Optimization for Cross Validation, Proceedings of the European Symposium on Artificial Neural Networks, Computational Intelligence, and Machine Learning, 2012, pp. 435-440. [cited by applicant]
Arnaldo et al., “Multiple Regression Genetic Programming,” Proceedings of the 2014 Annual Conf. Genetic and Evolutionary Comp., ACM 2014, pp. 879-886. [cited by applicant]
B. Boukhatem et al., Predicting concrete properties using neural networks (NN) with principal component analysis (PCA) technique, Computers and Concrete, vol. 10, No. 6, 2012, pp. 1-17. [cited by applicant]
B. Flyvbjerg et al., What Causes Cost Overruns in Transport Infrastructure Projects?, Transport Reviews, vol. 24, 2004, pp. 1-40. [cited by applicant]
B.K. Behera et al., Aabric Quality Evaluation By Objective Measurement, Indian Journal of Fibre and Textile Research, vol. 19, 1994, pp. 168-171. [cited by applicant]
Barresse, Microsoft Excel 2013—Flash Fill. Microsoft Excel and Access Experts Blog. Jul. 28, 2012. retrieved from https://excelandaccess.wordpress.com/2012/07/28/excel-2013-flash-fill/[Mar. 3, 2016 1:45:25PM]. [cited by applicant]
C. Bordat et al., An Analysis of Cost Overruns and Time Delays of INDOT Projects, Final Report, FHWA/IN/JTRP-2004/7, Joint Transportation Research Program, Purdue University, 2004, 191 pages. [cited by applicant]
Du, “The ‘Weight’ of Models and Complexity”, in Complexity vol. 21, No. 3, pp. 21-35 (Oct. 2, 2014). [cited by applicant]
Ekart et al. 2000. A Metric for Genetic Programs and Fitness Sharing. Genetic Programming. Springer Berlin Heidelberg. EuroGP 2000, LNCS 1802. 2000. 259-70. [cited by applicant]
Ensemble Learning; https://en.wikipedia.org/wiki/Ensemble_learing; Creative Commons Attribution-ShareAlike License, Aug. 2, 2017; pp. 1-8. [cited by applicant]
F. Xiao et al., Prediction of Fatigue Life of Rubberized Asphalt Concrete Mixtures Containing Reclaimed Asphalt Pavement Using Artificial Neural Networks, Journal of Materials in Civil Engineering, 2007, 41 pages. [cited by applicant]
G. Biau et al., COBRA: A Nonlinear Aggregation Strategy, Cornell University Library, Nov. 2013, 40 pages. [cited by applicant]
G. Bortolin et al., On modeling of curl in multi-ply paperboard, Journal of Process Control, vol. 16, 2006, pp. 419-429. [cited by applicant]
G. Bortolin, On Modeling and Estimation of Curl and Twist in Multi-ply Paperboard, Licentiate Thesis, Optimization and Systems Theory, Department of Mathematics, Royal Institute of Technology, Stockholm, Sweden, 2002, 1… [cited by applicant]
Google Cloud Platform—Smart Autofill Spreadsheets Add On—Predication API. Apr. 23, 2015. retrieved from https://cloud.google.com/prediction/docs/smart_autofill_add_on. [Aug. 23, 2015], 9 pages. [cited by applicant]
I.M. Alsmadi et al., Evaluation of Cost Estimation Metrics: Towards a Unified Terminology, Journal of Computing and Information Technology, vol. 21, Mar. 2013, pp. 23-34. [cited by applicant]
Information Criterion; https://en.wikipedia.org/w/index.php?title=Informationcriterion&oldid=793390961; Creative Commons Attribution-ShareAlike License; Aug. 1, 2017; 1 pg. [cited by applicant]
International Preliminary Report on Patentability for International Application No. PCT/US2015/032203 dated Nov. 29, 2016. (9 pages). [cited by applicant]
International Search Report and Written Opinion in PCT/US2015/032203 dated Jul. 22, 2015, 11 pages. [cited by applicant]
International Search Report and Written Opinion in PCT/US2017/057753 dated Feb. 12, 2018, 14 pages. [cited by applicant]
K. Valkili et al., Finding Regression Outliers with FastRCS, Cornell University Library, Feb. 2014, 23 pages. [cited by applicant]
Kommenda et al., “Complexity Measures for Multi-objective Symbolic Regression,” in Intl. Conf. Comp. Aided Systems Theory, Springer, Cham, 2015, pp. 409-412, 415-416. [cited by applicant]
Koza et al., “Genetic Programming,” in Genetic Programming: On the Programming of Computers by Means of Natural Selection, vol. 1, MIT Press, 1992, pp. i-xl. [cited by applicant]
M. Claesen et al., Hyperparameter tuning in Python using Optunity, International Workshop on Technical Computing for Machine Learning and Mathematical Engineering (TCMM), Sep. 2014, 2 pages. [cited by applicant]
Mitra et al., 2006. Multi-objective evolutionary biclustering of gene expression data. Pattern Recognition. 2006;39(12):2464-77. [cited by applicant]
N. Deshpande et al., Modelling Compressive Strength of Recycled Aggregate Concrete by Artificial Neural Network, Model Tree and Non-linear Regression, International Journal of Sustainable Built Environment, Dec. 2014, p… [cited by applicant]
N. Deshpande et al., Modelling Compressive Strength of Recycled Aggregate Concrete Using Neural Networks and Regression, Concrete Research Letters, vol. 4(2), Jun. 2013, pp. 580-590. [cited by applicant]
N. Sharma et al., Incorporating Data Mining Techniques on Software Cost Estimation: Validation and Improvement, International Journal of Emerging Technology and Advanced Engineering, vol. 2, Mar. 2012, pp. 301-309. [cited by applicant]
Nikolaev, Regularization Approach to inductive Genetic Programming, Aug. 2001, IEEE, vol. 5, No. 4, pp. 359-375. [cited by applicant]
P. Ramesh, Prediction of Cost Overruns Using Ensemble Methods in Data Mining and Text Mining Algorithms, Master's Thesis, Graduate Program in Civil and Environmental Engineering, Rutgers University, Jan. 2014, 50 pages. [cited by applicant]
P.J. Edwards et al., The application of neural networks to the paper-making industry, Proceedings of the European Symposium on Artificial Neural Networks, Apr. 1999, 6 pages. [cited by applicant]
R. Strapasson et al., Tensile and impact behavior of polypropylene/low density polyethylene blends, Polymer Testing 24, 2005, pp. 468-473. [cited by applicant]
S. Zheng, Boosting Based Conditional Quantile Estimation for Regression and Binary Classification, Proceedings of the 9th Mexican international conference on Artificial intelligence, 2010, pp. 67-79. [cited by applicant]
S.C. Lhee et al., Development of a two-step neural network-based model to predict construction cost contingency, Journal of Information Technology in Construction, vol. 19, Sep. 2014, pp. 399-411. [cited by applicant]
S.T. Yousif et al., Artificial Neural Network Model for Predicting Compressive Strength of Concrete, Tikrit Journal of Engineering Sciences, vol. 16, 2009, pp. 55-63. [cited by applicant]
Schmidt et al., “Comparison of Tree and Graph Endcodings as Function of Problem Complexity,” in Proceedings of the 9th Annual Conf. Genetic and Evolutionary Comp., ACM, 2007, pp. 1674-1679. [cited by applicant]
Schmidt, Michael D. et al. “Automated refinement and inference of analytical models for metabolic networks,” Aug. 10, 2011, Physical Biology, vol. 8, No. 5. 36 pages. [cited by applicant]
Schmidt, Michael et al., “Distilling Free-Form Natural Laws from Experimental Data,” Apr. 3, 2009, Science, vol. 324, No. 5923, pp. 81-85. [cited by applicant]
Smits et al., “Pareto-Front Exploitation in Symbolic Regression”, in Genetic Programming Theory and Practice II, Springer, Boston, MA, 2005, pp. 283-299. [cited by applicant]
T. Kraska et al. MLbase: A distributed machine-learning system. In Proceedings of 6th Biennial Conference on Innovative Data Systems Research (CIDR'13), Jan. 6, 2013 (7 pages). Available at <http://cidrdb.org/cidr2013/P… [cited by applicant]
V. Chandwani et al., Applications of Soft Computing in Civil Engineering: A Review, International Journal of Computer Applications, vol. 81, Nov. 2013, pp. 13-20. [cited by applicant]
V. Chandwani et al., Modeling Slump of Ready Mix Concrete Using Genetically Evolved Artificial Neural Networks, Advances in Artificial Neural Systems, Nov. 2014, 9 pages. [cited by applicant]
Vanneschi et al., “Measuring Bloat, Overfitting and Functional Complexity in Genetic Programming,” in Proceedings of the 12th Annual Conf. Genetic and Evolutionary Comp., ACM, 2010, pp. 877-884. [cited by applicant]
Vladislavleva et al., Order of Nonlinearity as a Complexity Measure for Models Generated by Symbolic Regression via Pareto Genetic Programming. IEEE Transactions on Evolutionary Computation. 2008; 13(2): 333-49. [cited by applicant]
X. He et al., Practical Lessons from Predicting Clicks on Ads at Facebook, Proceedings of ADKDD'14, Aug. 2014, 9 pages. [cited by applicant]
Y. Shan et al., Machine Learning of Poorly Predictable Ecological Data, Ecological Modeling, vol. 195, 2006, pp. 129-138. [cited by applicant]
Y.X. Zhao et al., Concrete cracking process induced by steel corrosion—A review, Proceedings of the Thirteenth East Asia-Pacific Conference on Structural Engineering and Construction, Sep. 2013, pp. 1-10. [cited by applicant]
Z.A. Khalifelu et al., Comparison and evaluation of data mining techniques with algorithmic models in software cost estimation, Procedia Technology, vol. 1, 2012, pp. 65-71. [cited by applicant]
U.S. Appl. No. 17/112,729, filed Dec. 4, 2020, Achin et al. [cited by applicant]
U.S. Appl. No. 16/682,813, filed Nov. 13, 2019, Achin et al. [cited by applicant]
U.S. Appl. No. 16/722,831, filed Dec. 20, 2019, Achin et al. [cited by applicant]
U.S. Appl. No. 16/506,219, filed Jul. 9, 2019, Bledsoe et al. [cited by applicant]
G. James et al. An Introduction to Statistical Learning with Applications in R, 2013, 441 pages. [cited by applicant]
I-C. Yeh, Modeling of strength of high performance concrete using artificial neural networks, Cement and Concrete Research, vol. 28, No. 12, 1998, pp. 1797-1808. [cited by applicant]
P. Love et al., Determining the Probability of Project Cost Overruns, Journal of Construction Engineering and Management, vol. 139, Mar. 2013, pp. 321-330. [cited by applicant]
T. Hastie et al., The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed., Feb. 2009, 758 pages, available at http://web.stanford.edu/˜hastie/local.ftp/Springer/OLD/ESLII_print4.pdf. [cited by applicant]
Yang, F. et al., “Forecasting time series with genetic programming based on least square method,” J. Syst. Sci. Complex, vol. 27 (2014), pp. 117-129. [cited by applicant]
Zamani et al., “Benchmark problems solving with the complexity of genetic programming”, 1.1 Int'l J. Applies Res. Computer Sci. Info. Tech., pp. 13-26 (2012). [cited by applicant]