IP Library Granted Patent US 10,592,817
Granted Patent B2
US 10,592,817 · App. 14/797,777 · Granted Mar 17, 2020

Parameter-dependent model-blending with multi-expert based machine learning and proxy sites

Inventors: Hendrik F. Hamann (Yorktown Heights, NY); Youngdeok Hwang (White Plains, NY); Levente Klein (Tuckahoe, NY); Jonathan Lenchner (North Salem, NY); Siyuan Lu (Yorktown Heights, NY); Fernando J. Marianno (New York, NY); Gerald J. Tesauro (Croton-on-Hudson, NY); Theodore G. van Kessel (Millbrook, NY)
Assignee: INTERNATIONAL BUSINESS MACHINES CORPORATION
G06N20/00G06F16/285G06F17/10G06F17/5009
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,592,817
App. No.
14/797,777
Granted
Mar 17, 2020
Kind
B2
Abstract

A parameter-based multi-model blending method and system are described. The method includes selecting a parameter of interest among parameters estimated by each of a set of individual models, running the set of individual models with a range of inputs to obtain a range of estimates of the parameters from each of the set of individual models, and identifying, for each of the set of individual models, critical parameters among the parameters estimated, the critical parameters exhibiting a specified correlation with an error in estimation of the parameter of interest. For each subspace of combinations of the critical parameters, obtaining a parameter-based blended model is based on blending the set of individual models in accordance with the subspace of the critical parameters, the subspace defining a sub-range for each of the critical parameters.

Claims (30)

1. A method of performing parameter-based multi-model blending, the method comprising:

selecting a parameter of interest among parameters estimated by each of a set of individual models;

running, using a processor, the set of individual models with a range of inputs to obtain a range of estimates of the parameters from each of the set of individual models, wherein each individual model from the set of individual models is associated with a particular site of a set of sites and wherein each site from the set of sites has a corresponding latitude, longitude, and elevation;

identifying, for each of the set of individual models, critical parameters among the parameters estimated, the critical parameters exhibiting a specified correlation with an error in estimation of the parameter of interest, wherein the identifying the critical parameters includes examining first order dependence and second order dependence of the error in the estimation of the parameter of interest associated with each of the parameters estimated by each of the set of individual models and wherein examining the first order dependence includes translating a first order error dependence curve into two coefficients and examining the second order dependence includes translating a second order error dependence surface into six coefficients;

correlating a type associated with each site with the latitude, the longitude, and the elevation of the site by obtaining, for each subspace defined as a particular combination of sub-ranges of a range of values of the critical parameters, a parameter-based blended model by training a classification machine learning model to correlate one or more machine learning models with corresponding values derived from the subspace of the critical parameters, the subspace defining a sub-range for each of the critical parameters, and inputting the corresponding values derived from the subspace of the critical parameters into the trained classification machine learning model, wherein the correlating includes training the classification machine learning model to correlate the site type with the latitude, the longitude, and the elevation of the site;

forecasting or estimating the parameter of interest using the parameter-based blended model selected, by the trained classification machine learning model, from among the one or more machine learning models based on the critical parameters; and

selecting a site of interest from the set of sites such that the range of inputs correspond with the site of interest.

2. The method according to claim 1 , wherein the identifying the critical parameters includes calculating a variance from the first order dependence associated with each of the parameters estimated by each of the set of individual models.

3. The method according to claim 2 , wherein the identifying the critical parameters includes identifying parameters among the parameters estimated by each of the set of individual models with the associated variance exceeding a threshold value.

4. The method according to claim 1 , wherein the identifying the critical parameters also includes examining higher order dependence of the error in the estimation of the parameter of interest associated with combinations of parameters estimated by each of the set of individual models.

5. The method according to claim 1 , wherein the identifying the critical parameters also includes examining inter-model second order dependence of the error in the estimation of the parameter of interest associated, the inter-model second order dependent of the error referring to a correlation between error dependence in estimation of the parameter of interest by a first model among the set of individual models on a first parameter and error dependence in estimation of the parameter of interest by a second model among the set of individual models on a second parameter.

6. The method according to claim 1 , wherein the obtaining the parameter-based blended model for each subspace of combinations of the critical parameters includes determining, by the trained classification machine learning model, a most accurate machine learning model among the one or more machine learning models for each combination of critical parameters.

7. A system to perform parameter-based multi-model blending, the system comprising:

a memory device configured to store inputs, the inputs including the parameter of interest among parameters estimated by each of a set of individual models; and

a processor configured to run the set of individual models with a range of inputs to obtain a range of estimates of the parameters from each of the set of individual models, wherein each individual model from the set of individual models is associated with a particular site of a set of sites and wherein each site from the set of sites has a corresponding latitude, longitude, and elevation, identify, for each of the set of individual models, critical parameters among the parameters estimated, the critical parameters exhibiting a specified correlation with an error in estimation of the parameter of interest, wherein the processor identifies the critical parameters based on examining first order dependence and second order dependence of the error in the estimation of the parameter of interest associated with each of the parameters estimated by each of the set of individual models and wherein he first order dependence is examined based on translating a first order error dependence curve into two coefficients and wherein the second order dependence is examined based on translating a second order error dependence surface into six coefficients, correlate a type associated with each site with the latitude, the longitude, and the elevation of the site by obtaining, for each subspace defined as a particular combination of sub-ranges of a range of values of the critical parameters, a parameter-based blended model by training a classification machine learning model to correlate one or more machine learning models with corresponding values derived from the subspace of the critical parameters, the subspace defining a sub-range for each of the critical parameters, and inputting the corresponding values derived from the subspace of the critical parameters into the trained classification machine learning model, wherein the correlating includes training the classification machine learning model to correlate the site type with the latitude, the longitude, and the elevation of the site, forecast or estimate the parameter of interest using the parameter-based blended model selected, by the trained classification machine learning model, from among the one or more machine learning models based on the critical parameters; and select a site of interest from the set of sites such that the range of inputs correspond with the site of interest.

8. The system according to claim 7 , wherein the processor identifies the critical parameters based on calculating a variance from the first order dependence associated with each of the parameters estimated by each of the set of individual models.

9. The system according to claim 8 , wherein the processor identifies the critical parameters based on identifying parameters among the parameters estimated by each of the set of individual models with the associated variance exceeding a threshold value.

10. The system according to claim 7 , wherein the processor identifies the critical parameters based additionally on examining higher order dependence of the error in the estimation of the parameter of interest associated with combinations of parameters estimated by each of the set of individual models.

11. The system according to claim 7 , wherein the processor identifies the critical parameters based additionally on examining inter-model second order dependence of the error in the estimation of the parameter of interest associated, the inter-model second order dependent of the error referring to a correlation between error dependence in estimation of the parameter of interest by a first model among the set of individual models on a first parameter and error dependence in estimation of the parameter of interest by a second model among the set of individual models on a second parameter.

12. The system according to claim 7 , wherein the processor obtains the parameter-based blended model by determining, by the trained classification machine learning model, a most accurate machine learning model among the one or more machine learning models for each combination of critical parameters.

13. A non-transitory computer program product having computer readable instructions stored thereon which, when executed by a processor, cause the processor to implement a method of multi-model blending, the method comprising:

selecting a parameter of interest among parameters estimated by each of a set of individual models;

running the set of individual models with a range of inputs to obtain a range of estimates of the parameters from each of the set of individual models, wherein each individual model from the set of individual models is associated with a particular site of the set of sites and wherein each site from the set of sites has a corresponding latitude, longitude, and elevation;

identifying, for each of the set of individual models, critical parameters among the parameters estimated, the critical parameters exhibiting a specified correlation with an error in estimation of the parameter of interest, wherein the identifying the critical parameters includes examining first order dependence and second order dependence of the error in the estimation of the parameter of interest associated with each of the parameters estimated by each of the set of individual models and wherein examining the first order dependence includes translating a first order error dependence curve into two coefficients and examining the second order dependence includes translating a second order error dependence surface into six coefficients;

correlating a type associated with each site with the latitude, the longitude, and the elevation of the site by obtaining, for each subspace defined as a particular combination of sub-ranges of a range of values of the critical parameters, a parameter-based blended model by training a classification machine learning model to correlate one or more machine learning models with corresponding values derived from the subspace of the critical parameters, the subspace defining a sub-range for each of the critical parameters, and inputting the corresponding values derived from the subspace of the critical parameters into the trained classification machine learning model, wherein the correlating includes training the classification machine learning model to correlate the site type with the latitude, the longitude, and the elevation of the site;

forecasting or estimating the parameter of interest using the parameter-based blended model selected, by the trained classification machine learning model, from among the one or more machine learning models based on the critical parameters; and

selecting a site of interest from the set of sites such that the range of inputs correspond with the site of interest.

14. The non-transitory computer program product according to claim 13 , wherein the identifying the critical parameters also includes examining higher order dependence of the error in the estimation of the parameter of interest associated with combinations of parameters estimated by each of the set of individual models.

15. The non-transitory computer program product according to claim 13 , wherein the identifying the critical parameters also includes examining inter-model second order dependence of the error in the estimation of the parameter of interest associated, the inter-model second order dependent of the error referring to a correlation between error dependence in estimation of the parameter of interest by a first model among the set of individual models on a first parameter and error dependence in estimation of the parameter of interest by a second model among the set of individual models on a second parameter.

16. The non-transitory computer program product according to claim 13 , wherein the obtaining the parameter-based blended model for each subspace of combinations of the critical parameters includes determining, by the trained classification machine learning model, a most accurate machine learning model among the one or more machine learning models for each combination of critical parameters.

Assignments (3)
CONFIRMATORY LICENSE Recorded Sep 16, 2024
From: INTERNATIONAL BUSINESS MACHINES CORPORATION
To: UNITED STATES DEPARTMENT OF ENERGY
Reel/Frame 068961/0148 →
CONFIRMATORY LICENSE Recorded Jan 15, 2016
From: INTERNATIONAL BUSINESS MACHINES CORPORATION
To: ENERGY, UNITED STATES DEPARTMENT OF
Reel/Frame 037565/0132 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 13, 2015
From: HAMANN, HENDRIK F.; HWANG, YOUNGDEOK; KLEIN, LEVENTE; LENCHNER, JONATHAN; LU, SIYUAN; MARIANNO, FERNANDO J.; TESAURO, GERALD J.; VAN KESSEL, THEODORE G.
To: INTERNATIONAL BUSINESS MACHINES CORPORATION
Reel/Frame 036071/0236 →
Continuity (1)
Related Publication 20170017732A1 · Jan 19, 2017