Systems and methods for building a prediction model with varying levels of data availability
Systems, methods, and computer program products described herein for building a prediction model with varying availability. Embodiments described herein build a prediction model that is initially trained with a selected subset of data characteristics (e.g., data characteristics that are commonly available in training data), and then augmented via Bayesian estimation with additional data characteristics when such data characteristics are available.
1 . A method for training a machine learning prediction generation model including a base prediction model and a Bayesian estimation layer connected on top of the base prediction model with varying levels of data availability, the method comprising:
obtaining, via a communication interface, training data having a plurality of training data samples, each of the training data samples comprising a set of data characteristics;
determining a subset of data characteristics from the set of data characteristics for building the base prediction model;
training the base prediction model to generate a prediction based on the subset of data characteristics, using the plurality of data samples, each of the training data samples comprising only the subset of data characteristics, wherein the training further comprises:
generating a training input vector combining the subset of data characteristics of the at least one data sample, and
updating the base prediction model based on a loss corresponding to a training output that is generated by the base prediction model from the training input vector;
receiving a data query comprising information corresponding to the subset of data characteristics and additional information corresponding to at least one data characteristic outside the subset of data characteristics;
generating, by the trained base prediction model of the machine learning prediction generation model, a first prediction value based on an input of only the information corresponding to the subset of data characteristics comprised in the data query without the additional information corresponding to the at least one data characteristic outside the subset of data characteristics;
generating, by the Bayesian estimation layer of the machine learning prediction generation model, a Bayesian statistical estimate of a conditional probability of the additional information corresponding to the at least one data characteristic conditioned on the first prediction value;
computing a second prediction value based on the first prediction value and the Bayesian statistical estimate of the conditional probability;
implementing the trained machine learning prediction generation model including the trained base prediction model and the Bayesian estimation layer at an autonomous driving vehicle to predict a traffic condition, wherein the trained base prediction model is trained using the training data samples comprising only the subset of data characteristics;
refining, a predicted traffic condition generated by the trained base prediction model, based on the Bayesian statistical estimate of the conditional probability of the at least one data characteristic, when the additional information is available without retraining the base prediction model; and
controlling an auto-navigation of the autonomous driving vehicle assisted by the refined predicted traffic condition.
2 . The method of claim 1 , wherein the at least one characteristic is unavailable at a time of training the base prediction model.
3 . The method of claim 1 , wherein the second prediction value is computed by a product of the first prediction value and the statistical estimate of the conditional probability divided by a probability of the at least one characteristic.
4 . The method of claim 3 , wherein the probability of the at least one characteristic is generated from historical data of the at least one characteristic.
5 . The method of claim 1 , wherein the data query comprises another data characteristic that is also outside the subset of data characteristics, and the method comprises:
generating another statistical estimate of another conditional probability of the another data characteristic conditioned on the first prediction value; and
computing a third prediction value based on a product of the second prediction value and the another conditional probability.
6 . The method of claim 1 , wherein the data query comprises a plurality of data characteristics that are all outside the subset of data characteristics, and the plurality of data characteristics are independent from each other, and the method comprises:
generating, by the trained base prediction model, a base prediction value based on an input of the data query only including data characteristics that belong to the subset of data characteristics;
generating, for each data characteristic in the plurality of data characteristics, a respective statistical estimate of a respective conditional probability of the respective data characteristic conditioned on the base prediction value; and
computing an updated prediction value based on a product of statistical estimates of conditional probabilities and the base prediction value divided by a marginal probability of the plurality of data characteristics.
7 . The method of claim 1 , wherein the training data contains no data sample having the at least one data characteristic at a time when the base prediction model is trained.
8 . The method of claim 1 , wherein the data query comprises a plurality of data characteristics that are all outside the subset of data characteristics, and a first data characteristic and a second data characteristic from the plurality are correlated, and the method comprises:
generating, by the trained base prediction model, a base prediction value based on an input of the data query only including data characteristics that belong to the subset of data characteristics;
generating, for the first data characteristic and the second data characteristic, a statistical estimate of a joint conditional probability of the first data characteristic and the second data characteristic conditioned on the base prediction value;
generating, for other data characteristics in the plurality of data characteristics, a respective statistical estimate of a respective conditional probability of the respective data characteristic conditioned on the base prediction value; and
computing an updated prediction value based on a product of statistical estimates of conditional probabilities of independent data characteristics, the statistical estimate of the joint conditional probability and the base prediction value, divided by a marginal probability of the plurality of data characteristics.
9 . The method of claim 1 , wherein the subset of data characteristics for building the base prediction model are determined based on historical availability of the set of data characteristics.
10 . The method of claim 1 , wherein the subset of input characteristics for building the base prediction model are determined by:
building multiple prediction models based on different training datasets corresponding to different sets of data characteristics;
generating testing outputs from the multiple prediction models in response to a same testing input query; and
identifying relevant data characteristics for building the base prediction model depending on whether the testing outputs are within an accuracy range.
11 . The method of claim 10 , further comprising:
training a first prediction model based on a first input comprising a first set of data characteristics;
generating, by the trained first prediction model, a first testing output in response to a first testing input;
determining whether the first testing output is within an accuracy range compared with a first ground-truth output associated with the first testing input.
12 . The method of claim 11 , further comprising:
in response to determining that the first testing output is outside the accuracy range:
adding another data characteristics to the first set of data characteristics for training a second prediction model;
generating, by the second prediction model, a second testing output in response to a second testing input; and
determining whether the second testing output is within the accuracy range compared with a second ground-truth output associated with the second testing input.
13 . The method of claim 12 , further comprising:
progressively adding data characteristics to the first set of data characteristics for training a new prediction model until a resulting testing output is within the accuracy range; and
determining that the first set of data characteristics and the added data characteristics form the subset of data characteristics for building the base prediction model.
14 . A system training a machine learning model including a base prediction model and a Bayesian estimation layer connected on top of the base prediction model with varying levels of data availability, the system comprising:
a communication interface that obtaining training data having a plurality of training data samples, each of the training data samples comprising a set of data characteristics;
a memory storing a plurality of processor-executable instructions; and
a processor executing the instructions to perform operations comprising:
determining a subset of data characteristics from the set of data characteristics for building the base prediction model;
training the base prediction model using the plurality of data samples to generate a prediction based on the subset of data characteristics, each of the training data samples comprising only the subset of data characteristics, wherein the training further comprises:
generating a training input vector combining the subset of data characteristics of the at least one data sample, and
updating the base prediction model based on a loss corresponding to a training output that is generated by the base prediction model from the training input vector;
receiving a data query comprising information corresponding to the subset of data characteristics and additional information corresponding to at least one data characteristic outside the subset of data characteristics;
generating, by the trained base prediction model, a first prediction value based on an input of only the information corresponding to the subset of data characteristics comprised in the data query without the additional information corresponding to the at least one data characteristic outside the subset of data characteristics;
generating, by the Bayesian estimation layer of the machine learning prediction generation model, a Bayesian statistical estimate of a conditional probability of the additional information corresponding to the at least one data characteristic conditioned on the first prediction value;
computing a second prediction value based on the first prediction value and the Bayesian statistical estimate of the conditional probability;
implementing the trained machine learning prediction generation model including the trained base prediction model and the Bayesian estimation layer at an autonomous driving vehicle to predict a traffic condition, wherein the trained base prediction model is trained using the training data samples comprising only the subset of data characteristics;
refining, a predicted traffic condition generated by the trained base prediction model, based on the Bayesian statistical estimate of the conditional probability of the at least one data characteristic, when the additional information is available without retraining the base prediction model; and
controlling an auto-navigation of the autonomous driving vehicle assisted by the refined predicted traffic condition.
15 . The system of claim 14 , wherein the data query comprises a plurality of data characteristics that are all outside the subset of data characteristics, and the plurality of data characteristics are independent from each other, and the operations comprise:
generating, by the trained base prediction model, a base prediction value based on an input of the data query only including data characteristics that belong to the subset of data characteristics;
generating, for each data characteristic in the plurality of data characteristics, a respective statistical estimate of a respective conditional probability of the respective data characteristic conditioned on the base prediction value; and
computing an updated prediction value based on a product of statistical estimates of conditional probabilities and the base prediction value divided by a marginal probability of the plurality of data characteristics.
16 . The system of claim 14 , wherein the data query comprises a plurality of data characteristics that are all outside the subset of data characteristics, and a first data characteristic and a second data characteristic from the plurality are correlated, and the operations comprise:
generating, by the trained base prediction model, a base prediction value based on an input of the data query only including data characteristics that belong to the subset of data characteristics;
generating, for the first data characteristic and the second data characteristic, a statistical estimate of a joint conditional probability of the first data characteristic and the second data characteristic conditioned on the base prediction value;
generating, for other data characteristics in the plurality of data characteristics, a respective statistical estimate of a respective conditional probability of the respective data characteristic conditioned on the base prediction value; and
computing an updated prediction value based on a product of statistical estimates of conditional probabilities of independent data characteristics, the statistical estimate of the joint conditional probability and the base prediction value, divided by a marginal probability of the plurality of data characteristics.
17 . The system of claim 14 , wherein the subset of input characteristics for building the base prediction model are determined by:
building multiple prediction models based on different training datasets corresponding to different sets of data characteristics;
generating testing outputs from the multiple prediction models in response to a same testing input query; and
identifying relevant data characteristics for building the base prediction model depending on whether the testing outputs are within an accuracy range.
18 . The system of claim 14 , wherein the subset of input characteristics for building the base prediction model are determined by:
training a first prediction model based on a first input comprising a first set of data characteristics;
generating, by the trained first prediction model, a first testing output in response to a first testing input;
determining whether the first testing output is within an accuracy range compared with a first ground-truth output associated with the first testing input;
in response to determining that the first testing output is outside the accuracy range:
adding another data characteristics to the first set of data characteristics for training a second prediction model;
generating, by the second prediction model, a second testing output in response to a second testing input; and
determining whether the second testing output is within the accuracy range compared with a second ground-truth output associated with the second testing input;
progressively adding data characteristics to the first set of data characteristics for training a new prediction model until a resulting testing output is within the accuracy range; and
determining that the first set of data characteristics and the added data characteristics form the subset of data characteristics for building the base prediction model.
19 . A non-transitory processor-readable storage medium storing a plurality of processor-executable instructions for training a machine learning prediction generation model including a base prediction model and a Bayesian estimation layer connected on top of the base prediction model with varying levels of data availability, the instructions being executed by a processor to perform operations comprising:
obtaining, via a communication interface, training data having a plurality of training data samples, each of the training data samples comprising a set of data characteristics;
determining a subset of data characteristics from the set of data characteristics for building the base prediction model;
training the base prediction model using the plurality of data samples to generate a prediction based on the subset of data characteristics, each of the training data samples comprising only the subset of data characteristics, wherein the training further comprises:
generating a training input vector combining the subset of data characteristics of the at least one data sample, and
updating the base prediction model based on a loss corresponding to a training output that is generated by the base prediction model from the training input vector;
receiving a data query comprising information corresponding to the subset of data characteristics and additional information corresponding to at least one data characteristic outside the subset of data characteristics;
generating, by the trained base prediction model, a first prediction value based on an input of only the information corresponding to the subset of data characteristics comprised in the data query without the additional information corresponding to the at least one data characteristic outside the subset of data characteristics;
generating, by the Bayesian estimation layer of the machine learning prediction generation model, a Bayesian statistical estimate of a conditional probability of the additional information corresponding to the at least one data characteristic conditioned on the first prediction value;
computing a second prediction value based on the first prediction value and the Bayesian statistical estimate of the conditional probability;
implementing the trained machine learning prediction generation model including the trained base prediction model and the Bayesian estimation layer at an autonomous driving vehicle to predict a traffic condition, wherein the trained base prediction model is trained using the training data samples comprising only the subset of data characteristics;
refining, a predicted traffic condition generated by the trained base prediction model, based on the Bayesian statistical estimate of the conditional probability of the at least one data characteristic, when the additional information is available without retraining the base prediction model; and
controlling an auto-navigation of the autonomous driving vehicle assisted by the refined predicted traffic condition.