System and method for utilizing a model to process big data
View Patent ↗A model is implemented that includes one or more classes. For each of the one or more classes of the model, a set of input variables of the big data set are represented as a matrix with non-zero values only provided as diagonal entries. A most likely class for each input variable is determined based at least in part on inverting the matrix. One or more predictions are determined for one or more output variables based at least in part on the most likely class of one or more input variables from the set of input variables.
1. A method for determining one or more predictions for a big data set, the method being implemented by one or more processors and comprising:
receiving at least a portion of the big data set;
implementing a model that includes one or more classes, the model associating a probability with each class of the one or more classes;
representing, for each of the one or more classes of the model, a set of input variables of the big data set as a matrix with non-zero values only provided as diagonal entries;
determining a most likely class for each input variable based at least in part on inverting the matrix; and
determining one or more predictions for one or more output variables based at least in part on the most likely class of one or more input variables from the set of input variables.
2. The method of claim 1 , wherein the model is based at least in part on (i) a covariance matrix amongst output variables a set of output variables for each class of the one or more classes, and (ii) a covariance matrix between the output variables the input variables for each class of the one or more classes.
3. The method of claim 2 , wherein the matrix of the set of input variables is also a covariance matrix.
4. The method of claim 1 , further comprising generating an output based on the one or more predictions.
5. A method for developing a model for analyzing a big data set, the method being implemented by one or more processors and comprising:
(a) determining a set of classes for the model based on a probability distribution;
(b) determining a most likely class for a set of training data;
wherein determining the most likely class includes (i) representing, for each of the one or more classes of the model, a set of input variables of the model as a matrix with non-zero values only provided as diagonal entries, and (ii) inverting the matrix; and
(c) updating one or more input variables of the model using the set of training data and the most likely class determined for the training data.
6. The method of claim 5 , wherein the set of input variables are represented as a covariance matrix with the non-zero values only provided as diagonal entries.
7. The method of claim 5 , wherein (a) through (c) are repeated until the model is converged.
8. The method of claim 7 , further comprising determining when the model is converged by determining one or more of (i) a change in a value of one or more of the input parameters is less than a threshold between one or more successive iterations of (a) through (c); or (ii) a class assigned to a given set of training data remains the same between one or more successive iterations of (a) through (c).
9. The method of claim 5 , further comprising inferring a set of input variables that are missing from the model based on input variables that are not missing in the model.
10. A non-transitory computer-readable medium that stores instructions, which when executed by one or more processors, cause the one or more processors to perform operations that include:
receiving at least a portion of a big data set;
implementing a model that includes one or more classes; the model associating a probability with each class of the one or more classes;
representing, for each of the one or more classes of the model, a set of input variables of the big data set as a matrix with non-zero values only provided as diagonal entries;
determining a most likely class for each input variable based at least in part on inverting the matrix; and
determining one or more predictions for one or more output variables based at least in part on the most likely class of one or more input variables from the set of input variables.
11. The computer-readable medium of claim 10 , wherein the model is based at least in part on (i) a covariance matrix amongst output variables a set of output variables for each class of the one or more classes, and (ii) a covariance matrix between the output variables the input variables for each class of the one or more classes.
12. The computer-readable medium of claim 11 , wherein the matrix of the set of input variables is also a covariance matrix.
13. The computer-readable medium of claim 10 , further comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations that include generating an output based on the one or more predictions.