Population modeling system based on multiple data sources having missing entries
A neural network is used to model to model the joint distribution of attributes across multiple health surveys. These multiple health surveys include large scale survey datasets and small scale survey datasets. The neural network model is trained using a combined dataset of the large scale survey datasets and the small scale survey datasets. The large scale survey datasets and the small scale survey datasets may include missing value indicators. The joint distribution of attributes modeled by the neural network model are the used to impute substitute values for the missing values to thereby create an output large scale dataset that does not include missing values.
1. A method, comprising:
training a neural network model to model a joint distribution of attributes across multiple health surveys, where the multiple health surveys include a first scale survey datasets and a second scale survey datasets wherein the first scale survey datasets have at least 10 times the number of entries as the second scale survey datasets, the neural network model trained using a combined dataset of the first scale survey datasets and the second scale survey datasets that include missing value indicators; and
using the joint distribution of attributes modeled by the neural network model to impute substitute values for the missing value indicators to create an output first scale dataset that does not include missing value indicators, wherein
the neural network model includes a visible layer comprising visible layer nodes and a hidden layer comprising hidden layer nodes that are configured as a fully connected bipartite graph, and
training the neural network model includes estimating, based on current values of the hidden layer nodes, first values for the visible layer nodes corresponding to the missing value indicators.
2. The method of claim 1 , wherein the neural network model is a Restricted Boltzman machine.
3. The method of claim 1 , wherein the estimating first values for the visible layer nodes corresponding to the missing value indicators is based on sampling of the current values of the hidden nodes according to a first probability distribution function of p(v miss |v part ,h), where v miss are current values of the visible layer nodes corresponding to the missing value indicators, v part are current values of the visible layer nodes not corresponding to the missing value indicators, and h are the current values of the hidden nodes.
4. The method of claim 1 , wherein training the neural network model includes:
alternately Gibbs sampling the visible layer and the hidden layer for k iterations, where k>1.
5. The method of claim 1 , wherein imputing the substitute values for the missing value indicators includes:
estimating, based on the current values of the hidden layer nodes obtained from the trained neural network model, second values for the visible layer nodes corresponding to the missing value indicators.
6. The method of claim 5 , wherein the estimating second values is based on sampling of the current values of the hidden nodes obtained from the trained neural network model according to a second probability distribution function of p(v miss |v part ,h), where v miss are current values of the visible layer nodes corresponding to the missing value indicators, v part are current values of the visible layer nodes not corresponding to the missing value indicators, and h are the current values of the hidden nodes.
7. A system, comprising:
a neural network model operable to model a joint distribution of attributes across multiple health surveys, where the multiple health surveys include first scale survey datasets and second scale survey datasets wherein the first scale survey datasets have at least 10 times the number of entries as the second scale survey datasets, the neural network model trained using a combined dataset of the first scale survey datasets and the second scale survey datasets that include missing value indicators; and
an imputation module to use a joint distribution of attributes modeled by the neural network model to impute substitute values for the missing value indicators to create an output first scale dataset that does not include missing value indicators, wherein
the neural network model includes a visible layer comprising visible layer nodes and a hidden layer comprising hidden layer nodes that are configured as a fully connected bipartite graph, and
the neural network model training included, based on current values of the hidden layer nodes, estimating first values for the visible layer nodes corresponding to the missing value indicators.
8. The system of claim 7 , wherein the neural network model training included estimating first values based on random sampling of the current values of the hidden nodes according to a probability distribution function of p(v miss |v part ,h), where v miss are current values of the visible layer nodes corresponding to the missing value indicators, v part are current values of the visible layer nodes not corresponding to the missing value indicators, and h are the current values of the hidden nodes.
9. The system of claim 7 , wherein the neural network model training included alternately Gibbs sampling the visible layer and the hidden layer for k iterations, where k>1.
10. The system of claim 7 , wherein imputation of the substitute values for the missing value indicators included, based on current values of the hidden layer nodes obtained from the trained neural network model, estimating second values for the visible layer nodes corresponding to the missing value indicators.
11. The system of claim 7 , wherein the neural network is a Restricted Boltzman Machine.