Systems, methods, and devices for biophysical modeling and response prediction
Various systems and methods are disclosed. One or more of the methods disclosed uses machine learning algorithms to predict biophysical responses from biophysical data, such as heart rate monitor data, food logs, or glucose measurements. Biophysical responses may include behavioral responses. Additional systems and methods extract nutritional information from food items by parsing strings containing names of food items.
1. A computer-implemented method for training a reinforcement learning algorithm, the method comprising:
(a) providing a reaction model comprising a personalized learned body model configured to generate a simulated biophysical or behavioral response of a subject responsive to a plurality of inputs;
(b) until a convergence condition is achieved, iteratively:
(i) generating, using the reinforcement learning algorithm, a recommendation comprising a recommended meal or physical activity,
(ii) processing the recommendation using the reaction model to generate a predicted response of the subject to following the recommendation, wherein the predicted response comprises a biophysical or behavioral response,
(iii) applying a first reward function to the predicted response to generate a first reward value indicative of a benefit or detriment of the predicted response toward a health measurement of interest; and
(iv) training the reinforcement learning algorithm in a first stage using the first reward value;
(c) providing the recommendation to the subject;
(d) measuring the health measurement of interest of the subject responsive to following the recommendation;
(e) applying a second reward function to the measured health measurement of interest to generate a second reward value; and
(f) training the reinforcement learning algorithm in a second stage using the second reward value.
2. The method of claim 1 , further comprising encoding the measured health measurement of interest of the subject into a low-dimension latent space for providing to the second reward function.
3. The method of claim 1 , wherein the first reward function is the same as the second reward function.
4. The method of claim 1 , wherein generating the predicted response of the subject further comprises:
processing encoded historical data of the subject using a trained predictor configured to infer the predicted response;
processing the recommendation using an adherence model configured to evaluate an adherence of the subject to the recommendation; and
selectively processing outputs of the trained predictor and the adherence model using the personalized learned body model.
5. The method of claim 4 , wherein generating the predicted response further comprises processing the simulated biophysical response using an autoencoder and a generative adversarial network.
6. The method of claim 1 , wherein the convergence condition comprises a criterion based at least in part on the magnitude of the first reward.
7. The method of claim 1 , wherein the reinforcement learning algorithm comprises a neural network.
8. The method of claim 1 , wherein the first reward function generates the first reward at a higher rate than the second reward function generates the second reward.
9. The method of claim 1 , wherein the reaction model is trained on historical biophysical or behavioral response data from the subject.
10. The method of claim 9 , wherein the reaction model is updated during inference based on new biophysical or behavioral response data of the subject.
11. The method of claim 1 , wherein the reaction model comprises a supervised machine learning model.
12. The method of claim 1 , wherein the first reward function and the second reward function generate a positive reward when the recommendation results in a beneficial predicted response or measured health measurement of interest of the subject.
13. The method of claim 1 , wherein the first reward function and the second reward function generate a negative reward when the recommendation results in a detrimental predicted response or measured health measurement of interest of the subject.
14. The method of claim 1 , wherein the predicted response comprises the biophysical response.
15. The method of claim 14 , wherein the predicted response comprises a predicted glucose response, and wherein the health measurement of interest comprises a glucose level.
16. The method of claim 15 , further comprising measuring the glucose level of the subject using a continuous glucose monitor.
17. The method of claim 1 , wherein the predicted response comprises the behavioral response.
18. The method of claim 1 , wherein training the reinforcement learning algorithm in the first stage or the second stage comprises adjusting a set of weights or parameters of the reinforcement learning algorithm.
19. The method of claim 1 , further comprising using the personalized learned body model to generate the simulated biophysical or behavioral response of the subject in real-time.
20. The method of claim 19 , wherein the simulated biophysical or behavioral response of the subject is generated based at least in part on real-time sensor data of the subject.
21. The method of claim 19 , further comprising:
determining if the simulated biophysical or behavioral response of the subject is outside of a pre-determined range; and
if the simulated biophysical or behavioral response of the subject is outside of the pre-determined range, providing the recommendation to the subject in real-time, wherein the recommendation is selected to adjust an actual biophysical or behavioral response of the subject to be within the pre-determined range.
22. A system comprising one or more computer processors and one or more storage devices having instructions stored thereon that are operable, when executed by the one or more computers, to cause the one or more computer processors to perform operations comprising:
(a) providing a reaction model comprising a personalized learned body model configured to predict a biophysical or behavioral response of a subject responsive to a plurality of inputs;
(b) until a convergence condition is achieved, iteratively:
(i) generating, using the reinforcement learning algorithm, a recommendation comprising a recommended meal or physical activity,
(ii) processing the recommendation using the reaction model to generate a predicted response of the subject to following the recommendation, wherein the predicted response comprises a biophysical or behavioral response,
(iii) applying a first reward function to the predicted response to generate a first reward value indicative of a benefit or detriment of the predicted response toward a health measurement of interest; and
(iv) training the reinforcement learning algorithm in a first stage using the first reward value;
(c) providing the recommendation to the subject;
(d) measuring the health measurement of interest of the subject responsive to following the recommendation;
(e) applying a second reward function to the measured health measurement of interest to generate a second reward value; and
(f) training the reinforcement learning algorithm in a second stage using the second reward value.
23. A non-transitory computer storage medium having instructions stored thereon that are operable, when executed by one or more computer processors, to cause the one or more computer processors to perform operations comprising:
(a) providing a reaction model comprising a personalized learned body model configured to predict a biophysical or behavioral response of a subject responsive to a plurality of inputs;
(b) until a convergence condition is achieved, iteratively:
(i) generating, using the reinforcement learning algorithm, a recommendation comprising a recommended meal or physical activity,
(ii) processing the recommendation using the reaction model to generate a predicted response of the subject to following the recommendation, wherein the predicted response comprises a biophysical or behavioral response,
(iii) applying a first reward function to the predicted response to generate a first reward value indicative of a benefit or detriment of the predicted response toward a health measurement of interest; and
(iv) training the reinforcement learning algorithm in a first stage using the first reward value;
(c) providing the recommendation to the subject;
(d) measuring the health measurement of interest of the subject responsive to following the recommendation;
(e) applying a second reward function to the measured health measurement of interest to generate a second reward value; and
(f) training the reinforcement learning algorithm in a second stage using the second reward value.