Biomarker determination using optical flows
A computer-implemented method includes obtaining a video of a subject, the video including a plurality of frames; generating, based on the plurality of frames, a plurality of optical flows; and encoding the plurality of optical flows using an autoencoder to obtain a movement-based biomarker value of the subject.
1. A computer-implemented method comprising:
obtaining a video of a subject, the video comprising a plurality of frames;
generating, based on both a subject portion and a background portion of each frame of the plurality of frames, a plurality of optical flows representing movement in the plurality of frames; and
providing the plurality of optical flows as inputs to an autoencoder to obtain, as an output of the autoencoder, a representation having a reduced dimensionality compared to the plurality of optical flows, the representation including a movement-based biomarker value of the subject,
wherein the autoencoder is trained based on comparisons between training optical flows and reconstructed optical flows generated based on autoencoder processing of the training optical flows.
2. The computer-implemented method of claim 1 , wherein the movement-based biomarker value comprises a frequency of tremor of the subject.
3. The computer-implemented method of claim 2 , wherein the representation comprises a type of tremor of the subject.
4. The computer-implemented method of claim 3 , wherein the type of tremor comprises a hand position of the subject.
5. The computer-implemented method of claim 1 , wherein the representation comprises a biomarker type corresponding to the movement-based biomarker value.
6. The computer-implemented method of claim 5 , wherein the biomarker type comprises a facial muscle group of the subject, and wherein the movement-based biomarker value comprises a measure of intensity of movement of the facial muscle group or a measure of frequency of movement of the facial muscle group.
7. The computer-implemented method of claim 1 , further comprising:
generating a plurality of reconstructed optical flows using an adversarial autoencoder network, the plurality of reconstructed optical flows based on random samples drawn from a prior distribution used to train the autoencoder in an adversarial discrimination process, and
training the autoencoder using the plurality of reconstructed optical flows.
8. The computer-implemented method of claim 1 , further comprising:
obtaining a second plurality of optical flows, the second plurality of optical flows being labeled;
performing one or more of random translation, random rotation, or random scaling on the second plurality of optical flows, to generate an augmenting plurality of optical flows; and
training the autoencoder using the augmenting plurality of optical flows.
9. The computer-implemented method of claim 1 , further comprising training the autoencoder using an adversarial discriminator, comprising:
comparing, by the adversarial discriminator, the output of the autoencoder to a distribution; and
updating parameters of the autoencoder based on a difference between the output of the autoencoder and the distribution.
10. The computer-implemented method of claim 1 , comprising training the autoencoder using labeled data.
11. The computer-implemented method of claim 10 , wherein the labeled data comprises experimentally-derived data, the experimentally-derived data comprising data generated by stimulating a second subject with stimulation having a known frequency.
12. The computer-implemented method of claim 10 , wherein the labeled data is labeled with a biomarker type, and
wherein training the autoencoder comprises training the autoencoder to determine a biomarker value based on implicit training.
13. The computer-implemented method of claim 10 , wherein the labeled data is labeled with a biomarker value, and
wherein training the autoencoder comprises training the autoencoder to determine a biomarker type based on implicit training.
14. The computer-implemented method of claim 1 , wherein generating the plurality of optical flows comprises:
processing the video with one or more of filtering, noise-reduction, or standardization, to generate a plurality of processed video frames; and
generating the plurality of optical flows based on the plurality of processed video frames.
15. The computer-implemented method of claim 1 , comprising generating the plurality of optical flows based on respective pairs of frames of the plurality of frames.
16. The computer-implemented method of claim 1 , wherein providing the plurality of optical flows as inputs to the autoencoder comprises providing one or more optical flow maps as inputs to the autoencoder, the optical flow maps generated based on the plurality of optical flows.