System and method of facial analysis
A system for facial analysis includes a camera, a data storage device and a data processing system. The camera takes video of a subject's face, and the data storage device receives and stores the video. The data processing system extracts a pose of the subject's face, and a representation of the subject's facial gesture state. The pose includes the angle and position of the subject's face. The representation includes facial keypoints that are a collection of points on the subject's face. The system then concatenates each data stream to align the data streams in time, extracts a plurality of facial syllables from the aligned data streams, and compiles the facial syllables into a series of state sequences. Based on the series of state sequences, the system extracts a behavioral fingerprint for the subject that provides a summary of the subject's state over a given period of time.
1. A system for facial analysis comprising:
a camera configured to take a video of a subject's face, the video having a plurality of frames;
a data storage device configured to receive and store the video; and
a data processing system having a processor and program code which when executed:
(a) generates a pose data stream by extracting, from the plurality of frames, a plurality of poses of the subject's face, each pose of the plurality of poses including an angle and a position of the subject's face,
(b) generates a facial gestures state data stream by extracting, from the plurality of frames, a plurality of representations of the subject's facial gesture state, each representation of the plurality of representations including facial keypoints, the facial keypoints being a collection of points on the subject's face,
(c) concatenates the pose data stream and the facial gestures state data stream to align the data streams in time,
(d) extracts a plurality of facial syllables from the aligned data streams,
(e) compiles the plurality of facial syllables into a series of state sequences, and
(f) extracts a behavioral fingerprint for the subject based on the series of state sequences, the behavioral fingerprint providing a summary of the subject's state over a given period of time, wherein extracting the behavioral fingerprint includes calculating a histogram of state frequencies based on the series of state sequences, the histogram indicative of how often each of the plurality of facial syllables occurs in the series of state sequences.
2. A system according to claim 1 , wherein the camera is a two-dimensional camera or a three-dimensional camera.
3. A system according to claim 1 , wherein the data processing system also has program code that extracts a face region from each of the frames of the video prior to extracting the pose of the subject's face.
4. A system according to claim 1 , wherein the facial keypoints include at least one selected from the group consisting of the subject's nose, mouth, eyes and jaw line.
5. A system according to claim 1 , wherein the data processing system uses latent embeddings derived from artificial neural networks and/or deep learning models to extract the facial gesture state.
6. A system according to claim 1 , wherein the processing system is configured to analyze videos in aggregate to extract the facial syllables.
7. A system according to claim 1 , wherein the processing system is configured to predict, based on the behavioral fingerprint, at least one selected from the group consisting of a level of pain, a level of anxiety, a level of depression, a level of hunger, a level of satiety, and a level of fatigue.
8. A system according to claim 1 , wherein the processing system is configured to classify the behavioral summary as a pre-event summary or a post event summary.
9. The system according to claim 1 , wherein extracting the plurality of facial syllables includes applying a sticky hierarchical Dirichlet process (HDP), autoregressive-emission (AR) hidden Markov model (HMM), wherein an HDP component is usable to automatically discover the number of states, an AR component is usable to model facial expressions as a smoothly varying trajectory over time, an HMM component is usable to model a grammatical structure between syllables, and a sticky modifier to the HMM is usable to model syllable durations.
10. A method of facial analysis comprising:
recording a video of a subject's face using a camera, the video having a plurality of frames;
storing the video in a data storage device;
generating a pose data stream by extracting, from the plurality of frames, a plurality of poses of the subject's face, each pose of the plurality of poses including an angle and a position of the subject's face;
generating a facial gestures state data stream by extracting, from the plurality of frames, a plurality of representations of the subject's facial gesture state, each representation of the plurality of representations including facial keypoints, the facial keypoints being a collection of points on the subject's face;
concatenating the pose data stream and the facial gestures state data stream to align the data streams in time;
extracting a plurality of facial syllables from the aligned data streams;
compiling the plurality of facial syllables into a series of state sequences; and
extracting a behavioral fingerprint for the subject based on the series of state sequences, the behavioral fingerprint providing a summary of the subject's state over a given period of time, wherein extracting the behavioral fingerprint includes calculating a histogram of state frequencies based on the series of state sequences, the histogram indicative of how often each of the plurality of facial syllables occurs in the series of state sequences.
11. A method according to claim 10 , wherein the camera is a two-dimensional camera or a three-dimensional camera.
12. A method according to claim 10 , further comprising:
extracting a face region from each of the frames of the video prior to extracting the pose of the subject's face.
13. A method according to claim 10 , wherein the facial keypoints include at least one selected from the group consisting of the subject's nose, mouth, eyes and jaw line.
14. A method according to claim 10 , wherein extracting the facial gesture state includes using latent embeddings derived from artificial neural networks and/or deep learning models to extract the facial gesture state.
15. A method according to claim 10 , wherein extracting the facial syllables includes analyzing videos in aggregate.
16. A method according to claim 10 , further comprising predicting, based on the behavioral fingerprint, at least one selected from the group consisting of a level of pain, a level of anxiety, a level of depression, a level of hunger, a level of satiety, and a level of fatigue.
17. A method according to claim 10 , further comprising classifying the behavioral summary as a pre-event summary or a post event summary.
18. The method according to claim 10 , wherein extracting the plurality of facial syllables includes applying a sticky hierarchical Dirichlet process (HDP), autoregressive-emission (AR) hidden Markov model (HMM), wherein an HDP component is usable to automatically discover the number of states, an AR component is usable to model facial expressions as a smoothly varying trajectory over time, an HMM component is usable to model a grammatical structure between syllables, and a sticky modifier to the HMM is usable to model syllable durations.
19. A system for subject analysis comprising:
a camera configured to take a video of a portion of a subject, the video having a plurality of frames;
a data storage device configured to receive and store the video; and
a data processing system having a processor and program code which when executed:
(a) generates a pose data stream by extracting, from the plurality of frames, a plurality of poses of the portion of the subject, each pose of the plurality of poses including an angle and a position of the portion of the subject,
(b) generates a gestures state data stream by extracting, from the plurality of frames, a plurality of representations of the subject's gesture state, each representation of the plurality of representations including keypoints, the keypoints being a collection of points on the portion of the subject,
(c) concatenates the pose data stream and the gestures state data stream to align the data streams in time,
(d) extracts a plurality of syllables from the aligned data streams,
(e) compiles the plurality of syllables into a series of state sequences, and
(f) extracts a behavioral fingerprint for the subject based on the series of state sequences, the behavioral fingerprint providing a summary of the subject's state over a given period of time, wherein extracting the behavioral fingerprint includes calculating a histogram of state frequencies based on the series of state sequences, the histogram indicative of how often each of the plurality of syllables occurs in the series of state sequences.
20. A system according to claim 19 , wherein the camera is a two-dimensional camera or a three-dimensional camera.
21. A system according to claim 19 , wherein the portion of the subject is the subject's face.
22. A system according to claim 21 , wherein the pose of the portion of the subject is the pose of the subject's face, the pose including the angle and position of the subject's face.
23. A system according to claim 22 , wherein the representation of the subject's gesture state is a representation of the subject's facial gesture state, the keypoints being facial keypoints that are a collection of points on the subject's face.
24. A system according to claim 23 , wherein the plurality of syllables are a plurality of facial syllables.
25. A system according to claim 23 , wherein the facial keypoints include at least one selected from the group consisting of the subject's nose, mouth, eyes and jaw line.
26. A system according to claim 19 , wherein the data processing system use latent embeddings derived from artificial neural networks and/or deep learning models to extract the gesture state.
27. A system according to claim 19 , wherein the processing system is configured to analyze videos in aggregate to extract the syllables.
28. A system according to claim 19 , wherein the processing system is configured to predict, based on the behavioral fingerprint, at least one selected from the group consisting of a level of pain based, a level of anxiety, a level of depression, a level of hunger, a level of satiety, and a level of fatigue.
29. A system according to claim 19 , wherein the processing system is configured to classify the behavioral summary as a pre-event summary or a post event summary.
30. A system according to claim 19 , wherein the data processing system also has program code that extracts a region from each of the frames of the video prior to extracting the pose of the portion of the subject.
31. The system according to claim 19 , wherein extracting the plurality of facial syllables includes applying a sticky hierarchical Dirichlet process (HDP), autoregressive-emission (AR) hidden Markov model (HMM), wherein an HDP component is usable to automatically discover the number of states, an AR component is usable to model facial expressions as a smoothly varying trajectory over time, an HMM component is usable to model a grammatical structure between syllables, and a sticky modifier to the HMM is usable to model syllable durations.