IP Library › Granted Patent US 12,379,777
Granted Patent B2
US 12,379,777 · App. 18/627,673 · Granted Aug 5, 2025

Controlling reading of a head-mounted sensor according to facial expressions

Inventors: Gil Thieberger (Kiryat Tivon, IL); Ari M Frank (Haifa, IL); Tal Thieberger-Navon (Kiryat Tivon, IL)
Assignee: Facense Ltd.
G06F3/013A61B5/0205A61B5/02427A61B5/02438A61B5/14546A61B5/1455A61B5/6803G06V10/141G06V40/166G06V40/174H04N23/611H04N23/651H04N23/951A61B5/02416A61B5/7221H04N25/46
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,379,777
App. No.
18/627,673
Granted
Aug 5, 2025
Kind
B2
Abstract

Disclosed herein are systems and methods for tracking facial expressions efficiently. In one embodiment, a system includes a non-contact head-mounted electro-optical sensor (e.g., an inward-facing head-mounted camera), which measures reflections from a region on a user's head, which are indicative of facial expressions of the user. The system also includes a computer that detects, based on the reflections, a type of facial expression expressed by the user, which belongs to a group comprising neutral facial expressions and non-neutral facial expressions. The computer uses this detection to determine how to proceed to read the electro-optical sensor: the computer reads the electro-optical sensor at a first average bitrate (b 1 ) when the user expresses a facial expression from among the neutral facial expressions, and reads the electro-optical sensor at a second average bitrate (b 2 ) when the user expresses a facial expression from among the non-neutral facial expressions, where b 2 >b 1 .

Claims (34)

1. A system comprising:

a non-contact head-mounted electro-optical sensor configured to measure reflections from a region on a user's head; whereby the reflections are indicative of facial expressions of the user; and

a computer configured to:

detect, based on the reflections, a type of facial expression expressed by the user, which belongs to a group comprising neutral facial expressions and non-neutral facial expressions;

read the electro-optical sensor at a first average bitrate (b 1 ) when the user expresses a facial expression from among the neutral facial expressions; and

read the electro-optical sensor at a second average bitrate (b 2 ) when the user expresses a facial expression from among the non-neutral facial expressions, wherein b 2 >b 1 .

2. The system of claim 1 , wherein the group further comprises transitional facial expressions, and the computer is further configured to read the electro-optical sensor at a third average bitrate (b 3 ) during the transitional facial expressions, wherein b 3 >b 1 .

3. The system of claim 1 , wherein the electro-optical sensor comprises an inward-facing head-mounted camera configured to provide, based on the reflections, images of the region; and wherein the computer is configured to lower the bitrate from b 2 to b 1 by reading the camera using a higher binning value and/or using a smaller region of interest readout.

4. The system of claim 1 , wherein the electro-optical sensor comprises an inward-facing head-mounted camera configured to provide, based on the reflections, images of the region; and wherein the computer is further configured to determine locations of key facial landmarks associated with at least some of the non-neutral facial expressions, and to set the camera's region of interest (ROI) to be around at least some of the key facial landmarks; and wherein the ROI covers less than half of the region.

5. The system of claim 1 , wherein the electro-optical sensor comprises an inward-facing head-mounted camera configured to provide, based on the reflections, images of the region; and wherein the group further comprises transitional facial expressions, and the computer is further configured to determine locations of key facial landmarks associated with a subset of the transitional facial expressions transitioning from the neutral facial expressions, and to set the camera's region of interest (ROI) to be around at least some of said key facial landmarks while the user is in a neutral facial expression from among the neutral facial expressions; and wherein the ROI covers less than half of the region.

6. The system of claim 1 , wherein the electro-optical sensor comprises an inward-facing head-mounted camera configured to provide, based on the reflections, images of the region; and wherein the group further comprises transitional facial expressions, and the computer is further configured to determine locations of key facial landmarks associated with a subset of the transitional facial expressions that occur in transitions between certain non-neutral facial expressions and certain neutral facial expressions, and to set the camera's region of interest (ROI) to be around at least some of said key facial landmarks while the user is in a certain non-neutral facial expression selected from the non-neutral facial expressions; and wherein the ROI covers less than half of the region.

7. The system of claim 1 , wherein the electro-optical sensor comprises an inward-facing head-mounted camera configured to provide, based on the reflections, images of the region; and wherein the group further comprises transitional facial expressions, and the computer is further configured to determine locations of key facial landmarks associated with a subset of the transitional facial expressions that occur in transitions between certain non-neutral facial expressions and other non-neutral facial expressions belonging to the non-neutral facial expressions, and to set the camera's region of interest (ROI) to be around at least some of said key facial landmarks while the user is in a certain non-neutral facial expression selected from the certain non-neutral facial expressions; and wherein the ROI covers less than half of the region.

8. The system of claim 1 , wherein the electro-optical sensor comprises an inward-facing head-mounted camera configured to provide, based on the reflections, images of the region; and wherein the computer is further configured to perform the following responsive to the user expressing a happy smiling facial expression, from among the non-neutral facial expressions: determine expected locations, in the images, of skin wrinkles at the edges of the user's eyes while expressing the happy smiling facial expression, and to set the camera's region of interest (ROI) to include at least a portion of said expected locations of the skin wrinkles while the user expresses the happy smiling facial expression.

9. The system of claim 1 , wherein the electro-optical sensor comprises an inward-facing head-mounted camera configured to provide, based on the reflections, images of the region; and wherein the computer is further configured to perform the following responsive to the user expressing a smiling facial expression, from among the non-neutral facial expressions: determine expected locations, in the images, of the user's oral commissures while expressing the smiling facial expression, and to set the camera's region of interest (ROI) to include an expected location of at least one of the oral commissures while the user expresses the smiling facial expression.

10. The system of claim 1 , wherein the electro-optical sensor comprises an inward-facing head-mounted camera configured to provide, based on the reflections, images of the region; and wherein the computer is further configured to perform the following responsive to the user expressing an angry facial expression, from among the non-neutral facial expressions: determine expected locations, in the images, of the user's lips while expressing the angry facial expression, and to set the camera's region of interest (ROI) to be around at least a portion of an expected location of the lips while the user expresses the angry facial expression.

11. The system of claim 1 , wherein the computer is further configured to detect the type of facial expression utilizing a real-time facial expression finite-state machine that is implemented utilizing at least one of the following: a neural network, a Bayesian network, a rule-based classifier, a support vector machine, a hidden Markov model, a deep learning model, and a deep sparse autoencoder.

12. The system of claim 1 , wherein the electro-optical sensor comprises: light sources configured to emit light towards the region, and discrete photosensors, spread over more than 2 cm, configured to measure reflections of the light from the region.

13. The system of claim 12 , wherein the system further comprises a head-mounted movement sensor configured to measure movements, and the computer is further configured to read the electro-optical sensor at the first average bitrate when the movements are below a threshold, and to read the electro-optical sensor at the second average bitrate, which is higher than the first average bitrate, when the movements are above the threshold.

14. A method comprising:

measuring, by a non-contact head-mounted electro-optical sensor, reflections from a region on a user's head; whereby the reflections are indicative of facial expressions of the user;

detecting, based on the reflections, a type of facial expression expressed by the user, which belongs to a group comprising neutral facial expressions and non-neutral facial expressions;

reading the electro-optical sensor at a first average bitrate (b 1 ) when the user expresses a facial expression from among the neutral facial expressions; and

reading the electro-optical sensor at a second average bitrate (b 2 ) when the user expresses a facial expression from among the non-neutral facial expressions, wherein b 2 >b 1 .

15. The method of claim 14 , wherein the electro-optical sensor comprises an inward-facing head-mounted camera that provides images of the region by detecting the reflections, and further comprising determining locations of key facial landmarks associated with at least some of the non-neutral facial expressions, and setting the camera's region of interest (ROI) to be around at least some of the key facial landmarks; and wherein the ROI covers less than half of the region.

16. The method of claim 14 , wherein the electro-optical sensor comprises an inward-facing head-mounted camera that provides images of the region by detecting the reflections, the group further comprises transitional facial expressions, and further comprising determining locations of key facial landmarks associated with a subset of the transitional facial expressions transitioning from the neutral facial expressions, and setting the camera's region of interest (ROI) to be around at least some of said key facial landmarks while the user is in a neutral facial expression from among the neutral facial expressions; and wherein the ROI covers less than half of the region.

17. The method of claim 14 , wherein the electro-optical sensor comprises an inward-facing head-mounted camera that provides images of the region by detecting the reflections, the group further comprises transitional facial expressions, and further comprising determining locations of key facial landmarks associated with a subset of the transitional facial expressions that occur in transitions between certain non-neutral facial expressions and certain neutral facial expressions, and setting the camera's region of interest (ROI) to be around at least some of said key facial landmarks while the user is in a certain non-neutral facial expression selected from the non-neutral facial expressions; and wherein the ROI covers less than half of the region.

18. The method of claim 14 , wherein the electro-optical sensor comprises an inward-facing head-mounted camera that provides images of the region by detecting the reflections, the group further comprises transitional facial expressions, and further comprising determining locations of key facial landmarks associated with a subset of the transitional facial expressions that occur in transitions between certain non-neutral facial expressions and other non-neutral facial expressions belonging to the non-neutral facial expressions, and setting the camera's region of interest (ROI) to be around at least some of said key facial landmarks while the user is in a certain non-neutral facial expression selected from the certain non-neutral facial expressions; and wherein the ROI covers less than half of the region.

19. A facial expression capturing system, comprising:

an inward-facing head-mounted camera configured to capture images of a region on a user's head; and

a computer configured to:

detect, based on the images, a type of facial expression expressed by the user, which belongs to a group comprising neutral facial expressions and non-neutral facial expressions;

read from the camera images having a first average size (size 1 ) when the user expresses a facial expression from among the neutral facial expressions; and

read from the camera images having a second average size (size 2 ) when the user expresses a facial expression from among the non-neutral facial expressions, wherein size 2 >size 1 .

20. The facial expression capturing system of claim 19 , wherein the group further comprises transitional facial expressions, and the computer is further configured to read from the camera images having a third average size (size 3 ) during the transitional facial expressions, wherein size 3 >size 1 , and to control resolution of the images utilizing at least one of binning and windowing; and wherein image size is proportional to color depth, such that the color depth read from the camera when the user expresses a facial expression from among the non-neutral facial expressions is higher compared to the color depth read from the camera when the user expresses a facial expression from among the neutral facial expressions.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 14, 2024
From: THIEBERGER, GIL, MR.; FRANK, ARI M,, DR.; THIEBERGER-NAVON, TAL, MRS.
To: FACENSE LTD.
Reel/Frame 067725/0524 →
Continuity (6)
Continuation 18105829 · Feb 4, 2023
Continuation 17524411 · Nov 11, 2021
Provisional Application 63140453 · Jan 22, 2021
Provisional Application 63122961 · Dec 9, 2020
Provisional Application 63113846 · Nov 14, 2020
Related Publication 20240272710A1 · Aug 15, 2024
References Cited (39)
US 8270473B2 · Chen et al. · 2012 [cited by applicant]
US 8401248B1 · Moon et al. · 2013 [cited by applicant]
US 9836484B1 · Bialynicka-Birula et al. · 2017 [cited by applicant]
US 20170007165A1 · Jain · 2017 [cited by examiner]
US 20180137678A1 · Kaehler · 2018 [cited by applicant]
JP WO2017006872 · 2017 [cited by applicant]
U.S. Appl. No. 10/074,024, filed Sep. 11, 2018, el Kaliouby et al. [cited by applicant]
U.S. Appl. No. 10/335,045, filed Jul. 2, 2019, Sebe et al. [cited by applicant]
U.S. Appl. No. 10/665,243, filed May 26, 2020, Whitmire et al. [cited by applicant]
Al Chanti, Dawood and Alice Caplier. “Spontaneous facial expression recognition using sparse representation.” arXiv preprint arXiv:1810.00362 (2018). [cited by applicant]
Barrett, Lisa Feldman, et al. “Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements.” Psychological science in the public interest 20.1 (2019): 1-68. [cited by applicant]
Canedo, Daniel, and António JR Neves. “Facial expression recognition using computer vision: a systematic review.” Applied Sciences 9.21 (2019): 4678. [cited by applicant]
Cha, Jaekwang, Jinhyuk Kim, and Shiho Kim. “Hands-free user interface for AR/VR devices exploiting wearer's facial gestures using unsupervised deep learning.” Sensors 19.20 (2019): 4441. [cited by applicant]
Chew, Sien W., et al. “Sparse temporal representations for facial expression recognition.” Pacific-Rim Symposium on Image and Video Technology. Springer, Berlin, Heidelberg, 2011. [cited by applicant]
Corneanu, Ciprian Adrian, et al. “Survey on rgb, 3d, thermal, and multimodal approaches for facial expression recognition: History, trends, and affect-related applications.” IEEE transactions on pattern analysis and mac… [cited by applicant]
Gao, Xinbo, et al. “A review of active appearance models.” IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 40.2 (2010): 145-158. [cited by applicant]
Hickson, Steven, et al. “Eyemotion: Classifying facial expressions in VR using eye-tracking cameras.” 2019 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 2019. [cited by applicant]
Jia, Shan, et al. “Detection of Genuine and Posed Facial Expressions of Emotion: Databases and Methods.” Frontiers in Psychology 11 (2020): 3818. [cited by applicant]
Kerst, Ariane, Jürgen Zielasek, and Wolfgang Gaebel. “Smartphone applications for depression: a systematic literature review and a survey of health care professionals' attitudes towards their use in clinical practice.” … [cited by applicant]
Li, Hao, et al. “Facial performance sensing head-mounted display.” ACM Transactions on Graphics (ToG) 34.4 (2015): 1-9. [cited by applicant]
Li, Shan, and Weihong Deng. “Deep facial expression recognition: A survey.” IEEE transactions on affective computing (2020). [cited by applicant]
Masai, Katsutoshi, et al. “Evaluation of facial expression recognition by a smart eyewear for facial direction changes, repeatability, and positional drift.” ACM Transactions on Interactive Intelligent Systems (TiiS) 7.… [cited by applicant]
Masai, Katsutoshi, Yuta Sugiura, and Maki Sugimoto. “Facerubbing: Input technique by rubbing face using optical sensors on smart eyewear for facial expression recognition.” Proceedings of the 9th Augmented Human Interna… [cited by applicant]
Masai, Katsutoshi. “Facial Expression Classification Using Photo-reflective Sensors on Smart Eyewear.”, PhD Thesis, (2018). [cited by applicant]
Milborrow, Stephen, and Fred Nicolls. “Locating facial features with an extended active shape model.” European conference on computer vision. Springer, Berlin, Heidelberg, 2008. [cited by applicant]
Nakamura, Fumihiko, et al. “Automatic Labeling of Training Data by Vowel Recognition for Mouth Shape Recognition with Optical Sensors Embedded in Head-Mounted Display.” ICAT-EGVE. 2019. [cited by applicant]
Nakamura, Hiromi, and Homei Miyashita. “Control of augmented reality information volume by glabellar fader.” Proceedings of the 1st Augmented Human international Conference. 2010. [cited by applicant]
Perusquía-Hernández, Monica. “Are people happy when they smile?: Affective assessments based on automatic smile genuineness identification.” Emotion Studies 6.1 (2021): 57-71. [cited by applicant]
Ringeval, Fabien, et al. “AVEC 2019 workshop and challenge: state-of-mind, detecting depression with AI, and cross-cultural affect recognition.” Proceedings of the 9th International on Audio/Visual Emotion Challenge and… [cited by applicant]
Si, Jiaxin, Yanchen Wan, and Teng Zhang. “Facial expression recognition based on ASM and Finite Automata Machine.” (2016): 1306-1310. Proceedings of the 2016 2nd Workshop on Advanced Research and Technology in Industry … [cited by applicant]
Sugiura, Yuta, et al. “Behind the palm: Hand gesture recognition through measuring skin deformation on back of hand by using optical sensors.” 2017 56th Annual Conference of the Society of Instrument and Control Enginee… [cited by applicant]
Suk, Myunghoon, and Balakrishnan Prabhakaran. “Real-time facial expression recognition on smartphones.” 2015 IEEE Winter Conference on Applications of Computer Vision. IEEE, 2015. [cited by applicant]
Suzuki, Katsuhiro, et al. “Recognition and mapping of facial expressions to avatar by embedded photo reflective sensors in head mounted display.” 2017 IEEE Virtual Reality (VR). IEEE, 2017. [cited by applicant]
Terracciano, Antonio, et al. “Personality predictors of longevity: activity, emotional stability, and conscientiousness.” Psychosomatic medicine 70.6 (2008): 621. [cited by applicant]
Umezawa, Akino, et al. “e2-MaskZ: a Mask-type Display with Facial Expression Identification using Embedded Photo Reflective Sensors.” Proceedings of the Augmented Humans International Conference. 2020. [cited by applicant]
Valstar, Michel, et al. “Avec 2016: Depression, mood, and emotion recognition workshop and challenge.” Proceedings of the 6th international workshop on audio/visual emotion challenge. 2016. [cited by applicant]
Yamashita, Koki, et al. “CheekInput: turning your cheek into an input surface by embedded optical sensors on a head-mounted display.” Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology. 201… [cited by applicant]
Yamashita, Koki, et al. “DecoTouch: Turning the Forehead as Input Surface for Head Mounted Display.” International Conference on Entertainment Computing. Springer, Cham, 2017. [cited by applicant]
Zeng, Nianyin, et al. “Facial expression recognition via learning deep sparse autoencoders.” Neurocomputing 273 (2018): 643-649. [cited by applicant]