IP Library Granted Patent US 10,984,602
Granted Patent B1
US 10,984,602 · App. 16/861,798 · Granted Apr 20, 2021

Facial expression tracking during augmented and virtual reality sessions

Inventors: Darius A. Miranda (San Francisco, CA); Chris Kalaboukis (San Jose, CA)
Assignee: Wells Fargo Bank, N.A.
G06T19/006G06K9/00302
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,984,602
App. No.
16/861,798
Filed
Apr 29, 2020
Granted
Apr 20, 2021
Kind
B1
Art Unit
2666
USPC
382/190
Abstract

An example method for estimating an emotion based upon a facial expression of a user can include: receiving one or more captured facial expressions from the user at a visual computing device; comparing the one or more captured facial expressions to one or more known facial expressions; and assigning an emotion to the plurality of captured facial expressions based upon the comparing.

Claims (50)

1. A visual computing device that can be worn by a user, the visual computing device comprising:

a processing unit;

system memory; and

a camera oriented towards a face of the user to capture facial reactions of the user;

wherein the system memory encodes instructions that, when executed by the processing unit, cause the visual computing device to:

display a first image to the user, wherein the first image depicts a first item associated with a financial profile of the user's current purchasing interests;

capture, by the camera, a facial expression of the user as a result of the user viewing the first image, the facial expression comprising a reaction of the user to the first image;

identify an emotion of the user corresponding to the facial expression;

determine when the captured facial expression is a negative emotion; and

when the emotion corresponding to the captured facial expression is the negative emotion, display a second image to the user based on the user's financial profile, wherein the second image depicts a second item that is (i) similar to the first item, and (ii) more closely-aligned to the user's financial profile based upon a cost of the second item.

2. The visual computing device of claim 1 , wherein the camera is located within a field of view of the user.

3. The visual computing device of claim 1 , wherein the visual computing device is a virtual reality device.

4. The visual computing device of claim 1 , wherein the visual computing device is an augmented reality device.

5. The visual computing device of claim 4 , wherein the augmented reality device further includes one or more outwardly-facing cameras which are oriented away from the face of the user.

6. The visual computing device of claim 1 , wherein the first image is an image of an automobile, and the second image is an image of a more or less expensive automobile.

7. The visual computing device of claim 1 , wherein the first image is an image of a first property and the second image is an image of a second property for sale, wherein the first property and second property are different in at least one of: price, style, location, and size.

8. A method implemented on a virtual reality device, the method comprising:

displaying, by the virtual reality device, a first image to a user, wherein the first image depicts a first item associated with a financial profile of the user's current purchasing interests;

orienting one or more inwardly-facing cameras of the virtual reality device towards a face of the user;

capturing a facial expression of the user, using the one or more inwardly-facing cameras, as a result of the user viewing the first image on the virtual reality device, the facial expression comprising a reaction of the user to the first image;

sending the captured facial expression to an electronic computing device;

receiving, from the electronic computing device, a message identifying a user emotion associated with the facial expression, the user emotion associated with the facial expression being identified via a comparison of the captured facial expression with a previous facial expression of the user obtained during a training period;

determining whether the user emotion is negative; and

when the user emotion is negative, displaying a second image to the user based on the user's financial profile, wherein the second image depicts a second item that is (i) similar to the first item, and (ii) more closely-aligned to the user's financial profile based upon a cost of the second item.

9. The method of claim 8 , further comprising selecting the second image to produce a positive emotion by the user.

10. The method of claim 8 , wherein the visual computing device is an augmented reality device or a virtual reality device.

11. The method of claim 8 , further comprising:

displaying one or more possible actions to be taken by the user to improve the financial health status of the user; and

capturing a second facial expression relating to the one or more possible actions.

12. The method of claim 11 , further comprising determining the user emotion associated with the one or more possible actions based upon the second facial expression.

13. The method of claim 8 , wherein the first image is an automobile, and the second image is a more or less expensive automobile.

14. The method of claim 8 , wherein the first image is an image of a first property and the second image is an image of a second property, wherein the first property for sale and second property are different in at least one of: price, style, location, and size.

15. A visual computing device that can be worn by a user, the visual computing device comprising:

a processing unit;

system memory; and

a camera oriented towards a face of the user to capture facial reactions of the user;

wherein the system memory encodes instructions that, when executed by the processing unit, cause the visual computing device to:

present one or more questions associated with a plurality of topics to the user, the one or more questions directed at identifying an emotional connection of the user to the plurality of topics;

receive one or more responses from the user to the one or more questions;

based on the one or more responses, identify the emotion connection of the user to each of the plurality of topics;

display a first image, wherein the first image depicts a first item corresponding to one of the plurality of topics to the user and a profile of the user's current purchasing interests;

capture, by the camera, a facial expression of the user as a result of the user viewing the first image, the facial expression comprising a reaction of the user to the image;

identify an emotion of the user based on the captured facial expression and the one or more responses associated to the one or more questions associated with the topic corresponding to the first image;

determine when the captured facial expression is a negative emotion; and

when the emotion corresponding to the captured facial expression is determined to be negative, display a second image to the user based on the user's financial profile, wherein the second image depicts a second item that is (i) similar to the first item, and (ii) more closely-aligned to the user's financial profile based upon a cost of the second item.

16. The visual computing device of claim 15 , wherein the second image is selected, based on the one or more responses to the one or more questions associated with a plurality of topics, to result in a positive emotion from the user.

17. The visual computing device of claim 15 , wherein the visual computing device is an augmented reality device.

18. The visual computing device of claim 17 , wherein the augmented reality device further includes one or more outwardly-facing cameras which are oriented away from the face of the user.

19. The visual computing device of claim 15 , wherein the first image is an image of an automobile, and the second image is an image of a more or less expensive automobile.

20. The visual computing device of claim 15 , wherein the first image is an image of a first property and the second image is an image of a second property, wherein the first property and second property are different in at least one of: price, style, location, and size.

Assignments (2)
STATEMENT OF CHANGE OF ADDRESS OF ASSIGNEE Recorded Jun 17, 2025
From: WELLS FARGO BANK, N.A.
To: WELLS FARGO BANK, N.A.
Reel/Frame 071649/0465 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 29, 2020
From: MIRANDA, DARIUS A.; KALABOUKIS, CHRIS
To: WELLS FARGO BANK, N.A.
Reel/Frame 052526/0451 →
Continuity (1)
Continuation 15339440 · Oct 31, 2016