IP Library Granted Patent US 10,796,176
Granted Patent B2
US 10,796,176 · App. 16/173,160 · Granted Oct 6, 2020

Personal emotional profile generation for vehicle manipulation

Inventors: Rana el Kaliouby (Milton, MA); Abdelrahman N. Mahmoud (Somerville, MA); Seyedmohammad Mavadati (Watertown, MA); Gabriele Zijderveld (Somerville, MA)
Assignee: Affectiva, Inc.
G06K9/00845A61B5/18B60W40/08G06K9/00302G06K9/00315G06K9/6273G06N3/0445G06N3/0454G06N3/084G06Q30/0255G08G1/0112G08G1/0129G08G1/096716G08G1/096725G08G1/096741G08G1/096775H04L67/22H04L67/306G10L25/48
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,796,176
App. No.
16/173,160
Filed
Oct 29, 2018
Granted
Oct 6, 2020
Kind
B2
Art Unit
3661
USPC
701/36
Abstract

Personal emotional profile generation uses cognitive state analysis for vehicle manipulation. Cognitive state data is obtained from an individual. The cognitive state data is extracted, using one or more processors, from facial images of an individual captured as they respond to stimuli within a vehicle. The cognitive state data extracted from facial images is analyzed to produce cognitive state information. The cognitive state information is categorized, using one or more processors, against a personal emotional profile for the individual. The vehicle is manipulated, based on the cognitive state information, the categorizing, and the stimuli. The personal emotional profile is generated by comparing the cognitive state information of the individual with cognitive state norms from a plurality of individuals and is based on cognitive state data for the individual that is accumulated over time. The cognitive state information is augmented based on audio data collected from within the vehicle.

Claims (45)

1. A computer-implemented method for cognitive state analysis comprising:

obtaining cognitive state data from an individual within a vehicle, wherein the cognitive state data is extracted, using one or more processors, from facial images of the individual that are captured as the individual responds to stimuli;

analyzing the cognitive state data extracted from facial images to produce cognitive state information;

developing norms for the individual;

categorizing, using one or more processors, the cognitive state information against a personal emotional profile for the individual; and

manipulating the vehicle, based on the cognitive state information, the categorizing, and the stimuli.

2. The method of claim 1 wherein the personal emotional profile is generated by comparing the cognitive state information of the individual with cognitive state norms from a plurality of individuals.

3. The method of claim 1 wherein the personal emotional profile is generated based on cognitive state data for the individual that is accumulated over time.

4. The method of claim 3 wherein the personal emotional profile is further generated based on two or more vehicle journeys by the individual.

5. The method of claim 4 wherein at least two of the two or more vehicle journeys are accomplished in two or more vehicles.

6. The method of claim 1 wherein the stimuli occurs within the vehicle.

7. The method of claim 1 wherein the stimuli occurs outside of the vehicle.

8. The method of claim 1 further comprising pushing media content to the individual based on the categorizing.

9. The method of claim 1 wherein the categorizing is based on a plurality of expressions by the individual.

10. The method of claim 9 wherein the categorizing is further based on a rate of change in the plurality of expressions by the individual.

11. The method of claim 1 wherein the personal emotional profile further comprises off-line behavior.

12. The method of claim 1 further comprising correlating a component of the personal emotional profile to a purchase intent.

13. The method of claim 1 further comprising using the personal emotional profile to predict future reactions to stimuli.

14. The method of claim 1 further comprising using the personal emotional profile to categorize digital media.

15. The method of claim 1 further comprising matching the individual with one or more people from a plurality of people, based on the categorizing.

16. The method of claim 1 wherein the cognitive state data is collected while the individual views a collection of digital media.

17. The method of claim 16 wherein the cognitive state information from a plurality of people is based on the plurality of people viewing the collection of digital media.

18. The method of claim 1 further comprising using response differences in the individual, over an interval of time, from the norms for the individual to make a recommendation.

19. The method of claim 1 further comprising developing norms for a demographic group.

20. The method of claim 19 further comprising using response differences for the individual from the norms for the demographic group to refine a profile.

21. The method of claim 1 further comprising augmenting the cognitive state information based on audio data collected from within the vehicle, wherein the audio data is collected contemporaneously with the facial images.

22. The method of claim 21 wherein the audio data includes speech.

23. The method of claim 21 wherein the audio data includes non-speech vocalizations.

24. The method of claim 1 further comprising tagging the cognitive state data with sensor data.

25. The method of claim 24 wherein the sensor data includes one or more of vehicle interior temperature, vehicle exterior temperature, time of day, day of week, season, level of daylight, weather conditions, road conditions, traffic conditions, headlight activation, windshield wiper activation, vehicle settings, entertainment center selection, or entertainment center volume.

26. The method of claim 1 wherein the manipulating is based on a deviation from the norms.

27. A computer program product embodied in a non-transitory computer readable medium for cognitive state analysis, the computer program product comprising code which causes one or more processors to perform operations of:

obtaining cognitive state data from an individual within a vehicle, wherein the cognitive state data is extracted, using one or more processors, from facial images of the individual that are captured as the individual responds to stimuli;

analyzing the cognitive state data extracted from facial images to produce cognitive state information;

developing norms for the individual;

categorizing, using one or more processors, the cognitive state information against a personal emotional profile for the individual; and

manipulating the vehicle, based on the cognitive state information, the categorizing, and the stimuli.

28. A computer system for cognitive state analysis comprising:

a memory which stores instructions;

one or more processors attached to the memory wherein the one or more processors, when executing the instructions which are stored, are configured to:

obtain cognitive state data from an individual within a vehicle, wherein the cognitive state data is extracted, using one or more processors, from facial images of the individual that are captured as the individual responds to stimuli;

analyze the cognitive state data extracted from facial images to produce cognitive state information;

develop norms for the individual;

categorize, using one or more processors, the cognitive state information against a personal emotional profile for the individual; and

manipulate the vehicle, based on the cognitive state information, the categorizing, and the stimuli.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 27, 2018
From: EL KALIOUBY, RANA; MAHMOUD, ABDELRAHMAN N.; MAVADATI, SEYEDMOHAMMAD; ZIJDERVELD, GABRIELE
To: AFFECTIVA, INC.
Reel/Frame 047592/0615 →
Continuity (43)
Continuation In Part 15875644 · Jan 19, 2018
Continuation In Part 15273765 · Sep 23, 2016
Continuation In Part 14796419 · Jul 10, 2015
Continuation In Part 14460915 · Aug 15, 2014
Continuation In Part 14328554 · Jul 10, 2014
Continuation In Part 13153745 · Jun 6, 2011
Continuation In Part 13153745 · Jun 6, 2011
Provisional Application 62679825 · Jun 3, 2018
Provisional Application 62637567 · Mar 2, 2018
Provisional Application 62625274 · Feb 1, 2018
Provisional Application 62611780 · Dec 29, 2017
Provisional Application 62593440 · Dec 1, 2017
Provisional Application 62593449 · Dec 1, 2017
Provisional Application 62557460 · Sep 12, 2017
Provisional Application 62541847 · Aug 7, 2017
Provisional Application 62524606 · Jun 25, 2017
Provisional Application 62503485 · May 9, 2017
Provisional Application 62469591 · Mar 10, 2017
Provisional Application 62448448 · Jan 20, 2017
Provisional Application 62370421 · Aug 3, 2016
Provisional Application 62301558 · Feb 29, 2016
Provisional Application 62273896 · Dec 31, 2015
Provisional Application 62265937 · Dec 10, 2015
Provisional Application 62222518 · Sep 23, 2015
Provisional Application 62128974 · Mar 5, 2015
Provisional Application 62082579 · Nov 20, 2014
Provisional Application 62047508 · Sep 8, 2014
Provisional Application 62023800 · Jul 11, 2014
Provisional Application 61972314 · Mar 30, 2014
Provisional Application 61953878 · Mar 16, 2014
Provisional Application 61927481 · Jan 15, 2014
Provisional Application 61924252 · Jan 7, 2014
Provisional Application 61916190 · Dec 14, 2013
Provisional Application 61867007 · Aug 16, 2013
Provisional Application 61844478 · Jul 10, 2013
Provisional Application 61467209 · Mar 24, 2011
Provisional Application 61447464 · Feb 28, 2011
Provisional Application 61447089 · Feb 27, 2011
Provisional Application 61439913 · Feb 6, 2011
Provisional Application 61414451 · Nov 17, 2010
Provisional Application 61388002 · Sep 30, 2010
Provisional Application 61352166 · Jun 7, 2010
Related Publication 20190073547A1 · Mar 7, 2019
Cited By (2)
US 12,511,095 US 12,614,557