IP Library › Granted Patent US 11,889,178
Granted Patent B2
US 11,889,178 · App. 17/682,388 · Granted Jan 30, 2024

Electronic device with automatic eye gaze tracking and camera adjustment

Inventors: Amit Kumar Agrawal (Bangalore, IN); Rahul Bharat Desai (Hoffman Estates, IL)
Assignee: Motorola Mobility LLC
H04N23/611G06F3/16H04N5/607
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,889,178
App. No.
17/682,388
Granted
Jan 30, 2024
Kind
B2
Abstract

An electronic device, a method and a computer program product for camera adjustments based on eye gaze tracking. The method includes capturing, via a front facing camera, an image stream containing a face of a user and determining, via a processor, an eye gaze direction based on images retrieved from the image stream. The eye gaze direction corresponds to a location where the user is looking. In response to determining that the user is looking away from the front surface of the electronic device and towards a direction within a field of view of a rear facing camera, the method further includes identifying image characteristics of the location, generating camera settings of the rear facing camera based on the eye gaze direction/characteristics of the location and adjusting the rear facing camera using the generated camera settings such that the rear facing camera focuses on the location.

Claims (125)

1. An electronic device comprising:

a front surface and a rear surface;

at least one front facing camera and at least one rear facing camera;

a memory having stored thereon a camera selection and control module (CSCM) for controlling the at least one front facing camera and the at least one rear facing camera; and

at least one processor communicatively coupled to each of the at least one front facing camera, each of the at least one rear facing camera, and to the memory, the at least one processor executing program code of the CSCM, which enables the electronic device to:

capture, via the at least one front facing camera, a first image stream containing a face of a first user;

determine a first eye gaze direction of the first user based on a first image retrieved from the first image stream, the first eye gaze direction corresponding to a first location where the first user is looking; and

in response to determining that the first user is looking away from the front surface of the electronic device and towards a direction within a field of view (FOV) of the at least one rear facing camera:

identify image characteristics of the first location;

generate at least one camera setting of a selected rear facing camera at least partially based on the first eye gaze direction and identified characteristics of the first location;

adjust the selected rear facing camera using the generated at least one camera setting such that the selected rear facing camera focuses on the first location; and

activate the selected rear facing camera as an active camera.

2. The electronic device of claim 1 , wherein the at least one processor:

tracks the eye gaze direction by monitoring and periodically analyzing the first image corresponding to the first image stream to identify a current eye gaze direction;

determines if the current eye gaze direction is towards a current location that is different from the first location;

in response to determining that the current location is different from the first location, determines, based on the current eye gaze direction, if the first user is looking away from the front surface of the electronic device; and

in response to determining that the first user is looking away from the front surface:

identifies image characteristics of the current location;

generates at least one new camera setting of the active camera based on the current eye gaze direction and identified characteristics of the current location; and

adjusts the active camera using the generated at least one new camera setting such that the at least one rear facing camera focuses on the current location.

3. The electronic device of claim 1 , wherein to generate the at least one camera setting of the at least one rear facing camera based on a current eye gaze direction, the at least one processor is further enabled to:

process the current eye gaze direction through an artificial intelligence engine, which generates the at least one camera setting for the selected rear facing camera based at least partially on the current eye gaze direction; and

adjust at least a directional setting of the selected rear facing camera to focus the FOV of the active camera on a current location determined from the current eye gaze detection.

4. The electronic device of claim 1 , wherein to generate the at least one camera setting of the selected rear facing camera based on a current eye gaze direction, the at least one processor is further enabled to:

retrieve, from the memory, first camera parameters and first camera settings for the selected rear facing camera; and

generate the at least one second camera setting of the selected rear facing camera at least partially based on the first eye gaze direction, the identified image characteristics of the first location, the first camera parameters, and the first camera settings.

5. The electronic device of claim 1 , wherein the generated at least one second camera setting comprises at least one of:

a zoom level;

a focus distance;

a directional setting;

a digital crop level;

a focal distance of the focus module;

a directional audio zoom;

a modified electronic image stabilization; and

an exposure compensation.

6. The electronic device of claim 1 , wherein the identified image characteristics of the first location comprises at least one of:

light levels at the first location;

face detection at the first location;

scene detection at the first location;

contrast detection autofocus at the first location;

phase detection autofocus at the first location;

object of interest detection using computer vision at the first location;

object of interest detection using machine learning at the first location; and

object of interest detection using artificial intelligence at the first location.

7. The electronic device of claim 1 , further comprising:

at least one input device communicatively coupled to the at least one processor; and

the at least one processor:

receives first input from the at least one input device;

determines a first context based on the first input received, the first context associated with at least one camera setting using the selected rear facing camera as the active camera to capture images;

generates at least one camera setting for the active camera based at least partially on the first context; and

adjusts the active camera using the generated at least one camera setting.

8. The electronic device of claim 7 , wherein:

the at least one input device comprises an audio input device and the first input is audio input captured by the audio input device; and

the at least one processor processes the audio input via a natural language processor/context evaluation (NLP/CE) engine to determine the first context.

9. A method comprising:

capturing, via at least one front facing camera of an electronic device, a first image stream containing a face of a first user;

determining, via at least one processor, a first eye gaze direction of the first user based on a first image retrieved from the first image stream, the first eye gaze direction corresponding to a first location where the first user is looking; and

in response to determining that the first user is looking away from a front surface of the electronic device and towards a direction within a field of view (FOV) of at least one rear facing camera of the electronic device:

identifying image characteristics of the first location;

generating at least one camera setting of a selected rear facing camera at least partially based on the first eye gaze direction and identified characteristics of the first location;

adjusting the selected rear facing camera using the generated at least one camera setting such that the selected rear facing camera focuses on the first location; and

activating the selected rear facing camera as an active camera.

10. The method of claim 9 , further comprising:

tracking the eye gaze direction by monitoring and periodically analyzing the first image corresponding to the first image stream to identify a current eye gaze direction;

determining if the current eye gaze direction is towards a current location that is different from the first location;

in response to determining that the current location is different from the first location, determining, based on the current eye gaze direction, if the first user is looking away from the front surface of the electronic device; and

in response to determining that the first user is looking away from the front surface:

identifying image characteristics of the current location;

generating at least one new camera setting of the active camera based on the current eye gaze direction and identified characteristics of the current location; and

adjusting the active camera using the generated at least one new camera setting such that the at least one rear facing camera focuses on the current location.

11. The method of claim 9 , wherein to generate the at least one camera setting of the at least one rear facing camera based on a current eye gaze direction, the method further comprises:

processing the current eye gaze direction through an artificial intelligence engine, which generates the at least one camera setting for the selected rear facing camera based at least partially on the current eye gaze direction; and

adjusting at least a directional setting of the selected rear facing camera to focus the FOV of the active camera on a current location determined from the current eye gaze detection.

12. The method of claim 9 , wherein to generate the at least one camera setting of the selected rear facing camera based on a current eye gaze direction, the method further comprises:

retrieving, from the memory, first camera parameters and first camera settings for the selected rear facing camera; and

generating at least one second camera setting of the selected rear facing camera at least partially based on the first eye gaze direction, the identified image characteristics of the first location, the first camera parameters and the first camera settings.

13. The method of claim 9 , wherein the generated at least one second camera setting comprises at least one of:

a zoom level;

a focus distance;

a directional setting;

a digital crop level;

a focal distance of the focus module;

a directional audio zoom;

a modified electronic image stabilization; and

an exposure compensation.

14. The method of claim 9 , wherein the identified image characteristics of the first location comprises at least one of:

light levels at the first location;

face detection at the first location;

scene detection at the first location;

contrast detection autofocus at the first location;

phase detection autofocus at the first location;

object of interest detection using computer vision at the first location;

object of interest detection using machine learning at the first location; and

object of interest detection using artificial intelligence at the first location.

15. The method of claim 9 , wherein at least one input device is communicatively coupled to the at least one processor, the method further comprising:

receiving first input from the at least one input device;

determining a first context based on the first input received, the first context associated with at least one camera setting using the selected rear facing camera as the active camera to capture images;

generating at least one camera setting for the active camera based at least partially on the first context; and

adjusting the active camera using the generated at least one camera setting.

16. The method of claim 15 , wherein:

the at least one input device comprises an audio input device and the first input is audio input captured by the audio input device; and

the at least one processor processes the audio input via a natural language processor/context evaluation (NLP/CE) engine to determine the first context.

17. A computer program product comprising:

a non-transitory computer readable storage device having stored thereon program code which, when executed by at least one processor of an electronic device having a front surface, a rear surface, at least one front facing camera, at least one rear facing camera, and a memory, enables the electronic device to complete functionality of:

capturing, via the at least one front facing camera, a first image stream containing a face of a first user;

determining a first eye gaze direction of the first user based on a first image retrieved from the first image stream, the first eye gaze direction corresponding to a first location where the first user is looking; and

in response to determining that the first user is looking away from the front surface of the electronic device and towards a direction within a field of view (FOV) of the at least one rear facing camera:

identifying image characteristics of the first location;

generating at least one camera setting of a selected rear facing camera at least partially based on the first eye gaze direction and identified characteristics of the first location;

adjusting the selected rear facing camera using the generated at least one camera setting such that the selected rear facing camera focuses on the first location; and

activating the selected rear facing camera as an active camera.

18. The computer program product of claim 17 , wherein the program code for generating and adjusting camera settings comprises program code that further enables the electronic device to complete the functionality of:

tracking the eye gaze direction by monitoring and periodically analyzing the first image corresponding to the first image stream from the least one front facing camera to identify a current eye gaze direction;

determining if the current eye gaze direction is towards a current location that is different from the first location;

in response to determining that the current location is different from the first location, determining, based on the current eye gaze direction, if the first user is looking away from the front surface of the electronic device;

in response to determining that the first user is looking away from the front surface:

identifying image characteristics of the current location;

generating at least one new camera setting of the active camera based on the current eye gaze direction and identified characteristics of the current location; and

adjusting the active camera using the generated at least one new camera setting such that the at least one rear facing camera focuses on the current location.

19. The computer program product of claim 17 , wherein to generate the at least one camera setting of the at least one rear facing camera based on a current eye gaze direction, the program code for generating and adjusting camera settings comprises program code that further enables the electronic device to complete the functionality of:

processing the current eye gaze direction through an artificial intelligence engine, which generates the at least one camera setting for the selected rear facing camera based at least partially on the current eye gaze direction; and

adjusting at least a directional setting of the selected rear facing camera to focus the FOV of the active camera on a current location determined from the current eye gaze detection.

20. The computer program product of claim 17 , wherein to generate the at least one camera setting of the at least one rear facing camera based on a current eye gaze direction, the program code for generating and adjusting camera settings comprises program code that further enables the electronic device to complete the functionality of:

retrieving, from the memory, first camera parameters and first camera settings for the selected rear facing camera; and

generating at least one second camera setting of the selected rear facing camera at least partially based on the first eye gaze direction, the identified image characteristics of the first location, the first camera parameters and the first camera settings.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 28, 2022
From: AGRAWAL, AMIT KUMAR; DESAI, RAHUL BHARAT
To: MOTOROLA MOBILITY LLC
Reel/Frame 059118/0050 →
Continuity (1)
Related Publication 20230276116A1 · Aug 31, 2023
Cited By (1)
US 12,464,088