Granted Patent B2

US 12,260,676 · App. 17/527,382 · Granted Mar 25, 2025

Inventor: Toshikazu Ohno (Ibaraki, JP)

Assignees: SWALLOW INCUBATE CO., LTD.; PANASONIC HOLDINGS CORPORATION

G06V40/193G06T7/73G06V40/19G06V40/197G06T2207/10048

Abstract

Claims (117)

1. An image processing method of detecting pupil information in an image processing device, the image processing method comprising:

acquiring image data;

detecting a face region including at least a part of a face of a person from the image data;

setting a first region used for detection of an eye of the person in the face region having been detected;

setting a second region estimated to include a pupil or an iris by narrowing the first region based on a predetermined criterion;

detecting pupil information indicating the pupil or the iris in the second region;

outputting the pupil information having been detected; and

detecting face orientation information indicating an orientation of a face of the person from the face region;

wherein the predetermined criterion is a criterion for removing, in the first region, a predetermined amount of area from a boundary side of the first region, and

wherein in setting of the second region,

as a rightward orientation degree of the face indicated by the face orientation information becomes greater, an exclusion width from a right boundary in the first region is made narrower and an exclusion width from a left boundary in the first region is made wider, and

as a leftward orientation degree of the face indicated by the face orientation information becomes greater, an exclusion width from a left boundary in the first region is made narrower and an exclusion width from a right boundary in the first region is made wider.

2. The image processing method according to claim 1 , further comprising:

detecting a facial feature point from the face region having been detected;

detecting face orientation information indicating an orientation of a face of the person based on the facial feature point;

detecting eye gaze information indicating an eye gaze of the person based on the pupil information and the face orientation information having been detected; and

outputting the eye gaze information having been detected.

3. The image processing method according to claim 2 , wherein the eye gaze information includes information indicating an eye gaze plane that is a region of a predetermined range with respect to an eye gaze point of the person on a predetermined target plane.

4. The image processing method according to claim 1 , wherein

the image data is captured by a visible light camera, and

the pupil information includes information indicating a center position of the iris and a size of the iris.

5. The image processing method according to claim 1 , wherein

the image data is captured by an infrared light camera, and

the pupil information includes at least one of information indicating a center position of the pupil and a size of the pupil and information indicating a center position of the iris and a size of the iris.

6. The image processing method according to claim 5 , wherein the image data is image data captured by an infrared light camera using infrared light in a predetermined second wavelength band in which a spectral intensity of sunlight is attenuated more than a predetermined first wavelength.

7. An image processing method of detecting pupil information in an image processing device, the image processing method comprising:

acquiring image data;

detecting a face region including at least a part of a face of a person from the image data;

setting a first region used for detection of an eye of the person in the face region having been detected;

setting a second region estimated to include a pupil or an iris by narrowing the first region based on a predetermined criterion;

detecting pupil information indicating the pupil or the iris in the second region;

outputting the pupil information having been detected;

detecting an outer corner of an eye and an inner corner of an eye from the face region having been detected, wherein

when the outer corner of the eye and the inner corner of the eye have been detected,

a region that surrounds the outer corner of the eye and the inner corner of the eye and is smaller than a first region set in a case of having failed to detect the outer corner of the eye and the inner corner of the eye is set as the first region in setting of the first region, and

the pupil information is detected in the first region in detection of the pupil information.

8. An image processing method of detecting pupil information in an image processing device, the image processing method comprising:

acquiring image data;

detecting a face region including at least a part of a face of a person from the image data;

setting a first region used for detection of an eye of the person in the face region having been detected;

setting a second region estimated to include a pupil or an iris by narrowing the first region based on a predetermined criterion;

detecting pupil information indicating the pupil or the iris in the second region; and

outputting the pupil information having been detected;

detecting a facial feature point from the face region having been detected;

detecting face orientation information indicating an orientation of a face of the person based on the facial feature point;

detecting eye gaze information indicating an eye gaze of the person based on the pupil information and the face orientation information having been detected;

outputting the eye gaze information having been detected detecting at least one of an eyebrow and a corner of a mouth from the face region having been detected; and

estimating a degree of interest of the person based on the eye gaze information and at least one of a position of the eyebrow having been detected and a position of the corner of the mouth having been detected.

9. The image processing method according to claim 8 , further comprising:

estimating an expression of the person based on at least one of a position of the eyebrow having been detected and a position of the corner of the mouth having been detected; and

estimating a degree of interest of the person based on the eye gaze information and information indicating the estimated expression.

10. An image processing device comprising:

a memory; and

a processor that executes a program from the memory so as to operate as:

an acquisition unit acquiring image data;

a face detection unit detecting a face region including an entire or part of a face of a person from the image data;

a first setting unit setting a first region used for detection of an eye of the person in the face region having been detected;

a second setting unit setting a second region estimated to include a pupil or an iris by narrowing the first region based on a predetermined criterion;

a face orientation detection unit detecting face orientation information indicating an orientation of a face of the person from the face region; and

a pupil information detection unit detecting pupil information indicating the pupil or the iris in the second region, and outputting the pupil information,

wherein the predetermined criterion is a criterion for removing, in the first region, a predetermined amount of area from a boundary side of the first region, and

wherein in setting of the second region,

as a rightward orientation degree of the face indicated by the face orientation information becomes greater, an exclusion width from a right boundary in the first region is made narrower and an exclusion width from a left boundary in the first region is made wider, and

as a leftward orientation degree of the face indicated by the face orientation information becomes greater, an exclusion width from a left boundary in the first region is made narrower and an exclusion width from a right boundary in the first region is made wider.

11. A non-transitory computer readable storage medium storing an image processing program for causing a computer to function as:

an acquisition unit acquiring image data;

a face detection unit detecting a face region including an entire or part of a face of a person from the image data;

a first setting unit setting a first region used for detection of an eye of the person in the face region having been detected;

a second setting unit setting a second region estimated to include a pupil or an iris by narrowing the first region based on a predetermined criterion;

a face orientation detection unit detecting face orientation information indicating an orientation of a face of the person from the face region; and

a pupil information detection unit detecting pupil information indicating the pupil or the iris in the second region, and outputting the pupil information,

wherein the predetermined criterion is a criterion for removing, in the first region, a predetermined amount of area from a boundary side of the first region, and

wherein in setting of the second region,

as a rightward orientation degree of the face indicated by the face orientation information becomes greater, an exclusion width from a right boundary in the first region is made narrower and an exclusion width from a left boundary in the first region is made wider, and

as a leftward orientation degree of the face indicated by the face orientation information becomes greater, an exclusion width from a left boundary in the first region is made narrower and an exclusion width from a right boundary in the first region is made wider.

12. An image processing device comprising:

a memory; and

a processor that executes a program from the memory so as to operate as:

an acquisition unit acquiring image data;

a face detection unit detecting a face region including an entire or part of a face of a person from the image data;

a first setting unit setting a first region used for detection of an eye of the person in the face region having been detected;

a second setting unit setting a second region estimated to include a pupil or an iris by narrowing the first region based on a predetermined criterion; and

a pupil information detection unit detecting pupil information indicating the pupil or the iris in the second region, and outputting the pupil information,

wherein, the first setting unit detects an outer corner of an eye and an inner corner of an eye from the face region having been detected,

when the outer corner of the eye and the inner corner of the eye have been detected, the first setting unit sets a region that surrounds the outer corner of the eye and the inner corner of the eye and is smaller than a first region set in a case of having failed to detect the outer corner of the eye and the inner corner of the eye is set as the first region in setting of the first region, and

the pupil information detection unit detects the pupil information in the first region.

13. A non-transitory computer readable storage medium storing an image processing program for causing a computer to function as:

an acquisition unit acquiring image data;

a face detection unit detecting a face region including an entire or part of a face of a person from the image data;

a first setting unit setting a first region used for detection of an eye of the person in the face region having been detected;

a second setting unit setting a second region estimated to include a pupil or an iris by narrowing the first region based on a predetermined criterion; and

a pupil information detection unit detecting pupil information indicating the pupil or the iris in the second region, and outputting the pupil information,

wherein, the first setting unit detects an outer corner of an eye and an inner corner of an eye from the face region having been detected,

when the outer corner of the eye and the inner corner of the eye have been detected, the first setting unit sets a region that surrounds the outer corner of the eye and the inner corner of the eye and is smaller than a first region set in a case of having failed to detect the outer corner of the eye and the inner corner of the eye is set as the first region in setting of the first region, and

the pupil information detection unit detects the pupil information in the first region.

14. An image processing device comprising:

a memory; and

a processor that executes a program from the memory so as to operate as:

an acquisition unit acquiring image data;

a face detection unit detecting a face region including an entire or part of a face of a person from the image data;

a first setting unit setting a first region used for detection of an eye of the person in the face region having been detected;

a second setting unit setting a second region estimated to include a pupil or an iris by narrowing the first region based on a predetermined criterion;

a pupil information detection unit detecting pupil information indicating the pupil or the iris in the second region, and outputting the pupil information;

a feature point detection unit detecting a facial feature point from the face region having been detected;

a face orientation detection unit detecting face orientation information indicating an orientation of a face of the person based on the facial feature point;

an eye gaze information detection unit detecting eye gaze information indicating an eye gaze of the person based on the pupil information and the face orientation information having been detected, and outputting the eye gaze information having been detected; and

a degree of interest estimation unit detecting at least one of an eyebrow and a corner of a mouth from the face region having been detected, and estimating a degree of interest of the person based on the eye gaze information and at least one of a position of the eyebrow having been detected and a position of the corner of the mouth having been detected.

15. A non-transitory computer readable storage medium storing an image processing program for causing a computer to function as:

an acquisition unit acquiring image data;

a face detection unit detecting a face region including an entire or part of a face of a person from the image data;

a first setting unit setting a first region used for detection of an eye of the person in the face region having been detected;

a second setting unit setting a second region estimated to include a pupil or an iris by narrowing the first region based on a predetermined criterion;

a pupil information detection unit detecting pupil information indicating the pupil or the iris in the second region, and outputting the pupil information;

a feature point detection unit detecting a facial feature point from the face region having been detected;

a face orientation detection unit detecting face orientation information indicating an orientation of a face of the person based on the facial feature point;

an eye gaze information detection unit detecting eye gaze information indicating an eye gaze of the person based on the pupil information and the face orientation information having been detected, and outputting the eye gaze information having been detected; and

a degree of interest estimation unit detecting at least one of an eyebrow and a corner of a mouth from the face region having been detected, and estimating a degree of interest of the person based on the eye gaze information and at least one of a position of the eyebrow having been detected and a position of the corner of the mouth having been detected.

Assignments (2)

Priority Claims (1)

Continuity (2)

References Cited (30)