Eyeball observation device, eyewear terminal, line-of-sight detection method, and program
There is provided an eyeball observation device, which can stably detect a line of sight, the eyeball observation device including: at least one infrared light source configured to radiate polarized infrared light onto an eyeball of a user; and at least one imaging device configured to capture an image of the eyeball irradiated with the polarized infrared light and to be capable of simultaneously capturing a polarization image with at least three directions.
1. An eyeball observation device comprising:
at least one infrared light source configured to radiate polarized infrared light onto an eyeball of a user;
at least one imaging device configured to capture an image of the eyeball irradiated with the polarized infrared light and to be capable of simultaneously performing capturing with at least three polarization directions in order to capture a polarization image that includes polarization in at least three directions;
a polarization model generation unit configured to generate a polarization model representing a relation between a polarization direction and luminance with respect to each pixel of the polarization image acquired by the imaging device;
an arbitrary-phase polarization image generation unit configured to generate an arbitrary-phase polarization image having an arbitrary polarization direction from the polarization image acquired by the imaging device by using the polarization model, wherein the arbitrary-phase polarization image generation unit extracts a maximum luminance value by changing a polarization direction with respect to each pixel of the polarization image and generates a polarization image for bright spot detection having the maximum luminance value;
a bright spot detection unit configured to detect a bright spot in the polarization image for bright spot detection; and
an outlier removal unit configured to remove an outlier of a corneal reflection image by the bright spot on the basis of degrees of polarization of detected bright spots,
wherein the polarization model generation unit, the arbitrary-phase polarization image generation unit, the bright spot detection unit, and the outlier removal unit are each implemented via at least one processor.
2. The eyeball observation device according to claim 1 , wherein the arbitrary-phase polarization image generation unit extracts a minimum luminance value by changing a polarization direction with respect to each pixel of the polarization image and generates a polarization image for pupil detection having the minimum luminance value.
3. The eyeball observation device according to claim 1 , wherein the outlier removal unit removes a bright spot having the degree of polarization smaller than a predetermined value as an outlier among the detected bright spots.
4. The eyeball observation device according to claim 1 , further comprising:
a corneal reflection image identification unit configured to identify the infrared light source corresponding to a corneal reflection image acquired by removing an outlier by the outlier removal unit on the basis of a difference between phases of the polarization models,
wherein the corneal reflection image identification unit is implemented via at least one processor.
5. The eyeball observation device according to claim 1 , wherein, in a case in which the plurality of infrared light sources are provided, at least one of polarizers corresponding to the respective infrared light sources has a polarization direction different from polarization directions of the other polarizers.
6. The eyeball observation device according to claim 1 , further comprising:
a first arithmetic device configured to detect or infer a line of sight of the user on the basis of the captured image of the eyeball,
wherein the first arithmetic device is implemented via at least one processor.
7. The eyeball observation device according to claim 1 , further comprising:
a second arithmetic device configured to acquire eyeball state information on the basis of the captured image of the eyeball and detect or infer at least one of a health state and an awakened state of the user on the basis of the acquired eyeball state information,
wherein the second arithmetic device is implemented via at least one processor.
8. An eyewear terminal comprising:
a lens configured to be provided in front of an eye of a user; and
an eyeball observation device including:
at least one infrared light source configured to radiate polarized infrared light onto an eyeball of the user;
at least one imaging device configured to capture an image of the eyeball irradiated with the polarized infrared light and to be capable of simultaneously performing capturing with at least three polarization directions in order to capture a polarization image that includes polarization in at least three directions;
a polarization model generation unit configured to generate a polarization model representing a relation between a polarization direction and luminance with respect to each pixel of the polarization image acquired by the imaging device;
an arbitrary-phase polarization image generation unit configured to generate an arbitrary-phase polarization image having an arbitrary polarization direction from the polarization image acquired by the imaging device by using the polarization model, wherein the arbitrary-phase polarization image generation unit extracts a maximum luminance value by changing a polarization direction with respect to each pixel of the polarization image and generates a polarization image for bright spot detection having the maximum luminance value;
a bright spot detection unit configured to detect a bright spot in the polarization image for bright spot detection; and
an outlier removal unit configured to remove an outlier of a corneal reflection image by the bright spot on the basis of degrees of polarization of detected bright spots,
wherein the polarization model generation unit, the arbitrary-phase polarization image generation unit, the bright spot detection unit, and the outlier removal unit are each implemented via at least one processor.
9. The eyewear terminal according to claim 8 , further comprising:
a first arithmetic device configured to detect or infer a line of sight of the user on the basis of the captured image of the eyeball.
10. The eyewear terminal according to claim 8 , further comprising:
a second arithmetic device configured to acquire eyeball state information on the basis of the captured image of the eyeball and detect or infer at least one of a health state and an awakened state of the user on the basis of the acquired eyeball state information.
11. A line-of-sight detection method comprising:
capturing an image of an eyeball of a user irradiated with infrared light polarized with respect to the eyeball by at least one infrared light source and simultaneously performing capturing with at least three polarization directions in order to acquire a polarization image that includes polarization in at least three directions;
generating a polarization model representing a relation between a polarization direction and luminance with respect to each pixel of the polarization image;
generating an arbitrary-phase polarization image having an arbitrary polarization direction by using the polarization model;
extracting a maximum luminance value by changing a polarization direction with respect to each pixel of the polarization image;
generating a polarization image for bright spot detection having the maximum luminance value;
detecting a bright spot in the polarization image for bright spot detection; and
removing an outlier of a corneal reflection image by the bright spot on the basis of degrees of polarization of detected bright spots.
12. A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer causes the computer to execute a method, the method comprising:
capturing an image of an eyeball of a user irradiated with infrared light polarized with respect to the eyeball by at least one infrared light source and simultaneously performing capturing with at least three polarization directions in order to acquire a polarization image that includes polarization in at least three directions;
generating a polarization model representing a relation between a polarization direction and luminance with respect to each pixel of the polarization image;
generating an arbitrary-phase polarization image having an arbitrary polarization direction by using the polarization model;
extracting a maximum luminance value by changing a polarization direction with respect to each pixel of the polarization image;
generating a polarization image for bright spot detection having the maximum luminance value;
detecting a bright spot in the polarization image for bright spot detection; and
removing an outlier of a corneal reflection image by the bright spot on the basis of degrees of polarization of detected bright spots.
13. The line-of-sight detection method according to claim 11 , further comprising:
inferring a direction of a line of sight on the basis of the arbitrary-phase polarization image.
14. The non-transitory computer-readable medium according to claim 12 , wherein the further comprises:
inferring a direction of a line of sight on the basis of the arbitrary-phase polarization image.