Line-of-sight detection device and display device
A line-of-sight detection device capable of achieving further improvement in line-of-sight detection accuracy, achieving low latency, and achieving low power consumption is provided. A line-of-sight detection device includes an imaging element, an illumination unit that illuminates an eyeball with a plurality of beams of illumination light, an optical element that guides respective beams of reflected light in a plurality of directions from the eyeball caused by the plurality of beams of illumination light to a direction of the imaging element, an optical system that causes the imaging element to non-selectively receive the beams of reflected light in the plurality of directions from the eyeball, and a calculation unit that converts, on the basis of the beams of reflected light in the plurality of directions from the eyeball received by the imaging element, the beams of reflected light into information of eyeball movement and/or positional information of a pupil.
1. A line-of-sight detection device, comprising:
an imaging element;
an illumination unit configured to illuminate an eyeball with a plurality of beams of illumination light;
an optical element configured to guide a plurality of beams of reflected light to a direction of the imaging element, wherein
the plurality of beams of reflected light includes the plurality of beams of illumination light reflected in a plurality of directions from the eyeball,
the optical element includes a plurality of optical elements,
a first optical element of the plurality of optical elements has:
a substantial difference in diffraction efficiency from a diffraction efficiency of a second optical element of the plurality of optical elements, or
a substantial difference in reflectance amount from a reflectance amount of the second optical element, and
each of the plurality of optical elements is configured to guide a respective beam of reflected light of the plurality of beams of reflected light in the plurality of directions from the eyeball to the direction of the imaging element;
an optical system configured to cause the imaging element to non-selectively receive the plurality of beams of reflected light in the plurality of directions from the eyeball; and
a calculation unit configured to convert, based on the plurality of beams of reflected light in the plurality of directions from the eyeball received by the imaging element, the plurality of beams of reflected light into at least one of information of eyeball movement or positional information of a pupil.
2. The line-of-sight detection device according to claim 1 , wherein the imaging element has an event-driven function.
3. The line-of-sight detection device according to claim 1 , wherein the optical system is further configured to cause the imaging element to receive the plurality of beams of reflected light in the plurality of directions from the eyeball without spatial assignment.
4. The line-of-sight detection device according to claim 1 , wherein
at least one optical element of the plurality of optical elements is a diffraction grating.
5. The line-of-sight detection device according to claim 1 , wherein a beam of reflected light of the plurality of beams of reflected light which is reflected in one direction of the plurality of directions from the eyeball is a beam of light from an optical axis of the eyeball.
6. The line-of-sight detection device according to claim 1 , wherein
the substantial difference between the reflectance amount of the first optical element and the reflectance amount of the second optical element is 50% or more.
7. The line-of-sight detection device according to claim 1 ,
the substantial difference between the diffraction efficiency of the first optical element and the diffraction efficiency of the second optical element is 50% or more.
8. The line-of-sight detection device according to claim 1 , wherein
the plurality of beams of reflected light includes:
a first beam of reflected light that is reflected from an optical axis of the eyeball, and
a second beam of reflected light that is different from the first beam of the reflected light, and
an amount of light of the first beam of reflected light that reaches the imaging element is larger than an amount of light of the second beam of reflected light that reaches the imaging element.
9. The line-of-sight detection device according to claim 1 , further comprising
at least one of a polarizing plate, an ND filter, or a retardation film, wherein
based on the at least one of the polarizing plate, the ND filter, or the retardation film, an amount of light of a first beam of reflected light of the plurality of beams of reflected light is adjustable, and
the first beam of reflected light is reflected in at least one direction of the plurality of directions.
10. The line-of-sight detection device according to claim 1 , wherein
the calculation unit is further configured to:
separate information of the plurality of beams of reflected light in the plurality of directions received by the imaging element into information of beams of reflected light in respective directions of the plurality of directions; and
convert, based on the separated information of beams of reflected light in the respective directions, each of the separated information of beams of reflected light in the respective directions into the at least one of information of the eyeball movement or the positional information of the pupil.
11. The line-of-sight detection device according to claim 1 , wherein
the plurality of beams of reflected light includes:
a first beam of reflected light that is reflected in a first direction of the plurality of directions, and
a second beam of reflected light that is reflected in a second direction of the plurality of directions, wherein the second direction is different from the first direction,
the optical element is further configured to generate a difference in an amount of light of the first beam of reflected light that reaches the imaging element and an amount of light of the second beam of reflected light that reaches the imaging element, and
the calculation unit is further configured to:
separate information of the first beam of reflected light and the second beam of reflected light into first information of the first beam of reflected light that is reflected in the first direction and second information of the second beam of reflected light that is reflected in the second direction; and
convert, based on the first information and the second information, the first information and the second information into the at least one of the information of the eyeball movement or the positional information of the pupil.
12. The line-of-sight detection device according to claim 1 , wherein
the plurality of beams of reflected light includes:
a first beam of reflected light that is reflected in a first direction of the plurality of directions, and
a second beam of reflected light that is reflected in a second direction of the plurality of directions, wherein the second direction is different from the first direction,
the optical element that includes the plurality of optical elements is further configured to generate a difference in aberration of the first beam of reflected light and the second beam of reflected light,
the calculation unit includes a low-pass filter and a high-pass filter, and
the calculation unit is further configured to:
perform Fourier transform on information of the first beam of reflected light and information of the second beam of reflected light received by the imaging element;
separate the information of the first beam of reflected light and the information of the second beam of reflected light into first information of the first beam of reflected light that is reflected in the first direction and second information of the second beam of reflected light that is reflected in the second direction-based on the difference in the aberration, wherein the information of the first beam of reflected light and the information of the second beam of reflected light is separated based on the low-pass filter and the high-pass filter; and
convert, based on the first information and the second information, the first information and the second information into the at least one of the information of the eyeball movement or the positional information of the pupil.
13. The line-of-sight detection device according to claim 1 , wherein
the calculation unit is further configured to:
acquire a point group in a time direction based on each of pieces of information of beams of reflected light in at least two directions among pieces of information of the beams of reflected light in the plurality of directions;
generate at least two optical flows;
separate the information of the beams of reflected light in the at least two directions into information of beams of reflected light in respective directions of the at least two directions based on a difference between the at least two optical flows; and
convert, based on the separated information of the beams of reflected light in the respective directions of the at least two directions, the separated information of the beams of reflected light in the respective directions of the at least two directions into the at least one of the information of the eyeball movement or the positional information of the pupil.
14. The line-of-sight detection device according to claim 1 , wherein
the calculation unit is further configured to:
separate information of the plurality of beams of reflected light in the plurality of directions received by the imaging element into information of beams of reflected light in respective directions of the plurality of directions based on a deep neural network; and
convert, based on the separated information of the beams of reflected light in the respective directions of the plurality of directions, the separated information of the beams of reflected light in the respective directions of the plurality of directions into the at least one of the information of the eyeball movement or the positional information of the pupil.
15. The line-of-sight detection device according to claim 1 , wherein the imaging element is a CMOS image sensor.
16. The line-of-sight detection device according to claim 15 ,
wherein
the calculation unit includes a high-pass filter,
the calculation unit is further configured to:
convert information of the plurality of beams of reflected light in the plurality of directions received by the CMOS image sensor into edge information based on the high-pass filter;
separate the information of the plurality of beams of reflected light in the plurality of directions into information of beams of reflected light in respective directions of the plurality of directions based on the edge information; and
convert, based on the separated information, the separated information into the at least one of the information of the eyeball movement or the positional information of the pupil.
17. A display device, comprising:
a line-of-sight detection device comprising:
an imaging element;
an illumination unit configured to illuminate an eyeball with a plurality of beams of illumination light;
an optical element configured to guide a plurality of beams of reflected light to a direction of the imaging element, wherein
the plurality of beams of reflected light includes the plurality of beams of illumination light reflected in a plurality of directions from the eyeball,
the optical element includes a plurality of optical elements,
a first optical element of the plurality of optical elements has:
a substantial difference in diffraction efficiency from a diffraction efficiency of a second optical element of the plurality of optical elements, or
a substantial difference in reflectance amount from a reflectance amount of the second optical element, and
each of the plurality of optical elements is configured to guide a respective beam of reflected light of the plurality of beams of reflected light in the plurality of directions from the eyeball to the direction of the imaging element;
an optical system configured to cause the imaging element to non-selectively receive the plurality of beams of reflected light in the plurality of directions from the eyeball; and
a calculation unit configured to convert, based on the plurality of beams of reflected light in the plurality of directions from the eyeball received by the imaging element, the plurality of beams of reflected light into at least one of information of eyeball movement or positional information of a pupil.
18. A line-of-sight detection device, comprising:
an imaging element;
an illumination unit configured to illuminate an eyeball with a plurality of beams of illumination light;
an optical element configured to guide a plurality of beams of reflected light to a direction of the imaging element, wherein
the plurality of beams of reflected light includes the plurality of beams of illumination light reflected in a plurality of directions from the eyeball,
the optical element includes a plurality of optical elements,
the plurality of optical elements includes a first optical element and a second optical element, and
a difference between reflectance of the first optical element and reflectance of the second optical element is 50% or more;
an optical system configured to cause the imaging element to non-selectively receive the plurality of beams of reflected light in the plurality of directions from the eyeball; and
a calculation unit configured to convert, based on the plurality of beams of reflected light in the plurality of directions from the eyeball received by the imaging element, the plurality of beams of reflected light into at least one of information of eyeball movement or positional information of a pupil.
19. A line-of-sight detection device, comprising:
an imaging element;
an illumination unit configured to illuminate an eyeball with a plurality of beams of illumination light;
an optical element configured to guide a plurality of beams of reflected light to a direction of the imaging element, wherein
the plurality of beams of reflected light includes the plurality of beams of illumination light reflected in a plurality of directions from the eyeball,
the optical element includes a plurality of optical elements,
the plurality of optical elements includes a first optical element and a second optical element, and
a difference between diffraction efficiency of the first optical element and diffraction efficiency of the second optical element is 50% or more;
an optical system configured to cause the imaging element to non-selectively receive the plurality of beams of reflected light in the plurality of directions from the eyeball; and
a calculation unit configured to convert, based on the plurality of beams of reflected light in the plurality of directions from the eyeball received by the imaging element, the plurality of beams of reflected light into at least one of information of eyeball movement or positional information of a pupil.
20. A line-of-sight detection device, comprising:
an imaging element;
an illumination unit configured to illuminate an eyeball with a plurality of beams of illumination light;
an optical element configured to guide a plurality of beams of reflected light to a direction of the imaging element, wherein the plurality of beams of reflected light includes the plurality of beams of illumination light reflected in a plurality of directions from the eyeball;
at least one of a polarizing plate, an ND filter, or a retardation film, wherein
based on the at least one of the polarizing plate, the ND filter, or the retardation film, an amount of light of a first beam of reflected light of the plurality of beams of reflected light is adjustable, and
the first beam of reflected light is reflected in at least one direction of the plurality of directions;
an optical system configured to cause the imaging element to non-selectively receive the plurality of beams of reflected light in the plurality of directions from the eyeball; and
a calculation unit configured to convert, based on the plurality of beams of reflected light in the plurality of directions from the eyeball received by the imaging element, the plurality of beams of reflected light into at least one of information of eyeball movement or positional information of a pupil.