Gaze point detection device and gaze point detection method

1. A gaze point detection device which inputs a first image and first depth information obtained by a first photographing unit that obtains at least an image of an area in which a second photographing unit is included and depth information, and a second image and second depth information obtained by the second photographing unit which is positioned opposite to the first photographing unit and obtains at least an image of an area in which the first photographing unit is included and depth information, wherein the gaze point detection device comprises:

a memory configured to store instructions;

at least one processor configured to execute the instructions to:

detect a face portion of an object person from the first image and the first depth information,

estimate a line of sight direction of the object person using the face portion detected by the face detecting means,

estimate a three-dimensional spatial position of a pupil of the object person and three-dimensional spatial positions of the first photographing unit and the second photographing unit, based on the first image, the first depth information, the second image and the second depth information, and

identify a gaze point of the object person, based on the three-dimensional spatial position of the pupil of the object person and the line of sight direction.

2. The gaze point detection device according to claim 1 , wherein

the at least one processor further configured to execute the instructions to display the gaze point of the object person in the image photographed by the second photographing unit.

3. The gaze point detection device according to claim 1 , wherein the at least one processor further configured to execute the instructions to

obtain an image including a front side of the object person as the first image, and

obtain an image including the front side of the object person as the first image and an image including a back side of the object person as the second image.

4. The gaze point detection device according to claim 1 , wherein

the at least one processor further configured to execute the instructions to calculate a rotation matrix that relates the position of the first photographing unit in the second image to the position of the second photographing unit in the first image.

5. The gaze point detection device according to claim 1 , wherein

the at least one processor further configured to execute the instructions to correct the three-dimensional spatial position of the pupil of the object person and the three-dimensional spatial positions of the first photographing unit and the second photographing unit, based on a third image obtained by a third photographing unit that is installed to be able to photograph an image of an area in which at least the first photographing unit and the second photographing unit are included.

6. A gaze point detection method comprising:

inputting a first image and first depth information obtained by a first photographing unit that obtains at least an image of an area in which a second photographing unit is included and depth information, and a second image and second depth information obtained by the second photographing unit which is positioned opposite to the first photographing unit and obtains at least an image of an area in which the first photographing unit is included and depth information,

detecting a face portion of an object person from the first image and the first depth information,

estimating a line of sight direction of the object person using the detected face portion,

estimating a three-dimensional spatial position of a pupil of the object person and three-dimensional spatial positions of the first photographing unit and the second photographing unit, based on the first image, the first depth information, the second image and the second depth information, and

identifying a gaze point of the object person, based on the three-dimensional spatial position of the pupil of the object person and the line of sight direction.

7. The gaze point detection method according to claim 6 , further comprising

displaying the gaze point of the object person in the image photographed by the second photographing unit.

8. The gaze point detection method according to claim 6 , further comprising

calculating a rotation matrix that relates the position of the first photographing unit in the second image to the position of the second photographing unit in the first image.

9. The gaze point detection method according to claim 6 , further comprising

correcting the three-dimensional spatial position of the pupil of the object person and the three-dimensional spatial positions of the first photographing unit and the second photographing unit, based on a third image obtained by a third photographing unit that is installed to be able to photograph an image of an area in which at least the first photographing unit and the second photographing unit are included.

10. A non-transitory computer-readable information recording medium storing a gaze point detection program, when executed by a processor, performing:

inputting a first image and first depth information obtained by a first photographing unit that obtains at least an image of an area in which a second photographing unit is included and depth information, and a second image and second depth information obtained by the second photographing unit which is positioned opposite to the first photographing unit and obtains at least an image of an area in which the first photographing unit is included and depth information,

detecting a face portion of an object person from the first image and the first depth information,

estimating a line of sight direction of the object person using the detected face portion,

estimating a three-dimensional spatial position of a pupil of the object person and three-dimensional spatial positions of the first photographing unit and the second photographing unit, based on the first image, the first depth information, the second image and the second depth information, and

identifying a gaze point of the object person, based on the three-dimensional spatial position of the pupil of the object person and the line of sight direction.

11. The information recording medium according to claim 10 , wherein the gaze point detection program, when executed by the processor, further performs

displaying the gaze point of the object person in the image photographed by the second photographing unit.

12. The information recording medium according to claim 10 , wherein the gaze point detection program, when executed by the processor, further performs

calculating a rotation matrix that relates the position of the first photographing unit in the second image to the position of the second photographing unit in the first image.

13. The information recording medium according to claim 10 , wherein the gaze point detection program, when executed by the processor, further performs

correcting the three-dimensional spatial position of the pupil of the object person and the three-dimensional spatial positions of the first photographing unit and the second photographing unit, based on a third image obtained by a third photographing unit that is installed to be able to photograph an image of an area in which at least the first photographing unit and the second photographing unit are included.