Depth acquisition device and depth acquisition method including estimating a depth of a dust region based on a visible light image

1. A depth acquisition device, comprising:

a memory; and

a processor, wherein

the processor performs:

acquiring timing information indicating a timing at which a light source irradiates a subject with infrared light;

acquiring an infrared light image stored in the memory, the infrared light image being generated by imaging a scene including the subject with the infrared light according to the timing indicated by the timing information;

acquiring a visible light image stored in the memory, the visible light image being generated by imaging a substantially same scene as the scene of the infrared light image, with visible light from a substantially same viewpoint as a viewpoint of the imaging the infrared light image at a substantially same time as an imaging time of imaging the infrared light image;

detecting a dust region showing dust from the infrared light image; and

estimating a depth of the dust region based on the visible light image and the dust region,

in the detecting of the dust region, the processor performs detecting, as the dust region, a higher luminance region having a luminance not less than a first threshold in the infrared light image, when the higher luminance region satisfies a first condition which is predetermined, and

the first condition is that a center of gravity of the higher luminance region is located on a straight line or an arc, the straight line or the arc intersecting (i) a center of gravity of each of at least two higher luminance regions other than the higher luminance region in the infrared light image and (ii) a Focus of Expansion (FOE) of one of the infrared light image and the visible light image.

2. The depth acquisition device according to claim 1 , wherein

in the estimating of the depth of the dust region, the processor performs:

estimating first depth information indicating a depth at each position in the infrared light image;

estimating second depth information indicating a corrected depth at each position in the dust region, the corrected depth being obtained by correcting, based on the visible light image, a depth at each position in the dust region which is indicated in the first depth information; and

generating third depth information indicating (i) a depth at each position in a region other than the dust region in the infrared light image which is indicated in the first depth information and (ii) a depth at each position in the dust region in the infrared light image which is indicated in the second depth information.

3. The depth acquisition device according to claim 1 , wherein

in the detecting of the dust region, the processor performs

detecting the higher luminance region as the dust region, when the higher luminance region further satisfies a second condition, which is predetermined.

4. The depth acquisition device according to claim 3 , wherein

the second condition is that a luminance of a position in the visible light image is less than a second threshold, the position in the visible light image corresponding to a center of gravity of the higher luminance region in the infrared light image.

5. The depth acquisition device according to claim 3 , wherein

the second condition is that a correlation coefficient between (i) a luminance in the higher luminance region in the infrared light image and (ii) a luminance in a region in the visible light image is less than a predetermined threshold, the region in the visible light image corresponding to the higher luminance region.

6. A depth acquisition device, comprising:

a memory; and

a processor, wherein

the processor performs:

acquiring timing information indicating a timing at which a light source irradiates a subject with infrared light;

acquiring an infrared light image stored in the memory, the infrared light image being generated by imaging a scene including the subject with the infrared light according to the timing indicated by the timing information;

acquiring a visible light image stored in the memory, the visible light image being generated by imaging a substantially same scene as the scene of the infrared light image, with visible light from a substantially same viewpoint as a viewpoint of the imaging the infrared light image at a substantially same time as an imaging time of imaging the infrared light image;

detecting a dust region showing dust from the infrared light image; and

estimating a depth of the dust region based on the visible light image and the dust region, in the detecting of the dust region, the processor:

performs principal component analysis on each pixel of a higher luminance region to detect a first principal component axis as a principal axis of the higher luminance region, the higher luminance region having a luminance not less than a first threshold in the infrared light image, and

performs detecting of the higher luminance region as the dust region, when the principal axis of the higher luminance region satisfies a first condition which is predetermined, and

the first condition is that one of the principle axis of the higher luminance region and a line extending from the principle axis intersects a Focus of Expansion (FOE) of the infrared light image.

7. A depth acquisition method, comprising:

acquiring timing information indicating a timing at which a light source irradiates a subject with infrared light;

acquiring an infrared light image stored in a memory, the infrared light image being generated by imaging a scene including the subject with the infrared light according to the timing indicated by the timing information;

acquiring a visible light image stored in the memory, the visible light image being generated by imaging a substantially same scene as the scene of the infrared light image, with visible light from a substantially same viewpoint as a viewpoint of the imaging the infrared light image at a substantially same time as an imaging time of imaging the infrared light image;

detecting a dust region showing dust from the infrared light image; and

estimating a depth of the dust region based on the visible light image and the dust region, wherein

in the detecting of the dust region, the processor performs detecting, as the dust region, a higher luminance region having a luminance not less than a first threshold in the infrared light image, when the higher luminance region satisfies a first condition which is predetermined, and

the first condition is that a center of gravity of the higher luminance region is located on a straight line or an arc, the straight line or the arc intersecting (i) a center of gravity of each of at least two higher luminance regions other than the higher luminance region in the infrared light image and (ii) a Focus of Expansion (FOE) of one of the infrared light image and the visible light image.