PERIPHERY MONITORING DEVICE AND PERIPHERY MONITORING PROGRAM
A periphery monitoring device includes: an image obtaining unit configured to obtain a captured image captured by an image capturing unit provided in a vehicle so as to image an area including a road surface on periphery of the vehicle; and a restoration control unit configured to control, when the captured image includes a spotted area caused by a spot on an optical system of the image capturing unit, whether to execute restoration processing of outputting a restored image restored from the captured image so as to simulatively reproduce a state where the optical system of the image capturing unit does not have the spot by removing the spotted area, according to a positional relationship between a road surface area where the road surface is captured and the spotted area in the captured image.
1 . A periphery monitoring device comprising:
an image obtaining unit configured to obtain a captured image captured by an image capturing unit provided in a vehicle so as to image an area including a road surface on periphery of the vehicle; and
a restoration control unit configured to control, when the captured image includes a spotted area caused by a spot on an optical system of the image capturing unit, whether to execute restoration processing of outputting a restored image restored from the captured image so as to simulatively reproduce a state where the optical system of the image capturing unit does not have the spot by removing the spotted area, according to a positional relationship between a road surface area where the road surface is captured and the spotted area in the captured image.
2 . The periphery monitoring device according to claim 1 , wherein
the restoration control unit is configured to prevent the execution of the restoration processing as the spotted area is closer to a center of gravity of the road surface area in the captured image.
3 . The periphery monitoring device according to claim 2 , wherein
the restoration control unit is configured to switch whether to execute the restoration processing according to a comparison result between a threshold and an evaluation value which is calculated according to a degree of proximity between the road surface area and a center of gravity of the spotted area in the captured image.
4 . The periphery monitoring device according to claim 3 , wherein
the restoration control unit is configured to, when the restoration processing is not executed, compare the threshold with the evaluation value which is calculated based on a new captured image obtained by the image obtaining unit, after a spot removing unit provided in the vehicle is operated to try to physically remove the spot on the optical system of the image capturing unit.
5 . The periphery monitoring device according to claim 3 , wherein
the restoration control unit is configured to calculate the evaluation value based on spot data related to a position and a size of the spotted area included in the captured image, and weight data related to a weight predetermined for each area in the captured image such that a value of the weight varies according to a distance between each area and the center of gravity of the road surface area.
6 . The periphery monitoring device according to claim 5 , wherein
the restoration control unit is configured to obtain the spot data using a spot detection neural network pre-trained by machine learning so as to output a possibility of corresponding to the spotted area of each area in the captured image according to input of the captured image.
7 . The periphery monitoring device according to claim 1 , wherein
the restoration control unit is configured to execute the restoration processing using a restoration neural network pre-trained by machine learning so as to output the restored image corresponding to the captured image according to the input of the captured image.
8 . The periphery monitoring device according to claim 1 , wherein
the restoration control unit is configured to control whether to execute the restoration processing when substitution control for substituting at least a part of driving operation of a driver on the vehicle is executed.
9 . The periphery monitoring device according to claim 1 , wherein
the restoration control unit is configured to control whether to execute the restoration processing according to a positional relationship between a central lower area of the captured image and the spotted area on the premise that the center of gravity of the road surface area is in the central lower area of the captured image.
10 . The periphery monitoring device according to claim 1 , further comprising:
a road surface estimation unit configured to estimate the road surface area from the captured image, wherein
the restoration control unit is configured to control whether to execute the restoration processing according to a positional relationship between the road surface area estimated by the road surface estimation unit and the spotted area.
11 . A non-transitory computer readable medium storing a periphery monitoring program for causing a computer to execute:
an image obtaining step of obtaining a captured image captured by an image capturing unit provided in a vehicle so as to image an area including a road surface on periphery of the vehicle; and
a restoration control step of controlling, when the captured image includes a spotted area caused by a spot on an optical system of the image capturing unit, whether to execute restoration processing of outputting a restored image restored from the captured image so as to simulatively reproduce a state where the optical system of the image capturing unit does not have the spot by removing the spotted area, according to a positional relationship between a road surface area where the road surface is captured and the spotted area in the captured image.