Device and method for image processing
An image processing apparatus for estimating a depth field over a field of view. The apparatus comprises one or more processors configured to receive a captured polarisation image representing a polarisation of light received at a first set of multiple locations over the field of view; process the captured polarisation image using a first trained neural network to form a first estimate of depths to one or more locations over the field of view; receive ranging data representing environmental distances from a datum to one or more locations over the field of view; and process the ranging data using a second trained neural network to form a second estimate of depths to a second set of multiple locations over the field of view.
1 . An image processing apparatus for estimating a depth over a field of view, the apparatus comprising one or more processors configured to:
receive a captured polarisation image representing a polarisation of light received at a first set of multiple locations over the field of view;
process the captured polarisation image using a first trained neural network to form a first estimate of depths at each location in the first set of multiple locations over the field of view;
receive ranging data representing environmental distances from a datum to a second set of multiple locations over the field of view;
process the ranging data using a second trained neural network to form a second estimate of depths to the second set of multiple locations over the field of view;
form a synthetic polarisation image representing estimates of the polarisation of light received at a third set of multiple locations over the field of view by processing one or both of the first and second estimates of depths using a first analytic formulae; and
form synthetic ranging data representing estimates of the environmental distances to the third set of multiple locations over the field of view by processing one or both of the first and second estimates of depths using a second analytic formulae.
2 . The image processing apparatus as claimed in claim 1 , wherein the polarisation image represents the polarisation of light received for multiple locations over the field of view at one or more predetermined polarisations.
3 . The image processing apparatus as claimed in claim 1 , wherein the image processing apparatus comprises an active sensor apparatus for generating the ranging data and the active sensor apparatus comprises a time-of-flight sensor.
4 . The image processing apparatus as claimed in claim 1 , the apparatus being configured to generate multiple synthetic polarisation images to be estimated by the first analytic formulae for a plurality of reflectiveness properties.
5 . The image processing apparatus as claimed in claim 4 , wherein the plurality of reflectiveness properties comprise diffuseness and specularity.
6 . The image processing apparatus as claimed in claim 1 , the apparatus being configured to generate multiple synthetic polarisation images to be estimated by the first analytic formulae for a plurality of polarisations.
7 . The image processing apparatus as claimed in claim 1 , the apparatus being configured to generate multiple synthetic polarisation images, the synthetic polarisation images being estimated by the first analytic formulae for a plurality of colours.
8 . The image processing apparatus as claimed in claim 4 , wherein the apparatus is further configured to compare the polarisation image with the synthetic polarisation image which comprises reducing the multiple synthetic polarisation images to the synthetic polarisation image by selecting, for each location in the field of view for which multiple of the synthetic polarisation images hold estimated polarisation information, the polarisation information having the smallest estimated error.
9 . The image processing apparatus as claimed in claim 1 , wherein the first analytic formulae are such as to form the estimates of the distances to multiple locations over the field of view in dependence on the intensity of at least one polarisation image at the respective locations.
10 . The image processing apparatus as claimed in claim 1 , wherein the second analytic formulae are used to generate for each of multiple locations over the field of view, multiple estimates of the distances to the respective location in dependence on respective phase offsets.
11 . The image processing apparatus as claimed in claim 1 , wherein the image processing apparatus comprises a camera for capturing the captured polarisation image.
12 . The image processing apparatus as claimed in claim 1 , wherein the captured polarisation image comprises stereoscopic polarisation images.
13 . The image processing apparatus as claimed in claim 1 , wherein the first and second neural networks are the same.
14 . The image processing apparatus as claimed in claim 1 , wherein the first analytic formula is used to calculate polarisation estimates in dependence on a trigonometric function of an angle formed from a first sub-angle calculated in dependence on a normal to a corresponding surface and a second sub-angle representing a candidate polarisation angle.
15 . The image processing apparatus as claimed in claim 1 , wherein the second analytic formula is used to calculate distance estimates in dependence on a trigonometric function of an angle formed from a phase value calculated in dependence on a depth to a corresponding surface.
16 . A computer-implemented method for estimating a depth over a field of view, the method comprising:
receiving a captured polarisation image representing a polarisation of light received at a first set of multiple locations over the field of view;
processing the captured polarisation image using a first trained neural network to form a first estimate of depths to the first set of multiple locations over the field of view;
receiving ranging data representing environmental distances from a datum to a second set of multiple locations over the field of view;
processing the ranging data using a second trained neural network to form the second estimate of depths at a second set of multiple locations over the field of view;
forming a synthetic polarisation image representing estimates of the polarisation of light received at a third set of multiple locations over the field of view by processing one or both of the first and second estimates of depths using a first analytic formulae; and
forming synthetic ranging data representing estimates of the environmental distances to the third set of multiple locations over the field of view by processing one or both of the first and second estimates of depths using a second analytic formulae.
17 . The computer-implemented method as claimed in claim 16 , wherein the polarisation image represents the polarisation of light received for multiple locations over the field of view at one or more predetermined polarisations.
18 . The computer-implemented method as claimed in claim 16 , wherein the image processing apparatus comprises an active sensor apparatus for generating the ranging data and the active sensor apparatus comprises a time-of-flight sensor.
19 . The computer-implemented method as claimed in claim 16 , the apparatus being configured to form multiple synthetic polarisation images, the synthetic polarisation images being estimated by the first analytic formulae for a plurality of reflectiveness properties.
20 . The computer-implemented method as claimed in claim 19 , wherein the plurality of reflectiveness properties comprise diffuseness and specularity.