Image processing device, moving apparatus, image processing method, and storage medium

1 . An image processing device comprising:

a plurality of imaging units each configured to include an optical system and an imaging element including a first photoelectric conversion unit and a second photoelectric conversion unit each of which generates a first image signal and a second image signal having a predetermined parallax from an optical image incident through the optical system;

at least one memory storing instructions; and

at least one processor or circuit capable of executing the instructions, the at least one processor or circuit configured to:

generate a plurality of pieces of image data based on outputs of the plurality of imaging units,

generate distance data for each pixel of the plurality of pieces of image data using an imaging plane phase difference method based on the first image signal and the second image signal generated by the first photoelectric conversion unit and the second photoelectric conversion unit of each of the plurality of imaging units,

generate a virtual viewpoint video viewed from a predetermined virtual viewpoint based on the plurality of pieces of image data and the distance data for each pixel of the plurality of pieces of image data,

generate a 3D model based on the distance data,

generate a textured 3D model by mapping the image data onto the 3D model,

generate a virtual viewpoint video based on an image of the textured 3D model viewed from the virtual viewpoint,

determine a reliability of the distance data based on characteristics of the optical system of the imaging unit,

determine the distance data to be used for 3D model generation based on the reliability and a result obtained by determining occlusion regions of the plurality of imaging units based on differences between distance data of the plurality of imaging units in an overlapping imaging region between the plurality of imaging units, and

perform region-division processing on at least one of the plurality of pieces of image data to obtain an object region, and, when the reliability of distance data of a predetermined region of the object is less than a predetermined value, correct the distance data of the predetermined region based on distance data of a region of the same object in which the reliability is equal to or greater than the predetermined value, the correction including linear interpolation using the distance data of the region having the reliability equal to or greater than the predetermined value.

2 . The image processing device according to claim 1 , wherein the plurality of imaging units are disposed apart from each other to image the surroundings of a moving apparatus, and

the at least one processor or circuit is configured to create the virtual viewpoint video of the surroundings of the moving apparatus.

3 . The image processing device according to claim 1 , wherein the characteristics of the optical system include a focal length.

4 . The image processing device according to claim 1 , wherein the at least one processor or circuit is configured to determine the reliability based on a contrast of the image data corresponding to the distance data.

5 . The image processing device according to claim 1 , wherein the at least one processor or circuit is configured to select the distance data to be used for 3D model generation from among the plurality of pieces of distance data based on the position of the virtual viewpoint.

6 . The image processing device according to claim 1 , wherein the imaging element includes one or two image sensors.

7 . A moving apparatus comprising:

a plurality of imaging units each configured to include an optical system and an imaging element including a first photoelectric conversion unit and a second photoelectric conversion unit each of which generates a first image signal and a second image signal having a predetermined parallax from an optical image incident through the optical system;

at least one memory storing instructions; and

at least one processor or circuit capable of executing the instructions, the at least one processor or circuit configured to:

generate a plurality of pieces of image data based on outputs of the plurality of imaging units,

generate distance data for each pixel of the plurality of pieces of image data using an imaging plane phase difference method based on the first image signal and the second image signal generated by the first photoelectric conversion unit and the second photoelectric conversion unit of each of the plurality of imaging units,

generate a virtual viewpoint video viewed from a predetermined virtual viewpoint based on the plurality of pieces of image data and the distance data for each pixel of the plurality of pieces of image data,

generate a 3D model based on the distance data,

generate a textured 3D model by mapping the image data onto the 3D model,

generate a virtual viewpoint video based on an image of the textured 3D model viewed from the virtual viewpoint,

determine a reliability of the distance data based on characteristics of the optical system of the imaging unit,

determine the distance data to be used for 3D model generation based on the reliability and a result obtained by determining occlusion regions of the plurality of imaging units based on differences between distance data of the plurality of imaging units in an overlapping imaging region between the plurality of imaging units,

perform region-division processing on at least one of the plurality of pieces of image data to obtain an object region, and, when the reliability of distance data of a predetermined region of the object is less than a predetermined value, correct the distance data of the predetermined region based on distance data of a region of the same object in which the reliability is equal to or greater than the predetermined value, the correction including linear interpolation using the distance data of the region having the reliability equal to or greater than the predetermined value, and

display the virtual viewpoint video.

8 . An imaging processing method comprising:

acquiring imaging outputs from a plurality of imaging units each configured to include an optical system and an imaging element including a first photoelectric conversion unit and a second photoelectric conversion unit each of which generates a first image signal and a second image signal having a predetermined parallax from an optical image incident through the optical system;

generating a plurality of pieces of image data based on the imaging outputs of the plurality of imaging units;

generating distance data for each pixel of the plurality of pieces of image data using an imaging plane phase difference method based on the first image signal and the second image signal generated by the first photoelectric conversion unit and the second photoelectric conversion unit each of the plurality of imaging units;

generating a virtual viewpoint video viewed from a predetermined virtual viewpoint based on the plurality of pieces of image data and the distance data for each pixel of the plurality of pieces of image data;

generating a 3D model based on the distance data;

generating a textured 3D model by mapping the image data onto the 3D model;

generating a virtual viewpoint video based on an image of the textured 3D model viewed from the virtual viewpoint;

determining a reliability of the distance data based on characteristics of the optical system of the imaging unit;

determining the distance data to be used for 3D model generation based on the reliability and a result obtained by determining occlusion regions of the plurality of imaging units based on differences between distance data of the plurality of imaging units in an overlapping imaging region between the plurality of imaging units; and

performing region-division processing on at least one of the plurality of pieces of image data to obtain an object region, and, when the reliability of distance data of a predetermined region of the object is less than a predetermined value, correcting the distance data of the predetermined region based on distance data of a region of the same object in which the reliability is equal to or greater than the predetermined value, the correction including linear interpolation using the distance data of the region having the reliability equal to or greater than the predetermined value.

9 . A non-transitory computer-readable storage medium configured to store a computer program comprising instructions for executing following processes:

acquiring imaging outputs from a plurality of imaging units each configured to include an optical system and an imaging element including a first photoelectric conversion unit and a second photoelectric conversion unit each of which generates a first image signal and a second image signal having a predetermined parallax from an optical image incident through the optical system;

generating a plurality of pieces of image data based on the imaging outputs of the plurality of imaging units;

generating distance data for each pixel of the plurality of pieces of image data using an imaging plane phase difference method based on the first image signal and the second image signal generated by the first photoelectric conversion unit and the second photoelectric conversion unit of each of the plurality of imaging units;

generating a virtual viewpoint video viewed from a predetermined virtual viewpoint based on the plurality of pieces of image data and the distance data for each pixel of the plurality of pieces of image data;

generating a 3D model based on the distance data;

generating a textured 3D model by mapping the image data onto the 3D model;

generating a virtual viewpoint video based on an image of the textured 3D model viewed from the virtual viewpoint;

determining a reliability of the distance data based on characteristics of the optical system of the imaging unit;

determining the distance data to be used for 3D model generation based on the reliability and a result obtained by determining occlusion regions of the plurality of imaging units based on differences between distance data of the plurality of imaging units in an overlapping imaging region between the plurality of imaging units; and

performing region-division processing on at least one of the plurality of pieces of image data to obtain an object region, and, when the reliability of distance data of a predetermined region of the object is less than a predetermined value, correcting the distance data of the predetermined region based on distance data of a region of the same object in which the reliability is equal to or greater than the predetermined value, the correction including linear interpolation using the distance data of the region having the reliability equal to or greater than the predetermined value.