Image processing apparatus, image processing method, and storage medium for lighting processing on image using model data

1. An image processing apparatus comprising:

an image acquisition unit configured to acquire image data representing an image including a subject;

a setting unit configured to set a position of a virtual light source;

a holding unit configured to hold face model data representing a three-dimensional shape of a predetermined face;

a detection unit configured to detect face information including information indicating a face region occupied by the face of the subject in the image represented by the image data;

a generation unit configured to generate correction face data that is normal data according to the face of the subject based on the face information detected by the detection unit and the face model data; and

a processing unit configured to perform lighting processing for correcting pixel values of pixels in the face region, based on the correction face data and the position of the virtual light source.

2. The image processing apparatus according to claim 1 , further comprising a distance acquisition unit configured to acquire distance information, which indicates a distance from an imaging device that has captured the image of the subject to the subject, for each pixel of the image data; and

a normal acquisition unit configured to acquire normal information indicating a normal direction on a surface of the subject for each pixel of the image data, based on the distance information,

wherein the processing unit performs the lighting processing based on the correction face data, the normal information, and the position of the light source.

3. The image processing apparatus according to claim 2 ,

wherein the face model data is normal image data in which a normal direction of a surface of the predetermined face is stored for each pixel of an image indicating the predetermined face, and

wherein the normal acquisition unit corrects, of the normal information, normal information of a region corresponding to the face detected by the detection unit, based on the correction face data, and

wherein the processing unit performs the lighting processing, based on the normal information corrected based on the correction face data, and the position of the light source.

4. The image processing apparatus according to claim 1 , wherein the face model data has a normal direction of a surface of the predetermined face, as a component in a direction of each of three coordinate axes orthogonal to each other, for each pixel of an image indicating the predetermined face, and

wherein the generation unit further causes a joint bilateral filter based on the pixel value of the image data to act on the transformed face model data, for each direction of the coordinate axes.

5. The image processing apparatus according to claim 1 , wherein the lighting processing is processing for performing addition or subtraction for the pixel value of the image data according to a distance between a subject position corresponding to each pixel of the image data and the position of the light source set by the setting unit.

6. The image processing apparatus according to claim 1 ,

wherein the processing unit acquires information indicating presence or absence of an electronic-flash emission when the image data is captured,

wherein, in a case where the electronic-flash emission is present when the image data is captured, the processing unit performs, as the lighting processing, processing for performing subtraction for the pixel value of the image data according to a distance between a subject position corresponding to each pixel of the image data and the position of the light source set by the setting unit, and

wherein, in a case where the electronic-flash emission is absent when the image data is captured, the processing unit performs, as the lighting processing, processing for performing addition for the pixel value of the image data according to the distance between the subject position corresponding to each pixel of the image data and the position of the light source set by the setting unit.

7. The image processing apparatus according to claim 1 , wherein the processing unit determines a correction degree for the pixel value of each pixel of the image data, based on a luminance value of each pixel of the image data, and the correction degree for the pixel value is smaller for the pixel of which the luminance value is larger.

8. The image processing apparatus according to claim 1 , wherein the generation unit transforms the face model data according to the face region detected by the detection unit.

9. The image processing apparatus according to claim 8 , wherein the detection unit detects organ position information indicating positions of parts included in the face as the face information,

wherein the generation unit transforms the face model data so that positions of parts in the face indicated by the face model data match the positions of parts included in the face detected by the detection unit.

10. The image processing apparatus according to claim 1 , further comprising a distance acquisition unit configured to acquire distance information, which indicates a distance from an imaging device that has captured the image of the subject to the subject, for each pixel of the image data,

wherein the processing unit performs lighting processing for correcting pixel values of pixels in the face region based on the distance information, the correction face data and the position of the virtual light source.

11. The image processing apparatus according to claim 10 , wherein the distance acquisition unit acquires distance image data that contains a distance value for each pixel from the imaging device, and corrects the distance image data based on the face information detected by the detection unit to acquire the distance information.

12. The image processing apparatus according to claim 1 , wherein the face model data includes normal image data in which a normal vector of the surface of the predetermined face, which corresponds to a face having the three-dimensional shape, is stored in a pixel value.

13. An image processing method comprising:

acquiring image data representing an image including a subject;

setting a position of a virtual light source;

holding face model data representing a three-dimensional shape of a predetermined face;

detecting face information including information indicating a face region occupied by the face of the subject in the image represented by the image data;

generating correction face data that is normal data according to the face of the subject based on the detected face and the face model data; and

performing lighting processing for correcting pixel values of pixels in the face region based on the correction face data and the position of the virtual light source.

14. A non-transitory computer readable storage medium storing a program for causing a computer to execute an image processing method, the method comprising:

acquiring image data representing an image including a subject;

setting a position of a virtual light source;

holding face model data representing a three-dimensional shape of a predetermined face;

detecting face information including information indicating a face region occupied by the face of the subject in the image represented by the image data;

generating correction face data that is normal data according to the face of the subject based on the detected face and the face model data; and

performing lighting processing for correcting pixel values of pixels in the face region based on the correction face data and the position of the virtual light source.