Image processing apparatus, image processing method, and storage medium
An image processing apparatus obtains data of a captured image, generates computer graphics corresponding to an appearance of a virtual object disposed in a virtual space from a virtual viewpoint, and generates a moving image by superimposing the computer graphics on the captured image. Generation information to be used in the case of generating a still image is outputted at time of acceptance of an obtainment request for data of the still image corresponding to a frame of the moving image. The generation information includes the data of the captured image and intermediate data to be generated in the case of generating the computer graphics, which are the data present at the time of acceptance of the obtainment request. The intermediate data is information used in lighting processing in the case of generating the computer graphics.
1. An image processing apparatus comprising:
one or more hardware processors; and
one or more memories storing one or more programs configured to be executed by the one or more hardware processors, the one or more programs including instructions for:
obtaining data of a captured image;
generating computer graphics corresponding to an appearance of a virtual object disposed in a virtual space from a virtual viewpoint;
generating a moving image by superimposing the computer graphics on the captured image;
accepting an obtainment request of data of a still image corresponding to a frame of the moving image; and
outputting generation information to be used in a case of generating the still image, wherein
the generation information includes
the data of the captured image at time of acceptance of the obtainment request, and
intermediate data to be generated in a case of generating the computer graphics, the intermediate data being present at the time of acceptance of the obtainment request, and
the intermediate data is information used in lighting processing in the case of generating the computer graphics.
2. The image processing apparatus according to claim 1 , wherein
the intermediate data includes at least any of
information indicating a color of each of one or more elements constituting the virtual object,
information indicating a distance of each of the one or more elements from the virtual viewpoint, and
information indicating reflection of each of the one or more elements.
3. The image processing apparatus according to claim 1 , wherein
in a case where the virtual object is formed from one or more polygons, the intermediate data includes at least any of
information indicating a color of each of the one or more polygons constituting the virtual object,
information indicating a distance of each of the one or more polygons from the virtual viewpoint,
information indicating reflection of each of the one or more polygons, and
information indicating an orientation of each of the one or more polygons.
4. The image processing apparatus according to claim 1 , wherein the one or more programs further include an instruction for:
causing a display device to display the moving image, wherein
the display device is the display device of a head mounted type, and
the moving image is displayed on the display device of the head mounted type.
5. The image processing apparatus according to claim 1 , wherein
data of the frame of the moving image at the time of acceptance of the obtainment request is outputted in addition to the generation information.
6. The image processing apparatus according to claim 1 , wherein
the generated computer graphics is generated based on rendering information indicating a rendering condition, and
the rendering information includes at least viewpoint information indicating a position of the virtual viewpoint, and
in addition to the viewpoint information, the rendering information includes at least any of
data of a shader code to be used as the rendering condition at the time of acceptance of the obtainment request, and
light source information to be used as the rendering condition at the time of acceptance of the obtainment request.
7. The image processing apparatus according to claim 1 , wherein
depth data indicating a depth of a space corresponding to at least part of an image capturing range in the captured image is further obtained,
the moving image is generated by using the obtained data of the captured image, the obtained depth data, and data of the generated computer graphics, and
the depth data is further outputted as the generation information.
8. The image processing apparatus according to claim 1 , wherein
RAW data corresponding to the data of the captured image is further obtained, and
the RAW data at the time of acceptance of the obtainment request is outputted as the generation information instead of the data of the captured image.
9. The image processing apparatus according to claim 1 , wherein
first computer graphics is generated,
the moving image is generated, and
the one or more programs further include instructions for:
obtaining the outputted generation information;
generating second computer graphics being different from the first computer graphics by carrying out the lighting processing by using the intermediate data included in the obtained generation information; and
generating the still image by superimposing the second computer graphics on the captured image by using the data of the captured image included in the obtained generation information and using data of the generated second computer graphics.
10. The image processing apparatus according to claim 9 , wherein
depth data indicating a depth of a space corresponding to at least part of an image capturing range in the captured image is further obtained,
the depth data at the time of acceptance of the obtainment request is further outputted as the generation information, and
the still image is generated by using the data of the captured image and the depth data included in the obtained generation information and using the data of the second computer graphics.
11. The image processing apparatus according to claim 9 ,
RAW data corresponding to the data of the captured image is further obtained,
the RAW data at the time of acceptance of the obtainment request is outputted as the generation information instead of the data of the captured image, and
the still image is generated by superimposing the second computer graphics on an image obtained by developing the RAW data included in the obtained generation information instead of the data of the captured image.
12. The image processing apparatus according to claim 9 , wherein
the second computer graphics is generated by further carrying out post-processing by using the intermediate data included in the obtained generation information.
13. An image processing method comprising the step of:
obtaining data of a captured image;
generating computer graphics corresponding to an appearance of a virtual object disposed in a virtual space from a virtual viewpoint;
generating a moving image by superimposing the computer graphics on the captured image;
accepting an obtainment request of data of a still image corresponding to a frame of the moving image; and
outputting generation information to be used in a case of generating the still image, wherein
the generation information includes
the data of the captured image at time of acceptance of the obtainment request, and
intermediate data to be generated in a case of generating the computer graphics, the intermediate data being present at the time of acceptance of the obtainment request, and
the intermediate data is information used in lighting processing in the case of generating the computer graphics.
14. A non-transitory computer readable storage medium storing a program for causing a computer to perform a control method of an image processing apparatus, the control method comprising the steps of
obtaining data of a captured image;
generating computer graphics corresponding to an appearance of a virtual object disposed in a virtual space from a virtual viewpoint;
generating a moving image by superimposing the computer graphics on the captured image;
accepting an obtainment request of data of a still image corresponding to a frame of the moving image; and
outputting generation information to be used in a case of generating the still image, wherein
the generation information includes
the data of the captured image at time of acceptance of the obtainment request, and
intermediate data to be generated in a case of generating the computer graphics, the intermediate data being present at the time of acceptance of the obtainment request, and
the intermediate data is information used in lighting processing in the case of generating the computer graphics.