Information processing apparatus, information processing method, and medium
An information processing apparatus is provided. A determination method for determining a color of an object in a virtual viewpoint video, which includes the object and is generated based on a captured image obtained by capturing an image of the object, is selected based on a movement of a virtual viewpoint corresponding to the virtual viewpoint video or a movement of the object. The virtual viewpoint video, which includes the object, is generated through determining the color of the object in the virtual viewpoint video with the selected determination method.
1. An information processing apparatus comprising one or more processors and one or more memories storing one or more programs which cause the one or more processors to:
select a determination method for determining a color of an object in a virtual viewpoint video, which includes the object and is generated based on a captured image obtained by capturing an image of the object, based on a movement of a virtual viewpoint corresponding to the virtual viewpoint video or a movement of the object, wherein the determination method is selected from:
a first determination method in which a color of a same object in the virtual viewpoint video changes between frames in accordance with a position of the virtual viewpoint selected in response to a movement amount of the virtual viewpoint or a movement amount of the object between frames exceeding a threshold, and
a second determination method in which the color of the same object in the virtual viewpoint video is constant between frames regardless of the position of the virtual viewpoint selected in response to the movement amount of the virtual viewpoint or the movement amount of the object between frames being less than or equal to the threshold; and
generate the virtual viewpoint video, which includes the object, through determining the color of the object in the virtual viewpoint video with the selected determination method.
2. The information processing apparatus according to claim 1 , wherein the one or more programs cause the one or more processor to:
select the determination method from out of the first determination method in which a color of an object in the virtual viewpoint video is determined in accordance with a position of the virtual viewpoint and the second determination method in which the color of the object in the virtual viewpoint video is determined regardless of the position of the virtual viewpoint.
3. The information processing apparatus according to claim 1 , wherein
in the first determination method, a captured image to be used for determining a color of an object in the virtual viewpoint video is selected in accordance with the position of the virtual viewpoint, and in the second determination method, the captured image to be used for determining the color of the object in the virtual viewpoint video is selected regardless of the position of the virtual viewpoint.
4. The information processing apparatus according to claim 3 , wherein
in the first determination method, a captured image, which has been captured by an image capturing apparatus whose line-of-sight direction from the image capturing apparatus to the object is closer to a line-of-sight direction from the virtual viewpoint to the object among a plurality of image capturing apparatuses capturing the captured image, is used for determining the color of the object in the virtual viewpoint video.
5. The information processing apparatus according to claim 1 , wherein
in the second determination method, the color of the object in the virtual viewpoint video is determined using color information that has been added to a three-dimensional model of the object.
6. The information processing apparatus according to claim 5 , wherein
the color information is added to the three-dimensional model based on the three-dimensional model and the captured image.
7. The information processing apparatus according to claim 6 , wherein
the color information is added to a portion which is of the three-dimensional model and which appears in the virtual viewpoint video.
8. The information processing apparatus according to claim 5 , wherein
among portions which are of the three-dimensional model and which appear in the virtual viewpoint video, for a portion for which color information has already been added, the color of the object is determined using that color information, and for a portion for which color information has not been added, the color of the object is determined based on the captured image.
9. The information processing apparatus according to claim 1 , wherein
in the second determination method, the color of the object in the virtual viewpoint video is determined based on a rendering result of a past frame.
10. The information processing apparatus according to claim 1 , wherein
the determination method is selected in accordance with whether an image capturing time of the object appearing in the virtual viewpoint video changes between frames.
11. The information processing apparatus according to claim 1 , wherein
the determination method is selected in accordance with a movement amount or a shape change amount between frames of the object appearing in the virtual viewpoint video.
12. The information processing apparatus according to claim 1 , wherein
the determination method is selected in accordance with a movement amount between frames of the virtual viewpoint.
13. The information processing apparatus according to claim 1 ,
wherein the virtual viewpoint video includes a plurality of objects, and
wherein the one or more programs cause the one or more processor to select a determination method for each of the plurality of objects.
14. The information processing apparatus according to claim 13 , wherein colors determined by a plurality of determination methods are combined.
15. The information processing apparatus according to claim 1 , wherein
the determination method is selected further based on whether the object includes a directional reflection region.
16. The information processing apparatus according to claim 15 , wherein
for the object including the directional reflection region, a determination method in which the color of the object in the virtual viewpoint video is determined in accordance with a position of the virtual viewpoint is selected regardless of a movement amount of the virtual viewpoint or the object.
17. An information processing method comprising:
selecting a determination method for determining a color of an object in a virtual viewpoint video, which includes the object and is generated based on a captured image obtained by capturing an image of the object, based on a movement of a virtual viewpoint corresponding to the virtual viewpoint video or a movement of the object, wherein the determination method is selected from:
a first determination method in which a color of a same object in the virtual viewpoint video changes between frames in accordance with a position of the virtual viewpoint selected in response to a movement amount of the virtual viewpoint or a movement amount of the object between frames exceeding a threshold, and
a second determination method in which the color of the same object in the virtual viewpoint video is constant between frames regardless of the position of the virtual viewpoint selected in response to the movement amount of the virtual viewpoint or the movement amount of the object between frames being less than or equal to the threshold; and
generating the virtual viewpoint video, which includes the object, through determining the color of the object in the virtual viewpoint video with the selected determination method.
18. A non-transitory computer-readable medium storing a program executable by a computer to perform a method comprising:
selecting a determination method for determining a color of an object in a virtual viewpoint video, which includes the object and is generated based on a captured image obtained by capturing an image of the object, based on a movement of a virtual viewpoint corresponding to the virtual viewpoint video or a movement of the object, wherein the determination method is selected from:
a first determination method in which a color of a same object in the virtual viewpoint video changes between frames in accordance with a position of the virtual viewpoint selected in response to a movement amount of the virtual viewpoint or a movement amount of the object between frames exceeding a threshold, and
a second determination method in which the color of the same object in the virtual viewpoint video is constant between frames regardless of the position of the virtual viewpoint selected in response to the movement amount of the virtual viewpoint or the movement amount of the object between frames being less than or equal to the threshold; and
generating the virtual viewpoint video, which includes the object, through determining the color of the object in the virtual viewpoint video with the selected determination method.