Generation apparatus and method for generating a virtual viewpoint image
A generation apparatus is configured to generate a virtual viewpoint image based on a plurality of images captured by a plurality of cameras for imaging a field from a plurality of different directions, the generation apparatus including an acquisition unit configured to acquire, based on a three-dimensional model of at least a portion of the field, correspondence information indicating correspondence between a coordinate of an image captured by at least one of the plurality of cameras and a coordinate related to a simple three-dimensional model less accurate than the three-dimensional model, and a generation unit configured to generate a virtual viewpoint image according to designation about a position and an orientation of a virtual viewpoint, by using an image captured by one or more of the plurality of cameras and the correspondence information acquired by the acquisition unit.
1. A generation apparatus comprising:
one or more memories storing instructions; and
one or more processors executing the instructions to:
acquire, based on a first three-dimensional model of an object captured by a plurality of cameras from a plurality of different directions, correspondence information indicating correspondence between a point of an image captured by at least one of the plurality of cameras and a point related to a second three-dimensional model of the object less accurate than the first three-dimensional model, wherein the correspondence information is acquired based on first correspondence relationship information and second correspondence relationship information, the first correspondence relationship information representing a correspondence relationship between a point of the image captured by at least one of the plurality of cameras and a point of the first three-dimensional model, and the second correspondence relationship information representing a correspondence relationship between a point of the first three-dimensional model and a point related to the second three-dimensional model; and
generate a virtual viewpoint image according to a position and an orientation of a virtual viewpoint, by using an image captured by one or more of the plurality of cameras and the acquired correspondence information.
2. The generation apparatus according to claim 1 , wherein the correspondence information indicates correspondence between a point of the image captured by at least one of the plurality of cameras and a point in a texture corresponding to the second three-dimensional model.
3. The generation apparatus according to claim 1 , wherein the correspondence information indicates correspondence between a point of the image captured by at least one of the plurality of cameras and a point located on a surface of a shape represented by the second three-dimensional model.
4. The generation apparatus according to claim 1 , wherein the one or more processors further execute the instructions to
determine, by using the correspondence information, pixel information to be in correspondence with a point in a texture of the second three-dimensional model, among pixel information of the images captured by the plurality of cameras, and
wherein the virtual viewpoint image is generated based on the position and the orientation of the virtual viewpoint, and the texture of the second three-dimensional model to which the pixel information is applied according to the determining.
5. The generation apparatus according to claim 1 , wherein the correspondence information is acquired from a storage device that stores the correspondence information.
6. The generation apparatus according to claim 1 , wherein the first correspondence relationship information is generated using a camera parameter about a position and an orientation of the at least one of the cameras.
7. The generation apparatus according to claim 1 , wherein the second correspondence relationship information is generated by executing projection processing for determining a point included in the second three-dimensional model corresponding to a point included in the first three-dimensional model.
8. The generation apparatus according to claim 1 , the one or more processors further execute the instructions to extract an area of a predetermined object from the plurality of images obtained by the plurality of cameras,
wherein an image of a background area excluding the predetermined object in the virtual viewpoint image is generated by using the image captured by one or more of the plurality of cameras and the correspondence information, and an area of the predetermined object in the virtual viewpoint image is generated based on data of an image corresponding to extracted area.
9. A generation method comprising:
acquiring, based on a first three-dimensional model of an object captured by a plurality of cameras from a plurality of different directions, correspondence information indicating correspondence between a point of an image captured by at least one of the plurality of cameras and a point related to a second three-dimensional model of the object less accurate than the first three-dimensional model, wherein the correspondence information is acquired based on first correspondence relationship information and second correspondence relationship information, the first correspondence relationship information representing a correspondence relationship between a point of the image captured by at least one of the plurality of cameras and a point of the first three-dimensional model, and the second correspondence relationship information representing a correspondence relationship between a point of the first three-dimensional model and a point related to the second three-dimensional model; and
generating a virtual viewpoint image according to a position and an orientation of a virtual viewpoint, by using an image captured by one or more of the plurality of cameras and the acquired correspondence information.
10. The generation method according to claim 9 , wherein the correspondence information indicates correspondence between a point of the image captured by at least one of the plurality of cameras and a point in a texture corresponding to the second three-dimensional model.
11. The generation method according to claim 9 , wherein the generating includes
determining, by using the correspondence information, pixel information to be in correspondence with a point in a texture of the second three-dimensional model, among pixel information of the images captured by the plurality of cameras, and
generating a virtual viewpoint image according to a position and an orientation of the virtual viewpoint, by using the texture of the second three-dimensional model to which the pixel information is applied according to the determining.
12. A non-transitory computer readable storage medium storing a program for causing a computer to execute a generation method, the generation method comprising:
acquiring, based on a first three-dimensional model of an object captured by a plurality of cameras from a plurality of different directions, correspondence information indicating correspondence between a point of an image captured by at least one of the plurality of cameras and a point related to a second three-dimensional model of the object less accurate than the first three-dimensional model, wherein the correspondence information is acquired based on first correspondence relationship information and second correspondence relationship information, the first correspondence relationship information representing a correspondence relationship between a point of the image captured by at least one of the plurality of cameras and a point of the first three-dimensional model, and the second correspondence relationship information representing a correspondence relationship between a point of the first three-dimensional model and a point related to the second three-dimensional model; and
generating a virtual viewpoint image according to a position and an orientation of a virtual viewpoint, by using an image captured by one or more of the plurality of cameras and the acquired correspondence information.