Generation apparatus, system and method for generating virtual viewpoint image

1. A generation apparatus comprising:

one or more memories storing instructions; and

one or more processors executing the instructions to:

acquire three-dimensional shape data corresponding to an object captured from a plurality of directions;

acquire three-dimensional shape data corresponding to a structure captured from a plurality of directions;

acquire background data corresponding to a background different at least from the object and the structure both captured from a plurality of directions;

acquire information indicating a virtual viewpoint; and

generate a virtual viewpoint image based on the acquired three-dimensional shape data corresponding to the object, the acquired three-dimensional shape data corresponding to the structure, the acquired background data, and the acquired information indicating the virtual viewpoint,

wherein the three-dimensional shape data corresponding to the structure is acquired by generation thereof based on a difference between a captured image obtained by first image capturing in a state where the object does not exist and the structure does not exist and a captured image obtained by second image capturing from the same direction as that of the first image capturing in a state where the structure exists and the object does not exist.

2. The generation apparatus according to claim 1 , wherein the three-dimensional shape data corresponding to the structure is acquired by generation thereof before starting image capturing of the object.

3. The generation apparatus according to claim 1 , wherein the three-dimensional shape data corresponding to the structure is generated based on an image which is based on a captured image obtained by the first image capturing and a captured image obtained by the second image capturing and which displays the structure area being distinguished from other areas.

4. The generation apparatus according to claim 1 , wherein the virtual viewpoint image is generated

by combining, in a case where a distance from a virtual viewpoint specified by the information indicating the acquired virtual viewpoint to the object is shorter than a distance from the specified virtual viewpoint to the structure, the three-dimensional shape data corresponding to the structure and the three-dimensional shape data corresponding to the object so that the object exists in front of the structure in the virtual viewpoint image; and

by combining, in a case where the distance from the specified virtual viewpoint to the structure is shorter than the distance from the specified virtual viewpoint to the object, the three-dimensional shape data corresponding to the object and the three-dimensional shape data corresponding to the structure so that the object exists behind the structure in the virtual viewpoint image.

5. The generation apparatus according to claim 1 , wherein the object is a moving object.

6. The generation apparatus according to claim 1 , wherein at least one of a person and a ball is the object.

7. The generation apparatus according to claim 1 , wherein the structure is an object whose still state continues.

8. The generation apparatus according to claim 1 , wherein at least one of a soccer goal and a corner flag, which are used in a soccer game, is the structure.

9. The generation apparatus according to claim 1 , wherein the structure is an object installed at a predetermined position.

10. The generation apparatus according to claim 1 , wherein at least a part of the structure is installed on a field on which a person, who is an object, plays a game.

11. The generation apparatus according to claim 1 , wherein the structure is a specified object.

12. A system comprising:

one or more memories storing instructions; and

one or more processors executing the instructions to:

generate, based on a plurality of captured images obtained by image capturing from a plurality of directions, a first image representing a structure area and an object area being distinguished from other areas;

generate, based on a plurality of captured images obtained by image capturing from a plurality of directions, a second image representing the object area being distinguished from other areas;

transmit the generated first image and the generated second image; acquire, based on the transmitted first image and the transmitted second image, three-dimensional shape data corresponding to the object;

acquire, based on the transmitted first image, three-dimensional shape data corresponding to the structure;

acquire background data corresponding to an area of a background different at least from the object and the structure;

acquire information indicating a virtual viewpoint; and

generate a virtual viewpoint image based on the acquired three-dimensional shape data corresponding to the object, the acquired three-dimensional shape data corresponding to the structure, the acquired background data, and the acquired information indicating the virtual viewpoint,

wherein the generated first image is transmitted at a frequency lower than that of the generated second image.

13. The system according to claim 12 , wherein the one or more processors further execute the instructions to generate, based on the transmitted first image and the transmitted second image, a third image representing the structure area being distinguished from other areas.

14. The system according to claim 13 , wherein in a case where the second image is transmitted, the three-dimensional shape data corresponding to the object and the three-dimensional shape data corresponding to the structure based on the transmitted second image are acquired and the third image is generated.

15. The system according to claim 12 , wherein the virtual viewpoint image is generated

by combining, in a case where a distance from a virtual viewpoint specified by the information indicating the acquired virtual viewpoint to the object is shorter than a distance from the specified virtual viewpoint to the structure, the three-dimensional shape data corresponding to the structure and the three-dimensional shape data corresponding to the object so that the object exists in front of the structure in the virtual viewpoint image; and

by combining, in a case where the distance from the specified virtual viewpoint to the structure is shorter than the distance from the specified virtual viewpoint to the object, the three-dimensional shape data corresponding to the object and the three-dimensional shape data corresponding to the structure so that the object exists behind the structure in the virtual viewpoint image.

16. A generation method of generating a virtual viewpoint image, the generation method comprising:

acquiring three-dimensional shape data corresponding to an object captured from a plurality of directions;

acquiring three-dimensional shape data corresponding to a structure captured from a plurality of directions;

acquiring background data corresponding to a background different at least from the object and the structure both captured from a plurality of directions; acquiring information indicating a virtual viewpoint; and

generating a virtual viewpoint image based on the acquired three-dimensional shape data corresponding to the object, the acquired three-dimensional shape data corresponding to the structure, the acquired background data, and the acquired information indicating the virtual viewpoint,

wherein the three-dimensional shape data corresponding to the structure is acquired by generation thereof based on difference between a captured image obtained by first image capturing in a state where the object does not exist and the structure does not exist and a captured image obtained by second image capturing from the same direction as that of the first image capturing in a state where the structure exists and the object does not exist.

17. A generation method of generating a virtual viewpoint image, the generation method comprising:

generating, based on a plurality of captured images obtained by image capturing from a plurality of directions, a first image representing a structure area and an object area being distinguished from other areas;

generating, based on a plurality of captured images obtained by image capturing from a plurality of directions, a second image representing the object area being distinguished from other areas;

transmitting the generated first image and the generated second image;

acquiring, based on the transmitted first image and the transmitted second image, three-dimensional shape data corresponding to the object;

acquiring, based on the transmitted first image, three-dimensional shape data corresponding to the structure;

acquiring background data corresponding to an area of a background different at least from the object and the structure;

acquiring information indicating a virtual viewpoint; and

generating a virtual viewpoint image based on the acquired three-dimensional shape data corresponding to the object, the acquired three-dimensional shape data corresponding to the structure, the acquired background data, and the acquired information indicating the virtual viewpoint,

wherein in the transmission, the first image is transmitted at a frequency lower than that of the second image.

18. A non-transitory computer-readable recording medium storing a program that causes a computer to execute a generation method of generating a virtual viewpoint image, the generation method comprising:

acquiring three-dimensional shape data corresponding to an object captured from a plurality of directions;

acquiring three-dimensional shape data corresponding to a structure captured from a plurality of directions;

acquiring background data corresponding to a background different at least from the object and the structure both captured from a plurality of directions;

acquiring information indicating a virtual viewpoint; and

generating a virtual viewpoint image based on the acquired three-dimensional shape data corresponding to the object, the acquired three-dimensional shape data corresponding to the structure, the acquired background data, and the acquired information indicating the virtual viewpoint,

wherein the three-dimensional shape data corresponding to the structure is acquired by generation thereof based on difference between a captured image obtained by first image capturing in a state where the object does not exist and the structure does not exist and a captured image obtained by second image capturing from the same direction as that of the first image capturing in a state where the structure exists and the object does not exist.

19. A non-transitory computer-readable recording medium storing a program that causes a computer to execute a generation method of generating a virtual viewpoint image, the generation method comprising:

generating, based on a plurality of captured images obtained by image capturing from a plurality of directions, a first image representing a structure area and an object area being distinguished from other areas;

generating, based on a plurality of captured images obtained by image capturing from a plurality of directions, a second image representing the object area being distinguished from other areas;

transmitting the generated first image and the generated second image;

acquiring, based on the transmitted first image and the transmitted second image, three-dimensional shape data corresponding to the object;

acquiring, based on the transmitted first image, three-dimensional shape data corresponding to the structure;

acquiring background data corresponding to an area of a background different at least from the object and the structure;

acquiring information indicating a virtual viewpoint; and

generating a virtual viewpoint image based on the acquired three-dimensional shape data corresponding to the object, the acquired three-dimensional shape data corresponding to the structure, the acquired background data, and the acquired information indicating the virtual viewpoint,

wherein in the transmission, the first image is transmitted at a frequency lower than that of the second image.