Display system for displaying mixed reality space image and processing method for use of display system

1. A display system comprising:

at least one processor and at least one memory coupled to the at least one processor and storing instructions that, when executed by the at least one processor, cause the at least one processor to function as:

a position-orientation acquisition unit configured to acquire a position and orientation of an image pickup apparatus based on a captured real space image;

a generation unit configured to generate a virtual space image based on the position and orientation;

a first image combining unit configured to generate a first mixed reality space image obtained by combining the captured real space image with the generated virtual space image; and

a second image combining unit configured generate a second mixed reality space image obtained by combining the captured real space image with the generated virtual space image,

wherein the position-orientation acquisition unit acquires the position and orientation of the image pickup apparatus based on a first frame of the captured real space image,

wherein the generation unit generates a first frame of the generated virtual space image based on the position and orientation of the image pickup apparatus in the first frame of the captured real space image,

wherein the first image combining unit combines a frame different from the first frame of the captured real space image with the first frame of the generated virtual space image, and

wherein the second image combining unit combines the first frame of the captured real space image with the first frame of the generated virtual space image.

2. The display system according to claim 1 ,

wherein the system includes an image pickup apparatus including an image pickup unit configured to capture the captured real space image and a display unit and a display apparatus,

wherein the image pickup apparatus includes a first position-orientation acquisition unit configured to acquire the position and orientation of the image pickup apparatus on a basis of the real space image,

a first generation unit configured to generate a first virtual space image on a basis of the position and orientation of the image pickup apparatus,

the first image combining unit configured to generate a first mixed reality space image by combining the real space image with the first virtual space image and display the first mixed reality space image, and

a first communication unit configured to transmit at least the real space image to the display apparatus,

wherein the first position-orientation acquisition unit acquires the position and orientation of the image pickup apparatus based on the first frame of the real space image,

wherein the first generation unit generates a first frame of the first virtual space image on the basis of the position and orientation of the image pickup apparatus based on the first frame of the real space image, and

wherein the first image combining unit combines the frame different from the first frame of the real space image with the first frame of the first virtual space image.

3. The display system according to claim 2 , wherein the display apparatus includes

a second communication unit configured to receive the captured real space image from the image pickup apparatus,

a second generation unit configured to generate a second virtual space image based on the position and orientation of the image pickup apparatus, and

the second image combining unit configured to generate a second mixed reality space image by combining the captured real space image received with the generated second virtual space image and display the second mixed reality space image,

wherein the second generation unit generates a first frame of the second virtual space image based on the position and orientation of the image pickup apparatus based on the first frame of the real space image, and

wherein the second image combining unit combines the first frame of the captured real space image with the first frame of the generated second virtual space image.

4. The display system according to claim 3 , wherein the first communication unit transmits, to the display apparatus, the first frame of the captured real space image and the position and orientation of the image pickup apparatus based on the first frame of the captured real space image,

wherein the second communication unit receives, from the image pickup apparatus, the first frame of the captured real space image and the position and orientation of the image pickup apparatus based on the first frame or the captured real space image, and

wherein the second generation unit generates a first frame of the second virtual space image based on the position and orientation of the image pickup apparatus in the first frame of the real space image received by the second communication unit.

5. The display system according to claim 4 , wherein the first position-orientation acquisition unit acquires the position and orientation of the image pickup apparatus at a capture time of the first frame of the real space image based on the first frame of the real space image,

wherein the first generation unit generates the first frame of the first virtual space image based on the position and orientation of the image pickup apparatus at a capture time of the first frame of the real space image,

wherein the first communication unit transmits, to the display apparatus, the first frame of the captured real space image and the position and orientation of the image pickup apparatus at a capture time of the first frame of the captured real space image,

wherein the second communication unit receives, from the image pickup apparatus, the first frame of the captured real space image and the position and orientation of the image pickup apparatus at a capture time of the first frame of the real space image, and

wherein the second generation unit generates a first frame of the second virtual space image based on the position and orientation of the image pickup apparatus at a capture time of the received first frame of the captured real space image.

6. The display system according to claim 3 , wherein the display apparatus includes a second position-orientation acquisition unit configured to acquire the position and orientation of the image pickup apparatus based on the first frame of the captured real space image of the first frame of the real space image received by the second communication unit, and

wherein the second generation unit generates a first frame of the second virtual space image based on the position and orientation of the image pickup apparatus in the first frame of the captured real space image acquired by the second position-orientation acquisition unit.

7. The display system according to claim 6 , wherein the first position-orientation acquisition unit acquires the position and orientation of the image pickup apparatus at a capture time of the first frame of the real space image based on the first frame of the real space image,

wherein the first generation unit generates the first frame of the first virtual space image based on the position and orientation of the image pickup apparatus at a capture time of the first frame of the captured real space image,

wherein the second position-orientation acquisition unit acquires the position and orientation of the image pickup apparatus at a capture time of the first frame of the real space image based on the received first frame of the real space image, and

wherein the second generation unit generates the first frame of the second virtual space image based on the position and orientation of the image pickup apparatus at a capture time of the acquired first frame of the captured real space image.

8. The display system according to claim 3 , wherein the first position-orientation acquisition unit acquires the position and orientation of the image pickup apparatus at a capture time of the first frame of the real space image,

wherein the first generation unit generates the first frame of the first virtual space image based on the position and orientation of the image pickup apparatus at a capture time of the first frame of the real space image, and

wherein the second generation unit generates a first frame of the second virtual space image based on the position and orientation of the image pickup apparatus at a capture time of the first frame of the real space image.

9. The display system according to claim 3 , wherein the first image combining unit combines a second frame subsequent to the first frame of the captured real space image with the first frame of the generated first virtual space image,

wherein the first position-orientation acquisition unit estimates the position and orientation of the image pickup apparatus at a capture time of the second frame of the real space image based on the first frame of the captured real space image,

wherein the first generation unit generates the first frame of the first virtual space image based on the position and orientation of the image pickup apparatus at a capture time of the second frame of the real space image, and

wherein the second generation unit generates the first frame of the second virtual space image based on the position and orientation of the image pickup apparatus at a capture time of the first frame of the real space image.

10. The display system according to claim 9 , wherein the first position-orientation acquisition unit acquires the position and orientation of the image pickup apparatus at a capture time of the first frame of the captured real space image and estimates the position and orientation of the image pickup apparatus at a capture time of the second frame of the real space image,

wherein the first generation unit generates the first frame of the first virtual space image based on the position and orientation of the image pickup apparatus at a capture time of the second frame of the real space image,

wherein the first communication unit transmits, to the display apparatus, the first frame of the captured real space image and the position and orientation of the image pickup apparatus at a capture time of the first frame of the captured real space image,

wherein the second communication unit receives, from the image pickup apparatus, the first frame of the real space image and the position and orientation of the image pickup apparatus at a capture time of the first frame of the capture real space image, and

wherein the second generation unit generates the first frame of the second virtual space image based on the position and orientation of the image pickup apparatus at a capture time of the received first frame of the captured real space image.

11. The display system according to claim 9 , wherein the display apparatus includes a second position-orientation acquisition unit configured to acquire the position and orientation of the image pickup apparatus at a capture time of the first frame of the real space image based on the first frame of the real space image received by the second communication unit, and

wherein the second generation unit generates the first frame of the second virtual space image based on the position and orientation of the image pickup apparatus at a capture time of the first frame of the real space image acquired by the second position-orientation acquisition unit.

12. The display system according to claim 3 , wherein the first image combining unit combines a second frame subsequent to the first frame of the real space image with the first frame of the first virtual space image,

wherein the first position-orientation acquisition unit acquires the position and orientation of the image pickup apparatus at a capture time of the first frame of the real space image based on the first frame of the real space image,

wherein the first generation unit generates a candidate frame of the first virtual space image at a capture time of the first frame of the real space image based on the position and orientation of the image pickup apparatus at the capture time of the first frame of the captured real space image,

wherein the first position-orientation acquisition unit acquires the position and orientation of the image pickup apparatus at a capture time of the second frame of the captured real space image based on the second frame of the real space image, and

wherein the first generation unit generates the first frame of the first virtual space image at a capture time of the second frame of the captured real space image based on a difference between the position and orientation of the image pickup apparatus at a capture time of the first frame of the captured real space image and the position and orientation of the image pickup apparatus at a capture time of the second frame of the captured real space image and the candidate frame of the first virtual space image.

13. The display system according to claim 12 , wherein the first communication unit transmits, to the display apparatus, the first frame of the real space image captured by the image pickup unit and the position and orientation of the image pickup apparatus at a capture time of the first frame of the captured real space image,

wherein the second communication unit receives, from the image pickup apparatus, the first frame of the captured real space image and the position and orientation of the image pickup apparatus at a capture time of the first frame of the real space image, and

wherein the second generation unit generates the first frame of the second virtual space image based on a basis of the position and orientation of the image pickup apparatus at a capture time of the first frame of the captured real space image received by the second communication unit.

14. The display system according to claim 12 , wherein the display apparatus includes a second position-orientation acquisition unit configured to acquire the position and orientation of the image pickup apparatus at a capture time of the first frame of the captured real space image based on the first frame of the real space image received by the second communication unit, and

wherein the second generation unit generates the first frame of the second virtual space image based on the position and orientation of the image pickup apparatus at a capture time of the acquired first frame of the captured real space image.

15. The display system according to claim 1 , wherein the first image combining unit combines a second frame subsequent to the first frame of the captured real space image with the first frame of the virtual space image.

16. The display system according to claim 15 , wherein the second frame of the real space image is the most recent frame of the real space image.

17. A processing method for use of a display system, the method comprising:

acquiring a position and orientation of an image pickup apparatus on a basis of a real space image;

generating a virtual space image on a basis of the position and orientation;

generating a first mixed reality space image obtained by combining the real space image with the virtual space image; and

generating a second mixed reality space image obtained by combining the real space image with the virtual space image,

wherein in acquiring a position and orientation of the image pickup apparatus, the position and orientation of the image pickup apparatus based on the first frame of the real space image is acquired,

wherein in generating a virtual space image, a first frame of the virtual space image is generated on a basis of the position and orientation of the image pickup apparatus based on the first frame of the real space image,

wherein in displaying a first mixed reality space image on the display unit, a frame different from the first frame of the real space image is combined with the first frame of the virtual space image, and

wherein in displaying a second mixed reality space image on the display apparatus, the first frame of the real space image is combined with the first frame of the virtual space image.

18. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor performs a method for use of a display system, the method comprising:

acquiring a position and orientation of an image pickup apparatus on a basis of a real space image;

generating a virtual space image on a basis of the position and orientation;

generating a first mixed reality space image obtained by combining the real space image with the virtual space image; and

generating a second mixed reality space image obtained by combining the real space image with the virtual space image,

wherein in acquiring a position and orientation of the image pickup apparatus, the position and orientation of the image pickup apparatus based on the first frame of the real space image is acquired,

wherein in generating a virtual space image, a first frame of the virtual space image is generated on a basis of the position and orientation of the image pickup apparatus based on the first frame of the real space image,

wherein in displaying a first mixed reality space image on the display unit, a frame different from the first frame of the real space image is combined with the first frame of the virtual space image, and

wherein in displaying a second mixed reality space image on the display apparatus, the first frame of the real space image is combined with the first frame of the virtual space image.