Terminal device, system, program and method for compositing a real-space image of a player into a virtual space

1. A terminal device comprising:

an interface configured to communicate with an external device including a display and a real space camera that are disposed in a real space;

a sensor configured to detect an instruction input of a real player in the real space;

a memory configured to store computer-readable instructions; and

a processor configured to execute the computer-readable instructions so as to:

cause the real space camera to capture a real space image of a real space target including the real player, the real space image includes a real space color image and a real space depth image, each pixel of the real space color image having a color value, each pixel of the real space depth image having a depth value;

obtain the real space image via the interface so as to store the real space image in the memory;

cause a virtual space camera to capture a virtual space image of a virtual space target including a virtual object, the virtual space camera being disposed in a virtual space of a game application, the real player performing the instruction input to the virtual object, the virtual space image includes a virtual space color image and a virtual space depth image, each pixel of the virtual space color image having a corresponding color value that is stored in the memory, each pixel of the virtual space depth image has a corresponding depth value that is stored in the memory;

obtain the virtual space image via the interface so as to store the virtual space image in the memory;

convert each of the depth values of a first group of the pixels of the real space depth image to a predetermined depth value, the first group of the pixels being spaced apart from the real space camera by a first distance or more;

create a player image of the real player from the real space image by excluding the pixels having the predetermined depth values, the player image being stored in the memory;

create a composite image that is formed by composing part of the virtual space image stored in the memory and the player image stored in the memory; and

output the composite image to the display via the interface so that the display is configured to display the composite image,

wherein the processor is configured to synchronize the real space camera with the virtual space camera so that the real space image and the virtual space image are captured at a same time, and

the processor is configured to start and end the game application in response to the detection of the instruction input of the real player by the sensor.

2. The terminal device according to claim 1 ,

wherein the real space camera is configured to measure the depth value of each of the pixels of the real space depth image, and

the real space camera is configured to generate the color value of each of the pixels of the real space color image.

3. The terminal device according to claim 1 ,

wherein the depth value corresponds to a distance from the real space camera to part of an imaging target in the real space image corresponding to each of the pixels of the real space depth image.

4. The terminal device according to claim 1 ,

wherein the color value corresponds to one of brightness and shade of a color of each of the pixels of the real space color image.

5. The terminal device according to claim 1 ,

wherein the real space camera is configured to capture the real space depth image at a first resolution, and

the real space camera is configured to capture the real space color image at a second resolution higher than the first resolution.

6. The terminal device according to claim 5 ,

wherein the processor is configured to convert the first resolution of the real space depth image to the second resolution.

7. The terminal device according to claim 6 ,

wherein when the processor determines that the color value of a first pixel of the pixels of the real space color image is a predetermined color value, the processor is configured to convert the depth value of a second pixel of the pixels of the real space depth image having the second resolution into the predetermined depth value that corresponds to an infinity value, and

the first and second pixels are located at a same coordinate position.

8. The terminal device according to claim 1 ,

wherein the processor is configured to compare the depth value for a first pixel of the pixels of the real space depth image and the depth value of a second pixel of the pixels of the virtual space depth image, and the first and second pixels are located at a same coordinate position,

when the depth value of the first pixel of the real space depth image is smaller than the depth value of the second pixel of the virtual space depth image, the processor is configured to select the color value of a third pixel of the pixels of the real space color image as the color value of a fourth pixel of pixels of the composite image, and the first, second, third, and fourth pixels are located at the same coordinate position, and

when the depth value of the second pixel of the virtual space depth image is smaller than the depth value of the first pixel of the real space depth image, the processor is configured to select the color value of a fifth pixel of the pixels of the virtual space color image as the color value of the fourth pixel of pixels of the composite image, and the first, second, third, fourth, and fifth pixels are located at the same coordinate position.

9. The terminal device according to claim 1 ,

wherein the real space target further includes a background screen located at a rear side of the real player with respect to the real space camera, the background screen is spaced apart from the real space camera by a second distance that is shorter than the first distance, and

wherein the processor is configured to:

determine a second group of the pixels of the real space color image that has the color value of the background screen; and

convert each of the depth values of a third group of the pixels of the real space depth image to the predetermined depth value, the second group of the pixels and the third group of the pixels being located at a same coordinate position.

10. A system comprising:

a terminal device, the terminal device including:

an interface;

a memory configured to store computer-readable instructions; and

a processor configured to execute the computer-readable instructions;

a sensor configured to detect an instruction input of a real player in a real space;

a real space camera arranged in the real space, communicatively connected with the terminal device via the interface, and configured to capture a real space image of a real space target including the real player in the real space, the real player performing the instruction input to a virtual object arranged in a virtual space of a game application; and

a display communicatively connected with the terminal device via the interface and configured to display an image in which at least a part of a virtual space image of a virtual space target virtually imaged by a virtual camera arranged in the virtual space,

wherein the processor is configured to:

cause the real space camera to capture the real space image, the real space image includes a real space color image and a real space depth image, each pixel of the real space color image having a color value, each pixel of the real space depth image having a depth value;

obtain the real space image via the interface so as to store the real space image in the memory;

cause the virtual space camera to capture the virtual space image of the virtual space target including the virtual object, the virtual space image includes a virtual space color image and a virtual space depth image, each pixel of the virtual space color image having a corresponding color value that is stored in the memory, each pixel of the virtual space depth image has a corresponding depth value that is stored in the memory;

obtain the virtual space image via the interface so as to store the virtual space image in the memory;

convert each of the depth values of a first group of the pixels of the real space depth image to a predetermined depth value, the first group of the pixels being spaced apart from the real space camera by a first distance or more;

create a player image of the real player from the real space image by excluding the pixels having the predetermined depth values, the player image being stored in the memory;

create a composite image that is formed by composing part of the virtual space image stored in the memory and the player image; and

output the composite image to the display via the interface so that the display is configured to display the composite image,

wherein the processor is configured to synchronize the real space camera with the virtual space camera so that the real space image and the virtual space image are captured at a same time, and

the processor is configured to start and end the game application in response to the detection of the instruction input of the real player by the sensor.

11. The system according to claim 10 ,

wherein the real space target further includes a background screen located at a rear side of the real player with respect to the real space camera, the background screen is spaced apart from the real space camera by a second distance that is shorter than the first distance, and

wherein the processor is configured to:

determine a second group of the pixels of the real space color image that has the color value of the background screen; and

convert each of the depth values of a third group of the pixels of the real space depth image to the predetermined depth value, the second group of the pixels and the third group of the pixels being located at a same coordinate.

12. A non-transitory computer-readable storage medium having executable computer-readable instructions of a computer program stored thereon for causing a computer to execute a process by a processor, the computer configured to perform the steps of:

causing a sensor to detect an instruction input of a real player in a real space;

causing a real space camera to capture a real space image of a real space target including the real player, the real space camera being disposed in the real space, the real space image includes a real space color image and a real space depth image, each pixel of the real space color image having a color value, each pixel of the real space depth image having a depth value;

obtaining the real space image so as to store the real space image in a memory;

causing a virtual space camera to capture a virtual space image of a virtual space target including a virtual object, the virtual space camera being disposed in a virtual space of a game application, the real player performing the instruction input to the virtual object, the virtual space image includes a virtual space color image and a virtual space depth image, each pixel of the virtual space color image having a corresponding color value that is stored in the memory, each pixel of the virtual space depth image has a corresponding depth value that is stored in the memory;

obtaining the virtual space image so as to store the virtual space image in the memory;

converting each of the depth values of a first group of the pixels of the real space depth image to a predetermined depth value, the first group of the pixels being spaced apart from the real space camera by a first distance or more;

creating a player image of the real player from the real space image by excluding the pixels having the predetermined depth values, the player image being stored in the memory;

creating a composite image that is formed by composing part of the virtual space image stored in the memory and the player image; and

outputting the composite image to a display so that the display is configured to display the composite image,

wherein the processor is configured to synchronize the real space camera with the virtual space camera so that the real space image and the virtual space image are captured at a same time, and

the processor is configured to start and end the game application in response to the detection of the instruction input of the real player by the sensor.

13. The non-transitory computer-readable storage medium according to claim 12 ,

wherein the real space target further includes a background screen located at a rear side of the real player with respect to the real space camera, the background screen is spaced apart from the real space camera by a second distance that is shorter than the first distance, and

wherein the processor is configured to:

determine a second group of the pixels of the real space color image that has the color value of the background screen; and

convert each of the depth values of a third group of the pixels of the real space depth image to the predetermined depth value, the second group of the pixels and the third group of the pixels being located at a same coordinate.

14. A method for causing a processor in a computer to execute a process, the computer including a memory configured to store computer-readable instructions,

the method comprising executing the computer-readable instructions on the processor the steps of:

causing a sensor to detect an instruction input of a real player in a real space;

causing a real space camera to capture a real space image of a real space target including the real player, the real space camera being disposed in the real space, the real space image includes a real space color image and a real space depth image, each pixel of the real space color image having a color value, each pixel of the real space depth image having a depth value;

obtaining the real space image so as to store the real space image in the memory;

causing a virtual space camera to capture a virtual space image of a virtual space target including a virtual object, the virtual space camera being disposed in a virtual space, the real player performing the instruction input to the virtual object, the virtual space image includes a virtual space color image and a virtual space depth image, each pixel of the virtual space color image having a corresponding color value that is stored in the memory, each pixel of the virtual space depth image has a corresponding depth value that is stored in the memory;

obtaining the virtual space image so as to store the virtual space image in the memory;

converting each of the depth values of a first group of the pixels of the real space depth image to a predetermined depth value, the first group of the pixels being spaced apart from the real space camera by a first distance or more;

creating a player image of the real player from the real space image by excluding the pixels having the predetermined depth values, the player image being stored in the memory;

creating a composite image that is formed by composing part of the virtual space image stored in the memory and the player image; and

outputting the composite image to a display so that the display is configured to display the composite image,

wherein the processor is configured to synchronize the real space camera with the virtual space camera so that the real space image and the virtual space image are captured at a same time, and

the processor is configured to start and end a game application in response to the detection of the instruction input of the real player by the sensor.

15. The method according to claim 14 ,

wherein the real space target further includes a background screen located at a rear side of the real player with respect to the real space camera, the background screen is spaced apart from the real space camera by a second distance that is shorter than the first distance, and

wherein the processor is configured to:

determine a second group of the pixels of the real space color image that has the color value of the background screen; and

convert each of the depth values of a third group of the pixels of the real space depth image to the predetermined depth value, the second group of the pixels and the third group of the pixels being located at a same coordinate.