Image processing device, image processing method, and image provision system for the display of viewpoint images of a shared environment

1. An image processing device, comprising:

a central processing unit (CPU) configured to:

receive each of:

a first viewpoint image of a first user, wherein the first viewpoint image is viewable from a viewpoint of the first user in a virtual space,

a second viewpoint image of a second user, wherein the second viewpoint image is viewable from a viewpoint of the second user in the virtual space,

viewpoint position information that indicates a position of the second user, and

line-of-sight information indicating a line-of-sight direction of the second user;

control superimposition of the second viewpoint image of the second user on the first viewpoint image of the first user based on a positional relationship of the first user and the second user in the virtual space, wherein the positional relationship is based on the viewpoint position information and the line-of-sight information of the second user;

control display of each of the first viewpoint image, and the second viewpoint image, wherein the second viewpoint image is superimposed on the first viewpoint image as a sub-screen on a display screen of the first user; and

control change of at least one of a position of the displayed second viewpoint image, a size of the displayed second viewpoint image, or transparency of the displayed second viewpoint image based on line-of-sight information of the first user, wherein the line-of-sight information of the first user indicates a line-of-sight direction of the first user with respect to the sub-screen.

2. The image processing device according to claim 1 , wherein the CPU is further configured to:

obtain specific information from the line-of-sight information of the first user and the line-of-sight information of the second user; and

control the display of the second viewpoint image based on the specific information.

3. The image processing device according to claim 2 , wherein

the specific information indicates a parallax associated with the line-of-sight information of the first user, and

the CPU is further configured to control movement of the displayed second viewpoint image to a center of the displayed first viewpoint image based on decrease in the parallax.

4. The image processing device according to claim 3 , wherein, in a case where the parallax is one of equal to or smaller than a first threshold, the CPU is further configured to control increase in the size of the displayed second viewpoint image.

5. The image processing device according to claim 4 , wherein the CPU is further configured to control, based on increase in the parallax, the movement of the displayed second viewpoint image in a direction out of a visual field of the first user.

6. The image processing device according to claim 5 , wherein

the CPU is further configured to control end of the display of the second viewpoint image in a case where the parallax is one of equal to or larger than a second threshold, and

the second threshold is larger than the first threshold.

7. The image processing device according to claim 2 , wherein

the specific information indicates a ratio at which a visual field of the first user overlaps with a visual field of the second user, and

the CPU is further configured to:

control increase in at least one of the size or transmittance of the displayed second viewpoint image in a case where the ratio increases; and

reduce at least one of the size or the transmittance of the displayed second viewpoint image in a case where the ratio decreases.

8. The image processing device according to claim 7 , wherein the CPU is further configured to control end of the display of the second viewpoint image in a case where the ratio is one of equal to or larger than a threshold.

9. The image processing device according to claim 8 , wherein, after the end of the display of the second viewpoint image, the CPU is further configured to restart the display of the second viewpoint image in a case where the ratio is one of equal to or smaller than the threshold.

10. The image processing device according to claim 2 , wherein

the CPU is further configured to control movement of the displayed second viewpoint image toward a center of a visual field of the first user, and

the movement of the displayed second viewpoint image toward the center of the visual field of the first user is controlled based on movement of the line-of-sight direction of the first user to the displayed second viewpoint image.

11. The image processing device according to claim 10 , wherein

the CPU is further configured to control display of a timer on the displayed second viewpoint image based on a static state of the displayed second viewpoint image, and

the displayed second viewpoint image is in the static state at the center of the visual field of the first user.

12. The image processing device according to claim 11 , wherein the CPU is further configured to:

control end of the display of the second viewpoint image based on end of the timer; and

control the display of the first viewpoint image based on the end of the display of the second viewpoint image, wherein the first viewpoint image is the same as the second viewpoint image.

13. The image processing device according to claim 11 , wherein the CPU is further configured to:

control, based on a shift of a line-of-sight of the first user before the timer ends, end of the display of the second viewpoint image; and

control the display of the first viewpoint image based on the end of the display of the second viewpoint image.

14. The image processing device according to claim 13 , wherein the CPU is further configured to restart the display of the second viewpoint image based on:

movement of at least one of the viewpoint of the first user or the viewpoint of the second user, and

a change in a direction vector of the viewpoint of the second user with respect to the first user, wherein the change in the direction vector of the viewpoint of the second user is one of equal to or larger than a threshold.

15. The image processing device according to claim 2 , wherein

the CPU is further configured to obtain the line-of-sight information of the first user from a sensor, and

the sensor detects movement of the first user.

16. The image processing device according to claim 15 , wherein the sensor is on a head mounted display.

17. The image processing device according to claim 1 , wherein bit rate distribution of the second viewpoint image is changeable based on one of the size or transmittance of the second viewpoint image.

18. An image processing method, comprising:

receiving each of:

a first viewpoint image of a first user, wherein the first viewpoint image is viewable from a viewpoint of the first user in a virtual space,

a second viewpoint image of a second user, wherein the second viewpoint image is viewable from a viewpoint of the second user in the virtual space,

viewpoint position information that indicates a position of the second user, and

line-of-sight information indicating a line-of-sight direction of the second user;

controlling superimposition of the second viewpoint image of the second user on the first viewpoint image of the first user based on a positional relationship of the first user and the second user in the virtual space, wherein the positional relationship is based on the viewpoint position information and the line-of-sight information of the second user;

controlling display of each of the first viewpoint image, and the second viewpoint image, wherein the second viewpoint image is superimposed on the first viewpoint image as a sub-screen on a display screen of the first user; and

controlling change of at least one of a position of the displayed second viewpoint image, a size of the displayed second viewpoint image, or transparency of the displayed second viewpoint image based on line-of-sight information of the first user, wherein the line-of-sight information of the first user indicates a line-of-sight direction of the first user with respect to the sub-screen.

19. An image provision system, comprising:

an image processing device that includes a central processing unit (CPU) configured to:

receive each of:

a first viewpoint image of a first user, wherein the first viewpoint image is viewable from a viewpoint of the first user in a virtual space,

a second viewpoint image of a second user, wherein the second viewpoint image is viewable from a viewpoint of the second user in the virtual space,

viewpoint position information that indicates a position of the second user, and

line-of-sight information indicating a line-of-sight direction of the second user;

control superimposition of the second viewpoint image of the second user on the first viewpoint image of the first user based on a positional relationship of the first user and the second user in the virtual space, wherein the positional relationship is based on the viewpoint position information and the line-of-sight information of the second user;

control display of each of the first viewpoint image, and the second viewpoint image, wherein the second viewpoint image is superimposed on the first viewpoint image as a sub-screen on a display screen of the first user; and

control change of at least one of a position of the displayed second viewpoint image, a size of the displayed second viewpoint image, or transparency of the displayed second viewpoint image based on line-of-sight information of the first user, wherein the line-of-sight information of the first user indicates a line-of-sight direction of the first user with respect to the sub-screen; and

an image provision device that includes circuitry configured to transmit the first viewpoint image and the second viewpoint image to the image processing device at bit rate distribution based on the size or the transparency of the second viewpoint image.

20. An image processing device, comprising:

a central processing unit (CPU) configured to:

receive each of:

a first viewpoint image of a first user, wherein the first viewpoint image is viewable from a viewpoint of the first user in a virtual space,

a second viewpoint image of a second user, wherein the second viewpoint image is viewable from a viewpoint of the second user in the virtual space,

viewpoint position information that indicates a position of the second user, and

line-of-sight information indicating a line-of-sight direction of the second user;

control superimposition of the second viewpoint image of the second user on the first viewpoint image of the first user based on a positional relationship of the first user and the second user in the virtual space,

wherein the positional relationship is based on the viewpoint position information and the line-of-sight information of the second user;

control display of each of the first viewpoint image, and the second viewpoint image superimposed on the first viewpoint image; and

obtain specific information from line-of-sight information of the first user and the line-of-sight information of the second user, wherein the specific information indicates a ratio at which a visual field of the first user overlaps with a visual field of the second user;

control increase in at least one of a size or a transmittance of the displayed second viewpoint image in a case where the ratio increases;

reduce at least one of the size or the transmittance of the displayed second viewpoint image in a case where the ratio decreases; and

control end of the display of the second viewpoint image in a case where the ratio is one of equal to or larger than a threshold.