Head mounted display device and control method

1. A head mounted display (HMD) device, comprising:

a display;

a distance image sensor that includes a camera configured to capture an image in a capturing range, the distance image sensor configured to acquire distance information indicating a distance from the distance image sensor to a hand of a user wearing the HMD device and output distance image information that combines the captured image and the acquired distance information; and

a processor configured to:

extract an image of the hand of the user from the captured image based on the distance image information,

specify a shape of the hand from the image of the hand,

calculate an inclination of the hand based on the distance information corresponding to the image of the hand,

determine whether the specified shape is a predetermined shape,

determine a viewpoint motion vector for moving a viewpoint position of the user in a virtual space when it is determined that the specified shape is a predetermined shape,

generate image data after the viewpoint position is moved,

determine a transparency of the image of the hand based on the calculated inclination,

generate an superimpose image by superimposing the image of the hand on the generated image data, and

control the display to display the generated superimpose image.

2. The HMD device according to claim 1 , wherein the display is a non-transmissive display.

3. The HMD device according to claim 1 , wherein

the display is configured to display video, and

the processor is further configured to:

generate a partial image corresponding to the hand region from the image, and

control the display to display the partial image superimposed on the video.

4. The HMD device according to claim 3 , wherein the partial image is displayed transparently on the video.

5. The HMD device according to claim 4 , wherein when the shape of the hand is a second shape that is different from the predetermined shape, the processor is configured to control the display to display the partial image on another image within the video in which the viewpoint position has not been moved.

6. The HMD device according to claim 5 , wherein when the shape of the hand is the predetermined shape, the hand is in an open state and fingers of the hand are spread apart from one another.

7. The HMD device according to claim 5 , wherein when the shape of the hand is the second shape, the hand is in an open state and fingers of the hand are not spread apart from one another.

8. The HMD device according to claim 5 , wherein the transparency of the partial image when the shape of the hand is the predetermined shape is greater than the transparency of the partial image when the shape of the hand is the second shape.

9. The HMD device according to claim 1 , wherein the processor is further configured to:

determine a first reference point corresponding to a fingertip of a finger of the hand based on the hand image,

determine a second reference point corresponding to a finger base of the finger of the hand based on the hand image, and

identify the inclination of the hand based on the first reference point and the second reference point.

10. The HMD device according to claim 9 , wherein the inclination of the hand is based on a first distance from the distance image sensor to the first reference point, and a second distance from the distance image sensor to the second reference point.

11. The HMD device according to claim 10 , wherein the processor is configured to:

determine that the hand is inclined forward when the first distance is greater than the second distance, and

determine that the hand is inclined backward when the first distance is less than or equal to the second distance.

12. The HMD device according to claim 1 , wherein

the image data is generated in accordance with the viewpoint motion vector.

13. The HMD device according to claim 1 , wherein

the viewpoint motion vector is for moving the virtual viewpoint position in the forward direction when the hand is inclined forward, and

the viewpoint motion vector is for moving the virtual viewpoint position in a backward direction when the hand is inclined backward.

14. The HMD device according to claim 1 , wherein the processor is configured to

determine a length of the viewpoint motion vector in accordance with a degree of the inclination.

15. The HMD device according to claim 1 , wherein the processor is configured to determine, as the hand, a region where a distance from the distance image sensor is less than or equal to a threshold in the distance information.

16. The HMD device according to claim 1 , wherein the processor is configured to obtain the inclination of the hand by calculating an absolute value of a differential value between a distance for a fingertip point and a distance for a finger base point included in a region of the hand.

17. The HMD device according to claim 1 , wherein the image of the hand included in the superimpose image is based on the determined transparency.

18. A control method executed by a head mounted display (HMD) device that includes a display, a distance image sensor that includes a camera, and a processor, the control method comprising:

capturing, by the camera, an image in a capturing range,

acquiring, by the distance image sensor, distance information indicating a distance from the distance image sensor to a hand of a user wearing the HMD device;

outputting, by the distance image sensor, distance image information that combines the captured image and the acquired distance information;

extracting, by the processor, an image of the hand of the user from the captured image based on the distance image information;

specifying a shape of the hand from the image of the hand;

calculating an inclination of the hand based on the distance information corresponding to the image of the hand;

determining whether the specified shape is a predetermined shape;

determining a viewpoint motion vector for moving a viewpoint position of the user in a virtual space when it is determined that the specified shape is a predetermined shape;

generating image data after the viewpoint position is moved;

determining a transparency of the image of the hand based on the calculated inclination;

generating an superimpose image by superimposing the image of the hand on the generated image data; and

controlling the display to display the generated superimpose image.

19. A non-transitory computer-readable recording medium storing a program that causes a processor included in a head mounted display (HMD) device to execute a process, the HMD device including a display, a distance image sensor that includes a camera, the process comprising:

extracting an image of a hand of a user from an image captured by the camera based on distance information acquired by the distance image sensor;

specifying a shape of the hand from the image of the hand;

calculating an inclination of the hand based on the distance information corresponding to the image of the hand;

determining whether the specified shape is a predetermined shape;

determining a viewpoint motion vector for moving a viewpoint position of the user in a virtual space when it is determined that the specified shape is a predetermined shape;

generating image data after the viewpoint position is moved;

determining a transparency of the image of the hand based on the calculated inclination;

generating an superimpose image by superimposing the image of the hand on the generated image data; and

controlling the display to display the generated superimpose image.