Three-dimensional user interface device and three-dimensional operation processing method
A three-dimensional user interface device includes a coordinate setting unit setting a three-dimensional coordinate space (3DCS) on the basis of a line-of-sight image, a virtual data generation unit generating three-dimensional area data representing a transparent virtual three-dimensional operation area (V3DOA) arranged in an arbitrary position in the 3DCS in a state in which at least a boundary of the area is visible, a display processing unit displaying a V3DOA represented by the generated three-dimensional area data by using a visible space in the 3DCS corresponding to a space shown on the line-of-sight image as a display reference, and an operation detection unit detecting an operation performed by the operator with the specific region in the V3DOA on the basis of the three-dimensional position acquired with respect to the specific region of the operator in the 3DCS and a position of the V3DOA in the 3DCS.
1. A three-dimensional user interface device, comprising:
one or more processors configured to
acquire a line-of-sight image from an operator, the line-of-sight image being captured by an image capturing device mounted on a head of the operator,
set three-dimensional coordinate space on the basis of the acquired line-of-sight image, and
generate three-dimensional area data representing a transparent virtual three-dimensional operation area arranged in an arbitrary position in the three-dimensional coordinate space in a state in which at least a boundary of the virtual three-dimensional operation area is visible, and generate display data of a function menu arranged in the three-dimensional coordinate space, the function menu representing a plurality of shapes taken by the virtual three-dimensional operation area; and
a display processor that causes a display mounted on the head of the operator to display the virtual three-dimensional operation area represented by the three-dimensional area data which is generated by the one or more processors by using a visible space in the three-dimensional coordinate space corresponding to a space shown on the line-of-sight image as a display reference, and causes the display to display the function menu,
wherein the one or more processors are configured to
acquire a three-dimensional position of a specific region of the operator in the three-dimensional coordinate space,
detect an operation performed by the operator with the specific region in the virtual three-dimensional operation area on the basis of the three-dimensional position which is acquired by the one or more processors and a position of the virtual three-dimensional operation area in the three-dimensional coordinate space, and detect an operation performed by the operator with the specific region to the function menu to cause the three-dimensional area data representing the virtual three-dimensional operation area which is arranged in a position in the three-dimensional space to be generated, the position determined according to a detected position of the operation and which has a shape selected by the operation, and
set a start point and an end point in the position in the three-dimensional coordinate space which is determined according to the detected position of the operation to the function menu, and set a shape of an area, where two spherical bodies intersect, to the virtual three-dimensional operation area which is indicated by new three-dimensional area data, the two spherical bodies having the start point and the end point as each center point and each having a spherical surface including a point to which a distance from the center point is half of a straight line connecting the start point and the end point.
2. The three-dimensional user interface device according to claim 1 , wherein the one or more processors are configured to apply at least one of resizing processing, rotation processing, modification processing, and movement processing corresponding to a specific operation which is detected by the one or more processors to the three-dimensional area data, when the one or more processors detect the specific operation to the virtual three-dimensional operation area itself.
3. The three-dimensional user interface device according to claim 2 , wherein
the one or more processors generate a plurality of the three-dimensional area data respectively representing a position in the three-dimensional coordinate space and an operation type with respect to each of a plurality of the virtual three-dimensional operation areas, and
specify a virtual three-dimensional operation area which is operated among the plurality of virtual three-dimensional operation areas, and acquires operation information corresponding to an operation type which is set in the specified virtual three-dimensional operation area.
4. The three-dimensional user interface device according to claim 1 , wherein
the one or more processors generate a plurality of the three-dimensional area data respectively representing a position in the three-dimensional coordinate space and an operation type with respect to each of a plurality of the virtual three-dimensional operation areas, and
specify a virtual three-dimensional operation area which is operated among the plurality of virtual three-dimensional operation areas, and acquires operation information corresponding to an operation type which is set in the specified virtual three-dimensional operation area.
5. The three-dimensional user interface device according to claim 1 , wherein the one or more processors synthesize the virtual three-dimensional operation area represented by the three-dimensional area data with the line-of-sight image showing the space which is the display reference on the basis of the three-dimensional coordinate space,
wherein the display processor causes the display to display an image obtained by the one or more processors.
6. A three-dimensional operation processing method executed by at least one computer, the method comprising:
acquiring a line-of-sight image from an operator, the line-of-sight image being captured by an image capturing device mounted on a head of the operator;
setting a three-dimensional coordinate space on the basis of the acquired line-of-sight image;
generating three-dimensional area data representing a transparent virtual three-dimensional operation area arranged in an arbitrary position in the three-dimensional coordinate space in a state in which at least a boundary of the virtual three-dimensional operation area is visible;
causing a display mounted on the head of the operator to display the virtual three-dimensional operation area represented by the generated three-dimensional area data by using a visible space in the three-dimensional coordinate space corresponding to a space shown on the line-of-sight image as a display reference;
acquiring a three-dimensional position of a specific region of the operator in the three-dimensional coordinate space;
detecting an operation performed by the operator with the specific region in the virtual three-dimensional operation area on the basis of the acquired three-dimensional position and a position of the virtual three-dimensional operation area in the three-dimensional coordinate space;
generating display data of a function menu arranged in the three-dimensional coordinate space, the function menu representing a plurality of shapes taken by the virtual three-dimensional operation area;
causing the display to display the function menu; and
detecting an operation performed by the operator with the specific region to the function menu,
wherein said generating the three-dimensional area data includes generating the three-dimensional area data representing the virtual three-dimensional operation area which is arranged in a position in the three-dimensional space, the position determined according to a detected position of the operation to the function menu, and which has a shape selected by the operation, and
said generating the three-dimensional area data includes setting a start point and an end point in the position in the three-dimensional coordinate space which is determined according to the detected position of the operation to the function menu and generating the three-dimensional area data representing the virtual three-dimensional operation area having a shape of an area, where two spherical bodies intersect, the two spherical bodies having the start point and the end point as each center point and each having a spherical surface including a point to which a distance from the center point is half of a straight line connecting the start point and the end point.
7. The three-dimensional operation processing method according to claim 6 , further comprising:
detecting a specific operation of the operator to the virtual three-dimensional operation area itself; and
applying at least one of resizing processing, rotation processing, modification processing, and movement processing corresponding to the detected specific operation to the three-dimensional area data.
8. The three-dimensional operation processing method according to claim 6 ,
wherein said generating the three-dimensional area data includes generating a plurality of the three-dimensional area data respectively representing a position in the three-dimensional coordinate space and an operation type with respect to each of a plurality of the virtual three-dimensional operation areas, and
said detecting the operation includes specifying a virtual three-dimensional operation area which is operated among the plurality of virtual three-dimensional operation areas and acquiring operation information corresponding to an operation type which is set in the specified virtual three-dimensional operation area.
9. The three-dimensional operation processing method according to claim 6 , further comprising:
synthesizing the virtual three-dimensional operation area represented by the three-dimensional area data with the line-of-sight image showing the space which is the display reference on the basis of the three-dimensional coordinate space,
wherein said causing displaying the virtual three-dimensional operation area includes causing the display to display an image obtained by the synthesizing.
10. A non-transitory computer-readable storage medium storing a program for causing at least one computer to execute a three-dimensional operation processing method, the method comprising:
acquiring a line-of-sight image from an operator, the line-of-sight image being captured by an image capturing device mounted on a head of the operator;
setting a three-dimensional coordinate space on the basis of the acquired line-of-sight image;
generating three-dimensional area data representing a transparent virtual three-dimensional operation area arranged in an arbitrary position in the three-dimensional coordinate space in a state in which at least a boundary of the virtual three-dimensional operation area is visible;
causing a display mounted on the head of the operator to display the virtual three-dimensional operation area represented by the generated three-dimensional area data by using a visible space in the three-dimensional coordinate space corresponding to a space shown on the line-of-sight image as a display reference;
acquiring a three-dimensional position of a specific region of the operator in the three-dimensional coordinate space;
detecting an operation performed by the operator with the specific region in the virtual three-dimensional operation area on the basis of the acquired three-dimensional position and a position of the virtual three-dimensional operation area in the three-dimensional coordinate space;
generating display data of a function menu arranged in the three-dimensional coordinate space, the function menu representing a plurality of shapes taken by the virtual three-dimensional operation area;
causing the display to display the function menu; and
detecting an operation performed by the operator with the specific region to the function menu,
wherein said generating the three-dimensional area data includes generating the three-dimensional area data representing the virtual three-dimensional operation area which is arranged in a position in the three-dimensional space, the position determined according to a detected position of the operation to the function menu, and which has a shape selected by the operation, and
said generating the three-dimensional area data includes setting a start point and an end point in the position in the three-dimensional coordinate space which is determined according to the detected position of the operation to the function menu and generating the three-dimensional area data representing the virtual three-dimensional operation area having a shape of an area, where two spherical bodies intersect, the two spherical bodies having the start point and the end point as each center point and each having a spherical surface including a point to which a distance from the center point is half of a straight line connecting the start point and the end point.
11. The non-transitory computer-readable storage medium according to claim 10 , further comprising:
detecting a specific operation of the operator to the virtual three-dimensional operation area itself; and
applying at least one of resizing processing, rotation processing, modification processing, and movement processing corresponding to the detected specific operation to the three-dimensional area data.
12. The non-transitory computer-readable storage medium according to claim 10 , wherein
said generating the three-dimensional area data includes generating a plurality of the three-dimensional area data respectively representing a position in the three-dimensional coordinate space and an operation type with respect to each of a plurality of the virtual three-dimensional operation areas, and
said detecting the operation includes specifying a virtual three-dimensional operation area which is operated among the plurality of virtual three-dimensional operation areas and acquiring operation information corresponding to an operation type which is set in the specified virtual three-dimensional operation area.
13. The non-transitory computer-readable storage medium according to claim 10 , further comprising:
synthesizing the virtual three-dimensional operation area represented by the three-dimensional area data with the line-of-sight image showing the space which is the display reference on the basis of the three-dimensional coordinate space,
wherein said causing to display the virtual three-dimensional operation area includes causing the display to display an image obtained by the synthesizing.