Mixed reality display device and mixed reality display method
The MR display device includes a controller and a display unit. The controller recognizes a reality object from a reality video imaged by a camera and links a predetermined virtual object to the reality object based on an operation of an experiencing person. The display unit displays a video of the linked virtual object. The controller acquires a recognition result of a reality object generated by the other MR display device and data of the linked virtual object, via a communication unit, synthesizes the acquired data and the own data, and displays a video obtained by the synthesis on the display unit. In addition, the controller allows an edit of the virtual object linked to the reality object on the video displayed on the display unit, based on an operation of the experiencing person.
1. A mixed reality display device that is worn by an experiencing person on a head and displays a video of a virtual object while being superimposed on a reality video visually recognized by the experiencing person, wherein
the mixed reality display device is configured to be connected to other mixed reality display devices worn by other experiencing persons and to cause the experiencing persons to share the reality video and the video of the virtual object with each other,
the mixed reality display device comprises:
a camera configured to image a reality video,
a distance measurement sensor configured to measure a distance to a reality object in the reality video,
a position and direction sensor configured to detect a position of the camera and an imaging direction,
a controller configured to recognize the reality object from the reality video imaged by the camera and link a predetermined virtual object to the recognized reality object based on an operation of the experiencing person,
a display configured to display a video of the linked virtual object,
a memory configured to store data of the virtual object linked to the reality object, and
a communicator configured to connect to the other mixed reality display devices and transmit and receive data to and from each other,
the controller is configured to:
acquire a recognition result of a reality object generated by the other mixed reality display device and data of the linked virtual object, via the communicator,
synthesize the acquired data and the data stored in the memory,
display a video obtained by the synthesis on the display, and
allow an edit of the virtual object linked to the reality object on the video displayed on the display, based on an operation of the experiencing person,
a reference point for the position of the camera and the imaging direction detected by the position and direction sensor is set to coincide with reference points of positions of cameras and imaging directions in the other mixed reality display devices,
when the data acquired from the other mixed reality display device and the data stored in the memory are synthesized, the controller viewpoint-transforms the video in accordance with the position of each of the cameras and the imaging direction and then performs synthesis, and
the controller updates the video to a new video based on a new position of the camera and a new imaging direction every time the experiencing person moves.
2. The mixed reality display device according to claim 1 ,
wherein, when the controller links the virtual object to the reality object, the controller assigns a priority flag indicating whether the other experiencing person is allowed to change the linking for the virtual object and stores a resultant of the assignment in the memory.
3. The mixed reality display device according to claim 1 ,
wherein the display is configured by a 3D projector that projects the video of the virtual object onto a transmissive screen, and
wherein the experiencing person is allowed to visually recognize the reality video seen through the transmissive screen and, at the same time, visually recognize the video of the virtual object on the transmissive screen.
4. The mixed reality display device according to claim 1 ,
wherein a 3D camera capable of imaging a reality video and measuring a distance to a reality object in the reality video is provided as the camera and the distance measurement sensor, and
wherein the display is a flat display that displays a result obtained by synthesizing a video of a virtual object with the reality video imaged by the 3D camera.
5. The mixed reality display device according to claim 1 ,
wherein the controller is capable of recognizing the reality object to be classified into an imaged object and a background object, based on the distance measured by the distance measurement sensor, and dividing the background object into a plurality of regions and linking the virtual object to the divided region.
6. The mixed reality display device according to claim 1 , wherein the controller is further configured to:
evaluate a distance between the reality object and the virtual object by the distance measurement sensor,
set a portion or an entirety of the virtual object not to be displayed when the reality object is in front of the virtual object, and
perform masking on the reality object by displaying the background object when the reality object indicates the other experiencing person.
7. A mixed reality display device that is worn by an experiencing person on a head and displays a video of a virtual object while being superimposed on a reality video visually recognized by the experiencing person, wherein
the mixed reality display device is configured to be connected to other mixed reality display devices worn by other experiencing persons and to cause the experiencing persons to share the reality video and the video of the virtual object with each other,
the mixed reality display device comprises:
a camera configured to image a reality video,
a distance measurement sensor configured to measure a distance to a reality object in the reality video,
a position and direction sensor configured to detect a position of the camera and an imaging direction,
a controller configured to recognize the reality object from the reality video imaged by the camera and link a predetermined virtual object to the recognized reality object based on an operation of the experiencing person,
a display configured to display a video of the linked virtual object,
a memory configured to store data of the virtual object linked to the reality object, and
a communicator configured to connect to the other mixed reality display devices and transmit and receive data to and from each other,
the controller is configured to:
acquire a recognition result of a reality object generated by the other mixed reality display device and data of the linked virtual object, via the communicator,
synthesize the acquired data and the data stored in the memory,
display a video obtained by the synthesis on the display, and
allow an edit of the virtual object linked to the reality object on the video displayed on the display, based on an operation of the experiencing person, and
the controller is capable of recognizing the reality object to be classified into an imaged object and a background object, based on the distance measured by the distance measurement sensor, and dividing the background object into a plurality of regions and linking the virtual object to the divided region.
8. The mixed reality display device according to claim 7 ,
wherein, when the controller links the virtual object to the reality object, the controller assigns a priority flag indicating whether the other experiencing person is allowed to change the linking for the virtual object and stores a resultant of the assignment in the memory.
9. The mixed reality display device according to claim 7 ,
wherein the display is configured by a 3D projector that projects the video of the virtual object onto a transmissive screen, and
wherein the experiencing person is allowed to visually recognize the reality video seen through the transmissive screen and, at the same time, visually recognize the video of the virtual object on the transmissive screen.
10. The mixed reality display device according to claim 7 ,
wherein a 3D camera capable of imaging a reality video and measuring a distance to a reality object in the reality video is provided as the camera and the distance measurement sensor, and
wherein the display is a flat display that displays a result obtained by synthesizing a video of a virtual object with the reality video imaged by the 3D camera.
11. A mixed reality display device that is worn by an experiencing person on a head and displays a video of a virtual object while being superimposed on a reality video visually recognized by the experiencing person, wherein
the mixed reality display device is configured to be connected to other mixed reality display devices worn by other experiencing persons and to cause the experiencing persons to share the reality video and the video of the virtual object with each other,
the mixed reality display device comprises:
a camera configured to image a reality video,
a distance measurement sensor configured to measure a distance to a reality object in the reality video,
a position and direction sensor configured to detect a position of the camera and an imaging direction,
a controller configured to recognize the reality object from the reality video imaged by the camera and link a predetermined virtual object to the recognized reality object based on an operation of the experiencing person,
a display configured to display a video of the linked virtual object,
a memory configured to store data of the virtual object linked to the reality object, and
a communicator configured to connect to the other mixed reality display devices and transmit and receive data to and from each other, and
the controller is configured to:
acquire a recognition result of a reality object generated by the other mixed reality display device and data of the linked virtual object, via the communicator,
synthesize the acquired data and the data stored in the memory,
display a video obtained by the synthesis on the display,
allow an edit of the virtual object linked to the reality object on the video displayed on the display, based on an operation of the experiencing person,
evaluate a distance between the reality object and the virtual object by the distance measurement sensor,
set a portion or an entirety of the virtual object not to be displayed when the reality object is in front of the virtual object, and
perform masking on the reality object by displaying the background object when the reality object indicates the other experiencing person.
12. The mixed reality display device according to claim 11 ,
wherein the controller is capable of recognizing the reality object to be classified into an imaged object and a background object, based on the distance measured by the distance measurement sensor, and dividing the background object into a plurality of regions and linking the virtual object to the divided region.
13. The mixed reality display device according to claim 11 ,
wherein, when the controller links the virtual object to the reality object, the controller assigns a priority flag indicating whether the other experiencing person is allowed to change the linking for the virtual object and stores a resultant of the assignment in the memory.
14. The mixed reality display device according to claim 11 ,
wherein the display is configured by a 3D projector that projects the video of the virtual object onto a transmissive screen, and
wherein the experiencing person is allowed to visually recognize the reality video seen through the transmissive screen and, at the same time, visually recognize the video of the virtual object on the transmissive screen.
15. The mixed reality display device according to claim 11 ,
wherein a 3D camera capable of imaging a reality video and measuring a distance to a reality object in the reality video is provided as the camera and the distance measurement sensor, and
wherein the display is a flat display that displays a result obtained by synthesizing a video of a virtual object with the reality video imaged by the 3D camera.
16. A mixed reality display device that is worn by an experiencing person on a head and displays a video of a virtual object while being superimposed on a reality video visually recognized by the experiencing person, wherein
the mixed reality display device is configured to be connected to other mixed reality display devices worn by other experiencing persons and to cause the experiencing persons to share the reality video and the video of the virtual object with each other,
the mixed reality display device comprises:
a camera configured to image a reality video,
a distance measurement sensor configured to measure a distance to a reality object in the reality video,
a position and direction sensor configured to detect a position of the camera and an imaging direction,
a controller configured to recognize the reality object from the reality video imaged by the camera and link a predetermined virtual object to the recognized reality object based on an operation of the experiencing person,
a display configured to display a video of the linked virtual object,
a memory configured to store data of the virtual object linked to the reality object, and
a communicator configured to connect to the other mixed reality display devices and transmit and receive data to and from each other, and
the controller is configured to:
acquire a recognition result of a reality object generated by the other mixed reality display device and data of the linked virtual object, via the communicator,
synthesize the acquired data and the data stored in the memory,
display a video obtained by the synthesis on the display, and
allow an edit of the virtual object linked to the reality object on the video displayed on the display, based on an operation of the experiencing person,
the virtual object linked to the reality object is configured by a plurality of components, and
the controller assigns a priority flag indicating an edit authority of the virtual object to each of the components and stores a resultant of the assignment.
17. The mixed reality display device according to claim 16 ,
wherein the controller is capable of recognizing the reality object to be classified into an imaged object and a background object, based on the distance measured by the distance measurement sensor, and dividing the background object into a plurality of regions and linking the virtual object to the divided region.
18. The mixed reality display device according to claim 16 , wherein the controller is further configured to:
evaluate a distance between the reality object and the virtual object by the distance measurement sensor,
set a portion or an entirety of the virtual object not to be displayed when the reality object is in front of the virtual object, and
perform masking on the reality object by displaying the background object when the reality object indicates the other experiencing person.
19. The mixed reality display device according to claim 16 ,
wherein the display is configured by a 3D projector that projects the video of the virtual object onto a transmissive screen, and
wherein the experiencing person is allowed to visually recognize the reality video seen through the transmissive screen and, at the same time, visually recognize the video of the virtual object on the transmissive screen.
20. The mixed reality display device according to claim 16 ,
wherein a 3D camera capable of imaging a reality video and measuring a distance to a reality object in the reality video is provided as the camera and the distance measurement sensor, and
wherein the display is a flat display that displays a result obtained by synthesizing a video of a virtual object with the reality video imaged by the 3D camera.