Mixed reality processing system and mixed reality processing method
A mixed reality processing method includes the following steps. A virtual scene is displayed by a head-mounted display. The virtual world coordinates of a physical object are obtained using a first SLAM map. A tracker uses a second SLAM map to obtain the real world coordinates of the physical object. The virtual world coordinates and the real world coordinates are merged. The virtual scene and the physical object are combined by a processor to generate a mixed reality image.
1. A mixed reality processing system, comprising:
a camera, configured to capture a two-dimensional image including a physical object;
a head-mounted display (HMD), configured to display a virtual scene, and obtain virtual world coordinates of the physical object in the virtual world through a first simultaneous localization and mapping map (SLAM map); wherein the virtual world coordinates are used to generate the virtual scene based on real world coordinates;
a tracker, configured to differentiate the physical object from the two-dimensional image, and obtain the real world coordinates of the physical object in the real world merely through a second simultaneous localization and mapping map, wherein the tracker constructs the second simultaneous localization and mapping map by obtaining environment information through a sensor, and uses the second simultaneous localization and mapping map to locate itself and thereby locate the physical object; and
a processor, configured to merge the virtual world coordinates and the real world coordinates, and merge the virtual scene and the physical object to generate a mixed reality image.
2. The mixed reality processing system of claim 1 , wherein the physical object is a human body, the tracker inputs the two-dimensional image into a segmentation model, and the segmentation model outputs a human body block, and the human body block is a part of the two-dimensional image.
3. The mixed reality processing system of claim 2 , wherein the tracker inputs the two-dimensional image into a skeleton model, and the skeleton model outputs a plurality of human skeleton points, the tracker generates a three-dimensional pose according to the human skeleton points, and the three-dimensional pose is configured to adjust a capture range of the human body block.
4. The mixed reality processing system of claim 2 , wherein the processor is located in the tracker or in an external computer, and the processor is configured to generate merging coordinates after respectively coinciding the origins and coordinate axes of the virtual world coordinates and the real world coordinates;
wherein the processor superimposes the human body block from the tracker on the virtual scene from the head-mounted display according to the merging coordinates to generate the mixed reality image.
5. The mixed reality processing system of claim 1 , wherein the tracker and the camera are located outside the head-mounted display, and outside-in tracking technology is applied to track the position of the head-mounted display.
6. A mixed reality processing method, comprising:
capturing a two-dimensional image including a physical object using a camera;
using a head-mounted display (HMD) to display a virtual scene and obtain virtual world coordinates of the physical object in the virtual world through a first simultaneous localization and mapping map (SLAM map), wherein the virtual world coordinates used to generate the virtual scene are based on real world coordinates;
using a tracker to differentiate the physical object from the two-dimensional image and obtain the real world coordinates of the physical object in the real world merely through a second simultaneous localization and mapping map, wherein the tracker constructs the second simultaneous localization and mapping map by obtaining environment information through a sensor, and uses the second simultaneous localization and mapping map to locate itself and hereby locate the physical object; and
using a processor to merge the virtual world coordinates and the real world coordinates and merge the virtual scene and the physical object to generate a mixed reality image.
7. The mixed reality processing method of claim 6 , wherein the physical object is a human body, the mixed reality processing method further comprising:
the tracker inputs the two-dimensional image into a segmentation model, and the segmentation model outputs a human body block;
wherein the human body block is a part of the two-dimensional image.
8. The mixed reality processing method of claim 7 , further comprising:
using the tracker to input the two-dimensional image into a skeleton model;
wherein the skeleton model outputs a plurality of human skeleton points, the tracker generates a three-dimensional pose according to the human skeleton points, the three-dimensional pose is configured to adjust the capture range of the human body block.
9. The mixed reality processing method of claim 7 , wherein the processor is located in the tracker or in an external computer, and the mixed reality processing method further comprises:
the processor generates merging coordinates after respectively coinciding the origins and coordinate axes of the virtual world coordinates and the real world coordinates;
using the processor to generate the mixed reality image by superimposing the human body block from the tracker on the virtual scene from the head-mounted display according to the merging coordinates.
10. The mixed reality processing method of claim 6 , wherein the tracker and the camera are located outside the head-mounted display, and outside-in tracking technology is applied to track the position of the head-mounted display.