System for blending extended reality images streamed from a plurality of application instances to an extended reality device
An extended reality (XR) streaming method of streaming XR images between a plurality of XR application instances and an XR device is described. The XR streaming method comprises streaming, with a first XR application instance first XR images to the XR device, wherein the first XR images streamed by the first XR application instance are associated with a first XR application. Streaming, by means of a second XR application instance, second XR images to the XR device, wherein the second XR images streamed by the second XR application instance are associated with a second XR application. Blending, by means of the XR device, the first XR images received from the first XR application instance with the second XR images received from the second XR application instance to obtain blended XR images for display on the XR device.
1 . An extended reality (XR) streaming method for streaming XR images between a plurality of XR application instances and an XR device, wherein the XR streaming method comprises:
streaming, from a first XR application instance of the plurality of XR application instances, first XR images to the XR device, wherein the first XR application instance comprises hardware or a combination of hardware and software that is configured to execute a first XR application, wherein the first XR application is an augmented reality, mixed reality, or virtual reality application, wherein the first XR images are associated with the first XR application;
streaming, from a second XR application instance of the plurality of XR application instances, second XR images to the XR device, wherein the second XR application instance comprises hardware or a combination of hardware and software that is configured to execute a second XR application, wherein the second XR application is an augmented reality, mixed reality, or virtual reality application, wherein the second XR images are associated with the second XR application, wherein the first XR application and the second XR application are different from each other;
streaming, first depth data from the first XR application instance and second depth data from the second XR application instance to the XR device, wherein the first depth data comprise information of distances of pixels of the first XR images and the second depth data comprise information of distances of pixels of the second XR images from the XR device in a virtual landscape; and
blending, by the XR device, the first XR images with the second XR images to obtain blended XR images to be displayed on the XR device, wherein blending comprises prioritizing each pixel of the blended XR images based on comparing the first depth data and the second depth data.
2 . The XR streaming method of claim 1 , wherein the first XR images received from the first XR application instance are blended with the second XR images received from the second XR application instance according to a positional blending technique that blends the first and second XR images based on an orientation of the XR device and/or based on a location of the XR device.
3 . The XR streaming method of claim 1 , wherein the plurality of XR application instances have respective priorities and wherein blending the first and second XR images is based on the priorities of the plurality of XR application instances.
4 . The XR streaming method of claim 3 , further comprising cutting out, with the XR device, those pixels of the first and second XR images having a predefined cutout color, wherein blending comprises, for each of the pixels of the first and second XR images, taking the pixel with a valid value of the XR application instance having the highest priority, wherein cutting out a pixel comprises determining the pixel has an invalid value.
5 . The XR streaming method of claim 1 , wherein alpha data is streamed to the XR device from the first XR application instance and the second XR application instance, respectively, wherein the alpha data comprises transparency information associated with pixels of the first and second XR images, and wherein the first XR images received from the first XR application instance are blended with the second XR images received from the second XR application instance based on the alpha data.
6 . The XR streaming method of claim 1 , wherein the first XR application instance employs a first rendering engine, wherein the second XR application instance employs a second rendering engine, and wherein the first rendering engine is different from the second rendering engine.
7 . The XR streaming method of claim 1 , wherein momentary position data is determined, wherein the momentary position data is associated with a momentary position of the XR device, wherein the momentary position data is forwarded to the first XR application instance and to the second XR application instance, and wherein the first XR application instance and the second XR application instance generate the XR images based on the momentary position data.
8 . The XR streaming method of claim 7 , wherein the momentary position data is forwarded to the first XR application instance and to the second XR application instance in a synchronized manner.
9 . The XR streaming method of claim 7 , wherein the momentary position data is associated with a position of at least one camera of the XR device.
10 . The XR streaming method of claim 1 , wherein the first XR images are reprojected before blending and/or wherein the second XR images are reprojected before blending.
11 . The XR streaming method of claim 1 , wherein XR image data associated with the first and second XR images comprise information on a view matrix and/or a projection matrix.
12 . An extended reality (XR) streaming system, comprising:
an XR device comprising a projection surface, a camera, communication circuitry, a processor, and one or more memories having stored therein instructions
executable by the processor to cause the XR device to,
determine momentary position data of the XR device;
transmit, via the communication circuitry, the momentary position data to one or more external computer devices implementing a plurality of XR application instances,
wherein the plurality of XR application instances comprise corresponding hardware or a combination of corresponding hardware and software that is configured to execute XR applications wherein the XR applications are augmented reality, mixed reality, or virtual reality applications, and
wherein the XR application instances of the plurality of XR application instances are different from each other;
receive, via the communication circuitry, streams of XR images from the plurality of XR application instances that have rendered the XR images based, at least in part, on the transmitted momentary position data;
blend the streamed XR images based, at least in part, on prioritizing each pixel of the blended XR image based on comparing first and second streamed depth data and at least one of a positional blending technique and priorities of the plurality of XR application instances,
wherein the first depth data are streamed to the XR device from a first XR application instance of the plurality of XR application instances, wherein the first depth data comprise information of distances of pixels of first XR images of the streamed XR images and the second depth data comprise information of distances of pixels of second XR images of the streamed XR images from the XR device in a virtual landscape; and
display the blended XR images on the projection surface.
13 . The system of claim 12 , wherein the one or more memories of the XR device further have stored therein instructions executable by the processor to cause the XR device to cutout those pixels of the streamed XR images that have a predefined cutout color, wherein the instructions to blend the streamed XR images based on priorities of the plurality of XR application instances comprise instructions executable by the processor to cause the XR device to take a pixel of a plurality of pixels of the streamed XR images corresponding to a highest priority if not cutout.
14 . The system of claim 12 , wherein the instructions executable by the processor to cause the XR device to transmit the momentary position data to the plurality of XR application instances comprises instructions executable by the processor to cause the XR device to transmit the momentary position data to the plurality of XR application instances in a synchronized manner.
15 . The system of claim 14 , wherein the XR device further comprises at least one camera, wherein the momentary position data is associated with a position of the at least one camera of the XR device.
16 . A non-transitory computer readable memories comprising instructions to:
determine momentary position data of an extended reality (XR) device;
transmit the momentary position data to instances of a plurality of XR applications,
wherein the plurality of XR application instances comprise corresponding hardware or a combination of corresponding hardware and software that is configured to execute XR applications wherein the XR applications are augmented reality, mixed reality, or virtual reality applications, and
wherein the XR application instances of the plurality of XR application instances are different from each other;
blend XR images streamed from the plurality of XR application instances based on streamed first and second depth data and at least one of a positional blending technique, priorities of the plurality of XR application instances, and alpha data from the plurality of XR application instances,
wherein the first depth data are streamed from a first XR application instance of the plurality of XR application instances and the second depth data are streamed from a second XR application instance of the plurality of XR application instances, wherein the first depth data comprise information of distances of pixels of first XR images of the streamed XR images and the second depth data comprise information of the distance of pixels of second XR images of the streamed XR images from the XR device in a virtual landscape; and
wherein the instructions to blend the XR images comprise further instructions to blend the XR images based on prioritization of each pixel of the blended XR images based on a comparison of the first and second depth data; and
display the blended XR images on a projection surface of the XR device.
17 . The non-transitory computer readable memories of claim 16 , wherein at least two different rendering engines are used by the plurality of XR application instances and the plurality of XR application instances render the XR images based, at least in part, on momentary position data of the XR device.
18 . The non-transitory computer readable memories of claim 16 , further comprising instructions to cutout those pixels of the streamed XR images that have a predefined cutout color, wherein the instructions to blend the streamed XR images based on priorities of the plurality of XR application instances comprise instructions to, for each pixel of corresponding XR images of different streams, take the pixel corresponding to a highest priority if not cutout.
19 . The non-transitory, computer readable memories of claim 16 , wherein the instructions to blend the XR images based on alpha data comprise instructions to blend the XR images based on alpha data streamed to the XR device from a first XR application instance and a second XR application instance of the plurality of XR application instances, wherein the alpha data comprise transparency information associated with pixels of the XR images, and wherein the first XR images received from the first XR application instance are blended with the second XR images received from the second XR application instance based on the alpha data.
20 . The non-transitory, computer readable memories of claim 16 , wherein the instructions to transmit the momentary position data to instances of a plurality of XR applications comprise instructions to transmit the momentary position data to the plurality of XR application instances in a synchronized manner.