Apparatus for creating shared virtual conversational space with AR and non-AR devices using edge processing
There is included a method and apparatus comprising computer code configured to cause a processor or processors to obtain video data from a non-AR device and from an AR device, the AR device being used by a first user in a first room, and the non-AR device being worn by a second user in a second room separate from the first room, determine, based on the video data, an orientation of the non-AR device relative to a position at which a second user is to be displayed in an AR scene in the first room, and stream, based on determining the orientation, to the non-AR device scene information.
1 . A method for augmented reality (AR) video streaming, the method comprising:
obtaining video data from a non-AR device and from an AR device respectively, the non-AR device being used by a first user in a first room and lacking AR rendering capability, and the AR device being worn by a second user in a second room separate from the first room;
obtaining an AR scene description, generated based on the video data, from the non-AR device that does not render an AR scene;
generating a virtual scene by a cloud device by parsing and rendering the AR scene description obtained from the non-AR device, the non-AR device provides the AR scene description to the cloud device, and the cloud device is a 5G edge server;
determining, based on the AR scene description obtained from the non-AR device, an orientation of the non-AR device relative to a position at which the second user is to be displayed in an AR scene in the first room; and
streaming, based on determining the orientation, the rendered virtual scene to the non-AR device.
2 . The method according to claim 1 ,
wherein the position at which the second user in the AR scene is to be displayed is determined based on a view selection of the first user via the non-AR device.
3 . The method according to claim 1 ,
wherein streaming the rendered virtual scene to the non-AR device comprises streaming at least one of a 360 video and a 2D video depending on a selection of the first user via the non-AR device.
4 . The method according to claim 1 ,
wherein the rendered virtual scene is generated in a cloud device separate from the non-AR device.
5 . The method according to claim 4 ,
wherein the cloud device implements AR rendering based on the video data and provides the rendered virtual scene to the non-AR device.
6 . The method according to claim 5 , wherein the rendered virtual scene comprises the second user virtually overlaid at a position in the first room.
7 . The method according to claim 6 , wherein the position at which the second user is virtually overlaid in the first room is a position in the first room at which at least one of the non-AR device and the cloud device has determined to be a dedicated location in the first room at which to overlay the second user during streaming of the rendered virtual scene.
8 . The method according to claim 5 ,
wherein the cloud device further provides updated scene information to the non-AR device based on the non-AR device switching views via the first user moving the non-AR device in the first room.
9 . The method according to claim 1 ,
wherein audio from the first room and from the second room is mixed and provided along with the rendered virtual scene to the non-AR device.
10 . The method according to claim 1 ,
wherein the second user of the AR device views the scene in an AR environment while the first user of the non-AR device views the scene in the non-AR environment according to the AR scene description.
11 . A apparatus for augmented reality (AR) video streaming, the apparatus comprising:
at least one memory configured to store computer program code;
at least one processor configured to access the computer program code and operate as instructed by the computer program code, the computer program code including:
obtaining code configured to cause the at least one processor to obtain video data from a non-AR device and from an AR device respectively, the non-AR device being used by a first user in a first room and lacking AR rendering capability, and the AR device being worn by a second user in a second room separate from the first room;
further obtaining code configured to cause the at least one process to obtain an AR scene description, generated based on the video data, from the non-AR device that does not render an AR scene;
generating a virtual scene by a cloud device by parsing and rendering the AR scene description obtained from the non-AR device, the non-AR device provides the AR scene description to the cloud device, and the cloud device is a 5G edge server;
determining code configured to cause the at least one processor to determine, based on the AR scene description obtained from the non-AR device, an orientation of the non-AR device relative to a position at which the second user is to be displayed in an AR scene in the first room; and
streaming code configured to cause the at least one processor to stream, the rendered virtual scene based on determining the orientation, to the non-AR device.
12 . The apparatus according to claim 11 ,
wherein the position at which the second user in the AR scene is to be displayed is determined based on a view selection of the first user via the non-AR device.
13 . The apparatus according to claim 11 ,
wherein streaming the rendered virtual scene to the non-AR device comprises streaming at least one of a 360 video and a 2D video depending on a selection of the first user via the non-AR device.
14 . The apparatus according to claim 11 ,
wherein the rendered virtual scene is generated in a cloud device separate from the non-AR device.
15 . The apparatus according to claim 14 ,
wherein the cloud device implements AR rendering based on the video data and provides the rendered virtual scene to the non-AR device.
16 . The apparatus according to claim 15 , wherein the rendered virtual scene comprises the second user virtually overlaid at a position in the first room.
17 . The apparatus according to claim 16 , wherein the position at which the second user is virtually overlaid in the first room is a position in the first room at which at least one of the non-AR device and the cloud device has determined to be a dedicated location in the first room at which to overlay the second user during streaming of the rendered virtual scene.
18 . The apparatus according to claim 15 ,
wherein the cloud device further provides updated scene information to the non-AR device based on the non-AR device switching views via the first user moving the non-AR device in the first room.
19 . The apparatus according to claim 11 ,
wherein audio from the first room and from the second room is mixed and provided along with the rendered virtual scene to the non-AR device.
20 . A non-transitory computer readable medium storing a program causing a computer to execute a process, the process comprising:
obtaining video data from a non-AR device and from an AR device respectively, the non-AR device being used by a first user in a first room and lacking AR rendering capability, and the AR device being worn by a second user in a second room separate from the first room;
obtaining an AR scene description, generated based on the video data, from the non-AR device that does not render an AR scene;
generating a virtual scene by a cloud device by parsing and rendering the AR scene description obtained from the non-AR device, the non-AR device provides the AR scene description to the cloud device, and the cloud device is a 5G edge server;
determining, based on the AR scene description obtained from the non-AR device, an orientation of the non-AR device relative to a position at which the second user is to be displayed in an AR scene in the first room; and
streaming, based on determining the orientation, the rendered virtual scene to the non-AR device.