Method and apparatus for real-time data communication in full-presence immersive platforms
Real-time two-way communication between a user immersed in a virtual reality (VR) environment and another party who is not in the VR environment. The sequence of video frames being presented to the VR user on, e.g., a head mounted display, i.e., the VR user's “field of view” in the VR environment, is shown to a guest user on their computing device. Simultaneously, video from a camera in a guest user computing device is shown to the VR user in a window in the VR environment. Audio is also exchanged in real time between the VR user and guest user by use of a microphone and loudspeaker in both the VR system and the guest user computing device. The VR user's system can also support both rotational and translational user movement thus properly changing the VR user's apparent distance from objects displayed in the VR environment.
1. A method of providing real-time, two-way communication between a virtual reality user, viewing on a virtual reality display device a displayed sequence of video frames comprising the virtual reality user's field of view of a virtual reality environment, and a guest user operating a computing device having a camera and microphone, comprising:
creating, by a processor, a connection between the virtual reality display device and the computing device of the guest user;
capturing, by the processor, the sequence of video frames displayed to the virtual reality user on the virtual reality display device;
generating, by a microphone, an audio signal from the virtual reality user;
obtaining, by the processor, permission from the virtual reality user before creating a connection between the virtual reality display device and the computing device of the guest user;
sending, by the processor, the captured sequence of video frames displayed to the virtual reality user and the generated audio signal from the virtual reality user to the computing device of the guest user;
displaying, by the computing device of the guest user, the captured sequence of video frames;
playing, by the computing device of the guest user, the generated audio signal from the virtual reality user;
capturing, by the camera of the computing device of the guest user, a video stream of the guest user;
generating, by the microphone of the computing device of the guest user, an audio signal from the guest user;
sending, by the computing device of the guest user, the captured video stream of the guest user and the generated audio signal from the guest user to the processor;
displaying, by the processor, the captured video stream of the guest user in a window in the virtual reality environment; and
playing, by the processor, the generated audio signal from the guest user.
2. The method of claim 1 wherein the virtual reality display device is a head-mounted display.
3. The method of claim 1 wherein the computing device of the guest user is a laptop computer, desktop computer, smartphone or computer tablet.
4. The method of claim 1 , further comprising authenticating the computing device of the guest user before obtaining permission from the virtual reality user.
5. A method of providing real-time, two-way communication between a virtual reality user, viewing on a virtual reality display device a displayed sequence of video frames comprising the virtual reality user's field of view of a virtual reality environment, and a guest user operating a computing device having a camera and microphone, comprising:
obtaining, by a processor, permission from the guest user before creating a connection between the virtual reality display device and the computing device of the guest user;
creating, by the processor, the connection between the virtual reality display device and the computing device of the guest user;
capturing, by the processor, the sequence of video frames displayed to the virtual reality user on the virtual reality display device;
generating, by a microphone, an audio signal from the virtual reality user;
sending, by the processor, the captured sequence of video frames displayed to the virtual reality user and the generated audio signal from the virtual reality user to the computing device of the guest user;
displaying, by the computing device of the guest user, the captured sequence of video frames;
playing, by the computing device of the guest user, the generated audio signal from the virtual reality user;
capturing, by the camera of the computing device of the guest user, a video stream of the guest user;
generating, by the microphone of the computing device of the guest user, an audio signal from the guest user;
sending, by the computing device of the guest user, the captured video stream of the guest user and the generated audio signal from the guest user to the processor;
displaying, by the processor, the captured video stream of the guest user in a window in the virtual reality environment; and
playing, by the processor, the generated audio signal from the guest user.
6. A method of providing real-time, two-way communication between a virtual reality user, viewing on a virtual reality display device a displayed sequence of video frames comprising the virtual reality user's field of view of a virtual reality environment, and a guest user operating a computing device having a camera and microphone, comprising:
creating, by a processor, a connection between the virtual reality display device and the computing device of the guest user;
capturing, by the processor, the sequence of video frames displayed to the virtual reality user on the virtual reality display device;
generating, by a microphone, an audio signal from the virtual reality user;
sending, by the processor, the captured sequence of video frames displayed to the virtual reality user and the generated audio signal from the virtual reality user to the computing device of the guest user;
displaying, by the computing device of the guest user, the captured sequence of video frames;
playing, by the computing device of the guest user, the generated audio signal from the virtual reality user;
capturing, by the camera of the computing device of the guest user, a video stream of the guest user;
generating, by the microphone of the computing device of the guest user, an audio signal from the guest user;
sending, by the computing device of the guest user, the captured video stream of the guest user and the generated audio signal from the guest user to the processor;
displaying, by the processor, the captured video stream of the guest user in a window in the virtual reality environment;
playing, by the processor, the generated audio signal from the guest user;
sending, by the computing device of the guest user, a second video stream from the guest user computing device that is not the captured video stream of the guest user and a second audio signal from the guest user computing device to the processor; and
displaying, by the processor, the second video stream from the guest user computing device in a window in the virtual reality environment and playing the second audio signal from the guest user computing device.
7. A system for providing real-time two-way communication between a virtual reality user, and a guest user operating a computing device containing a camera and microphone, comprising:
a virtual reality display device configured to display a sequence of video frames comprising the virtual reality user's field of view of a virtual reality environment;
a microphone configured to generate a first audio signal from the virtual reality user;
a processor configured to:
obtain permission from the virtual reality user before creating a connection between the virtual reality display device and the computing device of the guest user;
create the connection between the virtual reality display device and the computing device of the guest user;
capture the sequence of video frames displayed to the virtual reality user on the virtual reality display device;
send the captured sequence of frames displayed to the virtual reality user and the first audio signal to the computing device of the guest user;
receive a video stream and a second audio signal from the computing device of the guest user;
display the video stream from the computing device of the guest user in a window in the virtual reality environment; and
a transducer configured to play the second audio signal.
8. The system of claim 7 wherein the virtual reality display device is a head-mounted display.
9. The system of claim 7 wherein the computing device of the guest user is a laptop computer, desktop computer, smartphone or computer tablet.
10. The system of claim 7 , wherein the processor is further configured to authenticate the computing device of the guest user before obtaining permission from the virtual reality user.
11. A system for providing real-time two-way communication between a virtual reality user, and a guest user operating a computing device containing a camera and microphone, comprising:
a virtual reality display device configured to display a sequence of video frames comprising the virtual reality user's field of view of a virtual reality environment;
a microphone configured to generate a first audio signal from the virtual reality user;
a processor configured to:
obtain permission from the guest user before creating a connection between the virtual reality display device and the computing device of the guest user;
create the connection between the virtual reality display device and the computing device of the guest user;
capture the sequence of video frames displayed to the virtual reality user on the virtual reality display device;
send the captured sequence of frames displayed to the virtual reality user and the first audio signal to the computing device of the guest user;
receive a video stream and a second audio signal from the computing device of the guest user;
display the video stream from the computing device of the guest user in a window in the virtual reality environment; and
a transducer configured to play the second audio signal.
12. A system for providing real-time two-way communication between a virtual reality user, and a guest user operating a computing device containing a camera and microphone, comprising:
a virtual reality display device configured to display a sequence of video frames comprising the virtual reality user's field of view of a virtual reality environment;
a microphone configured to generate a first audio signal from the virtual reality user;
a processor is further configured to:
create the connection between the virtual reality display device and the computing device of the guest user;
capture the sequence of video frames displayed to the virtual reality user on the virtual reality display device;
send the captured sequence of frames displayed to the virtual reality user and the first audio signal to the computing device of the guest user;
receive a video stream and a second audio signal from the computing device of the guest user;
display the video stream from the computing device of the guest user in a window in the virtual reality environment;
receive a second video stream and a third audio signal from the guest user computing;
display the second video stream from the guest user computing device in a window in the virtual reality environment and play the third audio signal; and
a transducer configured to play the second audio signal.
13. A non-transitory computer readable storage medium having embodied thereon instructions for causing a computing device to execute a method for providing real-time, two-way communication between a virtual reality user, viewing on a virtual reality display device a displayed sequence of video frames comprising the virtual reality user's field of view of a virtual reality environment, and a guest user operating a computing device having a camera and microphone, the method comprising:
obtaining, by a processor, permission from the virtual reality user before creating a connection between the virtual reality display device and the computing device of the guest user;
creating, by the processor, the connection between the virtual reality display device and the computing device of the guest user;
capturing, by the processor, the sequence of video frames displayed to the virtual reality user on the virtual reality display device;
generating, by a microphone, an audio signal from the virtual reality user;
sending, by the processor, the captured sequence of video frames displayed to the virtual reality user and the generated audio signal from the virtual reality user to the computing device of the guest user;
displaying, by the computing device of the guest user, the captured sequence of video frames;
playing, by the computing device of the guest user, the generated audio signal from the virtual reality user;
capturing, by the camera of the computing device of the guest user, a video stream of the guest user;
generating, by the microphone of the computing device of the guest user, an audio signal from the guest user;
sending, by the computing device of the guest user, the captured video stream of the guest user and the generated audio signal from the guest user to the processor;
displaying, by the processor, the captured video stream of the guest user in a window in the virtual reality environment; and
playing, by the processor, the generated audio signal from the guest user.
14. A non-transitory computer readable storage medium having embodied thereon instructions for causing a computing device to execute a method for providing real-time, two-way communication between a virtual reality user, viewing on a virtual reality display device a displayed sequence of video frames comprising the virtual reality user's field of view of a virtual reality environment, and a guest user operating a computing device having a camera and microphone, the method comprising:
authenticating, by a processor, the computing device of the guest user before obtaining permission from the virtual reality user;
creating, by the processor, a connection between the virtual reality display device and the computing device of the guest user;
capturing, by the processor, the sequence of video frames displayed to the virtual reality user on the virtual reality display device;
generating, by a microphone, an audio signal from the virtual reality user;
sending, by the processor, the captured sequence of video frames displayed to the virtual reality user and the generated audio signal from the virtual reality user to the computing device of the guest user;
displaying, by the computing device of the guest user, the captured sequence of video frames;
playing, by the computing device of the guest user, the generated audio signal from the virtual reality user;
capturing, by the camera of the computing device of the guest user, a video stream of the guest user;
generating, by the microphone of the computing device of the guest user, an audio signal from the guest user;
sending, by the computing device of the guest user, the captured video stream of the guest user and the generated audio signal from the guest user to the processor;
displaying, by the processor, the captured video stream of the guest user in a window in the virtual reality environment; and
playing, by the processor, the generated audio signal from the guest user.
15. A non-transitory computer readable storage medium having embodied thereon instructions for causing a computing device to execute a method for providing real-time, two-way communication between a virtual reality user, viewing on a virtual reality display device a displayed sequence of video frames comprising the virtual reality user's field of view of a virtual reality environment, and a guest user operating a computing device having a camera and microphone, the method comprising:
obtaining, by a processor, permission from the guest user before creating a connection between the virtual reality display device and the computing device of the guest user;
creating, by the processor, the connection between the virtual reality display device and the computing device of the guest user;
capturing, by the processor, the sequence of video frames displayed to the virtual reality user on the virtual reality display device;
generating, by a microphone, an audio signal from the virtual reality user;
sending, by the processor, the captured sequence of video frames displayed to the virtual reality user and the generated audio signal from the virtual reality user to the computing device of the guest user;
displaying, by the computing device of the guest user, the captured sequence of video frames;
playing, by the computing device of the guest user, the generated audio signal from the virtual reality user;
capturing, by the camera of the computing device of the guest user, a video stream of the guest user;
generating, by the microphone of the computing device of the guest user, an audio signal from the guest user;
sending, by the computing device of the guest user, the captured video stream of the guest user and the generated audio signal from the guest user to the processor;
displaying, by the processor, the captured video stream of the guest user in a window in the virtual reality environment; and
playing, by the processor, the generated audio signal from the guest user.