System and method for augmented reality multi-view telepresence
Disclosed herein are systems and methods for augmented reality multi-view telepresence. An embodiment takes the form of a method that includes obtaining a session geometry of a multi-location telepresence session that includes a first-location participant at a first location and a second-location participant at a second location, each location having respective pluralities of cameras and display segments. The method includes selecting, according to the session geometry, both a first-to-second-viewpoint second-location camera from the plurality of second-location cameras as well as a first-to-second-viewpoint first-location display segment from the plurality of first-location display segments. The method includes receiving a first-to-second-viewpoint video stream captured by the selected first-to-second-viewpoint second-location camera, and further includes generating a line-of-sight augmented-reality experience for the multi-location telepresence session at least in part by rendering the received first-to-second-viewpoint video stream via the selected first-to-second-viewpoint first-location display segment.
1. A method comprising:
selecting, based on first location information and second location information, a camera from an array of cameras at a first location,
wherein the camera is configured to provide a line-of-sight video view of a first participant to display to a second participant,
wherein the first participant is at the first location and the second participant is at a second location, and
wherein the first location information comprises physical location information associated with the first participant and the second location information comprises physical location information associated with the second participant; and
transmitting, to the second location, a first video stream,
wherein the first video stream comprises the line-of-sight video view of the first participant captured by the camera selected.
2. The method of claim 1 , further comprising:
receiving a second video stream,
wherein the second video stream comprises a line-of-sight video view of the second participant to display to the first participant; and
displaying the second video stream in an AR display device of the first participant.
3. The method of claim 2 , wherein the second video stream is displayed in the AR display device of the first participant at a position based on the first location information and the second location information.
4. The method of claim 2 , wherein the second video stream is displayed in the AR display device of the first participant in alignment with a location of the selected camera from the array of cameras.
5. The method of claim 1 , further comprising:
determining, from the first location information and the second location information, corresponding locations in a virtual session geometry,
wherein selecting the camera from the array of cameras is further based on the determined locations in the virtual session geometry.
6. The method of claim 1 , further comprising cropping video content from the selected camera to produce the first video stream.
7. The method of claim 1 , wherein the array of cameras is a 1-D camera array.
8. The method of claim 1 , wherein the array of cameras is selected from the group consisting of a 2-D camera array and a 3-D camera array.
9. The method of claim 1 , wherein the array of cameras is a device comprising a plurality of cameras.
10. The method of claim 1 , further comprising transmitting an identifier of the first participant and the first location information associated with the first participant.
11. The method of claim 1 , further comprising:
transmitting a viewpoint of the first participant,
wherein the viewpoint specifies a desired viewing position of the first participant and the desired viewing position differs from a physical viewing position of the first participant at the first location; and
receiving a second video stream,
wherein the second video stream comprises a video view of the second location based on the transmitted viewpoint.
12. An apparatus comprising:
a processor; and
a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to:
select, based on first location information and second location information, a camera from an array of cameras at a first location,
wherein the camera is configured to provide a line-of-sight video view of a first participant to display to a second participant,
wherein the first participant is at the first location and the second participant is at a second location, and
wherein the first location information comprises physical location information associated with the first participant and the second location information comprises physical location information associated with the second participant; and
transmit, to the second location, a first video stream,
wherein the first video stream comprises the line-of-sight video view of the first participant captured by the camera selected.
13. The apparatus of claim 12 , wherein the instructions, when executed by the processor, are further operative to:
receive a second video stream,
wherein the second video stream comprises a line-of-sight video view of the second participant suitable for display to the first participant; and
display the second video stream in an AR display device of the first participant.
14. The apparatus of claim 13 , wherein the instructions, when executed by the processor, are further operative to display the second video stream in the AR display device of the first participant at a position based on the first location information and the second location information.
15. The apparatus of claim 13 , wherein the instructions, when executed by the processor, are further operative to display the second video stream in an AR display device of the first participant in alignment with a location of the selected camera from the array of cameras.
16. The apparatus of claim 12 ,
wherein the instructions, when executed by the processor, are further operative to determine, from the first location information and the second location information, corresponding locations in a virtual session geometry, and
wherein selecting the camera from the array of cameras is further based on the determined locations in the virtual session geometry.
17. The apparatus of claim 12 , wherein the instructions, when executed by the processor, are further operative to crop video content from the selected camera to produce the first video stream.
18. The apparatus of claim 12 , wherein the array of cameras is a device comprising a plurality of cameras.
19. The apparatus of claim 12 , wherein the instructions, when executed by the processor, are further operative to:
transmit a viewpoint of the first participant,
wherein the viewpoint specifies a desired viewing position of the first participant and the desired viewing position differs from a physical viewing position of the first participant at the first physical location; and
receive a second video stream,
wherein the second video stream comprises a video view of the second location based on the transmitted viewpoint.
20. A system comprising:
a camera array device comprising a plurality of cameras;
a processor; and
a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the system to:
determine physical location information associated with a first user and a second user;
select, based on the location information, a camera from the array of cameras,
wherein the selected camera is configured to provide a line-of-sight video view of the first user to display to the second user; and
transmit a first video stream,
wherein the first video stream comprises the line-of-sight video view of the first user captured by the selected camera.
21. The system of claim 20 , wherein the camera array device further comprises an AR marker in a position calibrated relative to at least one of the plurality of cameras of the camera array device.
22. The system of claim 21 , wherein the instructions operative, when executed by the processor, to cause the system to determine location information comprise instructions operative, when executed by the processor, to cause the system to:
capture an image of the camera array device using a camera associated with the first user;
identify the AR marker in the captured image; and
determine the location of the first user relative to the cameras of the camera array device based on the identification of the AR marker in the captured image.
23. The system of claim 20 , wherein the camera associated with the first user is part of a Head Mounted Display (HMD) device worn by the first user.