Methods and devices for video rendering for video see-through (VST) augmented reality (AR)
A method includes capturing an image and associating the image with a camera pose for each of multiple cameras. The method also includes determining, for each camera, a first contribution of the image for a first virtual view for display on a first display and a second contribution of the image for a second virtual view for display on a second display. The method further includes determining, for each camera, a first confidence map for the first virtual view based on the camera pose and a position of the camera in relation to a first virtual camera and a second confidence map for the second virtual view based on the camera pose and the position of the camera in relation to a second virtual camera. In addition, the method includes generating the first virtual view by combining the first contribution using the first confidence map for each of the cameras and the second virtual view by combining the second contribution using the second confidence map for each of the cameras.
1. A method for video rendering for video see-through (VST) augmented reality (AR) on an AR device, the method comprising:
for each of a plurality of cameras of the AR device:
capturing an image and associating the captured image with a camera pose;
determining (i) a first contribution of the captured image for a first virtual view for display on a first display of the AR device and (ii) a second contribution of the captured image for a second virtual view for display on a second display of the AR device; and
determining (i) a first confidence map for the first virtual view based on the camera pose and a position of the camera in relation to a first virtual camera and (ii) a second confidence map for the second virtual view based on the camera pose and the position of the camera in relation to a second virtual camera; and
generating (i) the first virtual view by combining the first contribution using the first confidence map for each of the plurality of cameras and (ii) the second virtual view by combining the second contribution using the second confidence map for each of the plurality of cameras.
2. The method of claim 1 , further comprising:
for each of the plurality of cameras, determining a first parallax map for the first virtual camera and a second parallax map for the second virtual camera by projecting a depth map associated with the captured image onto image planes of the first and second displays;
wherein the first parallax map is used in generating the first virtual view and the second parallax map is used in generating the second virtual view.
3. The method of claim 2 , wherein:
the first parallax map is used to identify a location of the first contribution in the first virtual view; and
the second parallax map is used to identify a location of the second contribution in the second virtual view.
4. The method of claim 2 , further comprising:
for each of the plurality of cameras, determining (i) a first parallax confidence map associated with the first parallax map for the first virtual view based on the camera pose and the position of the camera in relation to the first virtual camera and (ii) a second parallax confidence map associated with the second parallax map for the second virtual view based on the camera pose and the position of the camera in relation to the second virtual camera;
wherein the first parallax confidence map is used in generating the first virtual view and the second parallax confidence map is used in generating the second virtual view.
5. The method of claim 4 , wherein:
the first parallax confidence map is used to identify a priority of the first parallax map in generating the first virtual view; and
the second parallax confidence map is used to identify a priority of the second parallax map in generating the second virtual view.
6. The method of claim 5 , wherein a parallax map with a higher confidence than another parallax map is used in a front layer construction of a virtual view in relation to the other parallax map.
7. The method of claim 1 , wherein:
a confidence for the first or second confidence map is determined based on distance and orientation differences between the camera pose and the first or second virtual camera; and
the confidence increases as the distance and orientation differences between the camera and the first or second virtual camera decrease.
8. A video see-through (VST) augmented reality (AR) device comprising:
a plurality of cameras; and
at least one processing device operably coupled to the cameras, the at least one processing device configured to:
for each of the plurality of cameras:
capture an image using the camera and associate the captured image with a camera pose;
determine (i) a first contribution of the captured image for a first virtual view for display on a first display of the AR device and (ii) a second contribution of the captured image for a second virtual view for display on a second display of the AR device; and
determine (i) a first confidence map for the first virtual view based on the camera pose and a position of the camera in relation to a first virtual camera and (ii) a second confidence map for the second virtual view based on the camera pose and the position of the camera in relation to a second virtual camera; and
generate (i) the first virtual view by combining the first contribution using the first confidence map for each of the plurality of cameras and (ii) the second virtual view by combining the second contribution using the second confidence map for each of the plurality of cameras.
9. The VST AR device of claim 8 , wherein:
the at least one processing device is further configured, for each of the plurality of cameras, to determine a first parallax map for the first virtual camera and a second parallax map for the second virtual camera by projecting a depth map associated with the captured image onto image planes of the first and second displays; and
the first parallax map is used in generating the first virtual view and the second parallax map is used in generating the second virtual view.
10. The VST AR device of claim 9 , wherein:
the first parallax map is used to identify a location of the first contribution in the first virtual view; and
the second parallax map is used to identify a location of the second contribution in the second virtual view.
11. The VST AR device of claim 9 , wherein:
the at least one processing device is further configured, for each of the plurality of cameras, to determine (i) a first parallax confidence map associated with the first parallax map for the first virtual view based on the camera pose and the position of the camera in relation to the first virtual camera and (ii) a second parallax confidence map associated with the second parallax map for the second virtual view based on the camera pose and the position of the camera in relation to the second virtual camera; and
the first parallax confidence map is used in generating the first virtual view and the second parallax confidence map is used in generating the second virtual view.
12. The VST AR device of claim 11 , wherein:
the first parallax confidence map is used to identify a priority of the first parallax map in generating the first virtual view; and
the second parallax confidence map is used to identify a priority of the second parallax map in generating the second virtual view.
13. The VST AR device of claim 12 , wherein a parallax map with a higher confidence than another parallax map is used in a front layer construction of a virtual view in relation to the other parallax map.
14. The VST AR device of claim 8 , wherein:
a confidence for the first or second confidence map is determined based on distance and orientation differences between the camera pose and the first or second virtual camera; and
the confidence increases as the distance and orientation differences between the camera and the first or second virtual camera decrease.
15. A non-transitory machine readable medium containing instructions that when executed cause at least one processor to:
for each of a plurality of cameras of a video see-through (VST) augmented reality (AR) device:
capture an image and associate the captured image with a camera pose;
determine (i) a first contribution of the captured image for a first virtual view for display on a first display of the AR device and (ii) a second contribution of the captured image for a second virtual view for display on a second display of the AR device; and
determine (i) a first confidence map for the first virtual view based on the camera pose and a position of the camera in relation to a first virtual camera and (ii) a second confidence map for the second virtual view based on the camera pose and the position of the camera in relation to a second virtual camera; and
generate (i) the first virtual view by combining the first contribution using the first confidence map for each of the plurality of cameras and (ii) the second virtual view by combining the second contribution using the second confidence map for each of the plurality of cameras.
16. The non-transitory machine readable medium of claim 15 , further containing instructions that when executed cause the at least one processor, for each of the plurality of cameras, to determine a first parallax map for the first virtual camera and a second parallax map for the second virtual camera by projecting a depth map associated with the captured image onto image planes of the first and second displays;
wherein the first parallax map is used in generating the first virtual view and the second parallax map is used in generating the second virtual view.
17. The non-transitory machine readable medium of claim 16 , wherein:
the first parallax map is used to identify a location of the first contribution in the first virtual view; and
the second parallax map is used to identify a location of the second contribution in the second virtual view.
18. The non-transitory machine readable medium of claim 16 , further containing instructions that when executed cause the at least one processor, for each of the plurality of cameras, to determine (i) a first parallax confidence map associated with the first parallax map for the first virtual view based on the camera pose and the position of the camera in relation to the first virtual camera and (ii) a second parallax confidence map associated with the second parallax map for the second virtual view based on the camera pose and the position of the camera in relation to the second virtual camera; and
wherein the first parallax confidence map is used in generating the first virtual view and the second parallax confidence map is used in generating the second virtual view.
19. The non-transitory machine readable medium of claim 18 , wherein:
the first parallax confidence map is used to identify a priority of the first parallax map in generating the first virtual view;
the second parallax confidence map is used to identify a priority of the second parallax map in generating the second virtual view; and
a parallax map with a higher confidence than another parallax map is used in a front layer construction of a virtual view in relation to the other parallax map.
20. The non-transitory machine readable medium of claim 15 , wherein:
a confidence for the first or second confidence map is determined based on distance and orientation differences between the camera pose and the first or second virtual camera; and
the confidence increases as the distance and orientation differences between the camera and the first or second virtual camera decrease.