Efficient encoding of depth data across devices
One embodiment provides a method comprising determining a spatial relationship between an augmented reality (AR) device and a camera-equipped device. The AR device is worn by a user. The camera-equipped device is positioned within proximity of the user. The method comprises determining, based on the spatial relationship, minimum and maximum boundaries of a user volume encompassing a head and one or more extremities of the user. The minimum and maximum boundaries represent a reduced depth range that is narrower than a full depth range of a camera frustum of the camera-equipped device. The method further comprises encoding, based on the minimum and maximum boundaries, a depth image frame captured via the camera-equipped device, and transmitting the encoded depth image frame over a remote network connection. A reconstructed 3D image based in part on the encoded depth image frame is rendered by a different AR device.
1. A method, comprising:
determining a spatial relationship between an augmented reality (AR) device and a camera-equipped device, wherein the AR device is worn by a user, and the camera-equipped device is positioned within proximity of the user;
determining, based on the spatial relationship, minimum and maximum boundaries of a user volume encompassing a head and one or more extremities of the user, wherein the minimum and maximum boundaries represent a reduced depth range that is narrower than a full depth range of a camera frustum of the camera-equipped device;
encoding, based on the minimum and maximum boundaries, a depth image frame captured via the camera-equipped device; and
transmitting the encoded depth image frame over a remote network connection, wherein a reconstructed three-dimensional (3D) image based in part on the encoded depth image frame is rendered by a different AR device.
2. The method of claim 1 , wherein determining the spatial relationship comprises:
determining a set of depth values representing distances of the head and the one or more extremities of the user relative to a z-axis of the camera-equipped device, wherein the minimum and maximum boundaries are based on the set of depth values.
3. The method of claim 1 , further comprising:
for each subsequent depth image frame captured via the camera-equipped device:
determining an updated set of depth values representing distances of the head and the one or more extremities of the user relative to the z-axis of the camera-equipped device;
dynamically updating the minimum and maximum boundaries based on the update set of depth values; and
encoding the subsequent depth image frame based on the updated minimum and maximum boundaries.
4. The method of claim 1 , wherein the minimum and maximum boundaries are included in header values of the encoded depth image frame.
5. The method of claim 1 , wherein the one or more extremities of the user include at least one of a left hand, a right hand, a left foot, or a right foot of the user.
6. The method of claim 1 , wherein each AR device comprises a pair of AR glasses.
7. The method of claim 1 , wherein the camera-equipped device includes a Red, Green and Blue (RGB) camera and a depth camera.
8. The method of claim 1 , wherein the AR device is paired with the camera-equipped device over a local network connection.
9. A system, comprising:
at least one processor; and
a non-transitory processor-readable memory device storing instructions that when executed by the at least one processor causes the at least one processor to perform operations including:
determining a spatial relationship between an augmented reality (AR) device and a camera-equipped device paired with the AR device over a local network connection, wherein the AR device is worn by a user, and the camera-equipped device is positioned within proximity of the user;
determining, based on the spatial relationship, minimum and maximum boundaries of a user volume encompassing a head and one or more extremities of the user, wherein the minimum and maximum boundaries represent a reduced depth range that is narrower than a full depth range of a camera frustum of the camera-equipped device;
encoding, based on the minimum and maximum boundaries, a depth image frame captured via the camera-equipped device; and
transmitting the encoded depth image frame over a remote network connection, wherein a reconstructed three-dimensional (3D) image based in part on the encoded depth image frame is rendered by a different AR device.
10. The system of claim 9 , wherein determining the spatial relationship comprises:
determining a set of depth values representing distances of the head and the one or more extremities of the user relative to a z-axis of the camera-equipped device, wherein the minimum and maximum boundaries are based on the set of depth values.
11. The system of claim 9 , wherein the operations further comprise:
for each subsequent depth image frame captured via the camera-equipped device:
determining an updated set of depth values representing distances of the head and the one or more extremities of the user relative to the z-axis of the camera-equipped device;
dynamically updating the minimum and maximum boundaries based on the update set of depth values; and
encoding the subsequent depth image frame based on the updated minimum and maximum boundaries.
12. The system of claim 9 , wherein the minimum and maximum boundaries are included in header values of the encoded depth image frame.
13. The system of claim 9 , wherein the one or more extremities of the user include at least one of a left hand, a right hand, a left foot, or a right foot of the user.
14. The system of claim 9 , wherein each AR device comprises a pair of AR glasses.
15. The system of claim 9 , wherein the camera-equipped device includes a Red, Green and Blue (RGB) camera and a depth camera.
16. A non-transitory computer readable storage medium including instructions to perform a method comprising:
determining a spatial relationship between an augmented reality (AR) device and a camera-equipped device paired with the AR device over a local network connection, wherein the AR device is worn by a user, and the camera-equipped device is positioned within proximity of the user;
determining, based on the spatial relationship, minimum and maximum boundaries of a user volume encompassing a head and one or more extremities of the user, wherein the minimum and maximum boundaries represent a reduced depth range that is narrower than a full depth range of a camera frustum of the camera-equipped device;
encoding, based on the minimum and maximum boundaries, a depth image frame captured via the camera-equipped device; and
transmitting the encoded depth image frame over a remote network connection, wherein a reconstructed three-dimensional (3D) image based in part on the encoded depth image frame is rendered by a different AR device.
17. The computer program product of claim 16 , wherein determining the spatial relationship comprises:
determining a set of depth values representing distances of the head and the one or more extremities of the user relative to a z-axis of the camera-equipped device, wherein the minimum and maximum boundaries are based on the set of depth values.
18. The computer program product of claim 16 , wherein the method further comprises:
for each subsequent depth image frame captured via the camera-equipped device:
determining an updated set of depth values representing distances of the head and the one or more extremities of the user relative to the z-axis of the camera-equipped device;
dynamically updating the minimum and maximum boundaries based on the update set of depth values; and
encoding the subsequent depth image frame based on the updated minimum and maximum boundaries.
19. The computer program product of claim 16 , wherein the minimum and maximum boundaries are included in header values of the encoded depth image frame.
20. The computer program product of claim 16 , wherein the one or more extremities of the user include at least one of a left hand, a right hand, a left foot, or a right foot of the user.