Scene encoding for multi-user mixed-reality telepresence
A scene encoding subsystem for multi-user mixed-reality telepresence has at least one encoder that processes signals from a plurality of cameras. A scene manager determines relative priorities of the cameras and allocates CPU resources and GPU resources in the at least one encoder based on the relative priorities of the cameras, where (i) greater CPU resources are allocated to higher priority cameras than to lower priority cameras and (ii) greater GPU resources are allocated to lower priority cameras than to higher priority cameras. The scene manager instructs a CPU to process a skipped frame using a bounding-box expansion algorithm based on object motion determined using GPU object detection and segmentation processing of previous non-skipped frames. The scene manager multiplicatively decreases and additively increases CPU and GPU resources for a set of multiple cameras based on performance of any one camera in the set.
1 . A scene manager for a scene encoding subsystem for multi-user mixed-reality telepresence, the scene encoding subsystem comprising a plurality of cameras and at least one encoder adapted to process signals from the plurality of cameras, the at least one encoder comprising at least one central processing unit (CPU) and at least one graphics processing unit (GPU), the scene manager comprising:
at least one processor; and
at least one memory storing instructions that, upon being executed by the at least one processor, cause the scene manager at least to:
determine relative priorities of the cameras; and
allocate CPU resources and GPU resources in the at least one encoder based on the relative priorities of the cameras, wherein:
greater CPU resources are allocated to higher priority cameras than to lower priority cameras; and
greater GPU resources are allocated to lower priority cameras than to higher priority cameras.
2 . The scene manager of claim 1 , wherein the scene manager is adapted to modify the allocations of the CPU and GPU resources in the at least one encoder over time as the determined relative priorities of the cameras change over time.
3 . The scene manager of claim 1 , wherein the scene manager is adapted to determine the relative priorities of the cameras based on user position and field of view of each camera wherein cameras having fields of view that capture more of the user have higher priority than cameras having fields of view that capture less of the user.
4 . The scene manager of claim 3 , wherein the scene manager is adapted to instruct the at least one encoder to ignore signals from at least one camera whose field of view captures none of the user.
5 . The scene manager of claim 1 , wherein the scene manager is adapted to instruct a GPU to skip certain frames from at least one higher priority camera.
6 . The scene manager of claim 5 , wherein the scene manager is adapted to instruct a CPU to process a skipped frame using a bounding-box expansion algorithm based on object motion determined using GPU object detection and segmentation processing of previous non-skipped frames.
7 . The scene manager of claim 1 , wherein the scene manager is adapted to allocate higher point-cloud capture densities to higher-priority cameras.
8 . The scene manager of claim 1 , wherein the scene manager is adapted to perform an additive increase, multiplicative decrease (AIMD) technique to allocate the CPU and GPU resources based on the determined relative priorities of the cameras.
9 . The scene manager of claim 8 , wherein the scene manager is adapted to multiplicatively decrease and additively increase CPU and GPU resources for a set of multiple cameras based on performance of any one camera in the set.
10 . The scene manager of claim 1 , wherein the scene manager is adapted to:
receive feedback from the at least one encoder; and
control the processing of the at least one encoder based on the feedback.
11 . The scene manager of claim 10 , wherein the feedback comprises CPU and GPU processing latencies for different cameras.
12 . A method for a scene manager of a scene encoding subsystem for multi-user mixed-reality telepresence, the scene encoding subsystem comprising a plurality of cameras and at least one encoder adapted to process signals from the plurality of cameras, the at least one encoder comprising at least one CPU and at least one GPU, the method comprising the scene manager:
determining relative priorities of the cameras; and
allocating CPU resources and GPU resources in the at least one encoder based on the relative priorities of the cameras, wherein:
greater CPU resources are allocated to higher priority cameras than to lower priority cameras; and
greater GPU resources are allocated to lower priority cameras than to higher priority cameras.
13 . The method of claim 12 , wherein the scene manager modifies the allocations of the CPU and GPU resources in the at least one encoder over time as the determined relative priorities of the cameras change over time.
14 . The method of claim 12 , wherein the scene manager determines the relative priorities of the cameras based on user position and field of view of each camera wherein cameras having fields of view that capture more of the user have higher priority than cameras having fields of view that capture less of the user.
15 . The method of claim 12 , wherein the scene manager instructs a GPU to skip certain frames from at least one higher priority camera.
16 . The method of claim 15 , wherein the scene manager instructs a CPU to process a skipped frame using a bounding-box expansion algorithm based on object motion determined using GPU object detection and segmentation processing of previous non-skipped frames.
17 . The method of claim 12 , wherein the scene manager allocates higher point-cloud capture densities to higher-priority cameras.
18 . The method of claim 12 , wherein the scene manager performs an AIMD technique to allocate the CPU and GPU resources based on the determined relative priorities of the cameras.
19 . The method of claim 18 , wherein the scene manager multiplicatively decreases and additively increases CPU and GPU resources for a set of multiple cameras based on performance of any one camera in the set.
20 . The method of claim 12 , wherein the scene manager:
receives feedback from the at least one encoder; and
controls the processing of the at least one encoder based on the feedback.