HIGH DEFINITION BUBBLES FOR RENDERING FREE VIEWPOINT VIDEO
A “Dynamic High Definition Bubble Framework” allows local clients to display and navigate FVV of complex multi-resolution and multi-viewpoint scenes while reducing computational overhead and bandwidth for rendering and/or transmitting the FVV. Generally, the FVV is presented to the user as a broad area from some distance away. Then, as the user zooms in or changes viewpoints, one or more areas of the overall area are provided in higher definition or fidelity. Therefore, rather than capturing and providing high definition everywhere (at high computational and bandwidth costs), the Dynamic High Definition Bubble Framework captures one or more “bubbles” or volumetric regions in higher definition in locations where it is believed that the user will be most interested. This information is then provided to the client to allow individual clients to navigate and zoom different regions of the FVV during playback without losing fidelity or resolution in the zoomed areas.
1 . A computer-implemented process for generating navigable free viewpoint video (FVV), comprising using a computer to perform process actions for:
generating a geometric proxy from 3D image data of an overall volumetric space;
generating one or more geometric proxies for each of one or more sub-regions of the overall volumetric space;
registering one or more of the geometric proxies of the sub-regions with the geometric proxy of the overall volumetric space; and
rendering a multi-resolution user-navigable FVV from the registered geometric proxies and the geometric proxy of the overall volumetric space, wherein portions of the FVV corresponding to the sub-regions are rendered with a higher resolution than other regions of the FVV.
2 . The computer-implemented process of claim 1 wherein each sub-region is captured at a resolution greater than a resolution used to capture the overall volumetric space.
3 . The computer-implemented process of claim 1 wherein one or more of the sub-regions are captured using one or more moving camera arrays.
4 . The computer-implemented process of claim 1 wherein one or more of the sub-regions are captured using one or more fixed camera arrays.
5 . The computer-implemented process of claim 1 wherein rendering the multi-resolution user-navigable FVV further comprises process actions for:
determining a current view frustum corresponding to a current client viewpoint for viewing the FVV; and
transmitting appropriate geometric proxies within the current view frustum to the client for local rendering of video frames of the FVV.
6 . The computer-implemented process of claim 1 wherein one or more of the sub-regions move relative to the overall volumetric space during capture of the 3D image data for those sub-regions.
7 . The computer-implemented process of claim 1 wherein one or more of the sub-regions overlap within the overall volumetric space.
8 . A method for generating a navigable 3D representation of a volumetric space, comprising:
capturing 3D image data of an overall volumetric space and using this 3D image data to construct an environment model comprising a geometric proxy of the overall volumetric space;
capturing 3D image data for one or more sub-regions of the overall volumetric space and generating one or more geometric proxies of each sub-region;
registering one or more of the geometric proxies of each sub-region to the environment model;
determining a view frustum relative to the environment model; and
rendering frames of a multi-resolution user-navigable FVV from portions of the registered geometric proxies and environment model corresponding to the view frustum, wherein portions of the FVV corresponding to the sub-regions are rendered with a higher resolution than other regions of the FVV.
9 . The method of claim 8 wherein the view frustum is determined from a current viewpoint of a client viewing the FVV, and wherein the rendering is performed by the client from portions of the registered geometric proxies and environment model corresponding to the view frustum transmitted to the client.
10 . The method of claim 8 wherein zooming into portions of the FVV rendered with a higher resolution provides greater detail than when zooming into other regions of the FVV.
11 . The method of claim 8 wherein each sub-region is captured at a resolution greater than a resolution used to capture the overall volumetric space.
12 . The method of claim 8 wherein the sub-regions are captured using any combination of one or more moving camera arrays and one or more fixed camera arrays.
13 . The method of claim 8 wherein one or more of the sub-regions move relative to the overall volumetric space during capture of the 3D image data for those sub-regions.
14 . A computer-readable medium having computer executable instructions stored therein for generating a user navigable free viewpoint video (FVV), said instructions causing a computing device to execute a method comprising:
capturing 3D image data for an overall viewing area;
capturing 3D image data for one or more high definition bubbles within the overall viewing area;
generating a geometric proxy from the 3D image data of the overall viewing area;
generating one or more geometric proxies from the 3D image data of one or more of the high definition bubbles;
aligning one or more of the geometric proxies of the high definition bubbles with the geometric proxy of the overall viewing area; and
transmitting portions of any of the aligned geometric proxies corresponding to a current client viewpoint to a client for local client-based rendering of a multi-resolution user-navigable FVV, wherein portions of the FVV corresponding to the high definition bubbles are rendered with a higher resolution than other regions of the FVV.
15 . The computer-readable medium of claim 14 wherein each high definition bubble is captured at a resolution greater than a resolution used to capture the overall viewing area.
16 . The computer-readable medium of claim 14 wherein one or more of the high definition bubbles are captured using one or more moving camera arrays.
17 . The computer-readable medium of claim 14 wherein one or more of high definition bubbles are captured using one or more fixed camera arrays.
18 . The computer-readable medium of claim 14 wherein rendering the multi-resolution user-navigable FVV further comprises:
determining a current view frustum corresponding to a current client viewpoint for viewing the FVV; and
using portions of the aligned geometric proxies within the current view frustum for local rendering of video frames of the FVV.
19 . The computer-readable medium of claim 14 wherein one or more of the high definition bubbles move relative to the overall viewing area during capture of the 3D image data for those high definition bubbles.
20 . The computer-readable medium of claim 14 wherein one or more of the sub-regions overlap within the overall volumetric space.