Methods and system of multiview video rendering, preparing a multiview cache, and real-time multiview video conversion
Methods and system of multiview video rendering, preparing a multiview video cache, and real-time multiview video conversion employ video information having multiview frames corresponding to multiview video timestamps to generate views of a multiview frame using cache data entries of a multiview video cache corresponding to a target time stamp. The cache data entries may include a first cache data entry corresponding to a first multiview frame preceding the target timestamp and a second cache data entry corresponding to a second multiview frame following the target timestamp. The target timestamp for a particular multiview frame to be rendered may be based on a user input. The system may include a processor circuit and a memory that stores a plurality of instructions that, when executed by the processor circuit, cause the processor circuit to perform one or both of multiview video rendering and preparing a multiview video cache.
1 . A computer-implemented method of multiview video rendering, the method comprising:
receiving video information having a plurality of multiview frames using a computing device, each multiview frame of the multiview frame plurality corresponding to a respective multiview video timestamp and comprising a plurality of two-dimensional (2D) frames corresponding to different views of a scene within the multiview frame;
determining a target timestamp for a particular multiview frame to be rendered based on a user input;
retrieving from a multiview video cache a pair of cache data entries corresponding to the target timestamp, a first cache data entry of the pair of cache data entries comprising a respective first image frame group corresponding to a first multiview frame preceding the target timestamp and a second cache data entry of the pair of cache data entries comprising a respective second image frame group corresponding to a second multiview frame following the target timestamp, wherein each cache data entry comprises a tiled image and each tile in the tiled image corresponds to a different 2D frame of a respective multiview frame;
generating views of the particular multiview frame using the first and second image frame groups from the cache; and
rendering a multiview video for display by the computing device, the multiview video comprising the views of the particular multiview frame.
2 . The computer-implemented method of claim 1 , further comprising generating the cache with cache data entries corresponding to respective different timestamps.
3 . The computer-implemented method of claim 2 , wherein the tiles in a tiled image comprise a lower-resolution copy of a corresponding 2D frame from the video information.
4 . The computer-implemented method of claim 2 , wherein the cache data entries in the cache have non-time-adjacent timestamps.
5 . The computer-implemented method of claim 2 , wherein a number of the cache data entries in the cache is based on a timeline scrubbing resolution.
6 . The computer-implemented method of claim 2 , wherein generating the cache includes using 2D frames corresponding to fewer than all of the multiview frames of the video information.
7 . The computer-implemented method of claim 1 , comprising generating the cache with cache data entries corresponding to respective different timestamps, and at least one of the cache data entries comprises an image frame group representing fewer than all of the views of a corresponding multiview frame.
8 . The computer-implemented method of claim 1 , wherein generating views of the particular multiview frame includes using a virtual camera.
9 . The computer-implemented method of claim 1 , wherein generating the views of the particular multiview frame includes generating a respective view using 2D frames from each of the first and second cache data entries that correspond to the particular multiview frame.
10 . The computer-implemented method of claim 9 , wherein generating the respective view comprises blending information from the 2D frames from each of the first and second cache data entries.
11 . The computer-implemented method of claim 10 , wherein blending the information includes weighting an influence of each of the 2D frames proportional to a time difference of each of the cache data entries relative to the target timestamp.
12 . The computer-implemented method of claim 1 , wherein retrieving the pair of cache data entries from the cache includes identifying two cache data entries from the cache having timestamps that are nearest in time to the target timestamp.
13 . A computer-implemented method of preparing a multiview video cache, the method comprising:
receiving multiview video information comprising two-dimensional (2D) frames by a computing device, each 2D frame corresponding to a respective view of a multiview video and corresponding to a multiview video timestamp;
populating the cache with cache data entries corresponding to respective different multiview video timestamps, wherein each cache data entry comprises copies of the 2D frames for a particular respective timestamp; and
generating a first cache data entry of the cache corresponding to a first timestamp, wherein the first cache data entry comprises a tiled image, and tiles of the tiled image correspond to respective ones of the copies of the 2D frames for the first timestamp.
14 . The computer-implemented method of claim 13 , wherein the cache data entries correspond to non-time-adjacent multiview video timestamps.
15 . The computer-implemented method of claim 13 , wherein at least one of the cache data entries comprises fewer than all of the 2D frames from the video information for a particular timestamp.
16 . The computer-implemented method of claim 13 , wherein the tiles are lower-resolution copies of the 2D frames for the first timestamp.
17 . A system configured to perform real-time multiview video conversion, the system comprising:
a processor circuit; and
a memory that stores a plurality of instructions that, when executed by the processor circuit, cause the processor circuit to:
receive a target timestamp for a particular multiview frame to render;
retrieve, from a cache, a pair of cache data entries corresponding to the target timestamp, first and second cache data entries of the pair of cache data entries comprise respective first and second image frame groups, the first image frame group corresponds to a first multiview frame preceding the target timestamp and the second image frame group corresponds to a second multiview frame following the target timestamp; and
generate a multiview frame using information from the first and second image frame groups by blending the information from the first and second image frame groups, wherein the blending includes weighting an influence of each of the image frame groups proportional to a time difference of each of the cache data entries relative to the target timestamp,
wherein the multiview frame comprises a portion of a multiview video configured to be rendered on a multiview display.
18 . The system of claim 17 , wherein the memory stores instructions that cause the processor circuit to:
generate a user interface that is configured to be displayed on the multiview display; and
receive user input via the user interface, the user input indicating the target timestamp.
19 . The system of claim 17 , wherein the memory stores instructions that cause the processor circuit to generate the multiview frame by blending information from corresponding portions of the first and second image frame groups.