Methods and apparatus for encoding, communicating and/or using images
Methods and apparatus for capturing, communicating and using image data to support virtual reality experiences are described. Images, e.g., frames, are captured at a high resolution but lower frame rate than is used for playback. Interpolation is applied to captured frames to generate interpolated frames. Captured frames, along with interpolated frame information, are communicated to the playback device. The combination of captured and interpolated frames correspond to a second frame playback rate which is higher than the image capture rate. Cameras operate at a high image resolution but slower frame rate than images could be captured with the same cameras at a lower resolution. Interpolation is performed prior to delivery to the user device with segments to be interpolated being selected based on motion and/or lens FOV information. A relatively small amount of interpolated frame data is communicated compared to captured frame data for efficient bandwidth use.
1 . A method comprising:
obtaining a first stereoscopic frame pair from a stereoscopic frame set having a first frame rate;
obtaining motion vector data indicating motion for a portion of the first stereoscopic frame pair;
obtaining depth data for the portion of the first stereoscopic frame pair;
generating one or more additional frame pairs using the first stereoscopic frame pair, the motion vector data, and the depth data; and
generating a video sequence comprising the stereoscopic frame set and the one or more additional frame pairs, wherein the video sequence is associated with a second frame rate greater than the first frame rate.
2 . The method of claim 1 , wherein the video sequence comprises the first stereoscopic frame pair followed by the one or more additional frame pairs.
3 . The method of claim 1 , wherein generating the one or more additional frame pairs comprises:
filling an area of the one or more additional frame pairs using content from one or more preceding frames of the stereoscopic frame set.
4 . The method of claim 1 , wherein the motion vector data comprises one or more 3D motion vectors.
5 . The method of claim 1 , wherein the second frame rate corresponds to a display refresh rate supported by a display device.
6 . The method of claim 5 , wherein the display device is comprised in a head mounted device.
7 . The method of claim 6 , wherein the head mounted device supports a 3D viewing experience.
8 . A non-transitory computer readable medium comprising computer readable code executable by one or more processors to:
obtain a first stereoscopic frame pair from a stereoscopic frame set having a first frame rate;
obtain motion vector data indicating motion for a portion of the first stereoscopic frame pair;
obtain depth data for the portion of the first stereoscopic frame pair;
generate one or more additional frame pairs using the first stereoscopic frame pair, the motion vector data, and the depth data; and
generate a video sequence comprising the stereoscopic frame set and the one or more additional frame pairs, wherein the video sequence is associated with a second frame rate greater than the first frame rate.
9 . The non-transitory computer readable medium of claim 8 , wherein the video sequence comprises the first stereoscopic frame pair followed by the one or more additional frame pairs.
10 . The non-transitory computer readable medium of claim 8 , wherein the computer readable code to generate the one or more additional frame pairs comprises computer readable code to:
fill an area of the one or more additional frame pairs using content from one or more preceding frames of the stereoscopic frame set.
11 . The non-transitory computer readable medium of claim 8 , wherein the motion vector data comprises one or more 3D motion vectors.
12 . The non-transitory computer readable medium of claim 8 , wherein the second frame rate corresponds to a display refresh rate supported by a display device.
13 . The non-transitory computer readable medium of claim 12 , wherein the display device is comprised in a head mounted device.
14 . The non-transitory computer readable medium of claim 13 , wherein the head mounted device supports a 3D viewing experience.
15 . A system comprising:
one or more processors; and
one or more computer readable media comprising computer readable code executable by the one or more processors to:
obtain a first stereoscopic frame pair from a stereoscopic frame set having a first frame rate;
obtain motion vector data indicating motion for a portion of the first stereoscopic frame pair;
obtain depth data for the portion of the first stereoscopic frame pair;
generate one or more additional frames pairs using the first stereoscopic frame pair, the motion vector data, and the depth data; and
generate a video sequence comprising the stereoscopic frame set and the one or more additional frame pairs, wherein the video sequence is associated with a second frame rate greater than the first frame rate.
16 . The system of claim 15 , wherein the video sequence comprises the first stereoscopic frame pair followed by the one or more additional frame pairs.
17 . The system of claim 15 , wherein the computer readable code to generate the one or more additional frame pairs comprises computer readable code to:
fill an area of the one or more additional frame pairs using content from one or more preceding frames of the stereoscopic frame set.
18 . The system of claim 15 , wherein the motion vector data comprises one or more 3D motion vectors.
19 . The system of claim 15 , wherein the second frame rate corresponds to a display refresh rate supported by a display device.
20 . The system of claim 19 , wherein the display device is comprised in a head mounted device.