Methods and apparatus for encoding, communicating and/or using images
Methods and apparatus for capturing, communicating and using image data to support virtual reality experiences are described. Images, e.g., frames, are captured at a high resolution but lower frame rate than is used for playback. Interpolation is applied to captured frames to generate interpolated frames. Captured frames, along with interpolated frame information, are communicated to the playback device. The combination of captured and interpolated frames correspond to a second frame playback rate which is higher than the image capture rate. Cameras operate at a high image resolution but slower frame rate than images could be captured with the same cameras at a lower resolution. Interpolation is performed prior to delivery to the user device with segments to be interpolated being selected based on motion and/or lens FOV information. A relatively small amount of interpolated frame data is communicated compared to captured frame data for efficient bandwidth use.
1. A method comprising:
obtaining a stereoscopic frame set having a first frame rate, wherein the stereoscopic frame set comprises a series of image pairs;
for each of a series of left eye images and a series of right eye images of the series of image pairs, generating one or more intermediate frames to obtain one or more left eye intermediate frames and one or more right eye intermediate frames, comprising:
obtaining motion vector data indicating motion of a portion of the stereoscopic frame set,
obtaining depth data for one or more portions of the series of left eye images of the stereoscopic frame set, and
generating one or more intermediate frames using the stereoscopic frame set, the motion vector data, and the depth data; and
generating a left eye video sequence comprising the series of left eye images and the one or more left eye intermediate frames, and a right eye video sequence comprising the series of right eye images and the one or more right eye intermediate frames, wherein the left eye video sequence and the right eye video sequence are associated with a second frame rate greater than the first frame rate.
2. The method of claim 1 , wherein generating the one or more intermediate frames comprises:
generating a portion of a first intermediate frame from a first obtained frame based on the motion vector data; and
filling an area of the first intermediate frame for which interpolated segment data is not provided using content from one or more preceding frames of the stereoscopic frame set.
3. The method of claim 1 , wherein the motion vector data comprises one or more 3D motion vectors.
4. The method of claim 1 , wherein the left eye video sequence has a second frame rate which is higher than the first frame rate.
5. The method of claim 4 , further comprising:
outputting the left eye video sequence to a display device,
wherein the second frame rate corresponds to a display refresh rate supported by the display device.
6. The method of claim 5 , wherein the display device is comprised in a head mounted device.
7. The method of claim 5 , wherein the display device supports a 3D viewing experience.
8. A non-transitory computer readable medium comprising computer readable code executable by one or more processors to:
obtain a stereoscopic frame set having a first frame rate, wherein the stereoscopic frame set comprises a series of image pairs;
obtain motion vector data indicating motion of at least a portion of the stereoscopic frame set,
obtain depth data for one or more portions of a series of left eye images of the stereoscopic frame set, and
generate one or more left eye intermediate frames using the stereoscopic frame set, the motion vector data, and the depth data; and
generate a left eye video sequence comprising the series of left eye images and the one or more left eye intermediate frames, and a right eye video sequence comprising a series of right eye images of the stereoscopic frame set and one or more right eye intermediate frames, wherein the left eye video sequence and the right eye video sequence are associated with a second frame rate greater than the first frame rate.
9. The non-transitory computer readable medium of claim 8 , wherein the computer readable code to generate the one or more intermediate frames comprises computer readable code to:
generate a portion of a first intermediate frame from a first obtained frame based on the motion vector data; and
fill an area of the first intermediate frame for which interpolated segment data is not provided using content from one or more preceding frames of the stereoscopic frame set.
10. The non-transitory computer readable medium of claim 8 , wherein the motion vector data comprises one or more 3D motion vectors.
11. The non-transitory computer readable medium of claim 8 , wherein the left eye video sequence has a second frame rate which is higher than the first frame rate.
12. The non-transitory computer readable medium of claim 11 , further comprising computer readable code to:
output the left eye video sequence to a display device,
wherein the second frame rate corresponds to a display refresh rate supported by the display device.
13. The non-transitory computer readable medium of claim 12 , wherein the display device is comprised in a head mounted device.
14. The non-transitory computer readable medium of claim 12 , wherein the display device supports a 3D viewing experience.
15. A system comprising:
one or more processors; and
one or more computer readable media comprising computer readable code executable by the one or more processors to:
obtain a stereoscopic frame set having a first frame rate, wherein the stereoscopic frame set comprises a series of image pairs;
obtain motion vector data indicating motion of a portion of the stereoscopic frame set,
obtain depth data for one or more portions of a series of left eye images of the stereoscopic frame set, and
generate one or more left eye intermediate frames using the stereoscopic frame set, the motion vector data, and the depth data; and
generate a left eye video sequence comprising the series of left eye images and the one or more left eye intermediate frames, and a right eye video sequence comprising the series of right eye images and the one or more right eye intermediate frames, wherein the left eye video sequence and the right eye video sequence are associated with a second frame rate greater than the first frame rate.
16. The system of claim 15 , wherein the computer readable code to generate the one or more intermediate frames comprises computer readable code to:
generate a portion of a first intermediate frame from a first obtained frame based on the motion vector data; and
fill an area of the first intermediate frame for which interpolated segment data is not provided using content from one or more preceding frames of the stereoscopic frame set.
17. The system of claim 15 , wherein the left eye video sequence has a second frame rate which is higher than the first frame rate.
18. The system of claim 17 , further comprising computer readable code to:
output the left eye video sequence to a display device,
wherein the second frame rate corresponds to a display refresh rate supported by the display device.
19. The system of claim 18 , wherein the display device is comprised in a head mounted device.
20. The system of claim 18 , wherein the display device supports a 3D viewing experience.