Coding and decoding of interleaved image data
Sampled data is packaged in checkerboard format for encoding and decoding. The sampled data may be quincunx sampled multi-image video data (e.g., 3D video or a multi-program stream), and the data may also be divided into sub-images of each image which are then multiplexed, or interleaved, in frames of a video stream to be encoded and then decoded using a standardized video encoder. A system for viewing may utilize a standard video decoder and a formatting device that de-interleaves the decoded sub-images of each frame reformats the images for a display device. A 3D video may be encoded using a most advantageous interleaving format such that a preferred quality and compression ratio is reached. In one embodiment, the invention includes a display device that accepts data in multiple formats.
1. An apparatus comprising a non-transitory computer-readable medium for storing data accessible by a decoder, the computer-readable medium comprising:
a compressed bitstream stored in the computer-readable medium, the compressed bitstream representing:
pixel values of packed video frames, wherein each packed video frame comprises an interleaved arrangement of (i) samples of odd rows and odd columns of a first image, (ii) samples of even rows and even columns of the first image, (iii) samples of odd rows and even columns of a second image, and (iv) samples of even rows and odd columns of the second image, each forming a different portion of the packed video frame, respectively;
first side information indicating a sampling format of the interleaved arrangement wherein the first side information for a particular packed video frame indicates a quincunx sampling format; and
second side information indicating an arrangement of the pixel values of the first and second images in the interleaved arrangement, wherein the second side information for the particular packed video frame indicates a vertical arrangement,
wherein the particular packed video frame comprises (i) samples of the first image arranged as a top portion of the packed video frame, and (ii) samples of the second image arranged as a bottom portion of the packed video frame.
2. An apparatus comprising a non-transitory computer-readable medium for storing data accessible by a decoder, the computer-readable medium comprising:
a compressed bitstream stored in the computer-readable medium, the compressed bitstream representing:
pixel values of packed video frames, wherein each packed video frame comprises an interleaved arrangement of (i) samples of odd rows and odd columns of a first image, (ii) samples of even rows and even columns of the first image, (iii) samples of odd rows and even columns of a second image, and (iv) samples of even rows and odd columns of the second image, each forming a different portion of the packed video frame, respectively;
first side information indicating a sampling format of the interleaved arrangement, wherein the first side information for a particular packed video frame indicates a quincunx sampling format,
second side information indicating an arrangement of the pixel values of the first and second images in the interleaved arrangement, wherein the second side information for the particular packed video frame indicates a vertical arrangement, and
wherein the particular packed video frame comprises (i) samples of odd rows and odd columns of the first image arranged as a top-left portion of the packed video frame, (ii) samples of even rows and even columns of the first image arranged as a top-right portion of the packed video frame, (iii) samples of odd rows and even columns of the second image arranged as a bottom-left portion of the packed video frame, and (iv) samples of even rows and odd columns of the second image arranged as a bottom-right portion of the packed video frame.
3. An apparatus comprising a non-transitory computer-readable medium for storing data accessible by a decoder, the computer-readable medium comprising:
a compressed bitstream stored in the computer-readable medium, the compressed bitstream representing:
pixel values of packed video frames, wherein each packed video frame comprises an interleaved arrangement of (i) samples of odd rows and odd columns of a first image, (ii) samples of even rows and even columns of the first image, (iii) samples of odd rows and even columns of a second image, and (iv) samples of even rows and odd columns of the second image, each forming a different portion of the packed video frame, respectively;
first side information indicating a sampling format of the interleaved arrangement, wherein the first side information for a particular packed video frame indicates a quincunx sampling format,
second side information indicating an arrangement of the pixel values of the first and second images in the interleaved arrangement, wherein the second side information for the particular packed video frame indicates a vertical arrangement, and
wherein the particular packed video frame comprises (i) samples of odd rows and odd columns of the first image arranged as a top-left portion of the packed video frame, (ii) samples of even rows and even columns of the first image arranged as a bottom-right portion of the packed video frame, (iii) samples of odd rows and even columns of the second image arranged as a top-right portion of the packed video frame, and (iv) samples of even rows and odd columns of the second image arranged as a bottom-left portion of the packed video frame.
4. The apparatus of claim 1 , wherein the compressed bitstream comprises a stereoscopic (3D) video signal with a left view and a right view.
5. The apparatus of claim 1 , wherein the first side information or the second side information is changed adaptively in the compressed bitstream.
6. The apparatus of claim 1 , wherein the first side information or the second side information is changed adaptively at one or more of a sequence, picture, and slice level.
7. An apparatus to encode a sequence of multi-view frames into a compressed bitstream, the apparatus comprising:
an input port to receive a first view image and a second view image;
one or more processors configured to:
sample odd rows and odd columns of the first view image to generate a first sub-image of a packed video frame;
sample even rows and even columns of the first view image to generate a second sub-image of the packed video frame;
sample odd rows and even columns of the second view image to generate a third sub-image of the packed video frame;
sample even rows and odd columns of the second view image to generate a fourth sub-image of the packed video frame;
generate first side information for the packed video frame indicating a quincunx sampling format for the interleaved first and second view images;
generate second side information for the packed video frame indicating how the first, second, third, and fourth sub-images are tiled together to form the packed video frame, wherein the second side information indicates that (i) the first sub-image forms a top-left portion of the packed video frame, (ii) the second sub-image forms a bottom-left portion of the packed video frame, (iii) the third sub-image forms a top-right portion of the packed video frame, and (iv) the fourth sub-image forms a bottom-right portion of the packed video frame; and
an output port to output the packed video frame, the first side information, and the second side information.
8. An apparatus to encode a sequence of multi-view frames into a compressed bitstream, the apparatus comprising:
an input port to receive a first view image and a second view image;
one or more processors configured to:
sample odd rows and odd columns of the first view image to generate a first sub-image of a packed video frame;
sample even rows and even columns of the first view image to generate a second sub-image of the packed video frame;
sample odd rows and even columns of the second view image to generate a third sub-image of the packed video frame;
sample even rows and odd columns of the second view image to generate a fourth sub-image of the packed video frame;
generate first side information for the packed video frame indicating a quincunx sampling format for the interleaved first and second view images;
generate second side information for the packed video frame indicating how the first, second, third, and fourth sub-images are tiled together to form the packed video frame, wherein the second side information indicates that (i) the first sub-image forms a top-left portion of the packed video frame, (ii) the second sub-image forms a top-right portion of the packed video frame, (iii) the third sub-image forms a bottom-left portion of the packed video frame, and (iv) the fourth sub-image forms a bottom-right portion of the packed video frame.
9. An apparatus to encode a sequence of multi-view frames into a compressed bitstream, the apparatus comprising:
an input port to receive a first view image and a second view image;
one or more processors configured to:
sample odd rows and odd columns of the first view image to generate a first sub-image of a packed video frame;
sample even rows and even columns of the first view image to generate a second sub-image of the packed video frame;
sample odd rows and even columns of the second view image to generate a third sub-image of the packed video frame;
sample even rows and odd columns of the second view image to generate a fourth sub-image of the packed video frame;
generate first side information for the packed video frame indicating a quincunx sampling format for the interleaved first and second view images;
generate second side information for the packed video frame indicating how the first, second, third, and fourth sub-images are tiled together to form the packed video frame, wherein the second side information indicates a vertical arrangement, wherein (i) samples of the first view image form a top portion of the packed video frame, and (ii) samples of the second view image form a bottom portion of the packed video frame.
10. The apparatus of claim 7 , wherein the first side information or the second side information is changed adaptively based on characteristics of the first view image and the second view image.
11. The apparatus of claim 7 , wherein the first side information or the second side information is changed adaptively at one or more of a sequence, picture, and slice level.
12. An apparatus comprising a non-transitory computer-readable medium for storing data accessible by a decoder, the computer-readable medium comprising a compressed bitstream stored in the computer-readable medium, the compressed bitstream including:
pixel valued of packed video frames, wherein each frame comprises a first image and a second image in an interleaved arrangement;
first side information indicating a sampling format of the interleaved arrangement, wherein the first side information for a particular packed video frame indicates a quincunx sampling format; and
second side information indicating an arrangement of the pixel values of the first and second images in the interleaved arrangement, wherein the interleaved arrangement indicates tiling information for four sub-images for each packed video frame, wherein the four sub-images include (i) a first sub-image generated based on samples of odd rows and odd columns of the first image, (ii) a second sub-image generated based on samples of even rows and even columns of the first image, (iii) a third sub-image generated based on samples of odd rows and even columns of a second image, and (iv) a fourth sub-image generated based on samples of even rows and odd columns of the second image.
13. The apparatus of claim 12 , wherein the second side information indicates that (i) the first sub-image forms a top-left portion of the packed video frame, (ii) the second sub-image forms a bottom-left portion of the packed video frame, (iii) the third sub-image forms a top-right portion of the packed video frame, and (iv) the fourth sub-image forms a bottom-right portion of the packed video frame.
14. The apparatus of claim 12 , wherein the second side information indicates that (i) the first sub-image forms a top-left portion of the packed video frame, (ii) the second sub-image forms a top-right portion of the packed video frame, (iii) the third sub-image forms a bottom-left portion of the packed video frame, and (iv) the fourth sub-image forms a bottom-right portion of the packed video frame.
15. The apparatus of claim 12 , wherein the second side information indicates that (i) the first sub-image forms a top-left portion of the packed video frame, (ii) the second sub-image forms a bottom-right portion of the packed video frame, (iii) the third sub-image forms a top-right portion of the packed video frame, and (iv) the fourth sub-image forms a bottom-left portion of the packed video frame.
16. The apparatus of claim 12 , wherein the first side information or the second side information is changed adaptively based on characteristics of the first image and the second image.
17. The apparatus of claim 12 , wherein the first side information or the second side information is changed adaptively at one or more of a sequence, picture, and slice level.