Sphere projected motion estimation/compensation and mode decision
Techniques are disclosed for coding video data predictively based on predictions made from spherical-domain projections of input pictures to be coded and reference pictures that are prediction candidates. Spherical projection of an input picture and the candidate reference pictures may be generated. Thereafter, a search may be conducted for a match between the spherical-domain representation of a pixel block to be coded and a spherical-domain representation of the reference picture. On a match, an offset may be determined between the spherical-domain representation of the pixel block to a matching portion of the of the reference picture in the spherical-domain representation. The spherical-domain offset may be transformed to a motion vector in a source-domain representation of the input picture, and the pixel block may be coded predictively with reference to a source-domain representation of the matching portion of the reference picture.
1. A video coding method, comprising:
for a plurality of input pixel blocks from an input picture in a source-domain representation:
transforming a first input pixel block in the source-domain representation to a spherical-domain representation;
transforming a candidate reference picture from the source-domain representation to the spherical representation;
searching for a match between the spherical-domain representation of the first input pixel block and a portion of the spherical-domain representation of the candidate reference picture;
on a match, determining a spherical-domain motion vector including a spherical-domain rotational offset for predicting the first input pixel block in the spherical-domain representation from a matching portion of a reference picture in the spherical-domain representation;
transforming the spherical-domain motion vector to a two-dimensional source-domain motion vector and a source-domain rotational offset for predicting the first input pixel block in the source-domain representation of the input picture; and
predictively coding the first input pixel block in the source-domain including determining a source-domain prediction from the reference picture in the source-domain based on the two-dimensional source-domain motion vector and the source domain rotational offset.
2. The method of claim 1 , wherein the searching comprise searching for a best mode to optimize overall cost to send source-domain representation of the pixel block in terms of minimum spherical domain error and source-domain bit rate.
3. The method of claim 1 , further comprising, outputting coded data of the pixel blocks with source-domain motion vectors as coded data of the input picture.
4. The method of claim 1 , further comprising, outputting, with coded data of the pixel blocks, spherical-domain offsets as coded data of the input picture.
5. The method of claim 4 , further comprising, on the match:
outputting data identifying the rotational offset, wherein
the outputted spherical-domain offsets are represented differentially with respect to the rotational offset.
6. The method of claim 1 , wherein, for at least one pixel block, the searching includes estimating a relative radius ratio between the spherical-domain representation of the pixel block and the spherical-domain representation of the matching portion of the reference picture, and outputting data identifying the relative radius ratio as coded data of the respective input pixel block.
7. The method of claim 1 , wherein, for at least one pixel block, the searching includes estimating a relative offset between an origin of the spherical-domain representation of the pixel block and an origin of the spherical-domain representation of the matching portion of the of the reference picture, and outputting data identifying the relative offset as coded data of the respective input pixel block.
8. The method of claim 1 , wherein the searching comprises
estimating, for a plurality of candidate blocks from the reference pictures, a number of bits required to code the input pixel block with reference to each respective candidate block, and
a matching candidate block is selected based on a relative differences between the number of bits estimated as required from among the candidate blocks.
9. The method of claim 1 , wherein the searching comprises
estimating, for a plurality of candidate blocks from the reference pictures, an amount of distortion that could be created by coding the input pixel block with reference to each respective candidate block, and
a matching candidate block is selected based on a relative differences between the amount of distortion estimated for each of the candidate blocks.
10. The method of claim 1 , wherein the input picture is an equirectangular picture in its source-domain representation.
11. The method of claim 1 , wherein the input picture is a cube map picture in its source-domain representation.
12. The method of claim 1 , wherein the input picture is one of a truncated pyramid-based, tetrahedral-based, octahedral-based, dodecahedral-based and icosahedral-based image in its source-domain representation.
13. A video coder, comprising:
a pixel block coder, having an input for a pixel block of input data in a coding-domain and a reference pixel block,
a reference picture store, storing data of a plurality of reference pictures in the coding-domain,
a pair of spherical transform units, a first spherical transform unit having an input for the input pixel block in the coding-domain and a second spherical transform unit having an input for reference picture data in the coding-domain,
a predictor having spherical-domain inputs coupled to outputs from the first and second spherical transform units,
wherein, the predictor selects a spherical-domain motion vector including a spherical-domain rotational offset by searching the spherical transformed reference pictures for a prediction reference of the spherical transformed input pixel block, the predictor converts the spherical-domain motion vector to a coding-domain motion vector and a coding-domain rotational offset and outputs a prediction block in the coding-domain to the pixel block coder, and the pixel block coder codes a coding-domain residual between the pixel block in the coding-domain and the prediction in the coding domain.
14. The video coder of claim 13 , wherein the video coder outputs coded data of the pixel blocks with source-domain motion vectors as coded data of the input picture.
15. The video coder of claim 13 , wherein the video coder outputs coded data of the pixel blocks with spherical-domain offsets as coded data of the input picture.
16. The video coder claim 15 , further comprising a controller that:
outputs data identifying the rotational offset, wherein
the outputted spherical-domain offsets are represented differentially with respect to the rotational offset.
17. The video coder of claim 13 , further comprising a controller that estimates a relative radius ratio between the spherical-domain representation of the pixel block and the spherical-domain representation of the matching portion of the of the reference picture, and outputting data identifying the relative radius ratio as coded data of the respective input pixel block.
18. The video coder of claim 13 , further comprising a controller that estimates a relative offset between an origin of the spherical-domain representation of the pixel block and an origin of the spherical-domain representation of the matching portion of the of the reference picture, and outputting data identifying the relative offset as coded data of the respective input pixel block.
19. The video coder of claim 13 , wherein the predictor:
estimates, for a plurality of candidate blocks from the reference pictures, a number of bits required to code the input pixel block with reference to each respective candidate block, and
selects a matching candidate block based on a relative differences between the number of bits estimated as required from among the candidate blocks.
20. The video coder of claim 13 , wherein the predictor:
estimates, for a plurality of candidate blocks from the reference pictures, an amount of distortion that could be created by coding the input pixel block with reference to each respective candidate block, and
selects a matching candidate block based on a relative differences between the amount of distortion estimated for each of the candidate blocks.
21. The video coder of claim 13 , wherein the input picture is an equirectangular picture in its source-domain representation.
22. The video coder of claim 13 , wherein the input picture is a cube map picture in its source-domain representation.
23. A non-transitory computer readable medium storing program instructions that, when executed by a processing device, cause the device to:
for a plurality of input pixel blocks from an input picture in a source-domain representation:
transforming an first input pixel block in the source-domain representation to a spherical-domain representation;
transforming a candidate reference picture from the source-domain representation to the spherical representation;
searching for a match between the spherical-domain representation of the first input pixel block and a portion of the spherical-domain representation of the candidate reference picture;
on a match, determining a spherical-domain motion vector including a spherical-domain rotational offset for predicting the first input pixel block in the spherical-domain representation from a matching portion of a reference picture in the spherical-domain representation;
transforming the spherical-domain motion vector to a two-dimensional source-domain motion vector and a source-domain rotational offset for predicting the first input pixel block in the source-domain representation of the input picture; and
predictively coding the first input pixel block in the source-domain including determining a source-domain prediction from the reference picture in the source-domain based on the two-dimensional source-domain motion vector and the source domain rotational offset.
24. The medium of claim 23 , further comprising, on the match:
determining a rotational offset between a spherical-domain representation of the input picture and a spherical-domain representation of the reference picture,
outputting data identifying the rotational offset, and
outputting coded data of the pixel blocks with spherical-domain offsets as coded data of the input picture, wherein the spherical-domain offsets are represented differentially with respect to the rotational offset.
25. The medium of claim 23 , wherein, for at least one pixel block, the searching includes estimating a relative radius ratio between the spherical-domain representation of the pixel block and the spherical-domain representation of the matching portion of the of the reference picture, and outputting data identifying the relative radius ratio as coded data of the respective input pixel block.
26. The medium of claim 23 , wherein, for at least one pixel block, the searching includes estimating a relative offset between an origin of the spherical-domain representation of the pixel block and an origin of the spherical-domain representation of the matching portion of the of the reference picture, and outputting data identifying the relative offset as coded data of the respective input pixel block.
27. A video decoder, comprising:
a pixel block decoder having an input for coded image data and an output for reconstructed pixel blocks in a non-spherical projection domain;
a reference picture store for storing reconstructed reference pictures assembled from reconstructed pixel blocks of frames in the non-spherical projection domain represented by the coded video data; and
a predictor, responsive to motion vector data in a spherical-domain included in the coded image data, for converting the motion vector data in the spherical-domain to motion vector data in the non-spherical projection domain and providing reference block data in the non-spherical projection domain to the pixel block decoder based upon manipulations of a reference picture in a spherical domain;
wherein the motion vector data includes an identification of a rotational offset between a spherical domain representation of the coded picture and a spherical domain representation of a reference picture, and the predictor aligns the reference picture and the coded picture according to the rotational offset.
28. The video decoder of claim 27 , further comprising a transform unit for transforming reconstructed pictures from the spherical domain representation to a two-dimensional domain representation.
29. The video decoder of claim 27 , wherein motion vector data of a coded picture includes an identification of a ratio of radii between a spherical domain representation of the coded picture and a spherical domain representation of a reference picture identified by the motion vector, and the predictor aligns the spherical domain representation of the reference picture to the spherical domain representation of the coded picture according to the ratio.
30. The video decoder of claim 27 , wherein the motion vector includes an identification of an offset between an origin of a spherical domain representation of the coded picture and an origin of a spherical domain representation of a reference picture identified by the motion vector, and the predictor aligns the origins of the reference picture and the coded picture according to the offset.
31. A video coding method, comprising:
for an input pixel block from an input picture in a source-domain representation:
transforming the input pixel block in the source-domain representation to a spherical-domain representation;
transforming a candidate reference picture from the source-domain representation to the spherical representation;
searching for a match between the spherical-domain representation of the input pixel block and a portion of the spherical-domain representation of the candidate reference picture;
on a match, determining a spherical-domain motion vector between the pixel block in the spherical-domain representation and a matching portion of a reference picture in the spherical-domain representation;
transforming the spherical-domain motion vector to a source-domain motion vector for predicting the input pixel block in the source-domain representation of the input picture,
predictively coding the input pixel block with reference to a source-domain representation of the matching portion of the reference picture converted from the spherical-domain representation; and
outputting the coded pixel block and the spherical-domain offset.
32. A video coding method, comprising:
for a plurality of input pixel blocks from an input picture in a source-domain representation:
transforming a first input pixel block in the source-domain representation to a spherical-domain representation;
transforming a candidate reference picture from the source-domain representation to the spherical representation;
searching for a match between the spherical-domain representation of the first input pixel block and a portion of the spherical-domain representation of the candidate reference picture;
on a match, determining spherical-domain offsets including a rotational offset for predicting the first input pixel block in the spherical-domain representation from a matching portion of a reference picture in the spherical-domain representation; and
predictively coding the first input pixel block including predicting the first input pixel block in the spherical-domain using the spherical domain offsets including the rotational offset, and including in the coded output indications of the spherical domain offsets with the rotational offset.