IP Library › Granted Patent US 11,818,394
Granted Patent B2
US 11,818,394 · App. 17/207,267 · Granted Nov 14, 2023

Sphere projected motion estimation/compensation and mode decision

Inventors: Jae Hoon Kim (San Jose, CA); Xiaosong Zhou (Campbell, CA); Dazhong Zhang (Milpitas, CA); Hang Yuan (San Jose, CA); Jiefu Zhai (San Jose, CA); Chris Y. Chung (Sunnyvale, CA); Hsi-Jung Wu (San Jose, CA)
Assignee: APPLE INC.
H04N19/597H04N19/105H04N19/176H04N19/547
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,818,394
App. No.
17/207,267
Granted
Nov 14, 2023
Kind
B2
Abstract

Techniques are disclosed for coding video data predictively based on predictions made from spherical-domain projections of input pictures to be coded and reference pictures that are prediction candidates. Spherical projection of an input picture and the candidate reference pictures may be generated. Thereafter, a search may be conducted for a match between the spherical-domain representation of a pixel block to be coded and a spherical-domain representation of the reference picture. On a match, an offset may be determined between the spherical-domain representation of the pixel block to a matching portion of the of the reference picture in the spherical-domain representation. The spherical-domain offset may be transformed to a motion vector in a source-domain representation of the input picture, and the pixel block may be coded predictively with reference to a source-domain representation of the matching portion of the reference picture.

Claims (71)

1. A video decoding method, comprising:

decoding a reference frame in a source-domain from a coded video sequence;

decoding a spherical-domain motion vector for a pixel block of a current frame from the coded video sequence, the motion vector indicates a change in radii between a first radius of a first spherical domain projection of the reference frame and a second radius of a second spherical domain projection of the current frame, wherein the first radius and second radius are not the same;

transforming the reference frame from the source-domain to a spherical-domain;

generate a spherical-domain prediction of the pixel block from the transformed spherical-domain reference frame and the spherical-domain motion vector, the generating based on the change in the radius of the spherical-domain projection;

transforming the spherical-domain prediction into a source-domain prediction of the pixel block; and

generating a decoded current frame in the source-domain based on the source-domain prediction of the pixel block.

2. The video decoding method of claim 1 , wherein the spherical-domain motion vector specifies a rotation of the reference frame in three dimensions.

3. The video decoding method of claim 1 , further comprising:

decoding a global rotation value for the reference frame from the coded video sequence;

decoding a differential motion vector for the pixel block from the coded video sequence;

determining the spherical-domain motion vector for the pixel block based on the global rotation value and the differential motion vector.

4. The video decoding method of claim 1 , wherein the spherical domain motion vector include a relative ratio of radii specifying a ratio of a current frame radius used in the spherical-domain projection of the current frame and a reference frame radius used in the spherical-projection of the reference frame, and the spherical-domain prediction is based on the relative ratio of radii.

5. The video decoding method of claim 1 , further comprising:

decoding a reference spatial origin for the reference frame from the coded video sequence;

decoding a current spatial origin for the current frame from the coded video sequence;

and wherein the spherical-domain prediction is based on the reference spatial origin and the current spatial origin.

6. The method of claim 1 , wherein the change in radii indicates a change in size of elements of image content between the current frame and the reference frame.

7. A video decoding system, comprising:

a processor, and

a memory storing instructions that, when executed by the processor, cause:

decoding a reference frame in a source-domain from a coded video sequence;

decoding a spherical-domain motion vector for a pixel block of a current frame from the coded video sequence, the motion vector indicates a change in radii between a first radius of a first spherical domain projection of the reference frame and a second radius of a second spherical domain projection of the current frame, wherein the first radius and second radius are not the same;

transforming the reference frame from the source-domain to a spherical-domain;

generate a spherical-domain prediction of the pixel block from the transformed spherical-domain reference frame and the spherical-domain motion vector based on the change in the radius of the spherical-domain projection;

transforming the spherical-domain prediction into a source-domain prediction of the pixel block; and

generating a decoded current frame in the source-domain based on the source-domain prediction of the pixel block.

8. The video decoding system of claim 7 , wherein the spherical-domain motion vector specifies a rotation of the reference frame in three dimensions.

9. The video decoding system of claim 7 , wherein the instructions further cause:

decoding a global rotation value for the reference frame from the coded video sequence;

decoding a differential motion vector for the pixel block from the coded video sequence;

determining the spherical-domain motion vector for the pixel block based on the global rotation value and the differential motion vector.

10. The video decoding system of claim 7 , wherein the spherical domain motion vector include a relative ratio of radii specifying a ratio of a current frame radius used in the spherical-domain projection of the current frame and a reference frame radius used in the spherical-projection of the reference frame, and the spherical-domain prediction is based on the relative ratio of radii.

11. The video decoding system of claim 7 , wherein the instructions further cause:

decoding a reference spatial origin for the reference frame from the coded video sequence;

decoding a current spatial origin for the current frame from the coded video sequence;

and wherein the spherical-domain prediction is based on the reference spatial origin and the current spatial origin.

12. A video decoding method, comprising:

decoding a reference frame in a spherical-domain from a coded video sequence;

decoding a spherical-domain motion vector for a pixel block of a current frame from the coded video sequence, the motion vector indicates a change in radii between a first radius of a first spherical domain projection of the reference frame and a second radius of a second spherical domain projection of the current frame, wherein the first radius and second radius are not the same;

generating a spherical-domain prediction of the pixel block from the transformed spherical-domain reference frame and the spherical-domain motion vector based on the change in the radius of the spherical-domain projection;

generating a decoded current frame in the spherical-domain based on the spherical-domain prediction of the pixel block;

transforming the decoded current frame in spherical-domain into a source-domain; and

outputting the decoded current frame in the source-domain.

13. The video decoding method of claim 12 , wherein the spherical-domain motion vector specifies a rotation of the reference frame in three dimensions.

14. The video decoding method of claim 12 , further comprising:

decoding a global rotation value for the reference frame from the coded video sequence;

decoding a differential motion vector for the pixel block from the coded video sequence;

determining the spherical-domain motion vector for the pixel block based on the global rotation value and the differential motion vector.

15. The video decoding method of claim 12 , wherein the spherical domain motion vector include a relative ratio of radii specifying a ratio of a current frame radius used in the spherical-domain projection of the current frame and a reference frame radius used in the spherical-projection of the reference frame, and the spherical-domain prediction is based on the relative ratio of radii.

16. The video decoding method of claim 12 , further comprising:

decoding a reference spatial origin for the reference frame from the coded video sequence;

decoding a current spatial origin for the current frame from the coded video sequence;

and wherein the spherical-domain prediction is based on the reference spatial origin and the current spatial origin.

17. A non-transitory computer readable medium storing program instructions that, when executed by a processing device, cause the device to:

decode a reference frame in a spherical-domain from a coded video sequence;

decode a spherical-domain motion vector for a pixel block of a current frame from the coded video sequence, the motion vector indicates a change in radii, wherein the change in radii indicates a change between a first radius of a first spherical domain projection of the reference frame and a second radius of a second spherical domain projection of the current frame, wherein the first radius and second radius are not the same;

generate a spherical-domain prediction of the pixel block from the transformed spherical-domain reference frame and the spherical-domain motion vector based on the change in the radius of the spherical-domain projection;

generate a decoded current frame in the spherical-domain based on the spherical-domain prediction of the pixel block;

transform the decoded current frame in spherical-domain into a source-domain; and

output the decoded current frame in the source-domain.

18. The computer readable medium of claim 17 , wherein the spherical-domain motion vector specifies a rotation of the reference frame in three dimensions.

19. The computer readable medium of claim 17 , further comprising:

decoding a global rotation value for the reference frame from the coded video sequence;

decoding a differential motion vector for the pixel block from the coded video sequence;

determining the spherical-domain motion vector for the pixel block based on the global rotation value and the differential motion vector.

20. The computer readable medium of claim 17 , wherein the spherical domain motion vector include a relative ratio of radii specifying a ratio of a current frame radius used in the spherical-domain projection of the current frame and a reference frame radius used in the spherical-projection of the reference frame, and the spherical-domain prediction is based on the relative ratio of radii.

21. The computer readable medium of claim 17 , further comprising:

decoding a reference spatial origin for the reference frame from the coded video sequence;

decoding a current spatial origin for the current frame from the coded video sequence;

and wherein the spherical-domain prediction is based on the reference spatial origin and the current spatial origin.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 23, 2021
From: KIM, JAE HOON; ZHOU, XIAOSONG; ZHANG, DAZHONG; YUAN, HANG; ZHAI, JIEFU; CHUNG, CHRIS Y.; WU, HSI-JUNG
To: APPLE INC.
Reel/Frame 056959/0372 →
Continuity (2)
Continuation 15390202 · Dec 23, 2016
Related Publication 20210321133A1 · Oct 14, 2021
Cited By (1)
US 12,568,241