IP Library › Granted Patent US 12,003,734

Granted Patent B2

US 12,003,734 · App. 17/676,510 · Granted Jun 4, 2024

Machine learning based flow determination for video coding

Inventors: Ankitesh Kumar Singh (San Diego, CA); Hilmi Enes Egilmez (San Diego, CA); Muhammed Zeyd Coban (Carlsbad, CA); Marta Karczewicz (San Diego, CA)

Assignee: QUALCOMM Incorporated

H04N19/139 H04N19/172 H04N19/186 H04N23/632

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12,003,734
App. No.: 17/676,510
Granted: Jun 4, 2024
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

Systems and techniques are described herein for processing video data. In some aspects, a method can include obtain, by a machine learning system, input video data. The input video data includes one or more luminance components for a current frame. The method can include determining, by the machine learning system, motion information for the luminance component(s) of the current frame and motion information for one or more chrominance components of the current frame using the luminance component(s) for the current frame. In some cases, the method can include determining the motion information for the luminance component(s) based on the luma component(s) of the current frame and at least one reconstructed luma component of a previous frame. In some cases, the method can further include determining the motion information for the chrominance component(s) of the current frame using the motion information determined for the luminance component(s) of the current frame.

Claims (45)

1. A method of processing video data, the method comprising:

obtaining, by a machine learning system, a latent representation of at least one luminance component for a current frame, the latent representation of the at least one luminance component of the current frame being based on the at least one luminance component of the current frame and at least one reconstructed luma component of a previous frame;

determining, by the machine learning system, motion information for the at least one luminance component of the current frame using the latent representation of the at least one luminance component of the current frame; and

determining, using the machine learning system, motion information for one or more chrominance components of the current frame using the motion information determined for the at least one luminance component for the current frame.

2. The method of claim 1 , further comprising:

determining, by the machine learning system using the motion information for the at least one luminance component of the current frame and the motion information for the one or more chrominance components of the current frame, a warping parameter for the at least one luminance component of the current frame and one or more warping parameters for one or more chrominance components of the current frame; and

determining one or more inter-frame predictions for the current frame using the warping parameter for the at least one luminance component of the current frame and the one or more warping parameters for the one or more chrominance components of the current frame.

3. The method of claim 2 , wherein the one or more inter-frame predictions are determined at least in part by applying an interpolation operation using the warping parameter for the at least one luminance component of the current frame and the one or more warping parameters for the one or more chrominance components of the current frame.

4. The method of claim 3 , wherein the interpolation operation includes a trilinear interpolation operation.

5. The method of claim 2 , wherein the warping parameter for the at least one luminance component of the current frame and the one or more warping parameters for the one or more chrominance components of the current frame include space-scale flow (SSF) warping parameters.

6. The method of claim 5 , wherein the SSF warping parameters include learned scale-flow vectors.

7. The method of claim 1 , wherein the motion information for the one or more chrominance components of the current frame is determined using a convolutional layer of the machine learning system.

8. The method of claim 1 , wherein the motion information for the one or more chrominance components of the current frame is determined at least in part by sampling the motion information determined for the at least one luminance component of the current frame.

9. The method of claim 1 , wherein the current frame includes a video frame.

10. The method of claim 1 , wherein the one or more chrominance components include at least one chrominance-blue component and at least one chrominance-red component.

11. The method of claim 1 , wherein the current frame has a luminance-chrominance (YUV) format.

12. The method of claim 11 , wherein the YUV format is a YUV 4:2:0 format.

13. An apparatus for processing video data, comprising:

at least one memory; and

one or more processors coupled to the at least one memory, the one or more processors configured to:

obtain, using a machine learning system, a latent representation of at least one luminance component for a current frame the latent representation of the at least one luminance component of the current frame being based on the at least one luminance component of the current frame and at least one reconstructed luma component of a previous frame;

determine, using the machine learning system, motion information for the at least one luminance component of the current frame using the latent representation of the at least one luminance component of the current frame; and

determine, using the machine learning system, motion information for one or more chrominance components of the current frame using the motion information determined for the at least one luminance component for the current frame.

14. The apparatus of claim 13 , wherein the one or more processors are configured to:

determine, using the machine learning system based on the motion information for the at least one luminance component of the current frame and the motion information for the one or more chrominance components of the current frame, a warping parameter for the at least one luminance component of the current frame and one or more warping parameters for one or more chrominance components of the current frame; and

determine one or more inter-frame predictions for the current frame using the warping parameter for the at least one luminance component of the current frame and the one or more warping parameters for the one or more chrominance components of the current frame.

15. The apparatus of claim 14 , wherein the one or more inter-frame predictions are determined at least in part by applying an interpolation operation using the warping parameter for the at least one luminance component of the current frame and the one or more warping parameters for the one or more chrominance components of the current frame.

16. The apparatus of claim 15 , wherein the interpolation operation includes a trilinear interpolation operation.

17. The apparatus of claim 14 , wherein the warping parameter for the at least one luminance component of the current frame and the one or more warping parameters for the one or more chrominance components of the current frame include space-scale flow (SSF) warping parameters.

18. The apparatus of claim 17 , wherein the SSF warping parameters include learned scale-flow vectors.

19. The apparatus of claim 13 , wherein the motion information for the one or more chrominance components of the current frame is determined using a convolutional layer of the machine learning system.

20. The apparatus of claim 13 , wherein, to determine the motion information for the one or more chrominance components of the current frame, the one or more processors are configured to sample the motion information determined for the at least one luminance component of the current frame.

21. The apparatus of claim 13 , wherein the current frame includes a video frame.

22. The apparatus of claim 13 , wherein the one or more chrominance components include at least one chrominance-blue component and at least one chrominance-red component.

23. The apparatus of claim 13 , wherein the current frame has a luminance-chrominance (YUV) format.

24. The apparatus of claim 23 , wherein the YUV format is a YUV 4:2:0 format.

25. The apparatus of claim 13 , further comprising at least one camera configured to capture one or more frames.

26. The apparatus of claim 13 , further comprising at least one display configured to display one or more frames.

27. The apparatus of claim 13 , wherein the apparatus includes a mobile device.

28. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to:

obtain, using a machine learning system, a latent representation of at least one luminance component for a current frame, the latent representation of the at least one luminance component of the current frame being based on the at least one luminance component of the current frame and at least one reconstructed luma component of a previous frame;

29. The method of claim 1 , wherein the input video data does not include a latent representation of luminance component for a current frame.

30. The apparatus of claim 13 , wherein the input video data does not include a latent representation of luminance component for a current frame.

Assignments (1)

ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 18, 2022

From: SINGH, ANKITESH KUMAR; EGILMEZ, HILMI ENES; COBAN, MUHAMMED ZEYD; KARCZEWICZ, MARTA

To: QUALCOMM INCORPORATED

Reel/Frame 059627/0905 →

Continuity (2)

Provisional Application 63153475 · Feb 25, 2021

Related Publication 20220272355A1 · Aug 25, 2022