IP Library Granted Patent US 11,665,340
Granted Patent B2
US 11,665,340 · App. 17/558,276 · Granted May 30, 2023

Systems and methods for histogram-based weighted prediction in video encoding

Inventors: Junqiang Lan (Fremont, CA); Guogang Hua (San Ramon, CA); Harikrishna Madadi Reddy (San Jose, CA); Chung-Fu Lin (Campbell, CA); Xing Cindy Chen (Los Altos, CA); Sujith Srinivasan (Fremont, CA)
Assignee: Meta Platforms, Inc.
H04N19/105H04N19/172H04N19/196H04N19/51
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,665,340
App. No.
17/558,276
Granted
May 30, 2023
Kind
B2
Abstract

A disclosed computer-implemented method may include (1) selecting, from a video stream, a reference frame and a current frame, (2) collecting a reference histogram of the reference frame and a current histogram of the current frame, and (3) generating a smoothed reference histogram by applying a smoothing function to at least a portion of the reference histogram. In some examples, the computer-implemented method may also include (1) determining a similarity metric between the smoothed reference histogram and the current histogram and, (2) when the similarity metric is greater than a threshold value, applying weighted prediction during a motion estimation portion of an encoding of the video stream. Various other methods, systems, and computer-readable media are also disclosed.

Claims (58)

1. A computer-implemented method comprising:

selecting, from a video stream, a reference frame and a current frame;

collecting a reference histogram of the reference frame and a current histogram of the current frame;

determining, based on an energy change between the current frame and the reference frame, a weight value;

adjusting the reference histogram based on the determined weight value;

generating a smoothed reference histogram by applying a smoothing function to at least a portion of the adjusted reference histogram;

determining a similarity metric between the smoothed reference histogram and the current histogram; and

when the similarity metric is greater than a threshold value, applying weighted prediction during a motion estimation portion of an encoding of the video stream.

2. The computer-implemented method of claim 1 , wherein:

the computer-implemented method further comprises generating a smoothed current histogram from at least a portion of the current histogram; and

determining the similarity metric between the smoothed reference histogram and the current histogram comprises determining the similarity metric between the smoothed reference histogram and the smoothed current histogram.

3. The computer-implemented method of claim 1 , further comprising:

determining an offset value based on the determined weight value; and

adjusting the reference histogram by applying the offset value to the reference histogram.

4. The computer-implemented method of claim 3 , wherein determining the similarity metric comprises determining the similarity metric based on the adjusted reference histogram and the current histogram.

5. The computer-implemented method of claim 1 , wherein selecting the reference frame and the current frame comprises selecting the reference frame and the current frame during the motion estimation portion of the encoding of the video stream.

6. The computer-implemented method of claim 1 , wherein collecting the reference histogram of the reference frame and the current histogram of the current frame comprises collecting the reference histogram from a YUV plane of the reference frame and the current histogram from a YUV plane of the current frame.

7. The computer-implemented method of claim 1 , wherein determining the similarity metric comprises determining a correlation coefficient between the smoothed reference histogram and the current histogram.

8. The computer-implemented method of claim 7 , wherein determining the correlation coefficient comprises determining a Pearson correlation.

9. The computer-implemented method of claim 7 , wherein determining the correlation coefficient comprises determining at least one of:

a mutual information metric;

an intraclass correlation (ICC);

a polychoric correlation; or

a rank coefficient.

10. The computer-implemented method of claim 1 , wherein at least one of the reference histogram or the current histogram comprises at least 256 bins.

11. The computer-implemented method of claim 1 , wherein the smoothing function comprises a Gaussian smoothing function.

12. The computer-implemented method of claim 1 , wherein the smoothing function comprises at least one of:

a nearest-neighbor smoothing function;

a wavelet transform function;

a barycentric exponential smoothing function; or

a mean value smoothing function.

13. The computer-implemented method of claim 1 , wherein the reference histogram comprises at least 256 bins and the current histogram comprises at least 256 bins.

14. A system comprising:

a selecting module, stored in memory, that selects, from a video stream, a reference frame and a current frame;

a collecting module, stored in memory, that:

collects a reference histogram of the reference frame and a current histogram of the current frame;

determines, based on an energy change between the current frame and the reference frame, a weight value; and

adjusts the reference histogram based on the determined weight value;

a smoothing module, stored in memory, that generates a smoothed reference histogram by applying a smoothing function to at least a portion of the adjusted reference histogram;

a determining module, stored in memory, that determines a similarity metric between the smoothed reference histogram and the current histogram;

an applying module, stored in memory, that applies, when the determined similarity metric is greater than a threshold value, weighted prediction during a motion estimation portion of an encoding of the video stream; and

at least one physical processor that executes the selecting module, the collecting module, the smoothing module, the determining module, and the applying module.

15. The system of claim 14 , wherein:

the collecting module further collects the reference histogram by:

determining an offset value based on the determined weight value; and

further adjusting the reference histogram by applying the weight value and the offset value to the reference histogram.

16. The system of claim 14 , wherein the selecting module selects the reference frame and the current frame by selecting the reference frame and the current frame during the motion estimation portion of the encoding of the video stream.

17. The system of claim 14 , wherein the collecting module collects the reference histogram of the reference frame and the current histogram of the current frame by collecting the reference histogram from a YUV plane of the reference frame and the current histogram from a YUV plane of the current frame.

18. The system of claim 14 , wherein the determining module determines the similarity metric by determining a correlation coefficient between the reference histogram and the current histogram.

19. The system of claim 18 , wherein the determining module determines the correlation coefficient by determining a Pearson correlation.

20. A non-transitory computer-readable medium comprising computer-readable instructions that, when executed by at least one processor of a computing system, cause the computing system to:

select, from a video stream, a reference frame and a current frame;

collect a reference histogram of the reference frame and a current histogram of the current frame;

determine, based on an energy change between the current frame and the reference frame, a weight value;

adjust the reference histogram based on the determined weight value;

generate a smoothed reference histogram by applying a smoothing function to at least a portion of the reference histogram;

determine a similarity metric between the smoothed reference histogram and the current histogram; and

when the determined similarity metric is greater than a threshold value, apply weighted prediction during a motion estimation portion of an encoding of the video stream.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 5, 2022
From: LAN, JUNQIANG; HUA, GUOGANG; REDDY, HARIKRISHNA MADADI; LIN, CHUNG-FU; CHEN, XING CINDY; SRINIVASAN, SUJITH
To: META PLATFORMS, INC.
Reel/Frame 059179/0276 →