IP Library › Granted Patent US 12,244,814
Granted Patent B2
US 12,244,814 · App. 17/975,842 · Granted Mar 4, 2025

Method for processing a video sequence and apparatus for implementing the same

Inventors: Anthony Nasrallah (Puteaux, FR); Thomas Guionnet (Rennes, FR); Mohsen Abdoli (Thorigne Fouillard, FR)
Assignee: ATEME
H04N19/136H04N19/154H04N19/189
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,244,814
App. No.
17/975,842
Granted
Mar 4, 2025
Kind
B2
Abstract

A method of processing a video sequence is proposed, which comprises, for a video encoding tool of a set of video encoding tools configured for encoding the video sequence comprised in a video encoder, determining an efficiency value based on an encoding performance value and an encoding computational complexity value of the video encoding tool for encoding the video sequence, and determining, based on the efficiency value of the video encoding tool, whether to configure the video encoder to use the video encoding tool for the encoding of the video sequence.

Claims (80)

1. A method of processing a video sequence, comprising:

for a video encoding tool of a set of video encoding tools configured for encoding the video sequence comprised in a video encoder:

determining an encoding performance value of the video encoding tool for encoding the video sequence, wherein the encoding performance value measures an encoding performance of the video encoding tool for encoding the video sequence;

determining an encoding computational complexity value of the video encoding tool for encoding the video sequence, wherein the encoding computational complexity value measures the computational complexity incurred by using the video encoding tool for performing the encoding of the video sequence;

determining an efficiency value based on a ratio of the encoding performance value over the encoding computational complexity value of the video encoding tool for encoding the video sequence; and

determining, based on the efficiency value of the video encoding tool, whether to configure the video encoder to use the video encoding tool for the encoding of the video sequence,

wherein the method further comprises:

determining a plurality of global performance efficiency values of the video encoder respectively corresponding to configurations of the video encoder in which a respective set of video encoding tools is used for the encoding of the video sequence;

determining an optimum video encoding tool in the set of video encoding tools corresponding to an optimum global performance efficiency value in the plurality of global performance efficiency values of the video encoder;

wherein the determining whether to configure the video encoder to use the video encoding tool for the encoding of the video sequence is based on a comparison of the efficiency value with an efficiency value determined for the optimum video encoding tool.

2. The method of claim 1 , further comprising:

determining, based on the efficiency value of the video encoding tool, whether to activate the video encoding tool in a configuration of the video encoder for the encoding of the video sequence;

in a case where it has been determined to activate the video encoding tool, encoding, by the video encoder configured with the video encoding tool, the video sequence; and

in a case where it has been determined not to activate the video encoding tool, encoding, by the video encoder configured without the video encoding tool, the video sequence.

3. The method of claim 1 , further comprising:

determining a plurality of efficiency values respectively corresponding to a plurality of video encoding tools of the set of video encoding tools;

determining an ordered sequence of efficiency values based on the plurality of efficiency values;

wherein the determining whether to configure the video encoder to use the video encoding tool for the encoding of the video sequence is based on a position in the ordered sequence of the efficiency value determined for the video encoding tool.

4. The method of claim 1 , further comprising:

determining a plurality of efficiency values respectively corresponding to a plurality of video encoding tools of the set of video encoding tools;

determining an ordered sequence of efficiency values based on the plurality of efficiency values;

determining, based on the ordered sequence of efficiency values, a configuration of the video encoder to use a subset of the set of video encoding tools for encoding the video sequence.

5. The method of claim 1 , further comprising:

determining a plurality of global performance efficiency values of the video encoder respectively corresponding to configurations of the video encoder in which a respective set of video encoding tools is used for the encoding of the video sequence;

wherein the determining whether to configure the video encoder to use the video encoding tool for the encoding of the video sequence is based on a position, in an ordered sequence of the video encoding tools corresponding to the ordered sequence of efficiency values, of the video encoding tool with respect to a video encoding tool corresponding to a maximum value of the plurality of global performance efficiency values.

6. The method of claim 1 , wherein each of the respective efficiency values of a corresponding video encoding tool is based on a respective encoding performance value of the corresponding video encoding tool, and a respective encoding computational complexity value of the corresponding video encoding tool, and wherein the subset of video encoding tools to be used in the video encoder configuration is determined based on respective encoding performance values and respective encoding computational complexity values for corresponding video encoding tools.

7. The method of claim 1 , wherein the video sequence is divided into a plurality of subsequences, wherein the set of video encoding tools comprises one or more video encoding tools that can be determined to be used for encoding each subsequence of the video sequence, the method further comprising, for each of the subsequences and for each of the video encoding tool: determining whether to use the video encoding tool for the encoding the subsequence based on an efficiency value for encoding the subsequence determined for the video encoding tool.

8. A method for encoding a video sequence, comprising: training, using the method of claim 1 applied to training input data comprising a training input video sequence, a neural network to generate a prediction as to whether the video encoder is to use the video encoding tool for the encoding of the training input video sequence.

9. A method for encoding a video sequence by a video encoder which can be configured to use a video encoding tool for encoding the video sequence, the method comprising: determining, by the neural network of claim 8 , based on input data comprising the video sequence, a prediction as to whether the video encoder is to be configured to use the video encoding tool for encoding the video sequence.

10. A method for encoding a video sequence by a video encoder which can be configured with a video encoding tool for encoding the video sequence, the method comprising: determining, by the neural network of claim 8 , based on input data comprising a video sequence, a prediction of an efficiency value of the video encoding tool for encoding the video sequence.

11. An apparatus comprising a processor and a memory operatively coupled to the processor, wherein the processor is configured to perform a method of processing a video sequence, comprising:

for a video encoding tool of a set of video encoding tools configured for encoding the video sequence comprised in a video encoder;

determining an encoding performance value of the video encoding tool for encoding the video sequence, wherein the encoding performance value measures an encoding performance of the video encoding tool for encoding the video sequence;

determining an encoding computational complexity value of the video encoding tool for encoding the video sequence, wherein the encoding computational complexity value measures the computational complexity incurred by using the video encoding tool for performing the encoding of the video sequence;

determining an efficiency value based on a ratio of the encoding performance value over the encoding computational complexity value of the video encoding tool for encoding the video sequence; and

determining, based on the efficiency value of the video encoding tool, whether to configure the video encoder to use the video encoding tool for the encoding of the video sequence,

wherein the method further comprises:

determining a plurality of global performance efficiency values of the video encoder respectively corresponding to configurations of the video encoder in which a respective set of video encoding tools is used for the encoding of the video sequence;

determining an optimum video encoding tool in the set of video encoding tools corresponding to an optimum global performance efficiency value in the plurality of global performance efficiency values of the video encoder;

wherein the determining whether to configure the video encoder to use the video encoding tool for the encoding of the video sequence is based on a comparison of the efficiency value with an efficiency value determined for the optimum video encoding tool.

12. The apparatus of claim 11 , wherein the method further comprises:

determining, based on the efficiency value of the video encoding tool, whether to activate the video encoding tool in a configuration of the video encoder for the encoding of the video sequence;

in a case where it has been determined to activate the video encoding tool, encoding, by the video encoder configured with the video encoding tool, the video sequence; and

in a case where it has been determined not to activate the video encoding tool, encoding, by the video encoder configured without the video encoding tool, the video sequence.

13. The apparatus of claim 11 , wherein the method further comprises:

determining a plurality of efficiency values respectively corresponding to a plurality of video encoding tools of the set of video encoding tools;

determining an ordered sequence of efficiency values based on the plurality of efficiency values;

wherein the determining whether to configure the video encoder to use the video encoding tool for the encoding of the video sequence is based on a position in the ordered sequence of the efficiency value determined for the video encoding tool.

14. The apparatus of claim 11 , wherein the method further comprises:

determining a plurality of efficiency values respectively corresponding to a plurality of video encoding tools of the set of video encoding tools;

determining an ordered sequence of efficiency values based on the plurality of efficiency values;

determining, based on the ordered sequence of efficiency values, a configuration of the video encoder to use a subset of the set of video encoding tools for encoding the video sequence.

15. The apparatus of claim 11 , wherein the method further comprises:

determining a plurality of global performance efficiency values of the video encoder respectively corresponding to configurations of the video encoder in which a respective set of video encoding tools is used for the encoding of the video sequence;

wherein the determining whether to configure the video encoder to use the video encoding tool for the encoding of the video sequence is based on a position, in an ordered sequence of the video encoding tools corresponding to the ordered sequence of efficiency values, of the video encoding tool with respect to a video encoding tool corresponding to a maximum value of the plurality of global performance efficiency values.

16. A non-transitory computer-readable medium encoded with executable instructions which, when executed, causes an apparatus comprising a processor operatively coupled with a memory, to perform a method of processing a video sequence, comprising:

for a video encoding tool of a set of video encoding tools configured for encoding the video sequence comprised in a video encoder:

determining an encoding performance value of the video encoding tool for encoding the video sequence, wherein the encoding performance value measures an encoding performance of the video encoding tool for encoding the video sequence;

determining an encoding computational complexity value of the video encoding tool for encoding the video sequence, wherein the encoding computational complexity value measures the computational complexity incurred by using the video encoding tool for performing the encoding of the video sequence;

determining an efficiency value based on a ratio of the encoding performance value over the encoding computational complexity value of the video encoding tool for encoding the video sequence; and

determining, based on the efficiency value of the video encoding tool, whether to configure the video encoder to use the video encoding tool for the encoding of the video sequence,

wherein the method further comprises:

determining a plurality of global performance efficiency values of the video encoder respectively corresponding to configurations of the video encoder in which a respective set of video encoding tools is used for the encoding of the video sequence;

determining an optimum video encoding tool in the set of video encoding tools corresponding to an optimum global performance efficiency value in the plurality of global performance efficiency values of the video encoder;

wherein the determining whether to configure the video encoder to use the video encoding tool for the encoding of the video sequence is based on a comparison of the efficiency value with an efficiency value determined for the optimum video encoding tool.

17. The non-transitory computer-readable medium of claim 16 , wherein the method further comprises:

determining, based on the efficiency value of the video encoding tool, whether to activate the video encoding tool in a configuration of the video encoder for the encoding of the video sequence;

in a case where it has been determined to activate the video encoding tool, encoding, by the video encoder configured with the video encoding tool, the video sequence; and

in a case where it has been determined not to activate the video encoding tool, encoding, by the video encoder configured without the video encoding tool, the video sequence.

18. The non-transitory computer-readable medium of claim 16 , wherein the method further comprises:

determining a plurality of efficiency values respectively corresponding to a plurality of video encoding tools of the set of video encoding tools;

determining an ordered sequence of efficiency values based on the plurality of efficiency values;

wherein the determining whether to configure the video encoder to use the video encoding tool for the encoding of the video sequence is based on a position in the ordered sequence of the efficiency value determined for the video encoding tool.

19. The non-transitory computer-readable medium of claim 16 , wherein the method further comprises:

determining a plurality of efficiency values respectively corresponding to a plurality of video encoding tools of the set of video encoding tools;

determining an ordered sequence of efficiency values based on the plurality of efficiency values;

determining, based on the ordered sequence of efficiency values, a configuration of the video encoder to use a subset of the set of video encoding tools for encoding the video sequence.

20. The non-transitory computer-readable medium of claim 16 , wherein the method further comprises:

determining a plurality of global performance efficiency values of the video encoder respectively corresponding to configurations of the video encoder in which a respective set of video encoding tools is used for the encoding of the video sequence;

wherein the determining whether to configure the video encoder to use the video encoding tool for the encoding of the video sequence is based on a position, in an ordered sequence of the video encoding tools corresponding to the ordered sequence of efficiency values, of the video encoding tool with respect to a video encoding tool corresponding to a maximum value of the plurality of global performance efficiency values.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 3, 2025
From: NASRALLAH, ANTHONY; GUIONNET, THOMAS; ABDOLI, MOHSEN
To: ATEME
Reel/Frame 069734/0389 →
Priority Claims (1)
EP 21306511 · Oct 28, 2021 · regional
Continuity (1)
Related Publication 20230136187A1 · May 4, 2023
References Cited (24)
US 20230062752A1 · Lainema · 2023 [cited by examiner]
EP 3734972A1 · 2020 [cited by applicant]
Schwarz et al. “Coding Efficiency / Complexity Analysis of JEM 1.0 coding tools for the Random Access Configuration,” 114. MPEG Meeting; 20160222-20160226; San Diego; Motion Picture Expert Group or ISO/IEC JTC1/SC29/W G… [cited by examiner]
Aklouf et al., “Low complexity versatile video coding (VVC) for low bitrate applications,” 8th European Workshop on Visual Information Processing (EUVIP 2019), Rome, Italy, Oct. 2019, 7 pages. [cited by applicant]
Boyce et al., “JVET common test conditions and software reference configurations,” Document JVET-J1010, Ljubljana, Slovenia, Jul. 2018, 6 pages. [cited by applicant]
Brandenburg et al., “Towards fast and efficient VVC encoding,” 2020 IEEE 22nd International Workshop on Multimedia Signal Processing (MMSP), 2020, pp. 1-6. [cited by applicant]
Chen et al., “The joint exploration model (JEM) for video compression with capability beyond HEVC,” IEEE Transactions on Circuits and Systems for Video Technology, May 2020, vol. 30(5), pp. 1208-1225. [cited by applicant]
Chen et al., “Performance evaluation of JEM 1 tools by Qualcomm,” 2 JVET Meeting, Feb. 20-26, 2016, San Diego, CA, US, Feb. 2016, retrieved from internet: http://phenix.int-evry.fr/jvet No. JVET-B0045-v3 (retrieved on F… [cited by applicant]
Chen et al., “JVET AHG report: Tool procedure and testing (AHG13),” Document JVET-Q0013, 17th Meeting, Brussels, BE, Jan. 7-17, 2020, 13 pages. [cited by applicant]
Czyzak et al., “Pareto simulated annealing-A metaheuristic technique for multiple-objective combinatorial optimization,” Journal of Multi-Criteria Decision Analysis, 1998, vol. 7, pp. 34-47. [cited by applicant]
Da Silva et al., “Fast HEVC intra prediction mode decision based on edge direction information,” Proc. European Signal Processing Conference, IEEE, 2012, 5 pages. [cited by applicant]
Filippov et al., “Evaluation of some intra-coding tools of JEM1, ” 2 JVET Meeting, Feb. 20-26, 2016, San Diego, CA, US, No. JVET-B0047, Feb. 21, 2016, retrieve from Internet: http://phenix.int-evry.fr/jvet/doc_end_user/… [cited by applicant]
Fu et al., “Fast CU partitioning algorithm for H.266/VVC Intra-frame coding,” 2019 IEEE International Conference on Multimedia and Expo (ICME), 2019, pp. 55-60. [cited by applicant]
Howard et al., “MobileNets: Efficient convolutional neural networks for mobile vision applications,” arXiv preprint arXiv: 1704.04861, 2017, 10 pages. [cited by applicant]
Li et al., “DeepQTMT: A deep learning approach for fast QTMT-based CU partition of intra-mode VVC,” IEEE Transactions on Image Processing, 2021, vol. 30, pp. 5377-5390. [cited by applicant]
Sampaio et al., “Motion vectors merging: low complexity prediction unit decision heuristic for the inter-prediction of HEVC encoders,” Proc. IEEE International Conference on Multimedia and Expo, IEEE, 2012, 6 pages. [cited by applicant]
Schwarz et al., “Coding Efficiency/Complexity Analysis of JEM 1.0 coding tools for the Random Access Configuration, ”114 MPEG Meeting, Feb. 22-26, 2016, San Diego, CA, US, No. m37898, Feb. 23, 2016, retrieved from Inter… [cited by applicant]
Shen et al., “An effective CU size decision method for HEVC encoders,” IEEE transactions on Multimedia, Feb. 2013, vol. 15(2), pp. 465-470. [cited by applicant]
Sullivan et al., “Overview of the High Efficiency Video Coding (HEVC) standard,” IEEE Transactions on circuits and systems for video technology, Dec. 2012, vol. 22(12), pp. 1649-1668. [cited by applicant]
Tech et al., “NN-based parameter selection for fast VVC intra-picture encoding,” 2021 IEEE International Conference on Image Processing (ICIP), 2021, pp. 2109-2113. [cited by applicant]
Tissier et al., “CNN Oriented Complexity Reduction of VVC Intra Encoder,” IEEE International Conference on Image Processing (ICIP 2020), Abu Dhabi, United Arab Emirates, IEEE, Oct. 2020, 5 pages. [cited by applicant]
Zhao et al., “Fast mode decision algorithm for intra prediction in HEVC,” 2011 Visual Communications and Image Processing (VCIP), IEEE, 2011, pp. 1-4. [cited by applicant]
Zhao et al., “Adaptive CU split decision based on deep learning and multifeature fusion for H.266/VVC,” Scientific Programming, 2020, Article ID: 883214 , 2020, 11 pages. [cited by applicant]
Search Report issued in related application EP 21306511.3, Apr. 19, 2022, 13 pages. [cited by applicant]