Reference picture selection and coding type decision processing based on scene contents
An adaptive screen encoding method comprising: using a computer, creating and storing, in computer memory, a plurality of conditions for use by a server configured to determine which of picture coding type to select; detecting a current picture by a sender for a content type including textual content, graphical content, and natural image content; determining a percentage of static macroblocks corresponding to the current picture; selecting the picture coding type based on the content type, the plurality of conditions, and the percentage of static macroblocks, wherein the method is performed by one or more special-purpose computing devices.
1. A method, comprising:
determining a content type of a current picture of a data stream associated with real-time screen share content presented during a video conference;
determining a percentage of the current picture that includes static macro-blocks;
determining whether a scene change occurs between the current picture and a last played picture based on a comparison between the percentage and one or more thresholds;
determining a coding type for the current picture based on the content type and whether the scene change occurs; and
encoding the current picture according to the coding type, wherein a number of reference pictures to use to encode the current picture is based on the coding type for the current picture and the comparison between the percentage and the one or more thresholds.
2. The method of claim 1 , wherein, where a determination is made that the scene change occurs and the content type is a natural image picture type, the coding type is an I picture coding type and the current picture is marked as a long term reference picture for a new scene associated with the scene change.
3. The method of claim 1 , wherein, where a determination is made that the scene change does not occur and the content type is a natural image picture type, the coding type is a P picture coding type and the current picture is marked as a short term reference picture.
4. The method of claim 1 , wherein, where a determination is made that the scene change occurs and the content type is at least one of a textual content type or a graphical content type, the coding type is an I picture coding type and the current picture is marked as a long term reference picture for a new scene associated with the scene change.
5. The method of claim 1 , wherein, where the percentage is larger than a first threshold of the one or more thresholds and smaller than a second threshold of the one or more thresholds, the coding type is a P picture coding type and the current picture is marked as a long term reference picture.
6. The method of claim 1 , wherein, where the percentage is larger than a first threshold of the one or more thresholds and a second threshold of the one or more thresholds, the coding type is a P picture coding type and the current picture is marked as a short term reference picture.
7. The method of claim 1 , wherein, where the percentage is larger than a duplicate scene threshold of the one or more thresholds, the current picture is determined as a duplicate of a reference picture of a reference picture set available for encoding the current picture and replaces the reference picture within the reference picture set after the encoding of the current picture.
8. The method of claim 1 , wherein determining the percentage of the current picture that includes the static macro-blocks comprises:
calculating a sum of absolute differences value between a current block of the current picture and a collocated block of a reference picture; and
based on the sum of absolute differences value being equal to zero, determining that the current block is static.
9. The method of claim 1 , wherein the content type is at least one of a textual content type, a graphical content type, or a natural image content type.
10. The method of claim 1 , wherein multiple reference pictures of a reference picture list available for encoding the current picture are arranged based on scene content similarity values between the multiple reference pictures and the current picture.
11. A non-transitory computer readable medium storing instructions that, when executed by a processor, cause a performance of operations comprising:
determining a content type of a current picture of a data stream associated with real-time screen share content presented during a video conference;
determining a percentage of the current picture that includes static macro-blocks;
determining whether a scene change occurs between the current picture and a last played picture based on a comparison between the percentage and one or more thresholds;
determining a coding type for the current picture based on the content type and whether the scene change occurs; and
encoding the current picture according to the coding type, wherein a number of reference pictures to use to encode the current picture is based on the coding type for the current picture and the comparison between the percentage and the one or more thresholds.
12. The non-transitory computer readable medium of claim 11 , wherein the operations for determining the percentage of the current picture that includes the static macro-blocks comprise:
determining a scene content similarity between the current picture and a reference picture based on a sum of absolute differences value between a current block of the current picture and a collocated block of the reference picture.
13. The non-transitory computer readable medium of claim 11 , wherein the one or more thresholds includes one or more of a new scene threshold, a large difference scene threshold, or a duplicate scene threshold.
14. The non-transitory computer readable medium of claim 11 , the operations comprising:
replacing a reference picture of a reference picture list available for encoding the current picture with the current picture based on the percentage being greater than a threshold of the one or more thresholds.
15. The non-transitory computer readable medium of claim 11 , wherein the content type is at least one of a textual content type, a graphical content type, or a natural image content type.
16. A system, comprising:
a server device including a memory and a processor configured to execute instructions stored in the memory to:
determine a content type of a current picture of a data stream associated with real-time screen share content presented during a video conference;
determine a percentage of the current picture that includes static macro-blocks;
determine whether a scene change occurs between the current picture and a last played picture based on a comparison between the percentage and one or more thresholds;
determine a coding type for the current picture based on the content type and whether the scene change occurs; and
encode the current picture according to the coding type, wherein a number of reference pictures to use to encode the current picture is based on the coding type for the current picture and the comparison between the percentage and the one or more thresholds.
17. The system of claim 16 , wherein the processor is configured to execute the instructions to:
mark the current picture as a long term reference picture or a short term reference picture based on whether the scene change occurs.
18. The system of claim 16 , wherein the current picture is encoded using multiple reference pictures of a reference picture list ordered based on scene content similarity values relative to the current picture.
19. The system of claim 16 , wherein the processor is configured to execute the instructions to:
determine a size of a scene change region of the current picture based on the comparison between the percentage and the one or more thresholds.
20. The system of claim 16 , wherein the processor is configured to execute the instructions to:
determine that a macro-block of the current picture is a static macro-block based on a sum of absolute differences calculated between the macro-block and a collocated macro-block of a reference picture.