Encoding a video frame using different compression ratios for text blocks and non-text blocks
This document describes systems and techniques for encoding a video frame using different compression ratios or compression algorithms for text blocks and non-text blocks. The described systems and techniques can determine, using a machine-learned model, which blocks of a frame include and do not include text. The described systems and techniques can then use a different compression ratio or compression algorithm for text blocks than the compression ratio or compression algorithm used for non-text blocks. For example, the systems and techniques can encode the text blocks using a first compression ratio that results in higher video quality than a second compression ratio used on at least some non-text blocks. In this way, the described systems and techniques can improve text legibility in a video file without significantly increasing the bandwidth requirements to transmit the video file to remote computing devices.
1. A computer-implemented method for compressing a video file, the method comprising:
determining, using a machine-learned model and for each block of multiple blocks of which a frame of the video file is composed, text blocks and non-text blocks, the text blocks classified by the machine-learned model as including at least some text and the non-text blocks classified by the machine-learned model as not including text;
generating a segment map of the frame, the segment map identifying the text blocks or the non-text blocks of the frame; and
encoding, based on the segment map, the frame to generate an encoded frame of the video file in which the text blocks are encoded using a first compression ratio, a first subset of the non-text blocks are encoded using a second compression ratio, and a second subset of the non-text blocks are encoded using a third compression ratio, the first compression ratio resulting in higher video quality than the second compression ratio and the second compression ratio resulting in higher video quality than the third compression ratio.
2. The computer-implemented method of claim 1 , the method further comprising:
prior to determining the text blocks and the non-text blocks, dividing the frame of the video file into multiple portions, each of the multiple portions having two or more of the multiple blocks, wherein determining the text blocks and the non-text blocks is performed on the blocks of each portion.
3. The computer-implemented method of claim 2 , wherein:
the multiple portions comprise overlapping regions of the frame;
the multiple portions comprise non-overlapping regions of the frame;
the multiple portions comprise 256-by-256-pixel portions or 256-by-64-pixel portions; or
the blocks comprise eight-by-eight-pixel blocks, four-by-four-pixel blocks, or 32-by-32-pixel blocks.
4. The computer-implemented method of claim 1 , wherein encoding the frame to generate the encoded frame of the video file comprises:
masking, using the segment map, the non-text blocks;
encoding, using the first compression ratio, the text blocks of the segment map;
masking, using the segment map, the text blocks of the segment map; and
encoding, using the second compression ratio, the non-text blocks of the segment map.
5. The computer-implemented method of claim 1 , wherein the first subset of the non-text blocks comprises blocks adjacent to the text blocks.
6. The computer-implemented method of claim 1 , wherein the machine-learned model uses octave convolution and separable convolution to determine the text blocks and the non-text blocks.
7. The computer-implemented method of claim 1 , the method further comprising:
determining, using the segment map of the frame and the machine-learned model, for each block of the multiple blocks of a subsequent frame of the video file, the text blocks and non-text blocks;
generating a subsequent segment map of the subsequent frame, the subsequent segment map identifying the text blocks or the non-text blocks of the subsequent frame; and
encoding, based on the subsequent segment map, the subsequent frame to generate an encoded subsequent frame of the video file in which the text blocks are encoded using the first compression ratio and at least some of the non-text blocks are encoded using the second compression ratio.
8. The computer-implemented method of claim 7 , the subsequent segment map indicates differences between the subsequent segment map of the subsequent frame and the segment map of the frame.
9. The computer-implemented method of claim 7 , the method further comprising:
compiling the encoded frame and the encoded subsequent frame to generate an encoded video file and storing the encoded video file locally; or
compiling the encoded frame and the encoded subsequent frame to generate an encoded video file and transmitting the encoded video file to a remote device.
10. An apparatus comprising:
a memory storing a video file;
a text detection module configured to determine, using a machine-learned model and for each block of multiple blocks of which a frame of the video file is composed, text blocks and non-text blocks, the text blocks classified by the machine-learned model as including at least some text and the non-text blocks classified by the machine-learned model as not including text;
a segment map module configured to generate a segment map of the frame, the segment map identifying the text blocks or the non-text blocks of the frame; and
a video encoder module configured to encode, based on the segment map, the frame to generate an encoded frame of the video file in which the text blocks are encoded using a first compression ratio, a first subset of the non-text blocks are encoded using a second compression ratio, and a second subset of the non-text blocks are encoded using a third compression ratio, the first compression ratio resulting in higher video quality than the second compression ratio and the second compression ratio resulting in higher video quality than the third compression ratio.
11. The apparatus of claim 10 , wherein to generate the encoded frame of the video file, the video encoder is further configured to:
mask, using the segment map, the non-text blocks;
encode, using the first compression ratio, the text blocks of the segment map;
mask, using the segment map, the text blocks of the segment map; and
encode, using the second compression ratio, the non-text blocks of the segment map.
12. The apparatus of claim 10 , wherein the machine-learned model implements octave convolution and separable convolution to enable the text detection module to determine the text blocks and the non-text blocks.
13. The apparatus of claim 10 , wherein the video encoder module is further configured to:
compile the encoded frame and an encoded subsequent frame to generate an encoded video file and store the encoded video file locally; or
compile the encoded frame and an encoded subsequent frame to generate an encoded video file for transmission of the encoded video file to a remote device.
14. The apparatus of claim 10 , wherein the apparatus is configured as a hardware-accelerated video encoder or an integrated circuit.
15. The apparatus of claim 14 , wherein:
the hardware-accelerated video encoder comprises the machine-learned model; or
the integrated circuit comprises the machine-learned model.
16. A video encoding system comprising:
at least one processor; and
computer-readable storage media comprising processor-executable instructions that, responsive to execution by the at least one processor, implement operations comprising:
accessing a video file;
determining, using a machine-learned model and for each block of multiple blocks of which a frame of the video file is composed, text blocks and non-text blocks, the text blocks classified by the machine-learned model as including at least some text and the non-text blocks classified by the machine-learned model as not including text;
generating a segment map of the frame, the segment map identifying the text blocks or the non-text blocks of the frame; and
encoding, based on the segment map, the frame to generate an encoded frame of the video file in which the text blocks are encoded using a first compression ratio, a first subset of the non-text blocks are encoded using a second compression ratio, and a second subset of the non-text blocks are encoded using a third compression ratio, the first compression ratio resulting in higher video quality than the second compression ratio and the second compression ratio resulting in high video quality than the third compression ratio.
17. The video encoding system of claim 16 , wherein:
the operations further comprise, prior to determining the text blocks and the non-text blocks, dividing the frame of the video file into multiple portions, each of the multiple portions having two or more of the multiple blocks, and
determining the text blocks and the non-text blocks is performed on the blocks of each portion.
18. The video encoding system of claim 16 , wherein the operations further comprise:
determining, using the segment map of the frame and the machine-learned model, for each block of the multiple blocks of a subsequent frame of the video file, the text blocks and non-text blocks;
generating a subsequent segment map of the subsequent frame, the subsequent segment map identifying the text blocks or the non-text blocks of the subsequent frame; and
encoding, based on the subsequent segment map, the subsequent frame to generate an encoded subsequent frame of the video file in which the text blocks are encoded using the first compression ratio and at least some of the non-text blocks are encoded using the second compression ratio.
19. The video encoding system of claim 18 , wherein the operations further comprise:
receiving the video file from a remote device via a data interface; or
compiling the encoded frame and the encoded subsequent frame to generate an encoded video file and transmitting the encoded video file to a remote device.
20. The video encoding system of claim 18 , wherein the subsequent segment map indicates differences between the subsequent segment map of the subsequent frame and the segment map of the frame.