SYSTEMS AND METHODS FOR VIDEO ENCODING USING IMAGE SEGMENTATION
A video encoder is provided that includes an image detection and segmentation processor receiving the video frame and generating object recognition data and at least one image segmentation mask. A mask to coding block mapping processor maps the segmentation mask to the CTUs of the video frame and partitions at least one CTU into a plurality of coding units (CUs) based on a detected object boundary in the CTU. A video encoding processor receives the video frame, the object recognition data and the partitioned CUs and encodes the CUs with at least one of a resolution or quantization parameter determined at least in part on whether the CU includes an object.
1 . A method of encoding a video signal comprising:
receiving a video frame comprising a plurality of pixels;
partitioning the video frame into a plurality of coding tree units (CTUs);
performing object detection and image segmentation on the video frame to generate object recognition data and at least one segmentation mask identifying object boundaries;
overlay the segmentation mask with the plurality of CTUs; and
for a CTU in which an object boundary is identified, partition the CTU into at least two coding units (CUs) in which at least one CU contains the object of interest and at least one CU does not contain an object of interest.
2 . The method of claim 1 , further comprising encoding a CU with at least one of a resolution or quantization parameter determined at least in part by whether the CU contains an object of interest.
3 . The method of claim 1 , wherein the partition is selected from the group including a horizontal partition, a vertical partition, and a geometric partition.
4 . The method of claim 1 , wherein the image segmentation is selected from the group including semantic segmentation, instance segmentation, and panoptic segmentation.
5 . The method of claim 1 , wherein the object recognition data includes instance labels for each object detected in the video frame.
6 . The method of claim 1 , wherein the object recognition data includes instance labels for each of the pixels in the video frame.
7 . The method of claim 1 , wherein the object recognition data includes object class, object position in the frame.
8 . The method of claim 7 , wherein the object recognition data further comprises a bounding box of an object.
9 . The method of claim 2 , further comprising the step of motion estimation where the motion estimation is performed at least in part based on the object recognition data.
10 . A video encoder, the video encoder receiving video frame data comprising a plurality of pixels, the video frame being partitioned into a plurality of coding tree units (CTU). the encoder comprising:
an image detection and segmentation processor, the image detection and segmentation processor receiving the video frame and generating object recognition data and at least one image segmentation mask;
a mask to coding block mapping processor mapping the at least one segmentation mask to the CTUs of the video frame and partitioning at least one CTU into a plurality of coding units (CUs) based on a detected object boundary in the CTU;
a video encoding processor, the encoding processor receiving the video frame, the object recognition data and the partitioned CUs and encoding the CUs with at least one of a resolution or quantization parameter determined at least in part on whether the CU includes an object.
11 . The encoder of claim 10 , wherein the mask to coding block mapping processor partitions a CTU using a partition selected from the group including a horizontal partition, a vertical partition and a geometric partition.
12 . The encoder of claim 10 , wherein the image detection and segmentation processor applies an image segmentation method selected from the group including semantic segmentation, instance segmentation, and panoptic segmentation.
13 . The encoder of claim 10 , wherein the object recognition data includes instance labels for each object detected in the video frame.
14 . The encoder of claim 10 , wherein the object recognition data includes instance labels for each of the pixels in the video frame.
15 . The encoder of claim 10 , wherein the object recognition data includes object class, object position in the frame.
16 . The encoder of claim 10 , wherein the object recognition data further comprises a bounding box of an object.
17 . The encoder of claim 10 , further comprising motion estimation processing, wherein motion estimation is performed at least in part based on the object recognition data.
18 . The encoder of claim 10 , wherein the image detection and segmentation processor includes a neural network.