Method and system for optimizing image and video compression for machine vision
A method and a system described herein provide optimizing image and/or video compression for machine perception. According to an aspect, the method comprises receiving a raw image frame from a camera sensor; detecting a predefined object in the raw image frame and marking a region around the predefined object within the raw image frame as ROI. Based on the ROI, a partitioning scheme, a prediction mode, and quantization parameter are determined for improving coding efficiency. Machine perception efficiency is improved by selecting a quantization parameter table used for compressing and encoding the raw image or video frame based on a selected machine vision task. The selection of the quantization parameter table is based on training of the selected machine vision task using cost function optimization.
1. A computer-implemented method for optimizing image and/or video compression for machine perception, the method comprising:
receiving a raw image or video frame from a camera sensor;
selecting a machine vision task to be performed;
based on the selected machine vision task, selecting a quantization parameter table used for compressing and encoding the raw image or video frame; and
encoding the raw image or video frame using the selected quantization parameter table;
wherein selection of the quantization parameter table is based on training of the selected machine vision task using cost function optimization.
2. The computer-implemented method of claim 1 , wherein training of a machine vision task using cost function optimization comprises modifying an objective function of the machine vision task that is to be minimized, by adding additional terms that correspond to optimization of a quantization parameter table in the image or video encoder.
3. The computer-implemented method of claim 2 , wherein a combined objective function to be minimized is given by:
ℒ
=
ℒ
task
+
λ
1
Q
c
2
+
1
Q
l
2
wherein task is the objective function corresponding to the machine vision task, Q c is a quantization table for a chroma channel, Q l is a quantization table for a luma channel, and λ is a parameter for enabling a control mechanism to maintain a balance between necessary levels of task performance and required compression rate.
4. The computer-implemented method of claim 2 , wherein a combined objective function to be minimized is given by:
ℒ
=
ℒ
task
+
λ
c
2
1
Q
c
2
+
λ
l
2
1
Q
l
2
wherein task is the objective function corresponding to the machine vision task, Q c is a quantization table for a chroma channel, Q l is a quantization table for a luma channel, and λ c and Δ l are hyper parameters for controlling levels of chroma and luma compression level, respectively, and wherein the luma channel is compressed to a lower level while the chroma channels is compressed to a higher level.
5. The computer-implemented method of claim 2 , wherein a combined objective function to be minimized is given by:
ℒ
=
ℒ
task
+
λ
c
2
1
W
c
·
Q
c
2
+
λ
l
2
1
W
l
·
Q
l
2
wherein task is the objective function corresponding to the machine vision task, Q c is a quantization table for a chroma channel, Q l is a quantization table for a luma channel, W c and W l are weight matrices for ensuring optimizing quantizing of high frequency components, and λ c and λ l are hyper parameters for controlling levels of chroma and luma compression level, respectively.
6. The computer-implemented method of claim 1 , further comprising optimizing a transformation parameter for enabling performing the machine vision task to be performed on the encoded image or video data.
7. The computer-implemented method of claim 1 , wherein the method is implemented in an autonomous driving system, remote driving system and/or advanced driver assistance system.
8. A computer-readable medium comprising computer-readable instructions, that, when executed by at least one processor, cause the at least one processor to perform a method, the method comprising:
receiving a raw image or video frame from a camera sensor;
selecting a machine vision task to be performed;
based on the selected machine vision task, selecting a quantization parameter table used for compressing and encoding the raw image or video frame; and
encoding the raw image or video frame using the selected quantization parameter table;
wherein selection of the quantization parameter table is based on training of the selected machine vision task using cost function optimization.
9. The computer-readable medium of claim 8 , wherein training of a machine vision task using cost function optimization comprises modifying an objective function of the machine vision task that is to be minimized, by adding additional terms that correspond to optimization of a quantization parameter table in the image or video encoder.
10. The computer-readable medium of claim 9 , wherein a combined objective function to be minimized is given by:
ℒ
=
ℒ
task
+
λ
1
Q
c
2
+
1
Q
l
2
wherein task is the objective function corresponding to the machine vision task, Q c is a quantization table for a chroma channel, Q l is a quantization table for a luma channel, and λ is a parameter for enabling a control mechanism to maintain a balance between necessary levels of task performance and required compression rate.
11. The computer-readable medium of claim 9 , wherein a combined objective function to be minimized is given by:
ℒ
=
ℒ
task
+
λ
c
2
1
Q
c
2
+
λ
l
2
1
Q
l
2
wherein task is the objective function corresponding to the machine vision task, Q c is a quantization table for a chroma channel, Q l is a quantization table for a luma channel, and λ and λ l are hyper parameters for controlling levels of chroma and luma compression level, respectively, and wherein the luma channel is compressed to a lower level while the chroma channels is compressed to a higher level.
12. The computer-readable medium of claim 9 , wherein a combined objective function to be minimized is given by:
ℒ
=
ℒ
task
+
λ
c
2
1
W
c
·
Q
c
2
+
λ
l
2
1
W
l
·
Q
l
2
wherein task is the objective function corresponding to the machine vision task, Q c is a quantization table for a chroma channel, Q l is a quantization table for a luma channel, W c and W l are weight matrices for ensuring optimizing quantizing of high frequency components, and λ, and λ l are hyper parameters for controlling levels of chroma and luma compression level, respectively.
13. The computer-readable medium of claim 8 , further comprising optimizing a transformation parameter for enabling performing the machine vision task to be performed on the encoded image or video data.
14. An image/video pre-processing system for optimizing image and/or video compression for machine perception, the image/video pre-processing system being configured to:
receive a raw image or video frame from a camera sensor;
select a machine vision task to be performed;
based on the selected machine vision task, select a quantization parameter table used for compressing and encoding the raw image or video frame; and
encode the raw image or video frame using the selected quantization parameter table;
wherein selection of the quantization parameter table is based on training of the selected machine vision task using cost function optimization.
15. The image/video pre-processing system of claim 14 , wherein training of a machine vision task using cost function optimization comprises modifying an objective function of the machine vision task that is to be minimized, by adding additional terms that correspond to optimization of a quantization parameter table in the image or video encoder.
16. The computer-implemented method of claim 15 , wherein a combined objective function to be minimized is given by:
ℒ
=
ℒ
task
+
λ
1
Q
c
2
+
1
Q
l
2
wherein task is the objective function corresponding to the machine vision task, Q c is a quantization table for a chroma channel, Q l is a quantization table for a luma channel, and λ is a parameter for enabling a control mechanism to maintain a balance between necessary levels of task performance and required compression rate.
17. The computer-implemented method of claim 15 , wherein a combined objective function to be minimized is given by:
ℒ
=
ℒ
task
+
λ
c
2
1
Q
c
2
+
λ
l
2
1
Q
l
2
wherein task is the objective function corresponding to the machine vision task, Q c is a quantization table for a chroma channel, Q l is a quantization table for a luma channel, and λ and λ l are hyper parameters for controlling levels of chroma and luma compression level, respectively, and wherein the luma channel is compressed to a lower level while the chroma channels is compressed to a higher level.
18. The computer-implemented method of claim 15 , wherein a combined objective function to be minimized is given by:
ℒ
=
ℒ
task
+
λ
c
2
1
W
c
·
Q
c
2
+
λ
l
2
1
W
l
·
Q
l
2
wherein task is the objective function corresponding to the machine vision task, Q c is a quantization table for a chroma channel, Q l is a quantization table for a luma channel, W c and W l are weight matrices for ensuring optimizing quantizing of high frequency components, and λ and λ l are hyper parameters for controlling levels of chroma and luma compression level, respectively.
19. The computer-implemented method of claim 14 , further comprising optimizing a transformation parameter for enabling performing the machine vision task to be performed on the encoded image or video data.
20. The computer-implemented method of claim 14 , wherein the method is implemented in an autonomous driving system, remote driving system and/or advanced driver assistance system.