Joint objects image signal processing in temporal domain
The present disclosure relates to pre-processing of video images. In particular, the video images are pre-processed in an object-based manner, i.e., by applying different pre-processing to different objects detected in the image. Moreover, the pre-processing is applied to a group of images. As such, object detection is performed in a plurality of images and the pre-processing for the plurality of images may be adapted to the decoded images and is applied to the decoded images.
1 . A method for processing frames of a video sequence in raw image format, the method comprising:
performing pre-processing on at least two respective frames of the video sequence, wherein the pre-processing includes filtering the at least two respective frames with a filter of which parameters are set according to a size of an object to be detected in the at least two respective frames;
identifying the object included in image regions of the at least two respective frames of the video sequence, wherein identifying the object comprises:
detecting a location of the object within the at least two respective frames by distinguishing the object from other parts of the at least two respective frames; and
recognizing an identity of the object in the at least two respective frames based on detecting the location of the object within the at least two respective frames, wherein the performing pre-processing on the at least two respective frames of the video sequence is performed before the recognizing the identity of the object; and
processing the image regions in the at least two respective frames that include the object by a first image processing adapted to the object and different from a second image processing applied to image regions in the at least two respective frames not including the object, wherein the first image processing includes de-noising with a de-noising filter of which at least one parameter is determined based on an object bounding box of the object and a size of the object bounding box included in the at least two respective frames, and wherein the at least one parameter includes a volume in a number of samples of the object bounding box.
2 . The method according to claim 1 , wherein the recognizing the identity of the object comprises:
computing a plurality of feature vectors for a plurality of image regions in the at least two respective frames, wherein computing of a feature vector of the plurality of feature vectors for a given frame includes determining a value of at least one feature of the image region of the given frame; and
forming a cluster based on the feature vectors, wherein the cluster includes image regions of the at least two respective frames, the image regions including the object with the same recognized identity.
3 . The method according to claim 2 ,
wherein the forming the cluster is performed by K-means approach; and/or
wherein the forming the cluster is based on determining a similarity measure of feature vectors calculated for the image regions in different frames among the at least two respective frames, wherein the similarity measure employed is one of Euclidean distance, Chebyschev distance, or cosine similarity.
4 . The method according to claim 1 ,
wherein the identifying the object further comprises:
detecting one or more classes of the object; and
wherein the recognizing of the identity of the object is based on at least one of the one or more detected classes of the object.
5 . The method according to claim 4 , wherein the detecting of the location of the object and the detecting the one or more classes of the object is performed by a YOLO (You Only Look Once) object detection algorithm, a mobileNet object detection algorithm, a SSD (Single Shot Multibox Detector) object detection algorithm, a SSH (Single Stage Headless) face detection algorithm, or a MTCNN (Multi-task Cascaded Convolutional Neural Network) face detection algorithm.
6 . The method according to claim 1 , wherein the performing the pre-processing further includes:
filtering the at least two respective frames with a filter adapted to a type of the identity of the object.
7 . The method according to claim 1 , further comprising:
obtaining the at least two respective frames from an image sensor;
wherein the first image processing and/or the second image processing comprises performing at least one of:
defect pixel correction,
white balance,
de-noising,
demosaicing,
color space correction,
color enhancement,
contrast enhancement,
sharpening, or
color transformation.
8 . The method according to claim 1 , wherein the raw image format is a Bayer pattern and the performing pre-processing includes conversion of the at least two respective frames into an RGB (red-green-blue) image format.
9 . The method according to claim 1 , wherein the at least two respective frames are:
temporally adjacent frames; or
more than two frames equally spaced in a time domain.
10 . The method according to claim 1 , further comprising:
encoding the at least two respective frames of the video sequence by applying lossy and/or lossless compression.
11 . A non-transitory computer-readable storage medium that stores a computer program that, when executed on one or more processors, causes the one or more processors to execute operations comprising:
performing pre-processing on at least two respective frames of a video sequence, wherein the pre-processing includes filtering the at least two respective frames with a filter of which parameters are set according to a size of an object to be detected in the at least two respective frames;
identifying the object included in image regions of the at least two respective frames of the video sequence, wherein identifying the object comprises:
detecting a location of the object within the at least two respective frames by distinguishing the object from other parts of the at least two respective frames; and
recognizing an identity of the object in the at least two respective frames based on detecting the location of the object within the at least two respective frames, wherein the performing pre-processing on the at least two respective frames of the video sequence is performed before the recognizing the identity of the object; and
processing the image regions in the at least two respective frames that include the object by a first image processing adapted to the object and different from a second image processing applied to image regions in the at least two respective frames not including the object, wherein the first image processing includes de-noising with a de-noising filter of which at least one parameter is determined based on an object bounding box of the object and a size of the object bounding box included in the at least two respective frames, and wherein the at least one parameter includes a volume in a number of samples of the object bounding box.
12 . An apparatus for processing frames of a video sequence in raw image format, the apparatus comprising:
processing circuitry configured to:
perform pre-processing on at least two respective frames of the video sequence, wherein the pre-processing includes filtering the at least two respective frames with a filter of which parameters are set according to a size of an object to be detected in the at least two respective frames;
identify the object in image regions of the at least two respective frames of the video sequence, wherein identifying the object comprises:
detecting a location of the object within the at least two respective frames by distinguishing the object from other parts of the at least two respective frames; and
recognizing an identity of the object in the at least two respective frames based on detecting the location of the object within the at least two respective frames, wherein the performing pre-processing on the at least two respective frames of the video sequence is performed before the recognizing the identity of the object; and
process the image regions in the at least two respective frames that include the object by a first image processing adapted to the object and different from a second image processing applied to image regions in the at least two respective frames not including the object, wherein the first image processing includes de-noising with a de-noising filter of which at least one parameter is determined based on an object bounding box of the object and a size of the object bounding box included in the at least two respective frames, and wherein the at least one parameter includes a volume in a number of samples of the object bounding box.
13 . The apparatus according to claim 12 , further comprising:
an image sensor for capturing the video sequence in the raw image format.
14 . The method according to claim 1 , wherein the location of the object is a location of a bounding box framing the object or a pixel map.
15 . The computer-readable storage medium according to claim 11 , wherein the recognizing the identity of the object comprises:
computing a plurality of feature vectors for a plurality of image regions in the at least two respective frames, wherein computing of a feature vector of the plurality of feature vectors for a given frame includes determining a value of at least one feature of the image region of the given frame; and
forming a cluster based on the feature vectors, wherein the cluster includes image regions of the at least two respective frames, the image regions including the object with the same recognized identity.
16 . The computer-readable storage medium according to claim 15 ,
wherein the forming the cluster is performed by K-means approach; and/or
wherein the forming the cluster is based on determining a similarity measure of feature vectors calculated for the image regions in different frames among the at least two respective frames, wherein the similarity measure employed is one of Euclidean distance, Chebyschev distance, or cosine similarity.
17 . The computer-readable storage medium according to claim 11 ,
wherein the identifying the object further comprises:
detecting one or more classes of the object; and
wherein the recognizing of the identity of the object is based on at least one of the one or more detected classes of the object.
18 . The apparatus according to claim 12 , wherein the recognizing the identity of the object comprises:
computing a plurality of feature vectors for a plurality of image regions in the at least two respective frames, wherein computing of a feature vector of the plurality of feature vectors for a given frame includes determining a value of at least one feature of the image region of the given frame; and
forming a cluster based on the feature vectors, wherein the cluster includes image regions of the at least two respective frames, the image regions including the object with the same recognized identity.
19 . The apparatus according to claim 18 ,
wherein the forming the cluster is performed by K-means approach; and/or
wherein the forming the cluster is based on determining a similarity measure of feature vectors calculated for the image regions in different frames among the at least two respective frames, wherein the similarity measure employed is one of Euclidean distance, Chebyschev distance, or cosine similarity.
20 . The apparatus according to claim 12 ,
wherein the identifying the object further comprises:
detecting one or more classes of the object; and
wherein the recognizing of the identity of the object is based on at least one of the one or more detected classes of the object.