Method and apparatus for generating temporally consistent superpixels
View Patent ↗A method and an apparatus for generating superpixels for a sequence of frames. A feature space of the sequence of frames is separated into a color subspace and a spatial subspace. A clustering is then performed in the spatial subspace on a frame basis. In the color subspace a clustering is performed on stacked frames. An erroneous temporal consistency of a superpixel associated to a frame is detected by performing a similarity check between a corresponding superpixel in one or more past frames and one or more future frames using two or more metrics. The affected superpixels is future frames are corrected accordingly.
1. A method for generating temporally consistent superpixels associated to a frame of a sequence of frames, the method comprising:
separating a feature space of the sequence of frames into a color subspace and a spatial subspace;
performing a clustering in the spatial subspace on a frame basis;
performing a clustering in the color subspace on stacked frames; and
detecting an erroneous temporal consistency of a superpixel associated to a frame by performing a similarity check between a corresponding superpixel in one or more past frames and one or more future frames using two or more metrics.
2. The method according to claim 1 , wherein the two or more metrics include color histogram comparisons, descriptor matching, and color differences.
3. The method according to claim 2 , wherein chi-square distance or correlation is analyzed for color histogram comparison.
4. The method according to claim 2 , wherein descriptor matching is performed using scale-invariant feature transform descriptor matching.
5. The method according to claim 2 , wherein color differences are analyzed using Euclidean distances.
6. The method according to claim 1 , wherein the two or more metrics are compared with associated thresholds to detect an erroneous temporal consistency of a superpixel.
7. The method according to claim 1 , wherein in case an erroneous temporal consistency is detected for a superpixel a new label is assigned to this superpixel for future frames.
8. An apparatus configured to generate temporally consistent superpixels associated to frames of a sequence of frames, wherein the apparatus comprises:
a separator configured to separate a feature space of the sequence of frames into a color subspace and a spatial subspace;
a spatial clustering unit configured to perform a clustering in the spatial subspace on a frame basis;
a color clustering unit configured to perform a clustering in the color subspace on stacked frames; and
a detector configured to detect an erroneous temporal consistency of a superpixel associated to a frame by performing a similarity check between a corresponding superpixel in one or more past frames and one or more future frames using two or more metrics.
9. A computer readable non-transitory storage medium having stored therein instructions enabling generating temporally consistent superpixels associated to frames of a sequence of frames, which, when executed by a computer, cause the computer to:
separate a feature space of the sequence of frames into a color subspace and a spatial subspace;
perform a clustering in the spatial subspace on a frame basis;
perform a clustering in the color subspace on stacked frames;
detect an erroneous temporal consistency of a superpixel associated to a frame by performing a similarity check between a corresponding superpixel in one or more past frames and one or more future frames using two or more metrics.
10. The apparatus according to claim 8 , wherein the two or more metrics include color histogram comparisons, descriptor matching, and color differences.
11. The apparatus according to claim 10 , wherein the detector is configured to analyze chi-square distance or correlation for color histogram comparison.
12. The apparatus according to claim 10 , wherein the detector is configured to perform descriptor matching using scale-invariant feature transform descriptor matching.
13. The apparatus according to claim 10 , wherein the detector is configured to analyze color differences using Euclidean distances.
14. The apparatus according to claim 8 , wherein the detector is configured to compare the two or more metrics with associated thresholds to detect an erroneous temporal consistency of a superpixel.
15. The apparatus according to claim 8 , wherein the apparatus is configured to assign a new label to a superpixel for future frames in case an erroneous temporal consistency is detected for this superpixel.
16. The computer readable non-transitory storage medium according to claim 9 , wherein the two or more metrics include color histogram comparisons, descriptor matching, and color differences.
17. The computer readable non-transitory storage medium according to claim 16 , wherein the instructions cause the computer to analyze chi-square distance or correlation for color histogram comparison.
18. The computer readable non-transitory storage medium according to claim 16 , wherein the instructions cause the computer to perform descriptor matching using scale-invariant feature transform descriptor matching.
19. The computer readable non-transitory storage medium according to claim 16 , wherein the instructions cause the computer to analyze color differences using Euclidean distances.
20. The computer readable non-transitory storage medium according to claim 9 , wherein the instructions cause the computer to compare the two or more metrics with associated thresholds to detect an erroneous temporal consistency of a superpixel.
21. The computer readable non-transitory storage medium according to claim 9 , wherein the instructions cause the computer to assign a new label to a superpixel for future frames in case an erroneous temporal consistency is detected for this superpixel.