Graph-based segmentation integrating visible and NIR information
View Patent ↗A method for segmenting an image includes extracting unary potentials for pixels of the input image. These can be based for each of a set of possible labels, on information for a first channel in the image, such as in the visible range of the spectrum. Pairwise potentials are extracted for neighboring pairs of pixels of the image. These can be based on information for a second channel in the image, such as in the infrared range of the spectrum. An objective function is optimized over pixels of the input image to identify labels for the pixels. The objective function is based on a combination of ones of the extracted unary and pairwise potentials. The image is then segmented, based on the identified pixel labels. The method and system can provide an improvement in segmentation over methods which use only the visible information.
1. A method for segmenting an image comprising:
receiving an input image to be segmented; and
with a processor:
for each label in a set of labels, assigning unary potentials to pixels of the input image, the unary potentials being based on information for a first channel in the image;
for each of pairs of labels in the set of labels, assigning pairwise potentials for neighboring pixels of the image, the pairwise potentials being based on information for a second channel in the image, different from the first channel;
optimizing an objective function over pixels of the input image to identify labels for pixels of the image from the set of labels, the objective function being an aggregation of ones of the unary potentials and pairwise potentials; and
segmenting the input image based on the identified labels.
2. The method of claim 1 , further comprising outputting the segmented image or information based thereon.
3. The method of claim 1 , wherein the first channel is within a first range of the electromagnetic spectrum and the second channel is within a second range of the electromagnetic spectrum.
4. The method of claim 3 , wherein the first channel comprises a wavelength range in the visible range of the electromagnetic spectrum and the second channel comprises a wavelength range in the infrared range of the electromagnetic spectrum.
5. The method of claim 1 , wherein the first channel comprises only visible information from a visible range of the electromagnetic spectrum.
6. The method of claim 1 , wherein the second channel comprises only non-visible information from a non-visible range of the electromagnetic spectrum.
7. The method of claim 1 , wherein the unary potentials for a pixel express, for each of the labels in the set of labels, a cost of assigning the label to the pixel, based on the first channel information.
8. The method of claim 1 , wherein the pairwise potentials for a pair of neighboring pixels express, for each of the combinations of labels in the set of labels, a cost of assigning the combination of labels to the pair of neighboring pixels, based on the second channel information.
9. The method of claim 1 , wherein the assigning pairwise potentials for neighboring pixels comprises defining a graph over pixels of the image in which each pixel is joined by a set of edges to its neighboring pixels, each pixel having at least two neighboring pixels, the edges representing the pairwise potentials between the neighboring pixels.
10. The method of claim 9 , wherein each pixel has from 4-8 neighbors in the graph.
11. The method of claim 1 , wherein the objective function is optimized by minimizing an optionally weighted sum of terms given at least two labels in the set of labels, a first of the energy terms being based on the unary potentials and a second of the energy terms being based on the pairwise potentials.
12. The method of claim 1 , wherein the optimizing of the objective function over pixels of the input image comprises minimizing an objective function of the form:
E
(
L
)
=
∑
i
∈
V
u
i
(
l
i
)
+
λ
∑
(
i
,
j
)
∈
K
v
i
,
j
(
l
i
,
l
j
)
(
1
)
where E(L) represents the energy value to be minimized over the set of labels L;
∑
i
∈
V
u
i
(
l
i
)
represents a unary term, which is a sum over all pixels V of respective unary potentials, each unary potential being represented by a cost u i (l i ) of assigning a label l i to a pixel i where l i is selected from the set of labels L;
∑
(
i
,
j
)
∈
K
v
i
,
j
(
l
i
,
l
j
)
represents a pairwise term, which is a sum over all pairs of neighboring pixels of respective pairwise potentials, each pairwise potential being represented by a cost v i,j (l i ,l j ) of assigning a label l i to a pixel i and a label l j , selected from the set of labels L to a neighboring pixel j, where i,j are selected from a set of neighboring pixels; and
λ is a weighting parameter.
13. The method of claim 1 , wherein the objective function is optimized by an approximation method which computes an approximate joint labeling solution which optimizes the objective function.
14. The method of claim 1 , wherein the pairwise potential of neighboring pixels is lowest when the labels of the two pixels are the same and, where the two labels are not the same, the pairwise potential is computed as a function of the difference between observed pixel values in the second channel, such that the pairwise potential decreases as a difference between the observed pixel values in the second channel increases.
15. The method of claim 1 , wherein the labels correspond to object classes and the unary energy potentials are based on information provided by classifiers trained on images comprising regions of pixels labeled with labels selected from the set of labels.
16. The method of claim 1 , wherein the labels correspond to the labels relevant and non-relevant according to a given task image.
17. The method of claim 1 , wherein the labels correspond to the labels salient and non-salient, and the unary potentials are based on the output of a saliency detector on the first channel information of the input image.
18. The method of claim 1 , wherein the pixels of the input image each comprise a group of pixels in the input image.
19. A graph-based image segmentation system comprising:
memory which is configured for storing a graphical representation of an input image;
a unary component for assigning unary potentials for pixels of the image over labels from a set of labels, the unary potentials being based on information for a first channel in the image;
a pairwise component for assigning pairwise potentials for neighboring pixels of the image over pairs of labels from the set of labels, the neighboring pixels being defined by the graphical representation, the pairwise potentials being based on information for a second channel in the image, different from the first channel;
a segmentation component for:
identifying labels for pixels of the image, the labels being selected from a finite set of labels, by optimizing an objective function over pixels of the input image which is based on a combination of the unary and pairwise potentials, and
segmenting the input image based on the identified labels; and
a processor which implements the unary component, pairwise component, and segmentation component.
20. The system of claim 19 , further comprising a component for extracting the first and second channel information from the input image.
21. The system of claim 19 , wherein the objective function is optimized by minimizing an optionally weighted sum of energy terms given a set of at least two labels, a first of the energy terms being based on the unary potentials and a second of the energy terms being based on the pairwise potentials.
22. A graph-based image segmentation method comprising:
receiving an image to be segmented;
defining a graphical representation of pixels of the image in which pairs of neighboring pixels are identified by edges of the graphical representation;
assigning unary potentials to pixels in the graphical representation, the unary potentials being based on information for a visible channel in the image;
assigning pairwise potentials to the pairs of neighboring pixels of the graphical representation, the pairwise potentials being based on information for an infrared channel in the image;
identifying labels for pixels of the image, the labels being selected from a finite set of labels, by optimizing an objective function over the pixels in the graphical representation, the objective function aggregating ones of the unary potentials and the pairwise potentials, for labels from the set of labels; and
segmenting the input image based on the identified labels; and
wherein at least one of the defining, assigning unary potentials, assigning pairwise potentials, and identifying labels is performed with a processor.