Method for forming a depth image from digital image data
View Patent ↗A computer vision/image processing method generates a depth map useful in producing a foreground depth mask for 2D/3D image editing. The method uses image data from a plurality of scenes. Feature points on each of the vertical scan lines in each of the scene images are used to search for corresponding feature points on the corresponding vertical lines in other images. The corresponding feature-point search is performed by using a bipartite match network with a feature-point-ordering constraint and a disparity-limit constraint, and produces an individual feature-point depth map for each input image. A sparse feature-point depth map of the scene is obtained after applying a consistency test to all the individual depth maps. A complete feature-point depth map is produced by applying a color property assisted depth propagation process to the sparse feature-point depth map. Foreground and background separation is then conducted in the depth domain by using the order statistics of the depth data extracted the feature-point depth map. A foreground feature-point depth map is obtained from the separation operation. The final foreground depth mask is generated by applying a color aided eight-nearest-neighbor LMS interpolation process to the foreground feature-point depth map.
1. A method for generating a depth map of a scene from at least first and second intensity images of the same scene taken from different perspectives, said method comprising the steps of:
(a) identifying corresponding feature points characteristic of edges in the respective images;
(b) generating intensity segments characteristic of intensity values between feature points in each of the respective images:
(c) generating matches between intensity segments in said first intensity image and corresponding intensity segments in said second intensity image, to provide pairs of intensity segments;
(d) eliminating one or more unlikely pairs of intensity segments from step (c) to reduce computational complexity; and
(e) generating a depth map from the sparse matrix representation.
2. A method for generating a depth map of a scene from at least two intensity images of the same scene taken from different perspectives, said method comprising the steps of:
(a) identifying feature points characteristic of edges in the respective images;
(b) generating intensity segments characteristic of intensity values between feature points in the respective images: (c) generating matches between pairs of intensity segments in the respective images;
(d) introducing constraints to the step (c) in order to eliminate unlikely pairing and thereby reduce computational complexity; and
(e) generating a depth map from the sparse matrix representation;
wherein the step (d) of introducing constraints comprises use at least one of a disparity limit constraint based on the image locations of the features related to a given pair of intensity values and an ordering constraint for eliminating false positive matches.
3. A method for generating a depth map of a scene from at least two intensity images of the same scene taken from different perspectives, said method comprising the steps of:
(a) identifying feature points characteristic of edges in the respective images;
(b) generating intensity segments characteristic of intensity values between feature points in the respective images:
(c) generating matches between pairs of intensity segments in the respective images;
(d) introducing constraints to the step (c) in order to eliminate unlikely pairing and thereby reduce computational complexity; and
(e) generating a depth map from the sparse matrix representation;
wherein the step (e) comprises propagating additional values in the depth map based on the color properties of the intensity images, including at least one of the hue and color purity.
4. A method for forming a feature point depth map from at least two intensity images, said method comprising the steps of:
(a) identifying in each of the intensity images feature points that are edge points;
(b) selecting one of the images as a reference image;
(c) extracting an intensity segment between every two consecutive feature points on one of the vertical scan lines in the selected reference image;
(d) on a corresponding vertical scan line in another image, extracting intensity segments between each of the feature points to a point that makes each of the extracted segments have the same length as the segment stated in (c);
(e) pairing the segment in (c) with each segment in (d) and computing a match score for each pair;
(f) forming a score matrix for segments on the said scan line in the reference image;
(g) feeding the match score into a match network to find best matches for the segments on the said scan line in the reference image;
(h) deleting false positive matches based on a feature point ordering constraint;
(i) computing a feature-point depth map for each reference image based on the matched feature; and
(j) applying a consistency test to all the feature-point depth maps to form a feature-point map for the scene.
5. A method of forming a foreground depth mask from at least two intensity images for 2D/3D image editing, the method comprising the steps of:
(a) identifying in each of the intensity images feature points that are edge points;
(b) selecting one of the images as a reference image;
(c) extracting an intensity segment between every two consecutive feature points on one of the vertical scan lines in the selected reference image;
(d) on the corresponding vertical scan line in another image, extracting intensity segments between each of the feature points to a point that makes each of the extracted segments have the same length as the segment stated in (c);
(e) pairing the segment in (c) with each segment in (d) and computing a match score for each pair;
(f) forming a score matrix for segments on the said scan line in the reference image;
(g) feeding the match score into a match network to find best matches for the segments on the said scan line in the reference image;
(h) deleting false positive matches based on the feature point ordering constraint;
(i) computing a feature-point depth map for each reference image based on the matched feature points;
(j) applying a consistency test to all the feature-point depth maps to form a single feature-point map for the scene;
(k) forming a complete feature-point depth map by propagating additional depth values from feature points having depth values to those having no depth values based on a color similarity property;
(l) separating foreground objects from background objects based on the order statistics extracted from the feature-point depth map, thereby forming a foreground feature-point depth map; and
(m) forming a foreground depth mask by performing depth interpolation in the foreground feature-point depth map based on a color similarity property.
6. A method as claimed in claim 5 , further comprising determining, prior to step (e), whether a pair of segments is practically feasible.
7. A method as claimed in claim 6 , wherein the step of determining whether a pair of segments is practically feasible is performed by utilizing a disparity limit analysis.
8. A method as claimed in claim 5 , further comprising the following step prior to step (i): repeating step (c) through step (e) for substantially all of the scan lines in the reference image.
9. A method as claimed in claim 5 , further comprising the following step prior to step (j): repeating step (b) through step (i) for every image that has not been selected as a reference image.
10. The method as claimed in claim 5 , wherein the consistency testing is a logic AND operation.
11. The method as claimed in claim 5 , wherein the color similarity testing is conducted in the hue and purity space.
12. The method as claimed in claim 5 , wherein the depth interpolation is performed by using a length extendable eight-nearest-neighbor LMS estimation scheme.
13. A method for generating a depth map of a scene from at least two intensity images of the same scene taken from different perspectives, said method comprising the steps of:
(a) identifying feature points characteristic of edges in the respective images;
(b) generating intensity segments characteristic of intensity values between feature points in the respective images;
(c) generating matches between pairs of intensity segments in the respective images;
(d) eliminating one or more unlikely pairs of intensity segments from step (c) to reduce computational complexity; and
(e) generating a depth map from the sparse matrix representation;
wherein said eliminating further comprises use of a disparity limit constraint based on the image locations of the features related to a given pair of intensity values and an ordering constraint for eliminating false positive matches.
14. A method for generating a death map of a scene from at least two intensity images of the same scene taken from different perspectives, said method comprising the steps of:
(a) identifying feature points characteristic of edges in the respective images;
(b) generating intensity segments characteristic of intensity values between feature points in the respective images;
(c) generating matches between pairs of intensity segments in the respective images;
(d) eliminating one or more unlikely pairs of intensity segments from step (c) to reduce computational complexity; and
(e) generating a depth map from the sparse matrix representation;
wherein the step (e) comprises propagating additional values in the depth map based on the color properties of the intensity images, including at least one of the hue and color purity.