Dynamic depth recovery from multiple synchronized video streams

1. A method of generating a dynamic depth map of a scene from a sequence of sets of segmented images of the scene, including a current set of segmented images and a prior set of segmented images, each set of segmented images representing a plurality of different views of the scene at a respective instant, wherein each image in each set of segmented images includes a respective coordinate set, the method comprising the steps of:

a) determining a temporal correspondence between segments in at least one image in the current set of segmented images and corresponding segments in at least one image of the prior set of segmented images to obtain a temporal depth predicted value for each segment in the current set of segmented images;

b) determining a spatial correspondence among segments in the current set of segmented images to obtain a spatial depth predicted value for each segment in at least one image of the current set of images;

c) for each segment in the at least one image of the current set of segmented images, selecting one of the temporal depth predicted value and the spatial depth predicted value as an initial depth estimate of the segment;

d) generating a measure of difference between the at least one image in the current set of segmented images and each other image in the current set of segmented images;

e) repeating steps c) and d) selecting different initial depth values until a minimum measure of difference is generated in step d).

2. A method according to claim 1 , where in the step of generating the measure of difference between the at least one image in the current set of segmented images and each other image in the current set of segmented images includes the steps of:

warping the at least one image in the current set of segmented images into the coordinate system of each of the other images in the current set of segmented images to obtain a set of respective warped images;

measuring a difference between the warped images and their respective other image in the current set of segmented images and summing the result to obtain a first term;

measuring an average distance between points in corresponding segments of the warped images and the respective other images in the current set of segmented images and summing the result, to obtain a second term; and

summing the first term and the second term to obtain the measure of difference.

3. A method according to claim 1 , wherein:

the step of selecting one of the temporal depth predicted value and the spatial depth predicted value selects the value for one segment in the at least one image while holding the depth values of other, adjacent segments unchanged; and

the step of generating the measure of difference between the at least one image in the current set of images and each other image in the current set of images generates a difference only of the one segment and segments in the at least one image adjacent to the one segment.

4. A method according to claim 1 , wherein the step of determining a temporal correspondence between segments in the at least one image in the current set of segmented image and corresponding segments in the at least one image of the prior set of segmented images includes the steps of:

classifying each segment in the at least one image in the prior set of segmented images as a textured segment or an untextured segment;

for each untextured segment, associating at least one corresponding segment in the at least one image of the current set of segmented images;

for each segment in the at least one image of the current set of segmented images that is not associated with one of the untextured segments, identifying the segment as a textured segment and identifying each textured segment in the at least one image of the current set of images with a corresponding segment from the at least one image in the prior set of images using an optical flow technique; and

for each segment in the at least one image of the current set of segmented images, assigning a depth value of the corresponding segment of the at least one image in the prior set of images as the temporal depth predicted value.

5. A method according to claim 2 , wherein:

the step of selecting one of the temporal depth predicted value and the spatial depth predicted value selects the value for one segment in the at least one image while holding the depth values of other, adjacent segments unchanged; and

the step of generating the measure of difference between the at least one image in the current set of images and each other image in the current set of images generates a difference only of the one segment and segments in the at least one image adjacent to the one segment.

6. A method according to claim 5 , wherein the step of determining a temporal correspondence between segments in the at least one image in the current set of segmented image and corresponding segments in the at least one image of the prior set of segmented images includes the steps of:

classifying each segment in the at least one image in the prior set of segmented images as a textured segment or an untextured segment;

for each untextured segment, associating at least one corresponding segment in the at least one image of the current set of segmented images;

for each segment in the at least one image of the current set of segmented images that is not associated with one of the untextured segments, identifying the segment as a textured segment and identifying each textured segment in the at least one image of the current set of images with a corresponding segment from the at least one image in the prior set of images using an optical flow technique; and

for each segment in the at least one image of the current set of segmented images, assigning a depth value of the corresponding segment of the at least one image in the prior set of images as the temporal depth predicted value.