Method, apparatus and computer program product for disparity estimation of foreground objects in images

1. A method comprising:

facilitating receipt of a first image and a second image of a scene, the first image and the second image being different view images of the scene;

determining a cost volume between the first image and the second image for a set of foreground labels and a set of background labels, the cost volume comprising matching costs of pixels in the first image and corresponding pixels in the second image for the set of foreground labels and for the set of background labels;

determining a reduced cost volume from the cost volume, the reduced cost volume comprising matching costs of the pixels in the first image and the corresponding pixels in the second image for the set of foreground labels and for a background label of the set of background labels, wherein a matching cost of an individual pixel of pixels in the first image and a corresponding pixel of the individual pixel in the second image for the background label comprises a minimum matching cost from a set of matching costs of the individual pixel for the set of background labels; and

generating a disparity map by performing at least a cost aggregation of the reduced cost volume in the first image and the second image.

2. The method as claimed in claim 1 , wherein the cost aggregation is a tree based aggregation and wherein generating the disparity map comprises:

generating a first disparity map comprising disparity labels at pixels in the first image, wherein generating the first disparity map at least comprises:

performing the tree based aggregation of the reduced cost volume in the first image to determine a first aggregated cost volume; and

assigning the disparity labels at the pixels in the first image from the set of foreground labels and the background label based on the first aggregated cost volume;

generating a second disparity map comprising disparity labels at pixels in the second image, wherein generating the second disparity map at least comprises:

performing the tree based aggregation of the reduced cost volume in the second image to determine a second aggregated cost volume; and

assigning the disparity labels at the pixels in the second image from the set of foreground labels and the background label based on the second aggregated cost volume; and

generating the disparity map based on the first disparity map and the second disparity map.

3. The method as claimed in claim 2 , further comprising:

determining a first confidence map for the disparity labels at the pixels in the first image in the first disparity map based on the first aggregated cost volume;

determining a second confidence map for the disparity labels at the pixels in the second image in the second disparity map based on the second aggregated cost volume;

filtering the disparity labels at the pixels in the first image in the first disparity map based on the first confidence map to generate a first filtered disparity map; and

filtering the disparity labels at the pixels in the second image in the second disparity map based on the second confidence map to generate a second filtered disparity map.

4. The method as claimed in claim 3 , further comprising:

determining occlusion pixels by comparing the first filtered disparity map and the second filtered disparity map;

assigning disparity labels at the occlusion pixels; and

combining the first filtered disparity map, the second filtered disparity map and the assigned disparity labels at the occlusion pixels to generate the disparity map.

5. The method as claimed in claim 2 , wherein generating the disparity map based on the first disparity map and the second disparity map comprises:

generating a third disparity map from the first disparity map, wherein generating the third disparity map at least comprises:

determining one or more pixels in the first image that are labeled as the background label;

calculating a first minimum data cost and a second minimum data cost individually for the one or more pixels in the first image from the first aggregated cost volume;

determining first ratio of the second minimum data cost and the first minimum data cost individually for the one or more pixels in the first image;

determining, individually, whether the one or more pixels in the first image are correctly labeled as the background label based on respective first ratio; and

updating, a disparity label at an individual pixel of the one or more pixels in the first image if the individual pixel is incorrectly labeled as the background label, wherein the disparity label at the individual pixel is updated with a disparity label corresponding to a respective second minimum data cost for the individual pixel;

generating a fourth disparity map from the second disparity map, wherein generating the fourth disparity map at least comprises:

determining one or more pixels in the second image that are labeled as the background label;

calculating a first minimum data cost and a second minimum data cost individually for the one or more pixels in the second image from the second aggregated cost volume;

determining second ratio of the second minimum data cost and the first minimum data cost individually for the one or more pixels in the second image;

determining, individually, whether the one or more pixels in the second image are correctly labeled as the background label based on the respective second ratio; and

updating, a disparity label at an individual pixel of the one or more pixels in the second image if the individual pixel is incorrectly labeled as the background label, wherein the disparity label at the individual pixel is updated with a disparity label corresponding to a respective second minimum data cost for the individual pixel; and

generating the disparity map based on the third disparity map and the fourth disparity map.

6. The method as claimed in claim 5 , wherein generating the disparity map based on the third disparity map and the fourth disparity map comprises:

determining occlusion pixels by comparing the third disparity map and the fourth disparity map;

assigning disparity labels at the occlusion pixels; and

combining the third disparity map, the fourth disparity map and the assigned disparity labels at the occlusion pixels to generate the disparity map.

7. The method as claimed in claim 5 , wherein generating the disparity map based on the third disparity map and the fourth disparity map comprises:

determining a first confidence map for the disparity labels at the pixels in the first image in the third disparity map based on the first aggregated cost volume;

determining a second confidence map for the disparity labels at the pixels in the second image in the fourth disparity map based on the second aggregated cost volume;

filtering the disparity labels at the pixels in the first image in the third disparity map based on the first confidence map to generate a first filtered disparity map; and

filtering the disparity labels at the pixels in the second image in the fourth disparity map based on the second confidence map to generate a second filtered disparity map.

8. The method as claimed in claim 3 , wherein each of the first confidence map and the second confidence map comprises a plurality of confidence values for pixels in the first image and the second image, respectively, wherein a confidence value (C p ) for a pixel p of the pixels is determined based on an expression:

C

p

=

1

-

∑

d

≠

d

p

⁢

ⅇ

-

(

A

⁡

(

p

,

d

)

-

A

⁡

(

p

,

d

p

)

)

2

σ

2

S

-

1

where dp is a disparity label at the pixel p of the pixels, (A(p, d)) is an aggregated data cost of the pixel p at a disparity label d, (A(p, dp)) is an aggregated data cost of the pixel p at the disparity label dp, σ is a constant scaling factor, and S is a total number of disparity labels in the set of foreground labels and the background label.

9. The method as claimed in claim 3 , wherein each of the first filtered disparity map and the second filtered disparity map comprises a plurality of filtered disparity labels for pixels in the first image and the second image, respectively, wherein a filtered disparity label (d pf ) for a pixel p of the pixels is determined as per the expression:

d

pf

=

1

N

p

⁢

∑

q

∈

I

⁢

ⅇ

-

D

⁡

(

p

,

q

)

σ

⁢

d

q

⁢

C

q

where dpf is a filtered disparity label, Np is a normalizing term for the pixel p, D(p, q) is a distance between the pixel p and a pixel q in the tree based aggregation, σ is the constant scaling factor, dq is a disparity label at the pixel q of the pixels, and Cq is a confidence value for the pixel q.

10. The method as claimed in claim 7 , further comprising:

determining occlusion pixels by comparing the first filtered disparity map and the second filtered disparity map;

assigning disparity labels at the occlusion pixels; and

combining the first filtered disparity map, the second filtered disparity map and the assigned disparity labels at the occlusion pixels to generate the disparity map.

11. An apparatus comprising:

at least one processor; and

at least one memory comprising computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to at least perform:

facilitate receipt of a first image and a second image of a scene, the first image and the second image being different view images of the scene;

determine a cost volume between the first image and the second image for a set of foreground labels and a set of background labels, the cost volume comprising matching costs of pixels in the first image and corresponding pixels in the second image for the set of foreground labels and for the set of background labels;

determine a reduced cost volume from the cost volume, the reduced cost volume comprising matching costs of the pixels in the first image and the corresponding pixels in the second image for the set of foreground labels and for a background label of the set of background labels, wherein a matching cost of an individual pixel of pixels in the first image and a corresponding pixel of the individual pixel in the second image for the background label comprises a minimum matching cost from a set of matching costs of the individual pixel for the set of background labels; and

generate a disparity map by performing at least a cost aggregation of the reduced cost volume in the first image and the second image.

12. The apparatus as claimed in claim 11 , wherein the cost aggregation is a tree based aggregation and wherein for generating the disparity map, the apparatus is further caused, at least in part to:

generate a first disparity map comprising disparity labels at pixels in the first image, wherein to generate the first disparity map the apparatus is further configured at least to:

perform the tree based aggregation of the reduced cost volume in the first image to determine a first aggregated cost volume; and

assign the disparity labels at the pixels in the first image from the set of foreground labels and the background label based on the first aggregated cost volume;

generate a second disparity map comprising disparity labels at pixels in the second image, wherein generating the second disparity map at least comprises:

perform the tree based aggregation of the reduced cost volume in the second image to determine a second aggregated cost volume; and

assign the disparity labels at the pixels in the second image from the set of foreground labels and the background label based on the second aggregated cost volume; and

generate the disparity map based on the first disparity map and the second disparity map.

13. The apparatus as claimed in claim 12 , wherein the apparatus is further caused, at least in part to:

determine a first confidence map for the disparity labels at the pixels in the first image in the first disparity map based on the first aggregated cost volume;

determine a second confidence map for the disparity labels at the pixels in the second image in the second disparity map based on the second aggregated cost volume;

filter the disparity labels at the pixels in the first image in the first disparity map based on the first confidence map to generate a first filtered disparity map; and

filter the disparity labels at the pixels in the second image in the second disparity map based on the second confidence map to generate a second filtered disparity map.

14. The apparatus as claimed in claim 13 , wherein the apparatus is further caused, at least in part to:

determine occlusion pixels by comparing the first filtered disparity map and the second filtered disparity map;

assign disparity labels at the occlusion pixels; and

combine the first filtered disparity map, the second filtered disparity map and the assigned disparity labels at the occlusion pixels to generate the disparity map.

15. The apparatus as claimed in claim 12 , wherein for generating the disparity map based on the first disparity map and the second disparity map, the apparatus is further caused, at least in part to:

generate a third disparity map from the first disparity map, wherein to generate the third disparity map, the apparatus is further configured at least to:

determine one or more pixels in the first image that are labeled as the background label;

calculate a first minimum data cost and a second minimum data cost individually for the one or more pixels in the first image from the first aggregated cost volume;

determine first ratio of the second minimum data cost and the first minimum data cost individually for the one or more pixels in the first image;

determine, individually, whether the one or more pixels in the first image are correctly labeled as the background label based on respective first ratio; and

update, a disparity label at an individual pixel of the one or more pixels in the first image if the individual pixel is incorrectly labeled as the background label, wherein the disparity label at the individual pixel is updated with a disparity label corresponding to a respective second minimum data cost for the individual pixel;

generate a fourth disparity map from the second disparity map, wherein to generate the fourth disparity map, the apparatus is further configured at least to:

determine one or more pixels in the second image that are labeled as the background label;

calculate a first minimum data cost and a second minimum data cost individually for the one or more pixels in the second image from the second aggregated cost volume;

determine second ratio of the second minimum data cost and the first minimum data cost individually for the one or more pixels in the second image;

determine, individually, whether the one or more pixels in the second image are correctly labeled as the background label based on the respective second ratio; and

update, a disparity label at an individual pixel of the one or more pixels in the second image if the individual pixel is incorrectly labeled as the background label, wherein the disparity label at the individual pixel is updated with a disparity label corresponding to a respective second minimum data cost for the individual pixel; and

generate the disparity map based on the third disparity map and the fourth disparity map.

16. The apparatus as claimed in claim 15 , wherein for generating the disparity map based on the third disparity map and the fourth disparity map, the apparatus is further caused, at least in part to:

determine occlusion pixels by comparing the third disparity map and the fourth disparity map;

assign disparity labels at the occlusion pixels; and

combine the third disparity map, the fourth disparity map and the assigned disparity labels at the occlusion pixels to generate the disparity map.

17. The apparatus as claimed in claim 15 , wherein for generating the disparity map based on the third disparity map and the fourth disparity map, the apparatus is further caused, at least in part to:

determine a first confidence map for the disparity labels at the pixels in the first image in the third disparity map based on the first aggregated cost volume;

determine a second confidence map for the disparity labels at the pixels in the second image in the fourth disparity map based on the second aggregated cost volume;

filter the disparity labels at the pixels in the first image in the third disparity map based on the first confidence map to generate a first filtered disparity map; and

filter the disparity labels at the pixels in the second image in the fourth disparity map based on the second confidence map to generate a second filtered disparity map.

18. The apparatus as claimed in claim 17 , wherein the apparatus is further caused, at least in part to:

determine occlusion pixels by comparing the first filtered disparity map and the second filtered disparity map;

assign disparity labels at the occlusion pixels; and

combine the first filtered disparity map, the second filtered disparity map and the assigned disparity labels at the occlusion pixels to generate the disparity map.

19. A computer program product comprising at least one non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium comprising a set of instructions, which, when executed by one or more processors, cause an apparatus to at least perform:

facilitate receipt of a first image and a second image of a scene, the first image and the second image being different view images of the scene;

determine a cost volume between the first image and the second image for a set of foreground labels and a set of background labels, the cost volume comprising matching costs of pixels in the first image and corresponding pixels in the second image for the set of foreground labels and for the set of background labels;

determine a reduced cost volume from the cost volume, the reduced cost volume comprising matching costs of the pixels in the first image and the corresponding pixels in the second image for the set of foreground labels and for a background label of the set of background labels, wherein a matching cost of an individual pixel of pixels in the first image and a corresponding pixel of the individual pixel in the second image for the background label comprises a minimum matching cost from a set of matching costs of the individual pixel for the set of background labels; and

generate a disparity map by performing at least a cost aggregation of the reduced cost volume in the first image and the second image.

20. The computer program product as claimed in claim 19 , wherein the cost aggregation is a tree based aggregation and wherein for generating the disparity map, the apparatus is further caused, at least in part to:

generate a first disparity map comprising disparity labels at pixels in the first image, wherein generating the first disparity map at least comprises:

performing the tree based aggregation of the reduced cost volume in the first image to determine a first aggregated cost volume; and

assigning the disparity labels at the pixels in the first image from the set of foreground labels and the background label based on the first aggregated cost volume;

generate a second disparity map comprising disparity labels at pixels in the second image, wherein generating the second disparity map at least comprises:

performing the tree based aggregation of the reduced cost volume in the second image to determine a second aggregated cost volume; and

assigning the disparity labels at the pixels in the second image from the set of foreground labels and the background label based on the second aggregated cost volume; and

generate the disparity map based on the first disparity map and the second disparity map.

21. The computer program product as claimed in claim 20 , wherein the apparatus is further caused, at least in part to:

determine a first confidence map for the disparity labels at the pixels in the first image in the first disparity map based on the first aggregated cost volume;

determine a second confidence map for the disparity labels at the pixels in the second image in the second disparity map based on the second aggregated cost volume;

filter the disparity labels at the pixels in the first image in the first disparity map based on the first confidence map to generate a first filtered disparity map; and

filter the disparity labels at the pixels in the second image in the second disparity map based on the second confidence map to generate a second filtered disparity map.

22. The computer program product as claimed in claim 21 , wherein the apparatus is further caused, at least in part to:

determine occlusion pixels by comparing the first filtered disparity map and the second filtered disparity map;

assign disparity labels at the occlusion pixels; and

combine the first filtered disparity map, the second filtered disparity map and the assigned disparity labels at the occlusion pixels to generate the disparity map.