Stereo matching method and apparatus, image processing apparatus, and training method therefor

1. A stereo matching method comprising:

obtaining a first feature map associated with a first view image comprising a reference pixel and a second feature map associated with a second view image comprising candidate pixels using a neural network model-based feature extractor;

determining respective matching costs between the reference pixel of the first view image and the candidate pixels of the second view image based on the first feature map and the second feature map obtained from the neural network model-based feature extractor; and

determining a pixel corresponding to the reference pixel among the candidate pixels based on the determined matching costs,

wherein the determining of the respective matching costs comprises:

determining vector distances between a feature vector of the reference pixel included in the first feature map and respective feature vectors of the candidate pixels included in the second feature map, wherein the candidate pixels are pixels within a search range located on a line comprising a point in the second view image corresponding to the reference pixel of the first view image;

determining an initial matching cost based on a vector distance among the respective vector distances between the feature vector of the reference pixel and a respective feature vector of a current candidate pixel among the candidate pixels; and

determining a matching cost among the respective matching costs by adjusting the initial matching cost based on matching cost information associated with another reference pixel adjacent to the reference pixel.

2. The stereo matching method of claim 1 , wherein the determining of the pixel comprises determining, to be the pixel corresponding to the reference pixel, a candidate pixel having a lowest matching cost among the candidate pixels.

3. The stereo matching method of claim 1 , wherein the obtaining of the first feature map comprises:

receiving image information of the first view image as an input,

determining the feature vector of the reference pixel based on the input image information, and

obtaining the first feature map based on the feature vector of the reference pixel.

4. The stereo matching method of claim 1 , wherein the obtaining of the second feature map comprises:

receiving image information of the second view image as an input,

determining the respective feature vectors of the candidate pixels based on the input image information, and

obtaining the second feature map based on the respective feature vectors of the candidate pixels.

5. The stereo matching method of claim 1 , wherein the obtaining of the first feature map and the second feature map comprises:

obtaining the first feature map based on a patch region of the first view image, and

obtaining the second feature map based on patch regions of the second view image.

6. The stereo matching method of claim 5 , wherein the obtaining of the first feature map comprises:

receiving, as an input, information of a patch region comprising the reference pixel of the first view image,

determining the feature vector of the reference pixel based on the input information of the patch region, and

determining the first feature map based on the feature vector of the reference pixel.

7. The stereo matching method of claim 6 , wherein the information of the patch region comprises any one or any combination of any two or more of intensity information, color information, and gradient information of pixels included in the patch region comprising the reference pixel.

8. The stereo matching method of claim 5 , wherein the obtaining of the second feature map comprises:

receiving, as an input, information of patch regions respectively comprising the candidate pixels of the second view image,

determining the respective feature vectors of the candidate pixels based on the input information of the patch regions, and

obtaining the second feature map based on the respective feature vectors of the candidate pixels.

9. The stereo matching method of claim 8 , wherein the information of the patch regions comprises any one or any combination of any two or more of intensity information, color information, and gradient information of pixels included in the patch respectively comprising the candidate pixels.

10. The stereo matching method of claim 1 , wherein the first feature map comprises a feature vector of each of pixels included in the first view image, and

the second feature map comprises a feature vector of each of pixels included in the second view image.

11. The stereo matching method of claim 1 , further comprising determining depth information based on a disparity between the reference pixel and the pixel corresponding to the reference pixel.

12. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1 .

13. A stereo matching method comprising:

extracting a first feature vector of a reference pixel included in a first view image and respective second feature vectors of each of candidate pixels included in a second view image using a neural network model-based feature extractor; and

determining a pixel corresponding to the reference pixel among the candidate pixels based on the first feature vector and the second feature vectors obtained from the neural network model-based feature extractor,

wherein determining comprises:

determining a respective vector distance between the first feature vector and each of the second feature vectors, wherein the candidate pixels are pixels within a search range located on a line comprising a point in the second view image corresponding to the reference pixel of the first view image;

determining an initial matching cost based on a vector distance among the respective vector distances between the first feature vector and a respective second feature vector of a current candidate pixel among the candidate pixels;

determining a matching cost among respective matching costs by adjusting the initial matching cost based on matching cost information associated with another reference pixel adjacent to the reference pixel; and

determining the pixel corresponding to the reference pixel among the candidate pixels based on the respective matching costs.

14. An image processing apparatus comprising:

a processor configured to:

obtain a first feature map associated with a first view image comprising a reference pixel and a second feature map associated with a second view image comprising candidate pixels using a neural network model-based feature extractor,

determine respective matching costs between the reference pixel of the first view image and the candidate pixels of the second view image based on the first feature map and the second feature map obtained from the neural network model-based feature extractor, and

determine a pixel corresponding to the reference pixel among the candidate pixels based on the determined matching costs,

wherein the processor is further configured to:

determine vector distances between a feature vector of the reference pixel included in the first feature map and respective feature vectors of the candidate pixels included in the second feature map, wherein the candidate pixels are pixels within a search range located on a line comprising a point in the second view image corresponding to the reference pixel of the first view image, and

determine an initial matching cost based on a vector distance among the respective vector distances between the feature vector of the reference pixel and a respective feature vector of a current candidate pixel among the candidate pixels; and

determine a matching cost among the respective matching costs by adjusting the initial matching cost based on matching cost information associated with another reference pixel adjacent to the reference pixel.

15. The method of claim 1 , wherein the vector distances are Euclidian distances.

16. The method of claim 1 , wherein the determining of the pixel corresponding to the reference pixel comprises determining the pixel corresponding to the reference pixel from the search range of a resultant matching cost volume of the determined respective matching costs.