Granted Patent B2

US 12,373,959 · App. 17/678,904 · Granted Jul 29, 2025

Inventors: Raju Konda (Hyderabad, IN); Sampurnananda Mishra (Pune, IN)

Assignee: NVIDIA Corporation

G06T7/248G06T5/70G06T7/13G06T7/136

Abstract

Claims (80)

1. A method comprising:

obtaining an input image and a reference image, the input image comprising a plurality of image regions;

obtaining at least one penalty map comprising a set of penalty values for each of the plurality of image regions, the set of penalty values comprising, for each of the plurality of image regions, a penalty value for each of a plurality of directions intersecting the image region, wherein the at least one penalty map is obtained by at least:

denoising the input image;

producing an edge map by performing edge detection on the denoised input image, and

producing an object map by performing object detection on at least one of the denoised input image or the edge map; and

generating an optical flow map for the input and reference images based at least in part on the at least one penalty map.

2. The method of claim 1 , wherein the at least one penalty map comprises first and second penalty maps,

the optical flow map is generated using a Semi-Global Matching (“SGM”) process comprising a first penalty value and a second penalty value to determine a cost for a particular one of the plurality of image regions, a particular disparity, and a particular one of the plurality of directions,

the first penalty value is obtained using the first penalty map for the particular image region and the particular direction, and

the second penalty value is obtained using the second penalty map for the particular image region and the particular direction.

3. The method of claim 2 , wherein the optical flow map is generated by:

determining a plurality of costs, at least one cost of the plurality of costs being determined for one of the plurality of image regions, one of a plurality of disparities, and one of the plurality of directions;

obtaining a plurality of accumulated costs for each unique pair of one of the plurality of image regions and one of the plurality of disparities by summing those of the plurality of costs determined for the unique pair;

selecting, for each of the plurality of image regions, a smallest one of the plurality of accumulated costs obtained for the image region, the smallest accumulated cost having been obtained for a selected one of the plurality of disparities; and

placing, for each of the plurality of image regions, a value based at least in part on the selected disparity in the optical flow map in a location corresponding to the image region.

4. The method of claim 1 , wherein the at least one penalty map comprises first and second penalty maps,

the first penalty map is determined based at least in part on the object map, and

the second penalty map is determined based at least in part on the edge map.

5. The method of claim 1 , wherein obtaining the at least one penalty map further comprises producing a thresholded edge map by performing a thresholding process on the edge map to remove any edges having a width below a threshold value,

the at least one penalty map comprises first and second penalty maps,

the first penalty map is determined based at least in part on the object map, and

the second penalty map is determined based at least in part on both the edge map and the thresholded edge map.

6. The method of claim 1 , wherein the plurality of image regions comprises at least one of a set of feature points or a set of pixels.

7. The method of claim 1 , wherein the input image comprises a plurality of pixels, and

the plurality of image regions comprises a subset of the plurality of pixels, the subset including fewer than all of the plurality of pixels.

8. A system comprising:

one or more circuits to obtain a plurality of image regions from an input image, determine a set of penalty values for each of the plurality of image regions, and create an optical flow map for the input image and a reference image based at least in part on the set of penalty values determined for each of the plurality of image regions, the set of penalty values comprising, for each of the plurality of image regions, a penalty value for each of a plurality of directions intersecting the image region, wherein the optical flow map is created by:

denoising the input image;

producing an edge map by performing edge detection on the denoised input image; and

producing an object map by performing object detection on at least one of the denoised input image or the edge map.

9. The system of claim 8 , further comprising:

upstream hardware connected to the one or more circuits, the upstream hardware providing the input image to the one or more circuits.

10. The system of claim 8 , wherein the one or more circuits comprise:

at least one processor; and

memory storing instructions that are executable by the at least one processor, and when executed by the at least one processor cause the at least one processor to obtain the plurality of image regions, determine the set of penalty values for at least one image region of the plurality of image regions, and create the optical flow map.

11. The system of claim 10 , wherein when the instructions are executed by the at least one processor, the instructions cause the at least one processor to generate the optical flow map by:

determining a plurality of costs for unique combinations of the plurality of image regions, a plurality of disparities between the input and reference images, and the plurality of directions;

obtaining a plurality of accumulated costs by summing those of the plurality of costs determined for each unique pair of one of the plurality of image regions and at least one of the plurality of disparities;

selecting, for at least one portion of one or more of the plurality of image regions, a smallest one of the plurality of accumulated costs obtained for the image region, the smallest accumulated cost having been obtained for a selected one of the plurality of disparities; and

placing, for each of the portion of the plurality of image regions, a value based at least in part on the selected disparity in the optical flow map in a location corresponding to the image region.

12. The system of claim 11 , wherein the one or more circuits are further to determine a particular one of the plurality of costs for a particular one of the unique combinations comprising a particular one of the plurality of image regions, a particular one of the plurality of disparities, and a particular one of the plurality of directions by:

adding a first one of the set of penalty values to a matching term when a first disparity metric value of the particular image region differs by at most a predetermined amount from a second disparity metric value of a neighboring one of the plurality of image regions along the particular direction; and

adding a second one of the set of penalty values to the matching term when the first disparity metric value differs by more than the predetermined amount from the second disparity metric value along the particular direction, the first penalty value being less than the second penalty value.

13. The system of claim 12 , wherein the one or more circuits are further to determine the particular cost equals the matching term when the first disparity metric value equals the second disparity metric value.

14. The system of claim 8 , wherein the set of penalty values comprises a first set of penalty values and a different second set of penalty values,

each of the first set of penalty values is determined based at least in part on the object map, and

each of the second set of penalty values is determined based at least in part on the edge map.

15. The system of claim 10 , wherein when the instructions are executed by the at least one processor, the instructions cause the at least one processor to threshold the edge map to produce a thresholded edge map in which edges thinner than a threshold value have been removed,

the set of penalty values comprises a first set of penalty values and a different second set of penalty values,

each of the first set of penalty values is determined based at least in part on the object map, and

each of the second set of penalty values is determined based at least in part on both the edge map and the thresholded edge map.

16. The system of claim 10 , wherein the at least one processor comprises one or more parallel processing units.

17. The system of claim 10 , wherein when the instructions are executed by the at least one processor, the instructions cause the at least one processor to extract a set of feature points from the input image as the plurality of image regions.

18. The system of claim 10 , wherein when the instructions are executed by the at least one processor, the instructions cause the at least one processor to extract a set of pixels from the input image as the plurality of image regions.

19. The system of claim 18 , wherein the input image comprises a plurality of pixels, and the set of pixels comprises fewer than all of the plurality of pixels.

20. One or more non-transitory processor-readable media comprising instructions that are executable by at least one processor, and when executed by the at least one processor causing the at least one processor to:

obtain a set of disparity values for each of a plurality of image regions in an input image, the set of disparity values comprising, for each of the plurality of image regions, a disparity value for each of a plurality of directions intersecting the image region; and

use the set of disparity values to create an optical flow map for the input image and a different reference image, wherein the optical flow map is created by at least:

denoising the input image;

performing edge detection on the denoised input image to produce an edge map; and

producing an object map by performing object detection on at least one of the denoised input image or the edge map.

21. The one or more non-transitory processor-readable media of claim 20 , wherein when the instructions are executed by the at least one processor, the instructions cause the at least one processor to generate the optical flow map by:

determining a plurality of costs for unique combinations of the plurality of image regions, a plurality of disparities between the input and reference images, and the plurality of directions;

obtaining a plurality of accumulated costs by summing those of the plurality of costs determined for each unique pair of one of the plurality of image regions and one of the plurality of disparities;

selecting, for each of at least a portion of the plurality of image regions, a smallest one of the plurality of accumulated costs obtained for the image region, the smallest accumulated cost having been obtained for a selected one of the plurality of disparities; and

storing, for each of the portion of the plurality of image regions, a value based at least in part on the selected disparity in the optical flow map in a location corresponding to the image region.

22. The one or more non-transitory processor-readable media of claim 21 , wherein when the instructions are executed by the at least one processor, the instructions cause the at least one processor to store the set of disparity values as first and second disparity maps, and to determine a particular one of the plurality of costs for a particular one of the unique combinations comprising a particular one of the plurality of image regions, a particular one of the plurality of disparities, and a particular one of the plurality of directions by:

obtaining first and second values from the first and second disparity maps, respectively;

adding the first value to a matching term when a first disparity metric value of the particular image region differs by at most a predetermined amount from a second disparity metric value of a neighboring one of the plurality of image regions along the particular direction; and

adding the second value to the matching term when the first disparity metric value differs by more than the predetermined amount from the second disparity metric value along the particular direction, the first value being less than the second value.

23. The one or more non-transitory processor-readable media of claim 22 , wherein the at least one processor is further to determine the particular cost equals the matching term when the first disparity metric value equals the second disparity metric value.

24. The one or more non-transitory processor-readable media of claim 20 , wherein when the instructions are executed by the at least one processor, the instructions cause the at least one processor to extract a set of feature points or a set of pixels from the input image as the plurality of image regions.

25. The one or more non-transitory processor-readable media of claim 22 , wherein the at least one processor obtains the set of disparity values for each of a plurality of image regions based, at least in part, on:

assigning values to the first disparity map based at least in part on the object map; and

assigning values to the second disparity map based at least in part on the edge map.

26. The one or more non-transitory processor-readable media of claim 22 , wherein the at least one processor obtains the set of disparity values for each of a plurality of image regions based, at least in part, on:

thresholding the edge map to remove any edges that are not thicker than a threshold value to produce a thresholded edge map;

assigning values to the first disparity map based at least in part on the object map; and

assigning values to the second disparity map based at least in part on the edge map and the thresholded edge map.

Assignments (1)

Continuity (1)

References Cited (14)