Granted Patent B2

US 12,561,822 · App. 18/468,656 · Granted Feb 24, 2026

Inventors: Jisoo Jeong (San Diego, CA); Hong Cai (San Diego, CA); Babak Ehteshami Bejnordi (Amsterdam, NL); Risheek Garrepalli (San Diego, CA); Rajeev Yasarla (San Diego, CA); Fatih Murat Porikli (San Diego, CA)

Assignee: QUALCOMM Incorporated

G06T7/593G06T7/246G06T7/285G06T2207/10012G06T2207/20084G06T2207/20228

Abstract

Claims (90)

1 . A method for image processing, comprising:

correlating a first set of features obtained from a first viewpoint at a first time period with a second set of features obtained from a second viewpoint at the first time period to generate a first disparity cost volume;

correlating a third set of features obtained from the first viewpoint at a second time period with the first set of features to generate a first optical flow cost volume;

gating the first disparity cost volume to generate first intermediate disparity information;

gating the first optical flow cost volume to generate first intermediate optical flow information;

correlating the first set of features, the second set of features, and the first intermediate optical flow information to generate disparity information for output and a second disparity cost volume; and

correlating the third set of features, the first set of features, and the first intermediate disparity information to generate optical flow information for output and a second optical flow cost volume;

wherein correlating the first set of features, the second set of features, and the first intermediate optical flow information comprises:

combining the first set of features with the first intermediate optical flow information to refine the first set of features;

combining the second set of features with the first intermediate optical flow information to refine the second set of features; and

comparing the refined first set of features and the refined second set of features to generate the second disparity cost volume;

wherein:

combining the first set of features with the first intermediate optical flow information comprises concatenating the first set of features with the first intermediate optical flow information; and

combining the second set of features with the first intermediate optical flow information comprises concatenating the second set of features with the first intermediate optical flow information.

2 . The method of claim 1 , further comprising:

gating the second disparity cost volume to generate second intermediate disparity information;

gating the second optical flow cost volume to generate second intermediate optical flow information;

outputting disparity information based on the second intermediate disparity information; and

outputting optical flow information based on the second intermediate optical flow information.

3 . The method of claim 2 , further comprising:

concatenating the first intermediate optical flow information and the second intermediate optical flow information to generate concatenated optical flow information; and

concatenating the first intermediate disparity information and the second intermediate disparity information to generate concatenated disparity information.

4 . The method of claim 3 , wherein outputting disparity information and optical flow information comprising outputting the concatenated disparity information and the concatenated optical flow information.

5 . The method of claim 1 , wherein comparing the refined first set of features and the refined second set of features comprises determining a dot product between a feature of the refined first set of features and a feature of the refined second set of features.

6 . The method of claim 1 , wherein correlating the third set of features, the first set of features, and the first intermediate disparity information comprises:

combining the first set of features with the first intermediate disparity information to refine the first set of features;

combining the third set of features with the first intermediate disparity information to refine the second set of features; and

comparing the refined first set of features and the refined third set of features to generate the second optical flow cost volume.

7 . The method of claim 6 , wherein comparing the refined first set of features and the refined third set of features comprises determining a dot product between a feature of the refined first set of features and a feature of the refined third set of features.

8 . The method of claim 1 , wherein:

combining the first set of features with the first intermediate disparity information comprises concatenating the first set of features with the first intermediate disparity information; and

combining the third set of features with the first intermediate disparity information comprises concatenating the third set of features with the first intermediate disparity information.

9 . An apparatus for image processing, comprising:

at least one memory; and

at least one processor coupled to the at least one memory, the at least one processor being configured to:

correlate a first set of features obtained from a first viewpoint at a first time period with a second set of features obtained from a second viewpoint at the first time period to generate a first disparity cost volume;

correlate a third set of features obtained from the first viewpoint at a second time period with the first set of features to generate a first optical flow cost volume;

gate the first disparity cost volume to generate first intermediate disparity information;

gate the first optical flow cost volume to generate first intermediate optical flow information;

correlate the first set of features, the second set of features, and the first intermediate optical flow information to generate disparity information for output and a second disparity cost volume; and

correlate the third set of features, the first set of features, and the first intermediate disparity information to generate optical flow information for output and a second optical flow cost volume;

wherein, to correlate the first set of features, the second set of features, and the first intermediate optical flow information, the at least one processor is configured to:

combine the first set of features with the first intermediate optical flow information to refine the first set of features;

combine the second set of features with the first intermediate optical flow information to refine the second set of features; and

compare the refined first set of features and the refined second set of features to generate the second disparity cost volume;

wherein:

to combine the first set of features with the first intermediate optical flow information, the at least one processor is further configured to concatenate the first set of features with the first intermediate optical flow information; and

to combine the second set of features with the first intermediate optical flow information, the at least one processor is further configured to concatenate the second set of features with the first intermediate optical flow information.

10 . The apparatus of claim 9 , wherein the at least one processor is further configured to:

gate the second disparity cost volume to generate second intermediate disparity information;

gate the second optical flow cost volume to generate second intermediate optical flow information;

output disparity information based on the second intermediate disparity information; and

output optical flow information based on the second intermediate optical flow information.

11 . The apparatus of claim 10 , wherein the at least one processor is further configured to:

concatenate the first intermediate optical flow information and the second intermediate optical flow information to generate concatenated optical flow information; and

concatenate the first intermediate disparity information and the second intermediate disparity information to generate concatenated disparity information.

12 . The apparatus of claim 11 , wherein, to output disparity information and optical flow information, the at least one processor is configured to output the concatenated disparity information and the concatenated optical flow information.

13 . The apparatus of claim 9 , wherein, to compare the refined first set of features and the refined second set of features, the at least one processor is configured to determine a dot product between a feature of the refined first set of features and a feature of the refined second set of features.

14 . The apparatus of claim 9 , wherein, to correlate the third set of features, the first set of features, and the first intermediate disparity information, the at least one processor is configured to:

combine the first set of features with the first intermediate disparity information to refine the first set of features;

combine the third set of features with the first intermediate disparity information to refine the second set of features; and

compare the refined first set of features and the refined third set of features to generate the second optical flow cost volume.

15 . The apparatus of claim 14 , wherein:

to combine the first set of features with the first intermediate disparity information, the at least one processor is configured to concatenate the first set of features with the first intermediate disparity information; and

to combine the third set of features with the first intermediate disparity information, the at least one processor is configured to concatenate the third set of features with the first intermediate disparity information.

16 . The apparatus of claim 14 , wherein, to compare the refined first set of features and the refined third set of features, the at least one processor is further configured to determine a dot product between a feature of the refined first set of features and a feature of the refined third set of features.

17 . A non-transitory computer-readable medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to:

correlate a first set of features obtained from a first viewpoint at a first time period with a second set of features obtained from a second viewpoint at the first time period to generate a first disparity cost volume;

correlate a third set of features obtained from the first viewpoint at a second time period with the first set of features to generate a first optical flow cost volume;

gate the first disparity cost volume to generate first intermediate disparity information;

gate the first optical flow cost volume to generate first intermediate optical flow information;

correlate the first set of features, the second set of features, and the first intermediate optical flow information to generate disparity information for output and a second disparity cost volume; and

correlate the third set of features, the first set of features, and the first intermediate disparity information to generate optical flow information for output and a second optical flow cost volume;

wherein, to correlate the first set of features, the second set of features, and the first intermediate optical flow information, the at least one processor is configured to:

combine the first set of features with the first intermediate optical flow information to refine the first set of features;

combine the second set of features with the first intermediate optical flow information to refine the second set of features; and

compare the refined first set of features and the refined second set of features to generate the second disparity cost volume;

wherein:

to combine the first set of features with the first intermediate optical flow information, the at least one processor is further configured to concatenate the first set of features with the first intermediate optical flow information; and

to combine the second set of features with the first intermediate optical flow information, the at least one processor is further configured to concatenate the second set of features with the first intermediate optical flow information.

18 . The non-transitory computer-readable medium of claim 17 wherein the instructions cause the at least one processor to:

gate the second disparity cost volume to generate second intermediate disparity information;

gate the second optical flow cost volume to generate second intermediate optical flow information;

output disparity information based on the second intermediate disparity information; and

output optical flow information based on the second intermediate optical flow information.

19 . The non-transitory computer-readable medium of claim 18 , wherein the instructions cause the at least one processor to:

concatenate the first intermediate optical flow information and the second intermediate optical flow information to generate concatenated optical flow information; and

concatenate the first intermediate disparity information and the second intermediate disparity information to generate concatenated disparity information.

20 . The non-transitory computer-readable medium of claim 19 , wherein, to output disparity information and optical flow information, the instructions cause the at least one processor to output the concatenated disparity information and the concatenated optical flow information.

21 . The non-transitory computer-readable medium of claim 17 , wherein, to compare the refined first set of features and the refined second set of features, the instructions cause the at least one processor to determine a dot product between a feature of the refined first set of features and a feature of the refined second set of features.

Assignments (1)

Continuity (1)

References Cited (12)