Granted Patent B2

US 12,243,243 · App. 17/668,513 · Granted Mar 4, 2025

Inventors: Youngjun Kwak (Seoul, KR); Taekyung Kim (Daejeon, KR); Changick Kim (Daejeon, KR); Byeongjun Park (Daejeon, KR); Changbeom Park (Seoul, KR)

Assignees: Samsung Electronics Co., Ltd.; Korea Advanced Institute of Science and Technology

G06T7/248G06T3/18G06T3/4007G06T7/55G06V20/46G06T2207/10016

Abstract

Claims (71)

1. A processor-implemented scene flow estimation method, comprising:

receiving a first feature pyramid and a second feature pyramid by encoding a first frame and a second frame of an input image through a same encoder;

extracting a depth feature based on the received first feature pyramid;

extracting a motion feature based on the received first feature pyramid and the received second feature pyramid;

generating an overall feature based on the depth feature and the motion feature; and

estimating a scene flow based on the overall feature,

wherein the extracting of the motion feature comprises:

inputting a first level of the second feature pyramid to a warping layer;

inputting an output of the warping layer and a first level of the first feature pyramid to a correlation layer; and

concatenating an output of the correlation layer and the first level of the first feature pyramid and inputting a result of the concatenating to a correlation regularization module configured to perform a plurality of convolution operations,

wherein the warping layer is configured to adjust a position of each pixel of the first level of the second feature pyramid based on a result of a previous optical flow estimation performed based on the motion feature, and a result of a previous scene flow estimation performed based on the overall feature.

2. The method of claim 1 , further comprising:

estimating a depth of the input image based on the depth feature; and

estimating an optical flow of the input image based on the motion feature.

3. The method of claim 2 , wherein the extracting of the depth feature comprises:

performing decoding on a first level and a second level of the first feature pyramid; and

performing decoding on a third level of the first feature pyramid based on a result of the decoding on the first level and the second level of the first feature pyramid.

4. The method of claim 3 , wherein the performing of the decoding on the first level and the decoding on the second level comprises:

performing a first convolution operation on the first level;

upsampling a result of the first convolution operation, and concatenating a result of the upsampling and the second level; and

performing a second convolution operation on a result of the concatenating.

5. The method of claim 4 , wherein the performing of the decoding on the third level comprises:

concatenating the result of the decoding on the third level and a result of a previous scene flow estimation and performing a third convolution operation on a result of the concatenating;

upsampling a result of the third convolution operation and concatenating a result of the upsampling and the third level;

performing a fourth convolution operation on a result of the concatenating; and

concatenating a result of the fourth convolution operation and the overall feature and performing a fifth convolution operation on a result of the concatenating.

6. The method of claim 1 , wherein the correlation layer is configured to calculate a correlation of each pixel of the first level of the first feature pyramid with the output of the warping layer.

7. The method of claim 1 , wherein the generating of the overall feature comprises:

performing a nearest-neighbor interpolation (NNI) on the depth feature and concatenating a result of the NNI and the motion feature;

performing a convolution operation based on a concatenated feature obtained by the concatenating and extracting a spatial feature;

performing a global average pooling (GAP) operation on the concatenated feature and extracting a channel feature; and

obtaining the overall feature based on the spatial feature, the channel feature, and the motion feature.

8. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the scene flow estimation method of claim 1 .

9. A scene flow estimation apparatus, comprising:

a memory comprising instructions; and

one or more processors configured to execute the instructions,

wherein, when the instructions are executed by the one or more processors, the one or more processors are configured to:

receive a first feature pyramid and a second feature pyramid by encoding a first frame and a second frame of an input image through a same encoder;

extract a depth feature based on the received first feature pyramid;

extract a motion feature based on the received first feature pyramid and the received second feature pyramid;

generate an overall feature based on the depth feature and the motion feature; and

estimate a scene flow based on the overall feature,

wherein the one or more processors are further configured to:

input a first level of the second feature pyramid to a warping layer;

input an output of the warping layer and a first level of the first feature pyramid to a correlation layer; and

concatenate an output of the correlation layer and the first level of the first feature pyramid, and input a result of the concatenating to a correlation regularization module configured to perform a plurality of convolution operations,

wherein the warping layer is configured to adjust a position of each pixel of the first level of the second feature pyramid based on a result of a previous optical flow estimation performed based on the motion feature and a result of a previous scene flow estimation performed based on the overall feature.

10. The apparatus of claim 9 , wherein the one or more processors are further configured to:

estimate a depth of the input image based on the depth feature; and

estimate an optical flow of the input image based on the motion feature.

11. The apparatus of claim 10 , wherein the one or more processors are further configured to:

perform decoding on a first level and a second level of the first feature pyramid; and

perform decoding on a third level of the first feature pyramid based on a result of the decoding on the first level and the second level of the first feature pyramid.

12. The apparatus of claim 11 , wherein the one or more processors are further configured to:

perform a first convolution operation on the first level;

upsample a result of the first convolution operation and concatenate a result of the upsampling and the second level; and

perform a second convolution operation on a result of the concatenating.

13. The apparatus of claim 12 , wherein the one or more processors are further configured to:

concatenate the result of the decoding on the third level and a result of a previous scene flow estimation and perform a third convolution operation on a result of the concatenating;

upsample a result of the third convolution operation and concatenate a result of the upsampling and the third level;

perform a fourth convolution operation on a result of the concatenating; and

concatenate a result of the fourth convolution operation and the overall feature and perform a fifth convolution operation on a result of the concatenating.

14. The apparatus of claim 9 , wherein the correlation layer is configured to calculate a correlation of each pixel of the first level of the first feature pyramid with the output of the warping layer.

15. The apparatus of claim 9 , wherein the one or more processors are further configured to:

perform a nearest-neighbor interpolation (NNI) on the depth feature and concatenate a result of the NNI and the motion feature;

perform a convolution operation based on a concatenated feature obtained by the concatenating and extract a spatial feature;

perform a global average pooling (GAP) operation on the concatenated feature and extract a channel feature; and

obtain the overall feature based on the spatial feature, the channel feature, and the motion feature.

16. A scene flow estimation system, comprising:

a capturing apparatus configured to capture a monocular image; and

the scene flow estimation apparatus of claim 9 .

Assignments (1)

Priority Claims (2)

Continuity (1)

References Cited (24)