Granted Patent B2

US 12,347,161 · App. 17/868,660 · Granted Jul 1, 2025

Inventors: Qingfeng Liu (San Diego, CA); Mostafa El-Khamy (San Diego, CA)

Assignee: Samsung Electronics Co., Ltd.

G06V10/52G06T7/11G06V10/26G06V10/7715

Abstract

Claims (71)

1. A computer vision system comprising:

one or more processors; and

memory comprising instructions that, when executed by the one or more processors, cause the one or more processors to:

determine a semantic multi-scale context feature and an instance multi-scale context feature of an input scene;

generate a joint attention map based on the semantic multi-scale context feature and the instance multi-scale context feature;

refine the semantic multi-scale context feature and instance multi-scale context feature based on the joint attention map; and

generate a panoptic segmentation image based on the refined semantic multi-scale context feature and the refined instance multi-scale context feature.

2. The computer vision system of claim 1 , wherein to generate the joint attention map, the instructions further cause the one or more processors to calculate normalized correlations between the semantic multi-scale context feature and the instance multi-scale context feature.

3. The computer vision system of claim 2 , wherein to calculate the normalized correlations, the instructions further cause the one or more processors to:

convolve and reshape the semantic multi-scale context feature;

convolve, reshape, and transpose the instance multi-scale context feature;

apply a matrix multiplication operation between the reshaped semantic multi-scale context feature and the transposed instance multi-scale context feature; and

apply a softmax function to the output of the matrix multiplication operation.

4. The computer vision system of claim 1 , wherein to refine the semantic multi-scale context feature, the instructions further cause the one or more processors to:

convolve and reshape the semantic multi-scale context feature;

apply a matrix multiplication operation between the reshaped semantic multi-scale context feature and the joint attention map; and

reshape the output of the matrix multiplication operation.

5. The computer vision system of claim 1 , wherein to refine the instance multi-scale context feature, the instructions further cause the one or more processors to:

convolve and reshape the instance multi-scale context feature;

apply a matrix multiplication operation between the reshaped instance multi-scale context feature and the joint attention map; and

reshape the output of the matrix multiplication operation.

6. The computer vision system of claim 1 , wherein to generate the panoptic segmentation image, the instructions further cause the one or more processors to:

generate a semantic segmentation prediction based on the refined semantic multi-scale context feature;

generate an instance segmentation prediction based on the refined instance multi-scale context feature; and

fuse the semantic segmentation prediction with the instance segmentation prediction.

7. The computer vision system of claim 6 , wherein the instance segmentation prediction comprises:

an instance center prediction; and

an instance center offset prediction.

8. The computer vision system of claim 7 , wherein the instance center offset prediction is trained accordingly to an instance boundary aware regression loss function that applies more weights and penalties to boundary pixels defined in a ground truth boundary map than those applied to other pixels.

9. The computer vision system of claim 8 , wherein the ground truth boundary map is generated by comparing a Euclidean distance between a pixel and a boundary with a threshold distance.

10. A panoptic segmentation method, comprising:

determining a semantic multi-scale context feature and an instance multi-scale context feature of an input scene;

generating a joint attention map based on the semantic multi-scale context feature and the instance multi-scale context feature;

refining the semantic multi-scale context feature and instance multi-scale context feature based on the joint attention map; and

generating a panoptic segmentation image based on the refined semantic multi-scale context feature and the refined instance multi-scale context feature.

11. The method of claim 10 , wherein the generating of the joint attention map comprises calculating normalized correlations between the semantic multi-scale context feature and the instance multi-scale context feature.

12. The method of claim 11 , wherein the calculating of the normalized correlations comprises:

convolving and reshaping the semantic multi-scale context feature;

convolving, reshaping, and transposing the instance multi-scale context feature;

applying a matrix multiplication operation between the reshaped semantic multi-scale context feature and the transposed instance multi-scale context feature; and

applying a softmax function to the output of the matrix multiplication operation.

13. The method of claim 10 , wherein the refining of the semantic multi-scale context feature comprises:

convolving and reshaping the semantic multi-scale context feature;

applying a matrix multiplication operation between the reshaped semantic multi-scale context feature and the joint attention map; and

reshaping the output of the matrix multiplication operation.

14. The method of claim 10 , wherein the refining of the instance multi-scale context feature comprises:

convolving and reshaping the instance multi-scale context feature;

applying a matrix multiplication operation between the reshaped instance multi-scale context feature and the joint attention map; and

reshaping the output of the matrix multiplication operation.

15. The method of claim 10 , wherein the generating of the panoptic segmentation image comprises:

generating a semantic segmentation prediction based on the refined semantic multi-scale context feature;

generating an instance segmentation prediction based on the refined instance multi-scale context feature; and

fusing the semantic segmentation prediction with the instance segmentation prediction.

16. The method of claim 15 , wherein the instance segmentation prediction comprises:

an instance center prediction; and

an instance center offset prediction.

17. The method of claim 16 , wherein the instance center offset prediction is trained accordingly to an instance boundary aware regression loss function that applies more weights and penalties to boundary pixels defined in a ground truth boundary map than those applied to other pixels.

18. The method of claim 17 , wherein the ground truth boundary map is generated by comparing a Euclidean distance between a pixel and a boundary with a threshold distance.

19. A panoptic segmentation system comprising:

one or more processors; and

memory comprising instructions that, when executed by the one or more processors, cause the one or more processors to:

extract a semantic multi-scale context feature of an input scene;

extract an instance multi-scale context feature of the input scene;

generate a joint attention map based on the semantic multi-scale context feature and the instance multi-scale context feature;

refine the semantic multi-scale context feature based on the joint attention map;

refine the instance multi-scale context feature based on the joint attention map;

predict a semantic class label based on the refined semantic multi-scale context feature;

predict an instance center and an instance center offset based on the refined instance multi-scale context feature; and

generate a panoptic segmentation image based on the semantic class label, the instance center, and the instance center offset.

20. The panoptic segmentation system of claim 19 , wherein the predicting of the instance center and the instance center offset is based on an instance boundary aware regression loss function that applies more weights and penalties to boundary pixels defined in a ground truth boundary map than those applied to other pixels, and

wherein the ground truth boundary map is generated by comparing a Euclidean distance between a pixel and a boundary with a threshold distance.

Assignments (1)

Continuity (2)

References Cited (28)