Granted Patent B2

US 12,444,055 · App. 18/316,823 · Granted Oct 14, 2025

Inventors: Xin Li (San Diego, CA); Jiancheng Lyu (San Diego, CA); Yingyong Qi (San Diego, CA)

Assignee: QUALCOMM Incorporated

G06T7/11G06T2207/20084

Abstract

Claims (83)

1. A method for image processing, comprising:

obtaining an image of an environment;

extracting a first set of features at a first scale resolution of the image;

extracting a second set of features at a second scale resolution of the image, wherein the second scale resolution is lower than the first scale resolution;

performing a self-attention transform to generate similarity scores for the second set of features;

adding the similarity scores to the second set of features to generate a first feature extractor output;

up-sampling the first feature extractor output to generate a second feature extractor output;

adding the second feature extractor output to the first set of features to generate a third feature extractor output;

receiving an instance query for instances of a feature;

performing a cross-attention transform on the instance query and the first feature extractor output to generate a first set of weights;

performing a matrix multiplication on the first set of weights and the third feature extractor output to generate instance masks for the image; and

outputting the instance masks.

2. The method of claim 1 , wherein the feature received with the instance query is associated with a feature of the second set of features.

3. The method of claim 1 , wherein generating the first feature extractor output comprises performing a convolution operation on the added similarity scores and the second set of features.

4. The method of claim 1 , wherein the first set of features and the second set of features are extracted using a convolutional neural network based backbone.

5. The method of claim 1 , further comprising:

obtaining a set of probability scores based on the cross-attention transform;

associating probability scores of the set of probability scores with feature instances; and

outputting the probability scores.

6. The method of claim 1 , wherein the second set of features are extracted at a lowest scale resolution of all of the extracted sets of features for the image.

7. The method of claim 1 , further comprising performing a convolution operation and batch normalization operation on the second set of features.

8. The method of claim 1 , further comprising:

receiving a semantic query for a set of feature categories;

performing a cross-attention transform on the semantic query and the second feature extractor output to generate a second set of weights; and

performing a matrix multiplication on the second set of weights and the third feature extractor output to generate semantic masks for the image.

9. The method of claim 8 , further comprising outputting the semantic masks for the image.

10. The method of claim 1 , wherein the instance query indicates a number of instances of a feature to output.

11. An apparatus for image processing, comprising:

a memory; and

a processor coupled to the memory and configured to:

obtain an image of an environment;

extract a first set of features at a first scale resolution of the image;

extract a second set of features at a second scale resolution of the image, wherein the second scale resolution is lower than the first scale resolution;

perform a self-attention transform to generate similarity scores for the second set of features;

add the similarity scores to the second set of features to generate a first feature extractor output;

up-sample the first feature extractor output to generate a second feature extractor output;

add the second feature extractor output to the first set of features to generate a third feature extractor output;

receive an instance query for instances of a feature;

perform a cross-attention transform on the instance query and the first feature extractor output to generate a first set of weights;

perform a matrix multiplication on the first set of weights and the third feature extractor output to generate instance masks for the image; and

output the instance masks.

12. The apparatus of claim 11 , wherein the feature received with the instance query is associated with a feature of the second set of features.

13. The apparatus of claim 11 , wherein, to generate the first feature extractor output, the processor is configured to perform a convolution operation on the added similarity scores and the second set of features.

14. The apparatus of claim 11 , wherein the first set of features and the second set of features are extracted using a convolutional neural network based backbone.

15. The apparatus of claim 11 , wherein the processor is further configured to:

obtain a set of probability scores based on the cross-attention transform;

associate probability scores of the set of probability scores with feature instances; and

output the probability scores.

16. The apparatus of claim 11 , wherein the processor is further configured to extract the second set of features at a lowest scale resolution of all of the extracted sets of features for the image.

17. The apparatus of claim 11 , wherein the processor is further configured to perform a convolution operation and batch normalization operation on the second set of features.

18. The apparatus of claim 11 , wherein the processor is further configured to:

receive a semantic query for a set of feature categories;

perform a cross-attention transform on the semantic query and the second feature extractor output to generate a second set of weights; and

perform a matrix multiplication on the second set of weights and the third feature extractor output to generate semantic masks for the image.

19. The apparatus of claim 18 , wherein the processor is further configured to output the semantic masks for the image.

20. The apparatus of claim 11 , wherein the instance query indicates a number of instances of a feature to output.

21. A non-transitory computer-readable medium having stored thereon instructions that, when executed by a processor, cause the processor to:

obtain an image of an environment;

extract a first set of features at a first scale resolution of the image;

extract a second set of features at a second scale resolution of the image, wherein the second scale resolution is lower than the first scale resolution;

perform a self-attention transform to generate similarity scores for the second set of features;

add the similarity scores to the second set of features to generate a first feature extractor output;

up-sample the first feature extractor output to generate a second feature extractor output;

add the second feature extractor output to the first set of features to generate a third feature extractor output;

receive an instance query for instances of a feature;

perform a cross-attention transform on the instance query and the first feature extractor output to generate a first set of weights;

perform a matrix multiplication on the first set of weights and the third feature extractor output to generate instance masks for the image; and

output the instance masks.

22. The non-transitory computer-readable medium of claim 21 , wherein the feature received with the instance query is associated with a feature of the second set of features.

23. The non-transitory computer-readable medium of claim 21 , wherein, to generate the first feature extractor output, the instructions cause the processor to perform a convolution operation on the added similarity scores and the second set of features.

24. The non-transitory computer-readable medium of claim 21 , wherein the first set of features and the second set of features are extracted using a convolutional neural network based backbone.

25. The non-transitory computer-readable medium of claim 21 , wherein the instructions cause the processor to:

obtain a set of probability scores based on the cross-attention transform;

associate probability scores of the set of probability scores with feature instances; and

output the probability scores.

26. The non-transitory computer-readable medium of claim 21 , wherein the instructions cause the processor to extract the second set of features at a lowest scale resolution of all of the extracted sets of features for the image.

27. The non-transitory computer-readable medium of claim 21 , wherein the instructions cause the processor to perform a convolution operation and batch normalization operation on the second set of features.

28. The non-transitory computer-readable medium of claim 21 , wherein the instructions cause the processor to:

receive a semantic query for a set of feature categories;

perform a cross-attention transform on the semantic query and the second feature extractor output to generate a second set of weights; and

perform a matrix multiplication on the second set of weights and the third feature extractor output to generate semantic masks for the image.

29. The non-transitory computer-readable medium of claim 28 , wherein the instructions cause the processor to output the semantic masks for the image.

30. The non-transitory computer-readable medium of claim 21 , wherein the instance query indicates a number of instances of a feature to output.

Assignments (1)

Continuity (1)

References Cited (27)